Patricia García Majado

Scoring itself is by no means an unfamiliar tool for law or society; rather, it has been widely used for many years in both the public and private sectors. Consider, for example, that public examinations and competitive selection processes function as scoring systems, as do the competitive procedures for awarding public grants or even the points-based driving license system. Scoring is, therefore, a way to reduce the complexity inherent in the social environment and to prioritize/rank needs in decision-making, sometimes more efficiently, and at other times (or also), more objectively. However, with the advent of artificial intelligence, this mechanism has evolved exponentially. The vast amount of data accessible today, combined with the immense computing capabilities of such systems, has led to almost any element or sector of society becoming “scorable.” This has given rise to what is known as social or citizen scoring.
Citizen social scoring systems, following the approval of the Artificial Intelligence Act (AI Act), are subject to different legal frameworks depending on how they operate, the effects they produce and, also, the domains in which they are used. There is, therefore, no uniform treatment of these systems but rather a segmented approach, in which aligns with the «risk-based» approach adopted by the Regulation itself.
Article 5.1 c) of the AI Act explicitly prohibits citizen social scoring systems in certain circumstances. Although this provision was not the one that underwent the most modifications, it did undergo changes, mainly due to the amendments introduced by the European Parliament. First, the initial proposal prohibited the introduction of these systems «to public authorities or anyone acting on their behalf», a provision that was subsequently eliminated. Therefore, and quite logically, these systems cannot be used not only by public authorities but also by private companies, which, in many cases, are the primary users of such mechanisms. Secondly, the Commission’s proposal referred to systems that assess or rate individuals based on social behaviour or “known or predicted” personality traits. However, the final version also includes the possibility that such characteristics may be «inferred,» as such aspects can be derived from other, seemingly neutral, circumstances that have no direct connection to behaviour or personality traits. Indeed, AI is specifically used to infer. Third, whereas the initial proposal considered that the purpose of social scoring systems was to assess or classify the «trustworthiness» of individuals, the regulation refers only to «assessment or classification», regardless of its purpose, whether to measure their trustworthiness or any other variables. Fourth, the Commission’s text referred to the «assessment or classification of natural persons,» whereas, due to an amendment by Parliament, this assessment was ultimately extended to groups of persons for the purpose of their social scoring (workers, insured individuals, consumers, minors, etc.).

However, as with other prohibitions, the one we are dealing with here is also goal oriented. In this way, citizen scoring systems are not prohibited per se, but they are prohibited when: (a) they either cause harmful or unfavourable treatment in contexts different from where the data was generated or (b) when such treatment is unjustified or disproportionate with respect to social behaviour or its severity.
In relation to the first exclusion, it should be argued that evaluation or classification systems for individuals that provide a score based on data generated in the same context are not prohibited. An example in this regard might be insurance scoring systems, used to determine the risk level of each insured individual and thereby set appropriate premiums. The factors used are typically related to credit history, driving record, age, gender and claims history. Since all of these are variables related to the insurance context, they would not be excluded under Art. 5.1.c) of the AI Act.
However, systems that use data to create social scores in a context other than the one in which the data was generated would, however, be prohibited. Aside from other considerations, the specific basis for this prohibition appears to be the right to equality and non-discrimination. The regulation presumes that treating someone differently, in a given area, for reasons unrelated to that context (unrelated factors) constitutes unjustified differential treatment (and, therefore, discrimination). Why decide whether or not to promote an employee based on their credit score? This is a current and widely spread example, especially in the U.S. (although not exclusively), where creditworthiness examinations are used as a factor in deciding whom to hire, based on the assumption that those who are diligent and pay their bills on time are more likely to be good responsible employees, who show up on time, etc.
An example of this exclusion could be the Danish AI system created to monitor public benefit fraud, recently revealed by Amnesty International. This system uses data that includes information on residence status and movements, citizenship, place of birth and family relationships, all of which can be used to infer a person’s race, ethnicity or sexual orientation. For example, in order to detect benefit fraud in pension and children care plans, the Danish authorities use the Really Single algorithm to predict an individual’s family and relationship status. One of the parameters used by the Really Single fraud detection algorithm includes «unusual» or «atypical» life patterns or family compositions, such as married people not living together or multigenerational households. Another algorithm used (Abroad) employs citizenship as a parameter to detect fraud, through which the race, ethnicity and immigration status of an individual can be inferred, prioritizing the investigation of people who meet these criteria. In this way, it could be argued that the context in which the benefit operates (social benefits domain) is, in many cases, unrelated to the context from which the data is derived (family or personal situation, citizenship, etc.).
In relation to this first section of Art. 5.1.c) of the AI Act, an interpretation by the Commission of the concept of the context seems necessary, as its scope will vary depending on this interpretation. In addition, it will be necessary to examine how this concept operates in the world, such as that of AI, where based on known and context-relevant data, absolutely unpredictable correlations may arise and be used again for scoring. In short, when can we know that we are dealing with «the same context» in which the data was generated?
Regarding the second exclusion, social scoring systems that cause harmful or unfavourable treatment towards certain individuals or groups of people, which is unjustified or disproportionate in relation to their social behaviour or its severity, are prohibited. In other words, scoring systems that, even operating within the same informative context, are prohibited if they are disproportionate or unjustified. An example of this exclusion could be the food delivery platform Deliveroo. Here, workers can log in every Monday to reserve their work shifts in three time slots (11 a.m., 3 p.m. and 5 p.m.) according to their score in the Deliveroo ranking, which is managed by an algorithm called «Frank». The riders’ ranking depends on two indices: reliability and availability. While the former takes into account the number of times a rider fails to fulfil a previously booked shift, the latter takes into account the number of times the rider is available during peak hours (8 p.m. to 10 p.m. from Friday to Sunday). It seems obvious (or perhaps not so obvious) that the availability variable can lead to unjustified harmful treatment for those riders who, for example, are ill, have dependents or engage in union activities, issues that all reduce their total availability during these hours. The Ordinary Court of Bologna, which heard the case in 2020, considered it a case of indirect discrimination: an apparently neutral criterion (labour availability) ends up, in practice, discriminating against certain specific groups. Precisely for this reason, and considering the AI Act, it seems that the Frank algorithm would constitute unjustified harmful treatment for certain workers on the platform and, as such, discriminatory.
However, when social scoring systems operate in the same context and do not result in unjustified or disproportionate harmful treatment for individuals, they are permitted. Nevertheless, they are not exempt from risks. There are many, with the main ones concerning equality and non-discrimination, but also specific rights at stake: the right to work, social security, social assistance, etc. Hence, when used in certain areas or for certain purposes, they are considered, from the perspective of the AI Act, as high-risk systems and, therefore, subject to its specific safeguards, both ex ante and ex post. Annex III of the AI Act considers high risk those rating or evaluation systems used in the labour context (to assign tasks, monitor the performance or behaviour of employees) and also those used for rating or evaluation in the field of essential services, both public and private. This category includes AI systems used to assess the eligibility of individuals for essential public assistance benefits; those used to assess the solvency of individuals or establish their credit rating; or those used for risk assessment and pricing in life and health insurance.
Can there be scoring systems that are neither prohibited nor high risk? It is not the most common, but it could happen, provided that, while not being prohibited, they do not affect any of the areas covered by high-risk systems. Perhaps that could be the case of tenant scoring AI systems which aim to select the best candidate for rental by considering factors such as their employment contract, salary, whether they have had unpaid debts, the condition in which they left previous rented properties, etc.

Ultimately, the final regulation of the social scoring systems is more satisfactory than that initially proposed by the Commission. However, this classification is not definitive, and it is positive that this is the case, as it allows the law to adapt in the most legally correct way to the ongoing developments of AI. Social scoring systems currently prohibited may, in the future, be allowed if they can be legally regulated, or conversely, systems currently permitted could be banned if an unacceptable harmful potential to fundamental rights were identified. Similarly, new systems could emerge, perhaps unknown today, that would need to be assessed and “placed” accordingly. Indeed, the Commission is required to review both prohibited and high-risk practices (Annex III) on an annual basis. In addition, there are some aspects, already pointed out, relating to the interpretation of certain concepts used in Art. 5.1.c) of the AI Act that may require clarification. Although this may stem from the interpretative criteria developed by the Commission, in many cases it may not be resolved at the legislative level, and instead, it may come from the case law in the field, which will provide insights on how to redefine the scope of the prohibition.
P.S. This is an abridged version of the presentation made at the Seminar Mass Biometric Surveillance through AI: Corporeality, Identity and Fundamental Rights (UPV – November 25-26, 2024).

*This translation has been revised by María Amparo González Rúa from the original Spanish version, which can be consulted here.