PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrimination, artificial intelligence, and algorithmic decision-making

Frederik Zuiderveen Borgesius

arXiv:2510.13465v1cs.CY

TL;DR

The report examines how AI and algorithmic decision-making can generate discriminatory effects despite their benefits and widespread use. Through a literature review, it maps risks, safeguards, and organisational responses, finding that discrimination can appear across applications including pricing, image recognition, and translation. It concludes that existing legal tools require effective enforcement alongside more tailored regulation and organisational risk mitigation.

  • Problem

    AI can embed or perpetuate discrimination in decisions with far-reaching effects, while some unfair differentiation may escape laws focused on protected characteristics.

  • Method

    The report uses a literature review to examine discrimination risks, legal safeguards, and recommendations for organisations and oversight bodies.

  • Results

    AI-related discrimination is documented across domains, including higher offered prices for customers in areas with high Asian population density and higher image-recognition error rates for darker-skinned females.

  • Takeaways & Limitations

    Non-discrimination and data protection law can help address illegal discrimination when effectively enforced, while organisations should assess, mitigate, monitor, document, and sometimes report AI risks.

Abstract

from arXiv · show

Artificial intelligence (AI) has a huge impact on our personal lives and also on our democratic society as a whole. While AI offers vast opportunities for the benefit of people, its potential to embed and perpetuate bias and discrimination remains one of the most pressing challenges deriving from its increasing use. This new study, which was prepared by Prof. Frederik Zuiderveen Borgesius for the Anti-discrimination Department of the Council of Europe, elaborates on the risks of discrimination caused by algorithmic decision-making and other types of artificial intelligence (AI).

EXECUTIVE SUMMARY

The report examines how AI and algorithmic decision-making can advance important social goals while also embedding or producing discriminatory effects. It reviews existing safeguards and argues for stronger enforcement, additional sector-specific regulation, and continued research.

  • AI can produce discriminatory effects when systems learn from biased human decisions or when opaque recommendations are followed without adequate individual assessment.
  • AI-driven decisions can affect access to policing, benefits, jobs, credit, housing assistance, and prices, with small decisions accumulating into large effects.
  • Non-discrimination law and data protection law are the main existing legal tools, but effective enforcement of current non-discrimination norms is needed.
  • Some unfair differentiation may escape current laws when AI creates new classes unrelated to protected characteristics, especially where it reinforces social inequality.
  • Because AI uses vary across sectors and involve different values and problems, the report considers sector-specific rules rather than one general AI regime.
  • AI supports efficiency, health, economic growth, and services such as spam filtering, traffic planning, logistics, speech recognition, and disease diagnosis.

III. DISCRIMINATION RISKS

The report identifies multiple pathways through which AI decision-making can produce discrimination, from problem formulation and data practices to feature selection, proxies, and intentional use. Opaque systems can make discrimination difficult to detect, while biased data may reproduce or amplify existing inequalities.

  • AI systems are often opaque, making it difficult to determine why decisions occur or whether discrimination happened, including exclusion from targeted job advertising.
  • AI decision-making can create discrimination through target-variable and class-label definitions, training-data labelling and collection, feature selection, proxies, or intentional discriminatory use.
  • Defining target variables and class labels: Choosing target variables and class labels can disadvantage protected groups when the selected criterion reflects unequal social conditions.Using lateness as a label for a good employee could disadvantage people with immigrant backgrounds who face longer commutes.
  • Labelling training data: Biased training data can cause AI systems to reproduce discriminatory human decisions and outcomes.The medical-school example describes discrimination against women and people with an immigrant background.
  • Collecting training data: Biased sampling can over-represent groups targeted by prior policing and create feedback loops in predictive policing.
  • Selecting features: Feature selection requires simplifying people and situations into observable attributes, so the chosen features can shape discriminatory outcomes.

2. FIELDS IN WHICH AI BRINGS DISCRIMINATION RISKS

AI-related discrimination risks arise across public and private-sector systems, including criminal-risk assessment, employment, advertising, pricing, image recognition, translation, and systems that may expose existing inequality.

  • Public sector: COMPAS illustrates discriminatory risk assessment: it correctly predicts recidivism 61 percent of the time, while error patterns differ sharply between black and white defendants.Blacks were almost twice as likely to be labelled high risk without reoffending, while white defendants were more likely to be labelled low risk but later reoffend.
  • Public sector: Predictive-policing systems can reproduce and amplify existing discrimination when used to predict who will commit crime or where crime will occur.
  • Private sector: AI used to select employees or students can discriminate through biased training data; Amazon reportedly stopped a job-screening system that was not gender-neutral.
  • Private sector: Opaque advertising and pricing systems can exclude or charge groups differently: job ads may target only men, while Princeton Review offered higher prices 1.8 times as often in areas with high Asian-resident density.
  • Representational systems: Image recognition, image search, and translation systems can reproduce racial and gender stereotypes, including facial-recognition error rates up to 34.7% for darker-skinned females and male defaults in translation.
  • Detection: AI can also help discover inequality, for example by identifying gender stereotypes in collections of stock photographs that might otherwise remain hidden.

IV. LEGAL AND REGULATORY SAFEGUARDS

The report examines existing regulatory safeguards for AI-related discrimination, especially non-discrimination law and data protection law, while also considering other legal regimes and self-regulation.

  • Non-discrimination law and data protection law are the main legal regimes discussed as protections against AI-driven discrimination.
  • The safeguards chapter focuses on core principles and uses a broad-brush treatment, excluding some differences between Council of Europe member States and other issues of legal scope and enforcement.
  • The report also highlights other potentially relevant fields of law and self-regulation.

1. NON-DISCRIMINATION LAW

Non-discrimination law, particularly indirect-discrimination doctrine, can address many discriminatory effects of AI, but enforcement is difficult because the doctrine is open-ended and discrimination may remain hidden.

  • European human-rights and EU law prohibit direct and indirect discrimination based on protected characteristics such as racial origin.
  • Indirect discrimination focuses on disproportionate effects rather than discriminatory intent, although objective justification may rebut a prima facie case.
  • AI decisions can breach indirect-discrimination prohibitions when apparently neutral systems make people from a racial background pay more for goods or services.
  • Non-discrimination law is difficult to apply because indirect discrimination uses open-ended standards and requires evidence that a neutral practice disproportionately affects a protected group.
  • AI opacity can conceal discriminatory effects: a loan applicant may not know why an automated denial occurred or whether women are disproportionately denied loans.
  • The report concludes that non-discrimination law prohibits many discriminatory AI effects, but enforcement remains difficult and the law has weaknesses.

2. DATA PROTECTION LAW

Data protection law protects fairness and fundamental rights through principles governing personal-data processing, impact assessments, transparency, and oversight. The GDPR also regulates certain fully automated decisions, although enforcement and practical effects remain uncertain.

  • Data protection law aims to defend fairness and fundamental rights, including privacy and non-discrimination, through obligations on data controllers and rights for data subjects.
  • Its core principles require personal data to be processed lawfully, fairly, transparently, for specified purposes, and only as necessary, accurate, secure, and accountable.
  • Transparency duties can help researchers, journalists, and supervisory authorities identify potentially discriminatory AI processing and investigate it.
  • A DPIA is required for many AI systems that make decisions about people, and it must consider the risk of unfair or illegal discrimination.
  • The GDPR contains specific rules for automated individual decision-making, including an in-principle prohibition on certain solely automated decisions with legal or similarly significant effects.
  • The practical effect of automated-decision provisions remains uncertain, while data protection authorities face limited resources and, in many cases, weak sanctioning powers.

3. OTHER REGULATION

Other regulatory fields, sector-specific rules, and self-regulatory principles may help address AI-driven discrimination. However, abstract and non-binding principles cannot replace enforceable legal regulation protecting human rights.

  • Consumer, competition, administrative, and criminal law could help address different AI-related harms, including manipulative advertising and discrimination by monopolistic companies.
  • Several Council of Europe and European Union initiatives were considering regulatory measures, guidance, and standards for AI and automated data processing.
  • Existing rules for algorithmic trading illustrate how sectoral regulation can require compliance staff to understand algorithms and firms to establish governance arrangements.
  • Self-regulatory principles can inspire lawmakers and mitigate discrimination risks, but many are abstract and lack detailed guidance.
  • Human-rights protection cannot be left to self-regulation or soft law because such arrangements are non-binding and may distract from hard legal regulation.

V. RECOMMENDATIONS

The recommendations address how to mitigate discriminatory AI across fields, safeguards, organisational practice, and institutional oversight. They target organisations using AI, Equality Bodies, and human-rights monitoring bodies.

  • The report asks where algorithmic decision-making and AI create, or may create, discriminatory effects.
  • It develops recommendations for organisations, Equality Bodies, and human-rights monitoring bodies on mitigating discriminatory AI risks.

1. ORGANISATIONS USING AI

The report recommends that organisations prevent discriminatory AI through education, expertise, planning, multidisciplinary risk assessment, mitigation, monitoring, and suitable transparency. Public-sector systems require additional accountability, including possible sunset clauses.

  • Organisations should use education, technical and legal expertise, and careful planning to prevent discrimination in AI projects.
  • Education should make managers, lawyers, and computer scientists aware of accidental AI-driven discrimination risks.
  • Risk assessment and mitigation should involve multiple disciplines, document decisions, monitor implementation, and often report to the public or an oversight body.
  • Assessing discrimination risks requires active support for developers and sufficient time and money, because computer scientists working alone may face difficult value-laden choices.
  • Organisations should consider impact assessments, stakeholder participation, ethics committees, ongoing monitoring, and public or controlled access to information about AI systems.
  • Public-sector bodies should consider sunset clauses requiring evaluation after a defined period and possible abolition when results disappoint or risks are too great.

2. EQUALITY BODIES AND HUMAN RIGHTS MONITORING BODIES

The report recommends that Equality Bodies and human rights monitoring bodies build AI expertise, collaborate across institutions, and intervene early in AI-related decisions. Their work should combine consultation, education, technical scrutiny, public engagement, research, litigation, and regulatory advocacy.

  • Education and technical expertise: Education and awareness efforts should explain AI’s risks and benefits to officials, organisations, students, and the public without making individuals responsible for defending themselves against discrimination.The report recommends human-rights and ethics teaching in relevant university programmes and cautions against responsibilisation.
  • Education and technical expertise: Equality Bodies and monitoring bodies should obtain technical expertise, including by involving computer scientists, because they can identify certain AI risks better than lawyers alone.The report also notes that non-specialist computer scientists may conduct some investigations into AI-driven discrimination.
  • Prior consultation with Equality Bodies: Equality Bodies could require public bodies to consult them on AI projects, assess training-data bias, secure legal and technical expertise, and monitor risks regularly.Depending on the national situation, consultation could be suggested rather than required.
  • Prior consultation with Equality Bodies: Equality Bodies and monitoring bodies could develop a human rights and AI impact assessment method with stakeholders from different disciplines.The report says no specific AI impact-assessment method currently exists and suggests drawing inspiration from privacy and data-protection assessments.
  • Engage in public procurement processes: They should engage early in public-sector AI procurement to ensure systems address discrimination concerns, remain auditable, and include appropriate safeguards.Early involvement can build these requirements into systems before procurement is completed.
  • Cooperation and engagement: They should cooperate with Data Protection Authorities, academics, civil society, consumer groups, and digital-rights organisations to exchange expertise and improve scrutiny of discriminatory AI.The report highlights knowledge sharing, commissioned research, events, and possible strategic litigation or regulatory advocacy.

VI. IMPROVING REGULATION

The report argues that current law has weaknesses for AI-driven discrimination and that additional regulation is probably needed. It frames the regulatory task around enforcement, possible legal amendments, and the challenges of governing fast-changing technology.

  • VI. IMPROVING REGULATION: Current law has weaknesses when applied to AI-driven discrimination, so additional regulation is probably needed to protect against illegal discrimination and unfair differentiation.The section distinguishes improving enforcement from considering whether legal norms themselves should be amended.
  • VI. IMPROVING REGULATION: Regulatory proposals must address both existing non-discrimination norms and new forms of unfair differentiation enabled by AI decision-making.The report presents these as separate but related regulatory concerns.

1. REGULATION AND FAST-DEVELOPING TECHNOLOGY

Because AI and other technologies develop faster than statutes and treaties can be adopted, the report favors adaptable combinations of legal rules and regulatory guidance. It presents technology-neutral principles alongside more specific, amendable requirements as one possible model, while stressing democratic safeguards.

  • 1. REGULATION AND FAST-DEVELOPING TECHNOLOGY: Technology regulation is difficult because statutes and treaties may take years or decades to adopt while technology, markets, and society develop quickly.The report treats this as a general challenge of regulating fast-developing technology, not only AI.
  • 1. REGULATION AND FAST-DEVELOPING TECHNOLOGY: Policy-makers can combine technology-neutral statutes with broad principles and more specific regulator guidelines that are easier to amend.The report identifies this combination as a way to address changing technologies without revising statutes for every new development.
  • 1. REGULATION AND FAST-DEVELOPING TECHNOLOGY: Data protection law illustrates a combined approach through broadly phrased statutory provisions supplemented by interpretative guidance for particular situations.Examples mentioned include CCTV, workplace monitoring, profiling, big data, policing, and automated decision-making.
  • 1. REGULATION AND FAST-DEVELOPING TECHNOLOGY: The report cautions that data protection law should not automatically be treated as best practice for rapidly changing technological fields.It explicitly notes that data protection law has plenty to criticise.
  • 1. REGULATION AND FAST-DEVELOPING TECHNOLOGY: Co-regulation and other mixtures of statutory, regulatory, and self-regulatory rules are presented as additional possibilities for governing AI.The report says different types of rules can be combined, while entities issuing rules or guidelines require democratic legitimacy and checks and balances.

2. ENFORCEMENT

The report focuses enforcement on making AI systems more transparent, auditable, and investigable, while strengthening institutional capacity. It also recognizes that opacity, trade-secret protection, and limited expertise can obstruct accountability.

  • 2. ENFORCEMENT: Improving enforcement of current norms remains necessary even though AI can also create new forms of discrimination and differentiation that largely escape existing laws.The report presents stronger enforcement and further legal development as complementary responses.
  • 2. ENFORCEMENT: AI systems’ black-box character makes discrimination harder to discover, so regulation could require explainability and auditability, especially in public-sector systems.Similar requirements could be considered for private-sector decisions, and interpretability already exists for some algorithmic-trading systems.
  • 2. ENFORCEMENT: Public bodies could release underlying code where useful, but code audits work best when investigators have a defined question and standards for evaluating system behaviour or performance.The report also notes that complex systems may need to be examined in real-world use with real users and data.
  • 2. ENFORCEMENT: Trade secrets, intellectual-property rights, and company terms can hinder investigations by regulators, journalists, and academics, creating a case for carefully designed research-access rules.Any disclosure requirements would need to balance public-interest investigation against competing interests.
  • 2. ENFORCEMENT: The law could require risk assessment and oversight for public-sector AI, and potentially for private-sector systems used in decisions about insurance, credit, or employment.The report says further research is needed on who should conduct such audits and emphasizes the expertise required.
  • Investigation and enforcement powers: Member States should provide Equality Bodies and Data Protection Authorities with adequate funding and sufficient investigation and enforcement powers, because transparency alone may not produce accountability.The report treats institutional capacity as a condition for effective enforcement.

3. REGULATING NEW TYPES OF DIFFERENTIATION

AI can create unfair differentiation that current non-discrimination law does not address, including newly invented classes, financial status, and some predictive errors. The report therefore supports additional but sector-specific regulation, informed by each sector’s values, risks, and legal principles.

  • Legal gaps: Non-discrimination law leaves gaps because it generally covers protected characteristics, while AI may differentiate through newly invented classes or financial status.Data protection law can fill some, but not all, of these gaps.
  • Unfair differentiation: AI-driven price differentiation can reinforce social inequality when poorer consumers pay more, yet financial status is not generally a protected characteristic.Examples include higher online prices for rural consumers and higher insurance premiums for residents of poorer neighbourhoods.
  • Errors and harms: Predictive models can incorrectly deny loans to individuals who do not fit their group profile, and AI systems may make more errors for minority groups.A model identifying 80% late payers in a postal code would also deny loans to the 20% who pay on time.
  • Regulatory approach: Additional regulation should be considered because AI decisions outside non-discrimination law can remain unfair, but general AI rules would not fit its varied uses.The report contrasts predictive policing with chess software and recommends assessing risks within particular sectors.
  • Regulatory approach: Sector-specific rules should reflect each field’s underlying values, such as the presumption of innocence in criminal law and contractual freedom in consumer transactions.The proposed assessment asks which rules and rationales apply, how AI is used, what risks arise, and whether legal amendments are needed.
  • Evidence needs: More empirical research, policy debate, and interdisciplinary work are needed because the scale and uses of AI-driven decision-making remain unclear.The report concludes that further research should inform decisions about whether new rules are needed and which rules are appropriate.

VII. CONCLUSION

AI offers important societal benefits but can also produce opaque and discriminatory decisions with far-reaching effects in public and private settings. The report concludes that current legal safeguards should be better enforced, while new protections should be developed through sector-specific regulation, research, and debate.

  • Benefits and risks: AI supports goals including efficiency, health, and economic growth, but its decisions can be opaque and discriminatory.Discriminatory effects may arise when systems learn from data reflecting biased human decisions.
  • Scope of impact: AI-driven decisions can affect policing, sentencing, benefits, employment, housing, and credit, while many smaller decisions can accumulate into major effects.These consequences occur across both public and private sectors.
  • Existing safeguards: Non-discrimination law and data protection law are the main existing legal instruments for mitigating AI-driven discrimination, but enforcement of current norms should improve.The report specifically calls on member States, monitoring bodies, and Equality Bodies to pursue better enforcement.
  • Regulatory need: New types of unfair differentiation may escape current laws, especially when AI uses classes unrelated to protected characteristics and reinforces social inequality.The report identifies this gap as a reason to consider additional regulation.
  • Regulatory approach: Because AI uses and risks vary across sectors, the report recommends sector-specific rules rather than general AI regulation.It calls for more debate and interdisciplinary research to guide those choices.
Loading 2510.13465v1…