PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrimination, artificial intelligence, and algorithmic decision-making

Frederik Zuiderveen Borgesius

arXiv:2510.13465v1cs.CY

TL;DR

AI-driven decisions can have far-reaching effects, yet existing non-discrimination statutes may not cover unfair differentiation based on newly invented classes. This report addresses AI discrimination risks and legal tools for mitigating them, highlighting unfair differentiation that current laws may miss.

  • Problem

    AI decisions can have far-reaching effects, while existing non-discrimination statutes may not cover unfair differentiation based on newly invented classes.

    Source: EXECUTIVE SUMMARY · p. 5

  • Method

    The report addresses risks of AI-driven discrimination and considers non-discrimination and data-protection law as mitigation tools.

    Source: EXECUTIVE SUMMARY · p. 5

  • Results

    AI can enable unfair differentiation based on newly invented classes that current non-discrimination statutes do not cover.

    Source: EXECUTIVE SUMMARY · p. 5

  • Takeaways & Limitations

    The report calls for better enforcement of current norms and consideration of sector-specific rules rather than general AI regulation.

    Source: EXECUTIVE SUMMARY · p. 5

Abstract

from arXiv · show

Artificial intelligence (AI) has a huge impact on our personal lives and also on our democratic society as a whole. While AI offers vast opportunities for the benefit of people, its potential to embed and perpetuate bias and discrimination remains one of the most pressing challenges deriving from its increasing use. This new study, which was prepared by Prof. Frederik Zuiderveen Borgesius for the Anti-discrimination Department of the Council of Europe, elaborates on the risks of discrimination caused by algorithmic decision-making and other types of artificial intelligence (AI).

EXECUTIVE SUMMARY · I. INTRODUCTION · II. ARTIFICIAL INTELLIGENCE AND ALGORITHMIC DECISION-MAKING

AI can advance efficiency, health, and economic growth, yet algorithmic decision-making can also cause discrimination, including through biased data or decisions. The report surveys these risks, existing safeguards, and possible responses, while introducing key AI concepts.

EXECUTIVE SUMMARY

  • AI can support important goals but may produce unfair or illegal discrimination, including when systems learn from biased human decisions.The report notes that AI also serves uses such as spam filtering, traffic planning, logistics, speech recognition, and disease diagnosis.Source: EXECUTIVE SUMMARY · p. 5, §I · p. 7
  • AI-driven decisions can affect people in public and private services, while many small decisions, such as discriminatory pricing, can cumulatively burden groups.Examples include predictive policing, benefit eligibility, job selection, and credit decisions or interest rates.Source: EXECUTIVE SUMMARY · p. 5
  • Non-discrimination and data protection law can help if enforced, but the report calls for considering additional, sector-specific AI rules and further debate.It recommends better enforcement by Council of Europe member States, human rights monitoring bodies, and Equality Bodies; it cautions against one general rule set for varied AI uses.Source: EXECUTIVE SUMMARY · p. 5
  • AI may create unfair classes unrelated to protected characteristics, which current non-discrimination statutes may not cover even when they reinforce social inequality.The report therefore identifies a potential gap between existing protected-characteristic rules and unfair algorithmic differentiation.Source: EXECUTIVE SUMMARY · p. 5

I. INTRODUCTION

  • The introduction frames the report as a literature-review quick scan of discrimination risks, safeguards, and mitigation, excluding other AI concerns such as privacy and autonomous weapons.Its questions cover fields at risk, existing or considered safeguards and redress, recommendations for organisations and oversight bodies, and legal, regulatory, or self-regulatory action.Source: §I · p. 7

II. ARTIFICIAL INTELLIGENCE AND ALGORITHMIC DECISION-MAKING

  • The report notes that AI and algorithmic decision-making lack consensus definitions, and distinguishes an algorithmic procedure from its output, or decision.It introduces related terms including data mining, big data, and profiling, which involves automated data processing to develop profiles for decisions about people.Source: §II · pp. 8–9
  • Risks can persist in fully or partly automated decisions because people may blindly follow algorithmic recommendations or use them to minimise their own responsibility.The report describes this tendency to trust or follow computer advice as automation bias.Source: §II · p. 8
  • Expert systems relied on costly specialist-provided answers and could fail to fit messy reality, whereas machine learning learns patterns from examples and data to produce predictions.The report describes machine learning as a prominent AI type and data mining as a type of machine learning that discovers patterns in large data sets.Source: §II · pp. 8–9

Terminology in this report · III. DISCRIMINATION RISKS · 1. HOW AI CAN LEAD TO DISCRIMINATION

The report treats AI decisions as the result of organisational choices and explains that opaque decisions can make discrimination difficult to assess. It identifies six ways AI decision-making can produce discriminatory effects.

Terminology in this report

  • AI systems are not autonomous actors: organisations decide to use them, though design choices may predate deployment and their effects may emerge later.For readability, the report uses “AI” broadly, without specifying a particular technology, and sometimes speaks of organisations as if they were single actors.Source: Terminology in this report · pp. 9–10

III. DISCRIMINATION RISKS

  • The section asks where algorithmic decision-making and other AI create discriminatory effects or could do so in the foreseeable future.Source: §III · p. 10

1. HOW AI CAN LEAD TO DISCRIMINATION

  • Choosing a target variable or class label can disadvantage protected groups; defining “good” employees by punctuality, for example, may penalise immigrant-background workers facing longer commutes.Target variables specify what data miners seek, while class labels divide their possible values into categories.Source: §1 · pp. 10–11
  • When training labels reflect biased human decisions, AI can reproduce that bias, as a medical-school admissions program discriminated against women and people with immigrant backgrounds.The program used earlier admissions files and replicated the selection patterns in those records.Source: §1 · p. 11
  • Biased sampling can encode unequal policing or undercount poor communities, and crime predictions can intensify surveillance through a feedback loop.Street Bump data could under-represent faulty roads in poor neighbourhoods when fewer poor residents use smartphones, resulting in fewer repairs.Source: §1 · pp. 11–12
  • Feature choices can disadvantage groups, while apparently neutral proxies such as postcode may encode protected traits; removing such variables can sacrifice accuracy.A postcode correlated with racial origin could drive loan denials, and the report notes that preventing systematic disadvantage may require reducing overall decision accuracy.Source: §1 · pp. 12–13
  • Organisations may also use AI intentionally to discriminate, including by skewing training data or choosing proxies that conceal discriminatory aims.Proxy-based discrimination can be harder to detect than openly discriminatory decisions.Source: §1 · p. 13
  • The report identifies six routes: defining target variables and class labels, biased labelling or collection of training data, feature selection, proxies, and intentional discrimination.AI can also produce other unfair differentiation or errors, which the report returns to in chapter VI.Source: §1 · pp. 10, 14

2. FIELDS IN WHICH AI BRINGS DISCRIMINATION RISKS

This section gives examples of fields where AI decision-making has led or could lead to discrimination.

2. FIELDS IN WHICH AI BRINGS DISCRIMINATION RISKS

  • AI decision-making has led, or could lead, to discrimination in multiple fields.The section provides examples of those fields.Source: §2 · p. 14

Police, crime prevention · Selection of employees and students

The examples show that AI can reproduce or amplify discrimination in policing and selection, with unequal risk-score errors and differences in who sees ads or pays higher prices.

Police, crime prevention

  • COMPAS predicts recidivism correctly 61% of the time, but blacks are almost twice as likely as whites to be falsely labeled high-risk, while whites more often receive false low-risk labels.COMPAS does not use racial origin or skin colour as an input.Source: Police, crime prevention · p. 14
  • Black defendants were twice as likely as white defendants to be misclassified as high-risk for violent recidivism, while white violent recidivists were 63% more likely to be misclassified as low-risk.Source: Police, crime prevention · p. 14
  • Fairness standards can conflict: calibration means an 80%-risk group has 80% recidivists, but equalized error rates are mathematically impossible when underlying recidivism propensity differs.ProPublica and Northpointe disagreed mainly over which fairness standard should be used to assess COMPAS.Source: Police, crime prevention · pp. 14–15
  • Predictive policing systems make automated predictions about who will commit crime or when and where crime will occur, and can reproduce or amplify existing discrimination.Source: Police, crime prevention · p. 15

Selection of employees and students

  • Biased training data can discriminate in selection: a UK medical-school system was an example, and Amazon stopped an applicant screener that learned male candidates were preferable.Reuters reported that the system did not rate candidates for technical jobs in a gender-neutral way.Source: Selection of employees and students · p. 15
  • Google showed arrest-record-suggestive ads more often for African-American-sounding names and high-salary coaching ads more often to simulated men; opaque ad systems can hide exclusion and its cause.Researchers said their limited visibility into the ad ecosystem prevented them from determining who caused the gender disparity.Source: Selection of employees and students · pp. 15–16
  • Facebook enabled targeting by sensitive interests and racial exclusion, while job ads could be age-limited; Spanish researchers found 73% of EU users labeled with sensitive interests.Examples of sensitive interests included Islam, reproductive health, and homosexuality.Source: Selection of employees and students · p. 16
  • Customers in areas with a high density of Asian residents were 1.8 times as likely to be offered higher prices, regardless of income.Princeton Review charged different prices in different areas for online tutoring, ranging from 6600 to 8400 dollars.Source: Selection of employees and students · p. 16

Image search and analysis … IV. LEGAL AND REGULATORY SAFEGUARDS

AI systems can reflect discrimination present in data and society, producing biased image and translation outputs or misrecognizing faces; they may also help reveal existing inequality. The legal safeguards chapter identifies non-discrimination and data protection law as its main subjects.

Image search and analysis

  • A 2016 Google Images search for “three black teenagers” returned mugshots, while “three white kids” mostly returned pictures of happy white children.Google said image results reflect web content and how frequently images appear and are described; even if they reflect society rather than AI fault, they could influence beliefs.Source: Image search and analysis · pp. 16–17
  • Image searches for occupations slightly exaggerated gender stereotypes, portrayed the minority gender less professionally, and slightly under-represented women.These findings concern image search results for occupations.Source: Image search and analysis · p. 17
  • Darker-skinned females had error rates of up to 34.7%, compared with a maximum 0.8% for lighter-skinned males, in face-analysis systems.Examples include facial-tracking software failing to recognize dark-coloured faces and an app labelling an African-American couple as “gorillas”; the errors may stem from training on pictures of white men.Source: Image search and analysis · p. 17

Translation tools

  • Translating “He is a doctor. She is a nurse” through Turkish and back into English yielded “She is a nurse. He is a doctor.”The Turkish sentences are gender-neutral, as Turkish does not distinguish “he” from “she.”Source: Translation tools · p. 17
  • Tests of Google Translate across twelve gender-neutral languages found strong male defaults, exaggerated for fields associated with gender stereotypes such as STEM jobs.The authors concluded that AI-driven translation can reflect existing gender inequality and could influence people’s ideas.Source: Translation tools · p. 17

Nuancing the risks

  • AI systems do not necessarily perform worse than humans, and may help uncover existing inequality, such as gender stereotypes in a stock-photo collection.Some AI discrimination arises from data reflecting human discrimination; whether AI is compared with real-world or hypothetical discrimination-free human decisions therefore matters.Source: Nuancing the risks · p. 18

IV. LEGAL AND REGULATORY SAFEGUARDS

  • Non-discrimination law and data protection law are the main legal regimes the chapter discusses as potential protections against AI-driven discrimination.It also highlights other relevant fields of law and self-regulation, while excluding cross-state regulatory differences, territorial scope, and enforcement against organisations in other States.Source: §IV · p. 18

1. NON-DISCRIMINATION LAW

Non-discrimination law can reach AI discrimination through rules against direct and indirect discrimination, with indirect-discrimination analysis focused on effects rather than intent. Its protection is limited by difficult proof, case-specific justifications, opaque AI decisions, and gaps in the protected characteristics covered by law.

1. NON-DISCRIMINATION LAW

  • European human-rights law prohibits discrimination on specified and other-status grounds, but the European Convention does not directly regulate discrimination in the private sector.The Convention has some horizontal effect, but does not directly regulate private-sector discrimination.Source: §1 · p. 18
  • Direct discrimination concerns treatment based on a protected characteristic, while indirect discrimination arises when a neutral practice disproportionately disadvantages a protected group, without necessarily requiring discriminatory intent.The paper notes that indirect discrimination is called “disparate impact” in the United States.Source: §1 · pp. 18–19
  • An apparently neutral practice may be justified by a legitimate aim if its means are appropriate and necessary, but assessing justification requires a nuanced, case-specific proportionality test.Whether the practice breaches the prohibition is therefore not always clear.Source: §1 · pp. 19–20
  • AI-driven discrimination can be challenged under non-discrimination law, particularly for indirect effects, but enforcement is difficult and accidental indirect discrimination probably occurs more often than intentional discrimination.The paper concludes that indirect-discrimination rules prohibit many discriminatory effects of AI, while enforcement remains difficult.Source: §1 · pp. 19–20
  • Showing prima facie indirect discrimination requires proof that a neutral practice disproportionately affects a protected group, often through statistics, but opaque AI decisions can hide those effects from customers.A customer denied a loan may not know why the AI system rejected the application or whether it disproportionately denies loans to women.Source: §1 · pp. 19–20
  • Non-discrimination statutes focus on protected characteristics such as race, gender, or sexual orientation, leaving some apparently unfair AI-driven differentiation outside their scope.The paper identifies these exclusions as gaps in non-discrimination law.Source: §1 · p. 20

2. DATA PROTECTION LAW

Data protection law aims to protect fairness and fundamental rights while regulating personal-data processing. Its transparency duties, impact assessments, and enforcement powers can help address discrimination risks in AI decision-making.

2. DATA PROTECTION LAW

  • Data protection law protects fairness and fundamental rights, including privacy and non-discrimination, by granting rights to data subjects and imposing obligations on data controllers.110, 111, 112Source: §2 · p. 21
  • Personal data must be processed lawfully, fairly, and transparently, collected for specified purposes, and limited to what is necessary for those purposes.Source: §2 · p. 21
  • Transparency requirements mean organisations must disclose stages of AI decision-making that involve personal data, enabling researchers, journalists, and authorities to identify possible discriminatory effects and investigate.118Although most people do not read privacy notices, they may still help researchers, journalists, and supervisory authorities scrutinise processing practices.Source: §2 · p. 21
  • For many AI systems making decisions about people, a data protection impact assessment (DPIA) is required, and it must consider the risk of unfair or illegal discrimination.123, 124A DPIA supports informed decisions about whether to deploy an initiative and under what conditions; the GDPR requires one for practices likely to pose high risks to people’s rights and freedoms, and in some cases for fully automated decisions with legal or similar effects.121, 122Source: §2 · p. 22
  • Independent Data Protection Authorities can investigate controllers through audits, premises access, and orders for information and access to data-processing systems.125, 126, 127Source: §2 · p. 22

Rules on automated decisions

The section describes GDPR rules restricting certain fully automated decisions with significant effects, alongside safeguards and transparency duties, and notes limits on the rules’ reach.

Rules on automated decisions

  • The GDPR in-principle prohibits decisions based solely on automated processing, including profiling, when they produce legal or similarly significant effects for an individual.Four conditions must be met: there must be a decision, based solely on automated processing, with legal or similarly significant effects.Source: Rules on automated decisions · p. 23
  • The prohibition has exceptions for explicit consent, contractual necessity, or legal authorisation; consent and contract exceptions require safeguards including human intervention and an opportunity to contest.People may also express their point of view; for example, a bank can let customers request human reconsideration of an automatically denied loan.Source: Rules on automated decisions · p. 23
  • Controllers must disclose automated decision-making and, at least in covered cases, provide meaningful information about its logic, significance, and envisaged consequences.The GDPR’s automated-decision transparency duties are additional to its general transparency requirements.Source: Rules on automated decisions · pp. 23–24
  • Recital 71 suggests an individual right to explanation of AI decisions, but whether the GDPR creates such a right remains the subject of scholarly debate.Such a right could help protect fairness.Source: Rules on automated decisions · p. 24
  • Because Article 22 covers only decisions based solely on automated processing, an AI-assisted loan denial by an employee who does not rubber-stamp the recommendation falls outside its scope.Many types of automated decisions therefore remain outside the GDPR rules.Source: Rules on automated decisions · p. 24
  • The provisions’ practical effect remains uncertain: the GDPR predecessor was a dead letter, while Convention 108’s apparently broader requested-reasoning right awaits practical clarification.Despite uncertainty about practical effect, attention to the GDPR provisions has fostered interdisciplinary discussion on explaining AI decisions.Source: Rules on automated decisions · p. 24

Caveats · 3. OTHER REGULATION · Regulation under consideration

Data protection law may help fight illegal discrimination, but its reach and practical application face significant limits, while other legal and regulatory approaches remain underexplored or under consideration. Self-regulatory AI principles may help, but cannot replace binding human-rights protections.

Caveats

  • Compliance and enforcement are limited by under-resourced Data Protection Authorities and historically weak compliance, while the GDPR’s effect and data protection law’s anti-discrimination potential remain too early to assess.The GDPR gave EU authorities new powers, and compliance appears to have improved, but its effects are largely untested.Source: Caveats · pp. 24–25
  • Data protection law applies only when personal data are processed, so predictive models not linked to identifiable people fall outside its scope until applied to an individual.A model predicting that 80% of residents of a postal code pay bills late is not itself personal data.Source: Caveats · pp. 24–25
  • Data protection law’s open norms can adapt to new technologies and varied settings, but their abstraction can make them difficult to apply.The paper describes this flexible, cross-sector approach as an omnibus model.Source: Caveats · p. 25
  • Strict rules on sensitive data create tension because discrimination audits often assume access to such data, while organisations may avoid collecting them to comply with data protection law.This leaves a continuing conflict between respecting data protection rules and gathering sensitive data to assess and mitigate discrimination.Source: Caveats · p. 25
  • Even where explanations are legally required, complex AI decisions can be difficult to explain, and explanations may shift the burden to individuals; transparency tools and explainable AI are being explored.Transparency-enhancing technologies aim to help individuals and groups understand how information is collected, analysed, and used for decisions.Source: Caveats · p. 25

3. OTHER REGULATION

  • Consumer, competition, administrative, criminal, and freedom-of-information law could help address AI-related harms, but their application in this area is largely unexplored.The report gives examples including consumer protection against manipulative AI advertising and access to information about public-sector AI systems.Source: §3 · p. 26

Regulation under consideration

  • Regulatory measures under consideration include Council of Europe work, EU AI initiatives, a proposed ePrivacy Regulation, and governance requirements for algorithmic trading that might extend to other sectors.The EU initiatives include AI Ethics Guidelines; the trading regulation requires compliance staff to understand systems and formalised governance arrangements.Source: Regulation under consideration · pp. 26–27
  • Ethical AI principles, including proposals focused on discriminatory impacts, may inspire lawmakers, but non-binding and abstract self-regulation cannot ensure human-rights protection or replace possible hard law.The paper warns that ethics initiatives should not distract from the possible need for binding legal regulation.Source: Regulation under consideration · p. 27

V. RECOMMENDATIONS · 1. ORGANISATIONS USING AI · Risk assessment and mitigation

For organisations using AI, preventing discrimination calls for education, technical and legal expertise, and careful project planning. Risk assessment and mitigation should involve multiple disciplines, be documented, and continue through implementation and use.

V. RECOMMENDATIONS

  • Organisations using AI should combine education, technical and legal expertise, and careful project planning to prevent discrimination.Source: §1 · p. 28

1. ORGANISATIONS USING AI

  • Relevant managers, lawyers, and computer scientists should learn about accidental AI discrimination, which awareness might help prevent and mitigate automation bias.Source: §1 · p. 28

Risk assessment and mitigation

  • AI projects should assess and mitigate risks through multidisciplinary input, documented processes, implementation monitoring, and, often, outward reporting.Source: Risk assessment and mitigation · p. 28
  • Defining risks is difficult: computer scientists may face value-laden decisions and struggle to communicate risks, so development needs active support, time, and money.Source: Risk assessment and mitigation · p. 28
  • Risk assessment should involve experts familiar with the relevant sector, alongside ethics committees and potentially affected individuals, civil society groups, and academics.Source: Risk assessment and mitigation · p. 29
  • Organisations can use impact assessments informed by the GDPR’s DPIA requirement and consider publishing reports, especially in the public sector.Source: Risk assessment and mitigation · p. 29
  • Because modeled phenomena and impacts can change, organisations should monitor AI systems during use and consider publishing yearly monitoring reports.Source: Risk assessment and mitigation · p. 29
  • Organisations can reduce discriminatory effects by excluding input features that correlate with protected characteristics, such as distance to work correlating with race.Source: Risk assessment and mitigation · p. 29

Public sector bodies · 2. EQUALITY BODIES AND HUMAN RIGHTS MONITORING BODIES · Education and technical expertise

The recommendations call on public bodies, Equality Bodies, and human rights monitoring bodies to address AI’s discrimination risks through oversight, expertise, education, and awareness. They emphasize transparency and evaluation of public-sector systems while cautioning against shifting responsibility for preventing discrimination onto individuals.

Public sector bodies

  • Because public bodies have extra responsibilities as powerful State actors, they should design AI systems for transparency and offer controlled access for scrutiny when public release risks privacy or gaming.Secure environments for researchers or civil society are one possible form of controlled access.Source: Public sector bodies · p. 29
  • Public bodies introducing AI systems that decide matters about people should evaluate them after a set period, such as three years, and consider abolition if results disappoint or risks outweigh benefits.Private organisations can adopt similar measures.Source: Public sector bodies · p. 30

2. EQUALITY BODIES AND HUMAN RIGHTS MONITORING BODIES

  • The recommendations address Equality Bodies and human rights monitoring bodies, including ECRI, on mitigating AI-driven discrimination.Source: §2 · p. 30

Education and technical expertise

  • Equality Bodies and human rights monitoring bodies need education on AI’s basics, promises, and risks.Source: Education and technical expertise · p. 30
  • Equality Bodies and monitoring bodies should obtain technical expertise: computer scientists can investigate discrimination through simple input-output observation, and bodies can hire them by project or permanently.“Computer scientist” is shorthand that can include data scientists or others with sufficient AI knowledge.Source: Education and technical expertise · p. 30
  • Public awareness campaigns could help public- and private-sector organisations avoid discriminatory AI systems that they may use by accident.Source: Education and technical expertise · p. 30
  • Equality Bodies and monitoring bodies could help schools and universities teach human rights and ethics alongside computer science, data science, and AI.Source: Education and technical expertise · p. 30
  • Public awareness can support inclusive debate about AI’s risks and benefits, but must not make individuals responsible for defending themselves against discrimination.Policy-makers should avoid responsibilisation while promoting awareness.Source: Education and technical expertise · pp. 30–31

Prior consultation with Equality Bodies … Litigation and regulation

Equality Bodies and human rights monitoring bodies are urged to help prevent and address AI-driven discrimination through early oversight, cooperation, and engagement with external expertise. Depending on national circumstances, they could also pursue strategic litigation and advocate regulation.

Prior consultation with Equality Bodies

  • Equality Bodies could require public bodies to consult them on planned AI decision-making projects, maintain legal and technical expertise, and regularly assess discriminatory effects.Depending on national circumstances, they could suggest these measures rather than require them.Source: Prior consultation with Equality Bodies · p. 31
  • A specific human rights and AI impact-assessment method does not yet exist, and Equality Bodies and monitoring bodies could help develop one with diverse stakeholders and disciplines.Privacy and data-protection impact assessments could provide inspiration.Source: Prior consultation with Equality Bodies · p. 31

Engage in public procurement processes

  • Equality Bodies should seek early involvement in public-sector AI procurement so discrimination concerns shape systems that are auditable and subject to safeguards.They can pursue involvement through national provisions and processes or by lobbying for greater access.Source: Engage in public procurement processes · p. 31

Cooperate with Data Protection Authorities

  • Equality Bodies should cooperate with Data Protection Authorities to exchange expertise and information about AI systems that may pose discrimination risks.A voluntary network proposed by the European Data Protection Supervisor in 2016 could offer inspiration for cooperation between Equality Bodies and human rights monitoring bodies [215]. Depending on national circumstances, cooperation with consumer-protection and competition authorities may also be useful.Source: Cooperate with Data Protection… · pp. 31–32

Cooperate with academics

  • Equality Bodies and human rights monitoring bodies should build ties with academics, whose research has uncovered discriminatory AI decisions, through events, commissioned research, or a working party.They could attend conferences where researchers and regulators meet, or organize conferences and round tables to foster contact.Source: Cooperate with academics · p. 32
  • They should engage discrimination-focused, consumer, technology-policy, and digital-rights civil society groups, and foster contact between groups with different expertise.Greater contact is useful because many groups are interested in AI-driven discrimination.Source: Cooperate with academics · p. 32

Litigation and regulation

  • Depending on national circumstances, Equality Bodies could pursue strategic litigation on AI decision-making, while they and human rights monitoring bodies could advocate regulation to mitigate AI discrimination risks.Suggestions for improving regulation are discussed in the next chapter.Source: Litigation and regulation · p. 32

VI. IMPROVING REGULATION … Improving enforcement of current non-discrimination norms

The chapter argues that current law has weaknesses in addressing AI-driven discrimination and that additional regulation is probably needed. It offers starting points for discussion, including flexible rulemaking and stronger transparency, assessment, and oversight to enforce existing non-discrimination norms.

Which types of action (legal, regulatory, self-regulatory) can reduce risks?

  • The chapter frames its proposals as preliminary discussion points, with additional regulation potentially needed to protect against illegal discrimination and unfair differentiation.Its scope includes regulating fast-developing technology, enforcing current norms, and considering whether those norms should be amended for AI decision-making.Source: Which types of action (legal,… · p. 33

1. REGULATION AND FAST-DEVELOPING TECHNOLOGY

  • Because statutes and treaties can take years or decades while technology and society change quickly, regulation must account for rapid development.The chapter notes that regulating new technologies is difficult but possible and often necessary.Source: §1 · pp. 33–34
  • A combined approach can pair technology-neutral statutes and broad principles with more specific, amendable regulator guidance, or co-regulation.Data protection law illustrates the approach: broad provisions apply across technologies, while authorities issue more concrete guidance; guidance should be evaluated and updated regularly.Source: §1 · pp. 33–34

Improving enforcement of current non-discrimination norms

  • AI opacity both presents a problem in itself and makes discrimination harder to discover.The chapter suggests requiring public-sector systems—and potentially private-sector systems—to enable auditing and explainability.Source: Improving enforcement of current… · p. 34
  • Freedom-of-information laws could be adapted to let the public obtain and examine code from some public-sector AI systems.Code inspection can help where there is a clearly defined question and standards for assessing system behaviour, but code alone may reveal little about real-world effects.Source: Improving enforcement of current… · pp. 34–35
  • Law could enable research into AI systems protected by trade secrets or intellectual property and require disclosure to researchers on request.Any such rules would need to balance transparency’s public interest against commercial, privacy, and other interests in opacity.Source: Improving enforcement of current… · p. 35
  • Public bodies could be required to use only AI systems assessed for risks and capable of oversight and auditing; similar rules could cover some private-sector decisions.Examples include eligibility decisions for insurance, credit, or jobs; who should conduct audits remains open, and organisations need considerable expertise.Source: Improving enforcement of current… · p. 35

Investigation and enforcement powers … AI can lead to errors

The paper argues that AI oversight needs adequately funded bodies with effective enforcement powers, because existing laws miss some forms of AI-driven differentiation. Such differentiation can produce unfair outcomes, reinforce social inequality, and make errors that harm individuals or minority groups.

Investigation and enforcement powers

  • Adequately funded Equality Bodies and Data Protection Authorities need investigation and enforcement powers; monitoring bodies can press for rules enforcing current norms [251].Without enforcement, transparency alone may not lead to accountability [252].Source: Investigation and enforcement powers · p. 35

3. REGULATING NEW TYPES OF DIFFERENTIATION

  • Non-discrimination law often covers only specified protected traits, leaving financial status outside its scope; data protection fills some, but not all, gaps [253, 254].AI-driven differentiation can also fall outside current laws, opening the way to new forms of discrimination.Source: §3 · p. 35, Investigation and enforcement powers · p. 35
  • Newly invented classes such as browser type can evade non-discrimination law when they are not proxies for protected traits [255].Browser-based price discrimination is technically easy but has not been evidenced; a travel site did show more expensive hotels to Apple users than PC users [256].Source: §3 · pp. 35–36, AI can lead to errors · p. 36

AI can reinforce social inequality

  • Too much AI-driven insurance risk differentiation could make coverage unaffordable and undermine risk-pooling, while causing poor consumers to pay more [257].Risk differentiation can be necessary and fair in insurance, such as when high-risk customers pay higher premiums; the concern is its excessive use.Source: AI can reinforce social inequality · p. 36
  • Online price differentiation has led consumers in poorer US areas, including rural areas, to pay more; financial status is not protected by non-discrimination law [258, 259, 260].Rural consumers may face higher prices because they have fewer nearby competitors, and poorer people tend to live in the countryside.Source: AI can reinforce social inequality · p. 36, AI can lead to errors · p. 36

AI can lead to errors

  • AI predictions can wrongly deny loans to individuals who do not fit a group profile, and systems sometimes make more errors for minority groups [261, 262].For example, denying loans to everyone in a postal code because 80% pay late also denies loans to the 20% who pay on time.Source: AI can lead to errors · pp. 36–37

New rules? … VII. CONCLUSION

AI can produce opaque, discriminatory decisions with serious or cumulative effects, while existing law may not cover every unfair outcome. The paper calls for stronger enforcement, sector-specific consideration of new rules, and more empirical, technical, legal, and public debate.

New rules?

  • Assess possible rules sector by sector by examining existing rules and rationales, AI uses and risks, and whether law needs improvement to protect underlying principles.The relevant rationales can include human rights, equality, contractual freedom, fair-trial rights, and sector-specific economic considerations; the risk of a false positive in criminal law may be more harmful than an incorrect price-discrimination decision.Source: New rules? · p. 37

Empirical and technical research

  • More empirical research is needed because the scale of AI decision-making and its frequency of discrimination or other unfair differentiation remain unclear.The paper urges Council of Europe member States to support research by human rights monitoring bodies, Equality Bodies, and academics.Source: Empirical and technical research · pp. 37–38
  • Computer science research should explore how to design AI systems that promote human rights, fairness, and accountability and how to check training data for discrimination risks.The paper recommends that countries direct some AI research funding toward fairness and human-rights risks and their mitigation.Source: Empirical and technical research · p. 38

Normative and legal research

  • Further public debate and normative and legal research should address indirect discrimination, unfair differentiation beyond current law, group privacy, the rule of law, and whether new rules or legal tweaks are needed.The questions also include defining fairness across sectors, protecting against intersectional and structural discrimination, and identifying decisions that should never be made by computers.Source: Normative and legal research · p. 38

VII. CONCLUSION

  • AI decisions can be opaque and discriminatory, with far-reaching or cumulative effects across public and private services.Examples include predictive policing, benefits, employment, housing, and credit decisions, as well as targeted advertising and price differentiation that may affect groups when aggregated.Source: §VII · p. 38
  • Current non-discrimination and data protection laws could help fight illegal discrimination if enforced effectively, so States and equality bodies should strengthen enforcement.The paper names Council of Europe member States, human rights monitoring bodies, and Equality Bodies as actors in improving enforcement of current non-discrimination norms.Source: §VII · p. 38
  • Additional regulation may be needed for unfair differentiation beyond non-discrimination law, but rules should be sector-specific rather than apply to AI generally.Different sectors involve different risks, values, and legal principles; the conclusion calls for more debate and interdisciplinary research before choices are made.Source: New rules? · p. 37, §VII · p. 39

Related papers

Loading 2510.13465v1…