Source-linked AI summary
Strengthening legal protection against discrimination by algorithms and artificial intelligence
Frederik J. Zuiderveen Borgesius
TL;DR
Algorithmic decision-making can threaten non-discrimination rights, and the paper evaluates the adequacy of European legal protection against that risk. It examines existing non-discrimination and data protection law, finding that both can help but have serious weaknesses, and proposes improved enforcement and sector-specific regulation.
Problem
Algorithmic decision-making can produce discriminatory outcomes, raising the question of whether European law adequately protects people against algorithmic discrimination.
Method
The paper evaluates European non-discrimination and data protection law, assesses enforcement and possible new rules, and develops a sector-specific regulatory approach.
Results
Non-discrimination law, especially indirect discrimination, prohibits many discriminatory algorithmic effects, while data protection law can also help protect people despite serious weaknesses in both instruments.
Takeaways & Limitations
Effective enforcement of non-discrimination and data protection law could help protect people, while algorithmic regulation should be considered sector by sector.
Takeaways & Limitations
Data protection law does not cover predictive models that do not relate to identifiable persons, and enforcement remains constrained by compliance deficits and overburdened authorities.
Abstract
from arXiv · showhide
Algorithmic decision-making and other types of artificial intelligence (AI) can be used to predict who will commit crime, who will be a good employee, who will default on a loan, etc. However, algorithmic decision-making can also threaten human rights, such as the right to non-discrimination. The paper evaluates current legal protection in Europe against discriminatory algorithmic decisions. The paper shows that non-discrimination law, in particular through the concept of indirect discrimination, prohibits many types of algorithmic discrimination. Data protection law could also help to defend people against discrimination. Proper enforcement of non-discrimination law and data protection law could help to protect people. However, the paper shows that both legal instruments have severe weaknesses when applied to artificial intelligence. The paper suggests how enforcement of current rules can be improved. The paper also explores whether additional rules are needed. The paper argues for sector-specific - rather than general - rules, and outlines an approach to regulate algorithmic decision-making.
1. Introduction
Algorithmic decision-making is widespread across sectors but can produce unfair and illegal discrimination. The paper evaluates European legal protection against algorithmic discrimination and considers how it could be improved.
- Algorithmic systems are used across sectors including healthcare, logistics, traffic planning, and speech recognition, but can also produce unfair and illegal discrimination.
- The paper asks which European legal protections against algorithmic discrimination exist, what their limitations are, and how protection could be improved.
- It focuses on non-discrimination law and data protection law as the two main legal instruments for defending people against algorithmic discrimination.
- The paper contributes analysis of European Convention on Human Rights norms, assesses data protection law, and proposes sector-specific regulation.
- The analysis covers overarching European rules across the Council of Europe and excludes national rules, privacy, and freedom of expression.
- The paper examines discrimination risks, current legal protection, improved enforcement, possible amendments, and approaches to regulating algorithmic decision-making.
2. Algorithmic decision-making and artificial intelligence
The paper treats algorithmic decision-making as the process by which an algorithm produces an output, including fully automated and human-assisted decisions. It uses AI and related terms broadly for readability, while highlighting machine learning as an automated process that discovers data patterns for prediction or estimation.
- An algorithm is an abstract, formalized computational procedure, and algorithmic decision-making is the process by which it produces an output.
- Algorithmic decisions may be fully automated, such as spam filtering, or partly automated, such as bank lending decisions assisted by credit assessments.
- Discrimination risks can be similar in fully and partly automated decisions because people may follow computer recommendations that seem rational or infallible.
- Artificial intelligence is described as the study of designing intelligent agents, while machine learning discovers correlations or patterns in datasets to make predictions or estimates.
- For readability, the paper uses algorithmic decision-making, AI, and related terms without specifying whether they refer to machine learning or another technology.
3. Discrimination risks of algorithmic decision-making
Algorithmic decision-making can produce discriminatory effects in public and private sectors through biased data, feedback loops, opaque systems, and stereotyped outputs. These systems are not inherently discriminatory and can also help reveal existing discrimination.
- Algorithmic systems can create discriminatory effects when they learn from discriminatory human decisions or biased training data.
- COMPAS correctly predicts recidivism 61 percent of the time, yet blacks are almost twice as likely as whites to be labeled high risk without actually reoffending.
- Black defendants were twice as likely as white defendants to be misclassified as high risk for violent recidivism, while white violent recidivists were 63 percent more likely to be labeled low risk.
- Predictive policing can amplify existing discrimination through feedback loops when increased police attention generates more recorded crime and further policing.
- In private-sector selection, biased training data can reproduce discriminatory methods, as illustrated by an algorithmic system used to select medical students.
- Targeted advertising and image-search systems can produce gendered or racialized effects, while opaque decision-making makes discrimination and its causes harder to discover.
- Algorithmic decision-making can discriminate, but algorithms are not inherently discriminatory and may perform better than human decision-makers.
- Algorithmic systems can also help identify existing discrimination that might otherwise remain hidden.
4. Current regulation
European non-discrimination law and data protection law offer relevant protection against algorithmic discrimination, but both have important weaknesses in practice. Non-discrimination law can address discriminatory effects, while data protection law provides transparency-related tools but remains limited in scope, enforcement, and application.
- Non-discrimination law: Indirect discrimination can apply when a seemingly neutral algorithm disproportionately disadvantages people with a protected characteristic.Algorithmic systems may also use proxies to intentionally discriminate on the basis of characteristics such as ethnicity.
- Weaknesses of non-discrimination law: Black-box systems make enforcement difficult because affected people and even system developers may lack access to the logic behind algorithmic decisions.Trade or state secrets can further restrict access to relevant information.
- Weaknesses of non-discrimination law: Indirect discrimination remains difficult to apply because its open-ended standards depend on whether an alleged discriminator can establish an objective justification.Whether a practice breaches the prohibition depends on all the circumstances of the case.
- Non-discrimination law: Non-discrimination law prohibits many discriminatory effects of algorithmic decision-making, especially through indirect discrimination.The prohibition focuses on discriminatory effects rather than the alleged discriminator’s intention.
- Data protection law: Data protection law can reduce information asymmetry by requiring organisations to provide transparency about personal-data use throughout algorithmic decision-making.Its effects on automated-decision safeguards remain uncertain, although the rules have prompted discussion about explaining algorithmic decisions.
- Weaknesses of data protection law: Data protection law has substantial limits because enforcement is uncertain, predictive models may fall outside personal-data rules, and open norms can be difficult to apply.Its special-category data rules can also make it difficult to collect information needed to assess and mitigate discrimination, while explanations may be difficult or unhelpful in complex systems.
- Data protection law: More openness and explanation are desirable, but the effects of Convention 108 and the GDPR remain too early to assess conclusively.The paper characterises data protection law as largely untested but potentially useful against illegal discrimination.
- Other regulation: Self-regulation may help mitigate discrimination, but its non-binding nature, weak enforcement, and vague guidance mean human-rights protection cannot be left to voluntary measures.The paper warns that ethical principles should not distract legislators from considering new laws.
5. Improving regulation
The paper recommends improving enforcement of existing non-discrimination norms while considering targeted additional regulation for algorithmic decision-making. It argues that new rules should be sector-specific because risks, values, and legal principles vary across applications.
- Enforcement: Existing non-discrimination norms may need more effective enforcement in algorithmic decision-making.
- Transparency and auditing: Black-box systems hinder the discovery of discrimination, making transparency, auditing, and explainability important regulatory goals.
- Transparency and auditing: Code audits can clarify system behavior when regulators have a defined question and standards, but real-world testing may be necessary for complex programs.
- Transparency and auditing: Trade-secret and intellectual-property protections can obstruct investigation, so research exceptions and disclosure duties must balance transparency against commercial and privacy interests.
- Institutional enforcement: Public-sector systems, and potentially certain private-sector systems, could be required to enable oversight, undergo risk assessment, and receive expert auditing.
- Additional regulation: Additional rules should address risks that current law misses, including differentiation based on newly invented classes, social inequality, and inaccurate predictions.
- Sector-specific regulation: Because algorithmic risks and relevant legal principles differ across sectors, policymakers should assess each sector separately before deciding whether new rules are needed.
6. Concluding thoughts
The conclusion presents algorithmic decision-making as both beneficial and capable of producing serious effects across public and private domains. It finds that existing legal instruments help but leave gaps, supporting additional sector-specific regulation.
- Algorithmic decision-making can reproduce discrimination learned from discriminatory human decisions while offering substantial benefits.
- Algorithmic decisions can affect policing, sentencing, benefits, employment, housing, and credit, with individually small effects potentially accumulating into major consequences.
- Non-discrimination and data protection law can protect people when effectively enforced, but some algorithmic differentiation evades current legal protections.
- The paper therefore supports additional regulation, but argues that rules should be sector-specific rather than general because risks and relevant norms differ across sectors.
ORCID
Frederik J. Zuiderveen Borgesius is identified by an ORCID record.
- The paper identifies Frederik J. Zuiderveen Borgesius as its author.
- The ORCID entry provides a researcher identifier link for the author.
- The listed identifier is associated with the paper’s author information.
Notes on contributor
The contributor note describes Frederik J. Zuiderveen Borgesius as a professor whose research focuses on privacy, discrimination, and fundamental rights in new technologies.
- Frederik J. Zuiderveen Borgesius is Professor of ICT and Law at Radboud University Nijmegen.
- He is affiliated with Radboud University’s interdisciplinary hub on Security, Privacy, and Data Governance.
- His research interests include privacy, discrimination, and other fundamental rights, especially in new-technology contexts.