Source-linked AI summary
Enslaving the Algorithm: From a "Right to an Explanation" to a "Right to Better Decisions"?
Lilian Edwards, Michael Veale
TL;DR
The paper examines whether explanation rights can effectively challenge unfair decisions made by opaque machine-learning systems. It analyzes European legal provisions and newer governance mechanisms, concluding that explanation rights are technically possible but often uncertain, burdensome, and insufficient alone. The authors therefore emphasize impact assessments, representation, judicial review, and other upstream or collective tools.
Problem
Opaque machine-learning systems can reproduce unfairness, while individual users may struggle to challenge decisions and group harms are poorly addressed by existing rights.
Method
The paper analyzes European explanation rights alongside technical explainability, impact assessments, representation, secure data access, and judicial remedies.
Results
Explanation rights are uncertain, technically difficult, and often burdensome, while broader governance mechanisms can scrutinize systems before deployment and support collective redress.
Takeaways & Limitations
Rights-based remedies should remain part of the governance toolbox but receive support from impact assessments, representation, judicial review, and design-stage scrutiny.
Takeaways & Limitations
The GDPR remedy excludes many decision-support systems, burdens individual users, and handles aggregate harms poorly.
Abstract
from arXiv · showhide
As concerns about unfairness and discrimination in "black box" machine learning systems rise, a legal "right to an explanation" has emerged as a compellingly attractive approach for challenge and redress. We outline recent debates on the limited provisions in European data protection law, and introduce and analyze newer explanation rights in French administrative law and the draft modernized Council of Europe Convention 108. While individual rights can be useful, in privacy law they have historically unreasonably burdened the average data subject. "Meaningful information" about algorithmic logics is more technically possible than commonly thought, but this exacerbates a new "transparency fallacy"---an illusion of remedy rather than anything substantively helpful. While rights-based approaches deserve a firm place in the toolbox, other forms of governance, such as impact assessments, "soft law," judicial review, and model repositories deserve more attention, alongside catalyzing agencies acting for users to control algorithmic system design.
1 Introduction
Machine-learning systems increasingly influence high-stakes everyday decisions, while historical training data and opaque methods can reproduce discrimination that affected people struggle to detect or challenge.
- ML systems increasingly support decisions about criminal justice, medical treatment, welfare eligibility, entertainment, prices, and political information.
- Historical training data can reproduce unfair patterns, including disadvantaging women in hiring and creating unequal access to luxury-goods advertising.
- Opaque mathematical techniques make it difficult for affected people to identify data errors, improve future outcomes, or prove discrimination.
2 Enter the Right to an Explanation
European explanation rights remain uncertain and constrained, although French administrative law offers broader possibilities for decision-specific explanations. Technical explainability is feasible in some cases, but legal scope, intelligibility, and coverage remain limited.
- The GDPR’s Article 22 does not firmly mandate a general right to explanation; its safeguards and interpretation vary between binding text and contested recitals.
- Article 22 covers only solely automated decisions with legal or significant effects, excluding many decision-support systems and leaving meaningful human involvement unclear.
- The meaning of “significant effects” remains contested because individually minor advertising decisions may produce serious aggregate effects on groups or society.
- GDPR Article 15 may require only a model-based explanation rather than an explanation tied to an individual’s facts, and applies within a restricted personal-data framework.
- Trade-secret protections can limit disclosure, while model slices may offer a middle ground for explaining proprietary systems.
- The GDPR’s provisions remain largely modeled on a pre-Internet directive, contributing to uncertain and outdated coverage of modern algorithmic harms.
- French law can cover decision-support systems and, where appropriate, disclose factor weightings, suggesting a subject-based explanation, but applies only to administrative decisions.
- Extracting local algorithmic weightings is increasingly possible, but weights may remain unintelligible or impossible to retrofit in older or restricted systems.
3 Is a Right to an Explanation Our Best Remedy?
The paper argues that a right to an explanation has practical and conceptual flaws as a remedy for unfair algorithmic decisions, because it burdens individuals and may not reveal systemic discrimination. It therefore turns toward governance tools that act before deployment or decision-making.
- A right to an explanation places the primary burden of challenging bad decisions on individual users, who historically rarely make effective subject-access requests.The paper notes that such requests demand substantial time and persistence and are mainly used effectively by journalists and insiders.
- An explanation may not help individuals challenge discrimination because biased behavior can become apparent only across the full user population.Individual explanations do not necessarily expose patterns that require corpus-level analysis.
- Collective use of explanation rights could reveal model-wide effects, but current legal and technical mechanisms make coordinated challenge difficult.The paper uses targeted political advertising as an example where outsiders could not fully observe, count, or assess the influence of personalized ads.
- The authors warn that an explanation right could become a transparency fallacy: an apparently empowering formality that leaves users no better off.They compare this risk with the notice-and-choice fallacy surrounding unreadable and non-negotiable privacy policies.
- Because redress often arrives after an algorithmic system has harmed a person, the paper considers governance tools that shape systems before deployment or decision-making.The next sections examine regulatory tools intended to ensure, audit, or instigate fairer and less opaque algorithms.
4 Investigating before Deployment or Decision-Making
The paper examines upstream governance mechanisms that can assess and shape algorithmic systems before processing or deployment, rather than relying only on individual explanations after decisions. It focuses especially on DPIAs, privacy-by-design requirements, impact assessments, and certification, while identifying limits in their scope and enforcement.
- Investigating before Deployment or Decision-Making: Upstream governance is needed because transparency or redress after real-time decisions may provide little help to affected individuals.The paper therefore considers tools that influence systems while they are being designed or before deployment.
- Privacy by Design, Data Protection by Design, and Impact Assessments: The GDPR creates a regulatory environment intended to encourage less toxic automated systems rather than conferring only individual rights.These provisions build on privacy-by-design engineering and recognize that regulators cannot rely solely on top-down control.
- Privacy by Design, Data Protection by Design, and Impact Assessments: A DPIA is required before processing when new-technology processing is likely to create high risks to data subjects’ rights and freedoms.The assessment concerns the impact of envisaged processing operations on personal-data protection.
- Privacy by Design, Data Protection by Design, and Impact Assessments: DPIAs are likely to become obligatory precursors for many ML systems with substantial anticipated risks or consequences for individuals or groups.The paper connects this conclusion to GDPR provisions and guidance on high-risk processing.
- Privacy by Design, Data Protection by Design, and Impact Assessments: DPIAs can improve system design overall, but they are aimed primarily at builders and regulators rather than directly at users.They are not replacements for algorithmic explanations and are not generally required to be public documents.
- Privacy by Design, Data Protection by Design, and Impact Assessments: Discrimination is not expressly required as a DPIA trigger in Articles 35 and 36, although guidance and related impact-assessment practice support addressing it.The paper notes that early anti-discrimination language was relegated to recitals and that algorithmic impact assessments remain an immature field.
- Certification Systems: Certification could operationalize big-data due-process rights by evaluating algorithm design specifications, design processes, or monitored outputs.The paper also notes that certification standards could be tailored to particular sectors.
- Certification Systems: Private-sector certification and trust seals risk weakening scrutiny because schemes depend financially on member fees and may hesitate to punish members severely.The paper presents this as a recurring problem in privacy-domain self-regulation.
5 Enabling Review and Challenge Without Individual Burden
The paper argues that challenging algorithmic harms should not depend primarily on individual users, whose burdens are especially acute when harms affect groups. It therefore emphasizes representative action, judicial review, secure access infrastructures, and broader governance remedies alongside explanation rights.
- Representation and collective action: Individual challenge is burdensome, while systemic discrimination affecting whole classes may be better addressed by representative bodies.The GDPR allows some third-party involvement, but existing routes may still require users to notice breaches and seek help.
- Representation and collective action: GDPR Article 80(2) permits member states to authorize third-party bodies to pursue complaints without a data subject’s mandate.Such bodies could monitor sectors and controllers and pursue suspected infringements independently.
- Access and judicial scrutiny: Effective watchdogs need access to training data, input data, outputs, and models, but courts have generally been reluctant to order source-code disclosure.Proprietary intellectual-property concerns contribute to this access problem.
- Access and judicial scrutiny: No reported UK case has ordered disclosure of a decision-support system’s source code, and courts may remain reluctant to use training sets and models until they can confidently assess such evidence.A prior case instead excluded an automated calculation from influence because its generation was insufficiently evidenced.
- Access and judicial scrutiny: Judicial review could provide a valuable public-sector transparency tool because courts can review whether administrative acts and discretion are legal and reasonable.The paper presents this as an underused avenue for scrutinizing algorithmic systems.
- Access and judicial scrutiny: Secure, controlled data-access methods could support external analysis while protecting security and privacy, potentially extending to model and data depositories.The paper suggests archival or specialist libraries as possible homes for infrastructure to scrutinize models and code, subject to intellectual-property issues.