PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR

Sandra Wachter, Brent Mittelstadt, Chris Russell

arXiv:1711.00399v3cs.AI

TL;DR

Explaining automated decisions by opening algorithmic black boxes faces legal and technical barriers, while the GDPR provides limited support for understanding, contesting, or altering decisions. The paper assesses these aims and proposes unconditional counterfactual explanations that identify external changes leading to a desired outcome without revealing internal system logic.

  • Problem

    Opening algorithmic black boxes to explain automated decisions faces major legal and technical barriers, including limited GDPR coverage and the difficulty of explaining complex systems.

  • Method

    The paper compares understanding, contesting, and altering decisions with GDPR provisions and proposes unconditional counterfactual explanations based on external changes rather than internal logic.

  • Results

    The GDPR offers little support for understanding, contesting, or altering automated decisions and does not mandate algorithmic explainability for these purposes.

  • Takeaways & Limitations

    Controllers should offer counterfactual explanations whenever requested, regardless of the decision outcome or whether the decision was solely automated.

  • Takeaways & Limitations

    Counterfactual explanations may be insufficient for understanding system functionality or decision rationale and cannot provide statistical evidence for assessing fairness or racial bias.

Abstract

from arXiv · show

There has been much discussion of the right to explanation in the EU General Data Protection Regulation, and its existence, merits, and disadvantages. Implementing a right to explanation that opens the black box of algorithmic decision-making faces major legal and technical barriers. Explaining the functionality of complex algorithmic decision-making systems and their rationale in specific cases is a technically challenging problem. Some explanations may offer little meaningful information to data subjects, raising questions around their value. Explanations of automated decisions need not hinge on the general public understanding how algorithmic systems function. Even though such interpretability is of great importance and should be pursued, explanations can, in principle, be offered without opening the black box. Looking at explanations as a means to help a data subject act rather than merely understand, one could gauge the scope and content of explanations according to the specific goal or action they are intended to support. From the perspective of individuals affected by automated decision-making, we propose three aims for explanations: (1) to inform and help the individual understand why a particular decision was reached, (2) to provide grounds to contest the decision if the outcome is undesired, and (3) to understand what would need to change in order to receive a desired result in the future, based on the current decision-making model. We assess how each of these goals finds support in the GDPR. We suggest data controllers should offer a particular type of explanation, unconditional counterfactual explanations, to support these three aims. These counterfactual explanations describe the smallest change to the world that can be made to obtain a desirable outcome, or to arrive at the closest possible world, without needing to explain the internal logic of the system.

I. INTRODUCTION

The paper examines the debated GDPR right to explanation and argues that efforts to expose algorithmic black boxes face major legal and technical barriers. One identified barrier is that the GDPR does not establish a legally binding right to explanation.

  • The GDPR’s purported right to explanation has generated extensive debate over its existence, merits, and disadvantages.
  • Attempts to open algorithmic black boxes through explanations face four major legal and technical barriers.
  • The GDPR does not establish a legally binding right to explanation.

II. COUNTERFACTUALS · A. HISTORIC CONTEXT AND THE PROBLEM OF KNOWLEDGE

Counterfactual explanations state why a decision occurred by describing what would need to change for a desirable outcome, focusing on external facts rather than exposing algorithmic internals. The section situates this approach against conventional explanations concerned with conveying an algorithm’s internal logic and the broader problem of defining knowledge.

  • II. COUNTERFACTUALS: The literature cited in the section discusses counterfactual explanations as an alternative to approaches that attempt to open or approximate the decision-making system.The cited works include Wachter, Mittelstadt and Floridi; Mendoza and Bygrave; and Edwards and Veale.
  • II. COUNTERFACTUALS: A counterfactual pairs a decision with a statement of how the world would need to differ for a desirable outcome.For example, a denied loan might become available if annual income increased from £30,000 to £45,000.
  • II. COUNTERFACTUALS: Multiple counterfactuals may exist because several desirable outcomes and several ways of achieving each outcome are possible.The analysis therefore treats the smallest change producing a desirable result as the closest possible world.
  • II. COUNTERFACTUALS: The key distinction is that conventional explanations seek an algorithm’s internal logic, whereas counterfactuals describe dependencies on the external facts behind a decision.Modern machine-learning systems may contain millions of intricately connected variables, making their internal state difficult to convey to laypeople.
  • II. COUNTERFACTUALS: Machine-learning and legal communities have traditionally adopted relatively restricted views of what qualifies as an explanation.The machine-learning community has focused primarily on debugging and conveying algorithmic approximations to technical audiences.
  • A. HISTORIC CONTEXT AND THE PROBLEM OF KNOWLEDGE: The section’s historical context includes analytic philosophy’s longstanding analysis of the necessary conditions for propositional knowledge.This frames the problem of knowledge underlying debates about what an explanation must provide.

C. ADVERSARIAL PERTURBATIONS AND COUNTERFACTUAL EXPLANATIONS

Counterfactual explanations on deep networks closely relate to adversarial perturbations, which efficiently find nearby inputs that change classifier outputs. However, standard methods may lack human-interpretable sparsity and produce imperceptible changes outside the space of real images.

  • Relationship to adversarial perturbations: Adversarial perturbations generate synthetic data points close to existing ones but classified differently, making them counterfactuals by another name.DeepFool defines the perturbation as the smallest change to an image that changes its classification.
  • Computational tractability: Counterfactuals can be computed efficiently and effectively with standard techniques, including on cutting-edge neural-network architectures used for image labelling.Computer-vision systems use some of the largest and deepest neural networks, including architectures applied to ImageNet.
  • Limitations of standard methods: Standard adversarial-perturbation methods often lack appropriate distance functions and favour small changes across many variables rather than sparse, human-interpretable modifications.Differentiability in state-of-the-art algorithms nevertheless enables efficient computation and optimisation of these perturbations.
  • Plausibility and perceptibility: Adversarial perturbations can be barely perceptible to humans while causing drastic changes in classifier responses, suggesting that generated images may lie outside the space of real images.This highlights the importance of ensuring that counterfactuals are not merely close possible worlds but also plausible ones.

D. CAUSALITY AND FAIRNESS

The section examines counterfactual and causal approaches to algorithmic fairness, emphasizing both their evidentiary value for detecting discrimination and their limitations. It also cautions that understandable models and ostensibly fair features may worsen racial bias while reducing accuracy.

  • Counterfactual fairness requires decisions to remain unchanged when a subject’s race or sex is hypothetically changed.
  • Understandable models can mislead intuition, while predominantly using features perceived as fair slightly increased algorithmic racism and decreased accuracy.
  • Counterfactuals can show that an algorithmic decision is affected by a protected variable and may therefore be discriminatory.Changing race in the identified counterfactuals indicates race-dependent treatment.
  • The evidentiary relationship is asymmetric: counterfactuals that do not change a protected attribute cannot establish that treatment is independent of it.
  • For one classifier, proposed counterfactuals changed the race of ‘black’ people but did not suggest varying the race of ‘white’ people.

III. GENERATING COUNTERFACTUALS

The paper presents a method for computing counterfactuals by holding the classifier weights fixed and finding the closest input that achieves a target outcome. It recommends a robust, sparsity-inducing distance metric that changes few variables, while addressing optimisation and validity challenges.

  • III. GENERATING COUNTERFACTUALS: Counterfactuals x' are computed by minimizing distance from the original point x_i while requiring f_w(x') to equal a target y', with w held fixed.The method solves the related objective and iteratively adjusts the distance penalty until a sufficiently close solution is found.
  • III. GENERATING COUNTERFACTUALS: Optimiser choice is relatively unimportant: any optimiser capable of training the classifier appears to work equally well, and the experiments use ADAM.Different random initialisations address local minima, while distinct minima can provide diverse counterfactuals.
  • III. GENERATING COUNTERFACTUALS: The recommended distance is an L1 norm weighted by inverse median absolute deviation, refined for subject- and task-specific requirements.The metric uses MAD_k, the median absolute deviation of feature k over the point set P.
  • III. GENERATING COUNTERFACTUALS: The distance metric captures feature volatility, resists outliers through median absolute deviation, and encourages sparse changes that make counterfactuals easier to communicate.Its sparsity corresponds to changing only a small number of variables while leaving most features constant.
  • III. GENERATING COUNTERFACTUALS: The section highlights the need to ensure that synthetic counterfactuals are valid data points and discusses pitfalls and remedies for discrete features.It also illustrates how varying the distance function affects counterfactuals on the LSAT dataset.

A. LSAT DATASET

The LSAT experiments show that counterfactual explanations expose artefacts caused by unrealistic categorical changes and distance-function choices. Normalization and sparse weighting produce more plausible, accessible alterations while revealing dependence on race.

  • Dataset: The LSAT64 task predicts first-year average grades from race, prior GPA, and entrance-exam scores using data whose classifiers exhibit bias against ‘black’ people.
  • Model: A three-layer neural network with 941 weights and 40 interdependent neurons illustrates why communicating internal model state is challenging.
  • Unnormalized counterfactuals: Unweighted squared Euclidean distance assigns meaningless fractional or negative race values and changes GPA much more than exam results.
  • Normalized counterfactuals: Standard-deviation normalization keeps GPA more consistent and within expected ranges, while race changes for ‘black’ students, revealing decision dependence on race.
  • Sparse textual explanations: Inverse-median-absolute-deviation-weighted L1 counterfactuals are similar but sparser, leaving GPA unchanged and expressible as accessible statements about altered LSAT scores or race.

B. PIMA DIABETES DATABASE … 1. Broader possibilities with the right of access

The paper demonstrates that counterfactual explanations can provide personalized, actionable information without opening complex algorithmic black boxes or relying on detailed causal models. It then examines how this approach aligns with the GDPR’s limited and purpose-dependent explanation framework, including understanding decisions and access rights.

  • B. PIMA DIABETES DATABASE: A three-layer neural-network classifier predicts Pima women’s diabetes risk from eight variables, including pregnancies, age, and BMI, producing scores between 0 and 1.The counterfactuals are constrained to training-data ranges and use standard deviation rather than Euclidean distance.
  • B. PIMA DIABETES DATABASE: Counterfactuals typically alter only a few variables and automatically render those differences in human-readable text.Examples specify personalized changes in insulin or plasma glucose associated with a target score of 0.51.
  • C. CAUSAL ASSUMPTIONS AND COUNTERFACTUAL EXPLANATIONS: Unlike causal explanations, these counterfactuals assume variables are independent, enabling data subjects to audit the data for relevant inaccuracies.The paper argues that detailed causal models are unnecessary for counterfactual explanations to be useful, though accurate causal models may assist expert interventions.
  • IV. ADVANTAGES OF COUNTERFACTUAL EXPLANATIONS: Counterfactuals bypass the substantial technical challenge of explaining complex machine-learning systems’ internal workings while providing practically useful information to data subjects.The approach differs from existing machine-learning and legal proposals concerning the GDPR’s right to explanation.
  • IV. ADVANTAGES OF COUNTERFACTUAL EXPLANATIONS: Modern deep networks may compose functions more than 1,000 times and use more than 10 million parameters, making human-comprehensible explanations of their logic difficult.The paper presents this complexity as a reason counterfactuals can reduce regulatory burdens.
  • V. COUNTERFACTUAL EXPLANATIONS AND THE GDPR: Because the GDPR’s explanation language is legally non-binding and underspecified, explanation content should be tailored to the particular purpose it is intended to serve.The relevant information may need to vary in structure, complexity, and content for experts, non-experts, and affected individuals.
  • A. EXPLANATIONS TO UNDERSTAND DECISIONS: For understanding decisions, GDPR provisions support broad knowledge of automated decision-making but generally do not require disclosure of the rationale or internal logic of a specific decision.Transparency, notification, and access mechanisms create informational requirements about automated decision-making without explicitly requiring the black box to be opened.
  • 1. Broader possibilities with the right of access: The right of access can be invoked at any time by the data subject, creating broader possibilities for providing information than duties tied to notification before processing.The supplied passage introduces access as a potentially broader route for explanations, but does not specify its full content here.

2. Understanding through counterfactuals

Counterfactual explanations can satisfy GDPR transparency, notification, and access aims by giving data subjects simple information about specific automated decisions without exposing the system’s internal logic. They provide a minimal form of disclosure about the logic involved while minimizing interpretability and explanation burdens for controllers.

  • Understanding through counterfactuals: Counterfactual explanations meet and exceed the aims and requirements of the GDPR’s transparency, notification, and access mechanisms.These mechanisms provide data subjects with information to understand the scope of automated decision-making.
  • Understanding through counterfactuals: Simple “if-then” statements can inform data subjects about specific automated decisions without explaining how the entire algorithm works.They require only a specific, limited set of dependencies between variables and the decision.
  • Understanding through counterfactuals: Counterfactuals offer information about the rationale of specific automated decisions without opening the black box.The passage states that this approach need not explain the internal logic of how the decision was reached.
  • Understanding through counterfactuals: Counterfactuals provide a minimal disclosure about the “logic involved” in specific decisions, even though GDPR provisions do not require information about specific decisions.This disclosure concerns facts and variables relevant to a specific decision.
  • Understanding through counterfactuals: By avoiding the need to resolve complex interpretability problems or explain internal logic to non-experts, counterfactuals minimize the regulatory burden on data controllers.The supplied passage explicitly links this reduced burden to avoiding those technical and communication difficulties.

B. EXPLANATIONS TO CONTEST DECISIONS

Explanations may help individuals contest adverse automated decisions by revealing inaccuracies, problematic inferences, or flaws in algorithmic reasoning. However, the GDPR does not explicitly link explanation, contestation, and transparency rights, nor require disclosure of internal logic.

  • Models of contesting decisions: Article 22(3) safeguards provide rights to human intervention, expressing views, and contesting automated decisions.The safeguards may be interpreted as a unit or as rights invoked separately or in combination.
  • Models of contesting decisions: Four contestation models vary in whether humans replace the algorithm, assess its output, or monitor inputs while the algorithm decides.Human decisions fall outside Article 22(1)’s solely automated category, potentially removing safeguards against the subsequent decision.
  • Explanations and contestation: The usefulness of rationale explanations depends on the contestation model: they may inform assessment of reasons but have little practical effect when humans disregard the algorithm.Where algorithmic involvement remains, explanations can identify inaccurate input data, problematic inferences, or flaws in algorithmic reasoning.
  • GDPR support and limits: The GDPR does not explicitly require explanations to support contestation or disclose an algorithm’s internal logic.Recital 71 does not specify the right’s aim or required information, and contestation is not explicitly linked to transparency, notification, access, or explanation mechanisms.
  • GDPR support and limits: Articles 12–15 may assist contestation by informing data subjects about relevant rights, but their automated-decision information generally provides an overview rather than individual rationale.Articles 13–15 refer explicitly to automated decisions under Article 22(1) and (4), not clearly to its safeguards.

1. Contesting through counterfactuals

The GDPR’s information duties do little to facilitate contesting automated decisions and may not require disclosure of the system’s internal logic or the data on which a decision depended. Counterfactual explanations could strengthen the legally binding right to contest by revealing external factors and key variables without opening the black box.

  • GDPR support for contesting: Articles 13–15 do little to facilitate data subjects’ ability to challenge automated decisions.They do not provide information about Article 22(3) safeguards, including the right to contest.
  • GDPR support for contesting: The GDPR does not require disclosure of an automated system’s internal logic, despite provisions referring to meaningful information about its logic, significance, and consequences.Such information could facilitate contesting decisions but need not be provided.
  • Counterfactual support: Counterfactuals can support contesting by revealing the external factors and key variables that contributed to a specific decision without opening the black box.They may therefore provide greater protection than the GDPR currently envisions.
  • Counterfactual support: The right to contest is a legally binding safeguard regardless of the legal status of a right to explanation.Counterfactuals are proposed as information that helps data subjects exercise this safeguard.
  • Counterfactual support: Although Article 16 grants a right to correct inaccurate data used in decisions, data subjects need not be told which data the decision depended on.Without knowing the relevant data in a large corpus, correcting inaccurate inputs is difficult.

C. EXPLANATIONS TO ALTER FUTURE DECISIONS

Counterfactual explanations can guide data subjects toward changing key variables to obtain more favourable future automated decisions, even though this purpose is not directly addressed in the GDPR. Their usefulness depends on decision-model stability and may be reduced by unanticipated dependencies between changed attributes and other variables.

  • C. EXPLANATIONS TO ALTER FUTURE DECISIONS: Counterfactual explanations indicate what could be changed to obtain a desired future automated decision, including whether an immediate pay rise makes reapplying for a loan reasonable.Using explanations to alter behaviour is not directly addressed in the GDPR, but data subjects retain an interest in receiving desired results.
  • C. EXPLANATIONS TO ALTER FUTURE DECISIONS: By identifying key variables and close possible worlds with different outcomes, counterfactuals show which factors could be changed to receive the desired result.This guidance is especially applicable when the decision model and environment have low variability over time or are artificially frozen for the individual.
  • C. EXPLANATIONS TO ALTER FUTURE DECISIONS: Unanticipated dependencies between intentionally changed attributes and other variables can undermine counterfactual explanations as guides for future behaviour.An income increase resulting from a career change may affect other variables in ways the counterfactual does not anticipate.
  • C. EXPLANATIONS TO ALTER FUTURE DECISIONS: Counterfactual explanations can address the simultaneous impact of changing more than one variable on a model’s output.This capability may help account for dependencies that arise when altering attributes together.

CONCLUSION

The paper concludes that the GDPR offers little support for understanding, contesting, or altering automated decisions and does not require opening algorithmic black boxes. It therefore proposes unconditional counterfactual explanations as an efficient, action-oriented safeguard, while recognizing that broader interpretability remains necessary.

  • Conclusion: Counterfactual explanations identify which external facts could change to produce a desired outcome without revealing an algorithm’s internal logic.They are described as a lightweight form of explanation that is efficiently computable for many systems.
  • Conclusion: The GDPR provides little support for understanding, contesting, or altering automated decisions and does not clearly require explanations to open the black box.Recital 71 does not specify explanations’ purpose or content, and its explanations need not be a precondition for contesting decisions.
  • Conclusion: The authors propose providing counterfactual explanations whenever requested, regardless of the decision’s outcome or whether it meets GDPR automated-processing thresholds.This approach moves beyond the GDPR’s narrow definitions and conditions for automated decision-making.
  • Conclusion: Counterfactuals explain decisions, support contestation, and offer limited advice about changes that could produce a desired future result.Examples include identifying low income as a reason, exposing inaccurate income data, or indicating that an annual income increase of 4000 pounds would have changed the outcome.
  • Conclusion: Counterfactuals are an easy first step but cannot replace interpretability needed to understand system functionality or statistical evidence needed to assess fairness and racial bias.Future work should develop suitable distance metrics and requirements for diverse counterfactual sets across sectors and cases.

APPENDIX 1: SIMPLE LOCAL MODELS AS EXPLANATIONS · APPENDIX 2: EXAMPLE TRANSPARENCY INFOGRAPHIC

Appendix 1 shows that simple local models can change substantially with the scale of the domain, making their usefulness and counterfactual application difficult to determine. Appendix 2 indicates that proposed generic transparency icons reflected complex legislative communication expectations without requiring individual-level contextualised explanations.

  • APPENDIX 1: SIMPLE LOCAL MODELS AS EXPLANATIONS: Local approximation methods trade off approximation quality, interpretability, and the size of the domain they cover.LIME is cited as an example of generating simple models as local approximations of decisions.
  • APPENDIX 1: SIMPLE LOCAL MODELS AS EXPLANATIONS: Choosing the scale of an explanation is difficult because the appropriate region depends on how the explanation will be used.The passage illustrates this difficulty even for a simple one-variable function.
  • APPENDIX 1: SIMPLE LOCAL MODELS AS EXPLANATIONS: Generic automated explanations can vary wildly when the range of inputs changes.The appendix compares such explanations with route directions whose usefulness depends on the intended travel distance.
  • APPENDIX 1: SIMPLE LOCAL MODELS AS EXPLANATIONS: The direction and magnitude of a local linear approximation may change dramatically with the domain, making the most helpful approximation for a layperson nontrivial to choose.Figure 1 depicts opposing explanations in which the score either increases or decreases along the input axis.
  • APPENDIX 1: SIMPLE LOCAL MODELS AS EXPLANATIONS: Local approximations may be useless for computing counterfactuals or telling a data subject how to adjust their score.For a desired score of -10 or below, the appendix states that none of the local approximations would be useful.
  • APPENDIX 1: SIMPLE LOCAL MODELS AS EXPLANATIONS: Although local models and counterfactuals serve different purposes, counterfactuals can indicate whether reapplying after a pay rise is worthwhile.The passage contrasts this practical indication with the inability of counterfactuals to provide a local linear approximation around the data point.
  • APPENDIX 2: EXAMPLE TRANSPARENCY INFOGRAPHIC: Proposed European Parliament transparency icons were not adopted but suggest that EU legislators expected complex information to be communicated under Articles 13–14 without requiring individual-level contextualised information.Their generic form is presented as evidence that these provisions were not intended to create a de facto right to explanation comparable to another right discussed in the paper.
Loading 1711.00399v3…