Source-linked AI summary
The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons
Solon Barocas, Andrew D. Selbst, Manish Raghavan
TL;DR
Feature-highlighting explanations appear useful for explaining automated decisions, but their assumptions and subjective choices are often overlooked. This paper examines those assumptions and shows that their usefulness depends on connections between model features, real-world actions, autonomy, and model behavior that are not straightforward.
Problem
There is no natural, explicitly recognized way to choose between principal-reasons and counterfactual explanations, despite their different goals and outputs.
Method
The paper examines four overlooked assumptions underlying feature-highlighting explanations and explores consequences of acknowledging or addressing them.
Results
Feature-highlighting explanations lack clear connections to required real-world actions and can create autonomy paradoxes while granting decision makers greater power.
Takeaways & Limitations
Useful implementation requires understanding the disconnect between model features and real-world actions, alongside the subjective choices needed to address it.
Takeaways & Limitations
Feature changes do not clearly map to actions, which may allow alternative ways of changing features without taking the intended steps.
Abstract
from arXiv · showhide
Counterfactual explanations are gaining prominence within technical, legal, and business circles as a way to explain the decisions of a machine learning model. These explanations share a trait with the long-established "principal reason" explanations required by U.S. credit laws: they both explain a decision by highlighting a set of features deemed most relevant--and withholding others. These "feature-highlighting explanations" have several desirable properties: They place no constraints on model complexity, do not require model disclosure, detail what needed to be different to achieve a different decision, and seem to automate compliance with the law. But they are far more complex and subjective than they appear. In this paper, we demonstrate that the utility of feature-highlighting explanations relies on a number of easily overlooked assumptions: that the recommended change in feature values clearly maps to real-world actions, that features can be made commensurate by looking only at the distribution of the training data, that features are only relevant to the decision at hand, and that the underlying model is stable over time, monotonic, and limited to binary outcomes. We then explore several consequences of acknowledging and attempting to address these assumptions, including a paradox in the way that feature-highlighting explanations aim to respect autonomy, the unchecked power that feature-highlighting explanations grant decision makers, and a tension between making these explanations useful and the need to keep the model hidden. While new research suggests several ways that feature-highlighting explanations can work around some of the problems that we identify, the disconnect between features in the model and actions in the real world--and the subjective choices necessary to compensate for this--must be understood before these techniques can be usefully implemented.
1 Introduction
Feature-highlighting explanations aim to restore agency by directing attention to selected decision factors while preserving complex, undisclosed models. The paper argues that their promised utility depends on overlooked assumptions and subjective choices that create tensions around autonomy and model disclosure.
- Motivation: Explanations are presented as a way to restore agency by helping people navigate governing rules, contest decisions, and oversee automated decision making.These goals connect explanations to autonomy, recognition of objection opportunities, and algorithmic regulation.
- Approach: Counterfactual and principal-reason approaches are grouped as feature-highlighting explanations because both direct attention to a selected subset of decision features.The approaches differ in operation and motivation but share the practice of highlighting features considered most deserving of attention.
- Appeal: Their popularity reflects the apparent ability to support unconstrained model complexity while withholding full model disclosure and protecting proprietary interests.This approach also limits decision subjects’ ability to game the model.
- Assumptions: The paper argues that generating these explanations requires consequential subjective choices and rests on four easily overlooked, rarely justified assumptions.One assumption is that changing a feature value clearly maps to a real-world action; another concerns making features commensurate using training-data distributions.
- Tensions: Feature-highlighting explanations can produce paternalism in the name of autonomy because decision makers must determine what is best for decision subjects.Sensitivity to subjects’ needs and preferences can require further intrusion by the decision maker.
2 What are feature-highlighting explanations?
Feature-highlighting explanations educate decision subjects by identifying model features that matter to an individual decision without disclosing the full model. Counterfactual and principal-reason explanations share this feature-highlighting structure but differ in precision, comparison methods, and underlying goals.
- Definition: Feature-highlighting explanations identify specific model features that matter to an individual decision while withholding an exhaustive inventory of the model’s features.Counterfactual and principal-reason explanations both focus attention on a limited set of features rather than disclose the model in its entirety.
- Counterfactual explanations: Counterfactual explanations seek actionable guidance by identifying minimally changed features that could produce a different model outcome.Their optimization framing searches for the nearest hypothetical point classified differently from the current point, requiring a notion of distance across features.
- Principal-reason explanations: Principal-reason explanations derive from U.S. credit law, which requires creditors and others using credit information to provide reasons for adverse decisions.The approach is associated with the FCRA, ECOA, and Regulation B, whose requirements include explaining denials and other adverse credit decisions.
- Contrasting approaches: Principal reasons may compare applicants’ feature values with population averages, whereas counterfactuals compare them with a decision boundary or seek a minimally different classified point.Regulatory methods surface features farthest below averages for credit-receiving, passing, or general applicants rather than using the decision boundary.
- Contrasting approaches: Principal-reason explanations are less precise because legal ambiguity permits interpretation, while counterfactual explanations aim to specify the direction and magnitude of change needed for approval.Principal reasons can be vague and may omit both the magnitude and direction of necessary change, unlike counterfactual explanations.
- Contrasting approaches: Choosing between principal-reason and counterfactual explanations is not natural or merely technical because the methods produce different explanations and serve fundamentally different goals.Principal reasons emphasize which factors dominated a decision, while counterfactuals emphasize how choices affect outcomes and what actions could improve future results.
3 Feature-highlighting explanations in practice
Feature-highlighting explanations depend on hidden assumptions about how feature changes translate into real-world actions and how other features, costs, and incentives interact. These assumptions can fail because actions may change multiple features, encourage gaming or costly tradeoffs, and become unreliable as models and environments change.
- Real-world action mapping: Feature-highlighting explanations assume that suggested feature changes map clearly to actions, but concrete recommendations such as increasing income can involve several different, non-equivalent choices.Possible actions include seeking a new job, asking for a raise, or working more hours; these are not as simple as changing a feature value.
- Real-world action mapping: Feature changes may be impossible, delayed, or causally unclear, making explanations difficult to act on and potentially disconnected from the steps required in the real world.Length of employment is inherently time-dependent, and changing income or employment may require substantial time or effort.
- Feature interactions: “Gaming” arises because many actions can change highlighted features without following the decision maker’s intended causal sequence, while undisclosed feature changes can accidentally undermine the desired outcome.Decision subjects may pursue other life goals or career changes that alter features the explanation did not identify as constraints.
- Feature interactions: Feature-highlighting explanations assume other features can remain constant, but actions often affect multiple interacting features and may produce unintended outcomes.Changing jobs to increase income can negatively affect length of employment, while other actions may alter both features simultaneously.
- Costs and autonomy: Useful explanations should account for the cost of changing features, including dollars, effort, time, and opportunity costs across other areas of a person’s life.A large normalized feature change may be preferable when it is inexpensive, while following an explanation may require foregoing benefits in another domain.
- Model stability and behavior: Counterfactual explanations can become unstable as borrower behavior and environmental conditions change, and non-monotonic models can yield complex or counter-intuitive recommendations that differ across outcomes.Credit models may require retraining after recessions or collective behavioral shifts; qualifying for a high-interest loan may not reveal what is needed for a low-interest loan.
4 Unavoidable Tensions
Feature-highlighting explanations create unavoidable tensions: partial disclosure and real-world assumptions give decision makers substantial discretionary power, while collecting more information or increasing transparency can undermine autonomy, privacy, or model secrecy.
- Autonomy paradox: Respecting autonomy paradoxically requires assumptions about which information a decision subject values and can translate into feasible real-world action.Decision makers may not know how changing one feature affects other aspects of a person’s life.
- Autonomy paradox: Collecting and connecting comprehensive personal information could tailor explanations but would create a privacy disaster, producing an autonomy paradox because privacy is part of autonomy.The tension is framed through contextual integrity, which treats inappropriate information flows as privacy violations.
- Subjective disclosure: Feature-highlighting explanations cannot be completely formulaic because they require subjective choices about disclosure and assumptions about the real world.Their apparent automation therefore conceals human judgment.
- Decision-maker power: Partial explanations and unspecified choices about factors, feature costs, dependencies, and outcomes grant decision makers substantial power to pursue their own welfare.This power can be used for good or abused, including when selecting counterfactual interest-rate outcomes.
- Transparency and secrecy: Diverse counterfactuals, interactive exploration, and solicited preferences may address real-world limitations, but greater transparency risks enabling reconstruction of the underlying model.Firms concerned with intellectual property and gaming may therefore limit users’ freedom to explore.
5 Conclusion
Feature-highlighting explanations are disconnected from the real-world actions needed to change features, and attempts to fix this can undermine autonomy while increasing decision makers’ power. The paper proposes disclosing explanation methods, exploring fiduciary obligations, and conducting empirical research on effective actions and disclosures.
- Conclusion: Feature-highlighting explanations fail to account for action costs, decision subjects’ preferences, and effects on other areas of their lives.The paper argues that these explanations lack a connection to the real-world actions required to change features.
- Conclusion: Attempts to correct these deficiencies can violate decision subjects’ autonomy while granting decision makers more power.The paper identifies this as a central consequence of trying to improve feature-highlighting explanations.
- Proposed avenues: Explanation methods should be disclosed because decision subjects cannot understand how to realize their goals without knowing how explanations are generated.The paper recommends amending legal requirements for explanations and AANs to require this disclosure.
- Proposed avenues: Fiduciary obligations are worth exploring as a way to address the autonomy paradox.These obligations constrain people or entities in positions of trust to act in beneficiaries’ best interests.
- Proposed avenues: Empirical research should examine which actions people take and which disclosures help them act effectively when receiving feature-highlighting explanations.Suggested approaches include longitudinal data on successful paths and direct engagement with decision subjects.
- Conclusion: Further work in computer science, social science, and policy is needed to determine when and where feature-highlighting explanations are useful.The proposals address only some of the issues raised in the paper.