Source-linked AI summary
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Sahil Verma, Varich Boonsanong, Minh Hoang, Keegan E. Hines, John P. Dickerson, Chirag Shah
TL;DR
Machine learning decisions can be difficult for stakeholders to understand, motivating research on human-understandable explanations for consequential automated decisions. This paper reviews and categorizes more than 350 counterfactual-explanation papers using a common rubric, identifying comparative patterns, research themes, and gaps. The resulting synthesis supports understanding algorithmic trade-offs and choosing methods under application constraints.
Problem
Automated machine learning decisions can affect people while remaining difficult or impossible for human stakeholders to understand, creating a need for human-understandable explanations.
Method
The paper collects, reviews, and evaluates more than 350 counterfactual-explanation algorithms against a shared rubric of desirable properties.
Results
The rubric categorizes proposed approaches by their properties and assumptions, enabling comparison of their advantages and disadvantages and identifying research gaps.
Takeaways & Limitations
The review helps researchers and organizations understand counterfactual-explanation methods and choose algorithms suited to their application constraints.
Takeaways & Limitations
Counterfactual explanations can enhance privacy attacks, including model extraction and membership inference, especially when diverse explanations are provided.
Abstract
from arXiv · showhide
Machine learning plays a role in many deployed decision systems, often in ways that are difficult or impossible to understand by human stakeholders. Explaining, in a human-understandable way, the relationship between the input and output of machine learning models is essential to the development of trustworthy machine learning based systems. A burgeoning body of research seeks to define the goals and methods of explainability in machine learning. In this paper, we seek to review and categorize research on counterfactual explanations, a specific class of explanation that provides a link between what could have happened had input to a model been changed in a particular way. Modern approaches to counterfactual explainability in machine learning draw connections to the established legal doctrine in many countries, making them appealing to fielded systems in high-impact areas such as finance and healthcare. Thus, we design a rubric with desirable properties of counterfactual explanation algorithms and comprehensively evaluate all currently proposed algorithms against that rubric. Our rubric provides easy comparison and comprehension of the advantages and disadvantages of different approaches and serves as an introduction to major research themes in this field. We also identify gaps and discuss promising research directions in the space of counterfactual explainability.
1 INTRODUCTION
The review motivates counterfactual explanations as actionable, model-agnostic feedback for consequential automated decisions and evaluates proposed algorithms using a common rubric. It surveys more than 350 papers to compare methods, assumptions, advantages, disadvantages, and research gaps.
- Machine learning decisions affect people in domains including credit lending, talent sourcing, parole, and medical treatment.
- Explanations can help applicants understand decision drivers, challenge potentially unfair outcomes, receive actionable feedback, and help organizations inspect algorithmic bias.
- Counterfactual explanations identify nearby datapoints with different predictions, providing suggestions for changing features to reach a desired outcome.
- Unlike many explainability techniques, CFEs can apply to black-box models without model disclosure or complexity restrictions and can provide accurate feedback without approximating the model.
- The review collects and categorizes more than 350 papers against shared desiderata to compare algorithms and their assumptions.
- The rubric supports algorithm selection based on assumptions, generation speed, and generation quality, while introducing counterfactual explanations to beginners.
2 BACKGROUND
The background places explainability within efforts to address the social implications of machine learning and connects counterfactual explanations to broader traditions of causal and contrastive reasoning. Examples show that apparently accurate models may rely on misleading features.
- Fairness research seeks to prevent decisions from being biased against demographic groups defined by sensitive or protected features such as race, sex, and religion.
- Explainability research includes model-level explanations of global behavior and outcome-level explanations of individual predictions.
- The tank and husky examples show models using background conditions such as weather or snow rather than the intended object-related distinctions.
- The paper represents the prediction function as f: X^m→Y for assigning labels to unseen future datapoints.
- Counterfactual explanations have roots in philosophy, psychology, and social science, where counterfactuals are linked to causal reasoning and counterfactual dependence.
3 COUNTERFACTUAL EXPLANATIONS
Counterfactual explanations identify small changes to an input that would produce a desired model outcome, while seeking actionable, realistic, sparse, causally coherent, and computationally practical recommendations.
- Counterfactual explanations answer what changes could move an input across the classifier’s decision boundary, rather than only why the original prediction occurred.
- Desiderata: Validity requires the counterfactual to receive the desired label while remaining close to the original datapoint.
- Desiderata: Actionable counterfactuals change mutable features, avoid immutable or legally sensitive attributes, and may reflect user preferences among feasible changes.
- Desiderata: Sparsity favors changing fewer features, although minimizing the number of changed features can trade off against minimizing total change.
- Desiderata: Realistic counterfactuals stay near observed training data and preserve feature relationships, including known causal relations.
- Major themes: Amortized inference predicts counterfactuals for new inputs without solving a fresh optimization problem, while black-box and model-agnostic methods broaden deployment settings.
- Related terms: The review uses counterfactual explanation, CFE, and recourse interchangeably, although recourse explicitly emphasizes actionability.
4 ASSESSMENT OF THE APPROACHES ON COUNTERFACTUAL PROPERTIES
The review assesses counterfactual-generation algorithms using a rubric covering model access, model compatibility, computational amortization, and counterfactual attributes such as sparsity, realism, causality, and actionability.
- The authors evaluate collected algorithms against properties intended to make their advantages and disadvantages easy to compare and comprehend.
- Model access: Algorithms differ in required model access: complete internals, gradients, or only the prediction function.
- Model domain: Gradient-based methods require differentiable models, solver methods require suitable linear or piece-wise linear models, and black-box methods are model-agnostic.
- Optimization amortization: Most methods optimize separately for each input and often return one counterfactual, while some generate multiple counterfactuals per input or amortize inference.
- CF attributes: The rubric examines sparsity, data-manifold adherence, causality, feature actionability, and user preferences through varied constraints and modeling choices.
5 EVALUATION OF COUNTERFACTUAL GENERATION ALGORITHMS
The review describes the datasets and evaluation framework used across counterfactual-generation research, emphasizing common tabular and image benchmarks and comparative assessment of algorithmic properties.
- Datasets: Evaluation datasets are grouped into tabular and image data, with some methods unable to support image datasets.
- Datasets: Common image benchmarks include MNIST, EMNIST, CelebA, CheXpert, ImageNet, ISIC Skin Lesion, ADNI, and ChestX-ray8.
- Datasets: Common tabular benchmarks span income, credit, education, healthcare, lending, fairness, housing, insurance, and other decision settings.
- Assessment: The reviewed algorithms are generally evaluated on desirable counterfactual properties, with comparative assessments organized in Tables 1 and 2.
5.3 Other works
Other work studies desirable CFE properties, limitations and assumptions, causal formulations, model-specific methods, toolboxes, and practical constraints on actions and changing models.
- Desirable properties: Research proposes properties and metrics such as proximity, connectedness, stability, and E-chainability to relate explanations to training data.
- Issues: Studies identify unjustified explanations, privacy conflicts, and implicit assumptions as important issues in counterfactual generation.
- Causal CFEs: Causal-CFE work includes conditional-subspace VAEs, optimal-transport approximations to Pearl’s counterfactuals, and analyses of actual causation.
- Other approaches: Other approaches target Bayesian network classifiers, provide extensible comparison toolboxes, or incorporate real-world constraints and action costs.
6 COUNTERFACTUAL EXPLANATIONS FOR OTHER DATA MODALITIES
The survey focuses on counterfactual explanations for tabular data and directs readers to work addressing other data modalities.
- The survey restricts its coverage to counterfactual-explanation algorithms for tabular data.
- The section points readers toward research targeting data modalities outside the survey’s tabular-data scope.
- Related work addresses counterfactual explanations for graph neural networks, functional data, and behavioral data.
7 OTHER APPLICATIONS OF COUNTERFACTUAL EXPLANATIONS
Counterfactual explanations are used beyond explaining machine-learning models, including anomaly detection, dataset debugging, data augmentation, molecular design, bias detection, and other applications.
- Counterfactual explanations support anomaly and data-drift detection by identifying changes associated with drift, anomalies, or increased classifier confidence.
- They are used to debug training datasets and models, including diagnosing decision boundaries, mislabeled data, financial-model bugs, and responsible training examples.
- Researchers use counterfactual explanations for data augmentation addressing market-volatility prediction, class imbalance, robustness, and crop-growth prediction.
- Counterfactual methods have been applied to molecular design, model-bias detection, employment services, social-network community detection, dimensionality reduction, and student advice.
- A real-world deployment at Hired used a GAN-based approach to suggest changes in salary expectations, experience, and skills for marketplace approval.
8 OPEN QUESTIONS AND RESEARCH PROGRESS FOR SOLVING THEM
The survey identifies open challenges for counterfactual explanations and reviews progress toward more integrated, actionable, scalable, fair, robust, and broadly applicable methods.
- Research Challenge 1. Unify counterfactual explanations with traditional “explainable AI.”: Counterfactual explanations should complement traditional explainable-AI methods by indicating both actionable changes and reasons for the original decision.
- Research Challenge 2. Provide counterfactual explanations as discrete and sequential steps of actions.: Real-world recourse requires discrete, sequential actions rather than only a modified endpoint, motivating sequence-generation methods and user studies.
- Research Challenge 3. Extend counterfactual explanations beyond classification.: The survey calls for extending counterfactual explanations beyond classification and reports approaches for regression, graph neural networks, and time-series classification.
- Research Challenge 4. Counterfactual explanations as an interactive service to the applicants.: Interactive interfaces could update instructions as applicants modify their state, while existing work demonstrates interactive exploration and sequential counterfactual explanations.
- Research Challenge 5. The ability of counterfactual explanations to work with incomplete—or missing—causal graphs.: Causal counterfactuals must accommodate incomplete causal graphs because complete graphs and structural equations are rarely available in practice.
- Research Challenge 6. The ability of counterfactual explanations to work with missing feature values.: Counterfactual algorithms should handle missing feature values, which commonly occur in real-world data.
- Research Challenge 7. Scalability and throughput of counterfactual explanations generation.: Scalable generation remains necessary for industrial deployment because most approaches solve an optimization problem separately for different input datapoints.
- Research Challenges 8–9 and regulatory direction: Counterfactual explanations should account for classifier bias, robustness, and stability, while regulatory coordination could clarify their suitability for legal and consumer-facing uses.
9 CONCLUSIONS
The paper reviews a large body of counterfactual-explanation research using a common rubric to compare methods and identify open problems.
- The survey reviews more than 350 papers proposing algorithmic solutions for counterfactual explanations of machine-learning decisions.
- Applying one rubric helps compare approaches, understand their advantages and disadvantages, and select methods under application constraints.
- The review identifies gaps and provides an introduction and resource guide for researchers and practitioners entering counterfactual explainability.
A FULL TABLE
The authors reduce a larger categorization table to its most critical columns in Table 1, while making the complete table available separately.
- Table 1 presents the most critical columns from the authors’ initial, substantially larger categorization table.
- The initial categorization included more columns than the condensed Table 1.
- The full categorization table is available separately from the condensed presentation.
B.1 How we collected the paper to review?
The review assembled more than 350 papers through citation-based expansion and scholarly search, then restricted its scope to recent machine-learning studies of counterfactual explanations in classification settings.
- How we collected the paper to review?: More than 350 papers were collected for the review.
- How we collected the paper to review?: The updated search included papers citing Wachter et al. and the first version of this survey.
- How we collected the paper to review?: The authors searched Semantic Scholar and Google Scholar for “counterfactual explanations,” “recourse,” and “inverse classification.”
- How we collected the paper to review?: The collected papers were published before 31st May 2022, including work from conferences and arXiv.
- Scope of the review: The review covers recent counterfactual-explanation research in machine-learning classification settings, while recognizing related work in other fields.
- Scope of the review: The review situates its topic alongside legal requirements concerning explanations for automated decisions, including GDPR interpretations and US credit laws.