Source-linked AI summary
Taking the Whys Seriously: Limitations of Counterfactual Explanations in Justification and Recourse
Mattia Cerrato, Otto Sahlgren, Xenia Heilmann
TL;DR
Counterfactual explanations may not reliably answer the “why” questions required for justification and recourse because upstream design choices shape both decisions and explanations. Through empirical interventions across the ML pipeline, the paper shows these choices can substantially alter counterfactuals, limiting their role in real-world decision-making.
Problem
The paper examines whether counterfactual explanations can reliably support justification, contestation, and recourse across varied real-world decision settings.
Method
The authors measure counterfactual sensitivity by modeling upstream data, labeling, business, and model-selection choices as interventions across the ML pipeline.
Results
Upstream interventions substantially change generated counterfactuals, with effects larger than changing the validity threshold and comparable to changing the explanation method.
Takeaways & Limitations
Counterfactual explanations offer only partial justification and recourse because their contestable, potentially unstable upstream assumptions must also be evaluated.
Takeaways & Limitations
The analysis operationalizes a fixed-model assumption, although the model depends on decision-maker choices about success metrics, labels, and feature measurements.
Abstract
from arXiv · showhide
Counterfactual explanations (CEs) are widely used in explainable artificial intelligence (AI) to show how a model's outputs would change if the input features were manipulated. This technique is used for a range of tasks such as debugging models, explaining predictions, justifying decisions, and providing algorithmic recourse. In this paper, we explore the normative legitimacy of employing counterfactuals in real-life model deployment settings. We discuss the different stakes involved in these different purposes for which CEs are commonly employed, and find stricter requirements for justification and recourse. In particular, we find that naive application of CEs for justification and recourse can lead to ignoring contestable choices made throughout the machine learning (ML) pipeline, thus obfuscating that decisions and counterfactuals for those decisions are also artifacts of an organization's materialized design and governance choices. We demonstrate this with four empirical experiments involving interventions at stages of the ML pipeline ``upstream" of the explanation itself, and show that these affect the generated counterfactuals. We find that an organization's choices on measurement models for feature and labels, business requirements, model validation, and the metric of model success have as much or more impact on the generated counterfactuals as the specifics of the generating method. Our findings underline the need to account for such choices upon providing justification and recourse, providing a stark reminder of the relational nature of these tasks. As putative justifications or recourse recommendations, CEs do not provide adequate answers to some important "why"-questions because they preclude consideration of whether the decision-maker ought to have acted differently.
Introduction
Counterfactual explanations answer why-model questions by proposing input changes that would produce different decisions, but their reliability for justification, contestation, and reversal depends on choices made throughout the ML pipeline. The paper therefore studies upstream interventions and argues that CEs may fail to represent adequate “whys” in real-life settings.
- Counterfactual explanations: CEs propose input modifications that would incur a different model decision, making them a prominent technique for answering why a model returned its decision.They have been described as a “de-facto standard in explainable artificial intelligence.”
- Research motivation: The paper evaluates whether CEs can reliably support justification, contestation, and reversal across real-world decision-making settings.It treats sensitivity to expectable circumstantial variation as crucial to assessing whether CEs can robustly realize values central to practice.
- Empirical approach: The study measures CE sensitivity to systematic upstream interventions spanning data collection, model selection and fitting, and CE generation.It models certain business-side ML pipeline choices as interventions on a structural causal model.
- Main finding: Upstream interventions can dramatically alter downstream CEs, producing differences comparable in magnitude to those caused by changing the explanation method.The finding supports the paper’s theoretical claim that empirical and normative assumptions are needed for CEs to represent adequate “whys.”
Background and Related Work
Counterfactual explanations are local methods that answer “why” questions by identifying input changes that would alter a model’s prediction, and they are widely viewed as a means of explanation and algorithmic recourse. However, their validity, actionability, and robustness depend on changing data, models, causal structures, and upstream measurement choices.
- Counterfactual Explanations: Counterfactual explanations describe how a prediction would change after input changes, often by identifying minimal changes that flip the model’s output.They operationalize counterfactual search as an optimization problem balancing similarity and outcome change.
- Robustness and Validity: CE validity and recourse reliability can deteriorate when models, datasets, or causal structures change over time.Actions sufficient at time t may fail at t+1, undermining the actionability and validity of recommendations.
- Algorithmic Recourse: Algorithmic recourse concerns actionable changes that enable people to improve outcomes or reverse unfavorable algorithmic decisions.Recourse is associated with personal autonomy, agency, and social trust.
- CEs and Recourse: CEs are widely regarded as promising for helping people understand, appeal, contest, and potentially reverse unfavorable decisions.They are also discussed as a possible way to implement the right to explanation by stating which features should have differed.
- Upstream Design Choices: Recorded features and labels are outputs of measurement models and business rules, whose design choices causally affect the trained decision model and its counterfactual explanations.The framework treats these measurement processes as explicit choices rather than direct facts of the world.
Experiments
The experiments test whether upstream choices in the ML pipeline change counterfactual explanations more than the downstream stopping threshold. Across sensitivity and agreement analyses, label and feature-definition choices substantially alter explanations, while upstream interventions can reduce consistency as much as or slightly more than threshold changes.
- Experimental design: The study compares counterfactual sets after interventions on label definitions, feature operationalization, model-selection metrics, validation strategies, and stopping thresholds.Experiments use ACS microdata with seven features and compare DiCE-kdtree and NICE, both retrieval-based methods.
- RQ1: Sensitivity: Salary-cap sensitivity increases with distance from the $40k baseline, with $70k and $80k caps producing distances above the stopping-threshold band for both methods.The label defines positives as top-10% earners below the selected salary cap, representing an organizational affordability constraint.
- RQ1: Sensitivity: Three of four interventions move both methods clearly outside the stopping-threshold sensitivity band, indicating substantial effects from upstream pipeline choices.The reported comparisons use mean symmetric set-distance with 95% Student-t confidence intervals over ten seeds.
- RQ2: Consistency: Under upstream interventions, DiCE-kdtree’s mean Value Agreement falls from 0.39 across threshold values to 0.18–0.26, while NICE falls from 1.00 to 0.61–0.87.Jaccard agreement is slightly lower upstream, Sign Agreement is roughly equal, and cross-validation folds are the reported exception.
- RQ2: Consistency: Upstream interventions affect explanation consistency at the same level as, or slightly more than, interventions on the stopping threshold.This is the paper’s conclusion for RQ2.
Discussion
Upstream design and measurement choices significantly affect counterfactual explanations, challenging the fixed-model assumption underlying their use for justification and recourse. Consequently, CEs provide incomplete answers unless model assumptions, responsibilities, alternatives, and institutional relations remain open to evaluation.
- Overall implications: Upstream interventions significantly affect generated counterfactuals, with larger effects than changing the method’s validity threshold τ across two retrieval-based generators.The authors present this as evidence for their theoretical argument rather than a limitation of CEs per se.
- Overall implications: The fixed-model assumption conceals decision-makers’ contestable actions and upstream choices that help determine both decisions and their counterfactual explanations.Whether a fixed model is reasonable depends on actions by the decision-maker that are not presented as the underlying reason for the decision.
- Implications for Justification: CEs have limited justificatory force when models and their design choices remain open to evaluation, because justification requires assessing legitimate aims and reasonable alternative models.Even evidence that disparities are proportional to base-rate differences does not establish that the preferred model is legitimate, necessary, or appropriate.
- Implications for Recourse: CE-based recourse may lack long-term robustness, actionability, and normative determinacy because different defensible upstream choices can produce incompatible recommendations and different associated costs.Changes in feature measurements and model-selection metrics can alter properties such as minimality and actionability; fixed-model technical robustness does not guarantee stable recourse.
- Implications for Recourse: Effective recourse should be treated as a relational practice requiring attention to decision-makers’ responsibilities, affected subjects’ obligations, and broader algorithmic transparency.Relevant transparency measures include data-sharing structures and documentation, particularly when subjects cannot contest design choices without access to underlying information.
Conclusions
The paper finds that counterfactual explanations (CEs) vary substantially with upstream ML-pipeline design choices, even when generated using the same method. Because these choices may be temporally unstable and contestable, CEs have only a limited role in real-life explanation, debugging, justification, and recourse.
- Empirical findings: CEs generated with the same method vary substantially in response to design choices made earlier in the ML pipeline.The paper empirically measures CE sensitivity to systematic upstream interventions across the pipeline.
- Empirical findings: CEs track many factors in the ML pipeline that condition the model to which a CE method is applied.
- Practical implications: CEs can play only a limited role in real-life explanation, debugging, justification, and recourse provision.Their evaluation and practical application should account for the potential temporal instability and contestability of key design choices and their assumptions.
Hyperparameter Selection
Table 2 reports the hyperparameter grids and defaults used in Interventions 3 and 4, with defaults representing the non-interventional SCM setting ξ0.
- Hyperparameter Selection: Table 2 lists hyperparameter grids and defaults for model selection strategy and model success metric in Interventions 3 and 4.The “Default” column denotes the non-interventional choice, corresponding to SCM setting ξ0.
Counterfactual distance metrics
The paper measures counterfactual-set sensitivity with a two-layer distance: a mixed-type per-counterfactual cost and a symmetric set-level nearest-neighbour aggregator. Sensitivities are averaged across eligible queries and seeds, with 95% Student-t confidence intervals.
- Counterfactual distance metrics: The per-counterfactual cost rescales continuous-feature differences by training-set MAD and treats categorical differences as 0/1 mismatches.MAD is fitted separately for each seed, making the cost run-specific, unit-free, and outlier-robust.
- Counterfactual distance metrics: The set-level distance symmetrically averages nearest-neighbour distances between two counterfactual sets, providing a softened alternative to Hausdorff distance.Replacing the means with maxima recovers Hausdorff distance.
- Counterfactual distance metrics: The mean aggregator accommodates mildly different counterfactual-set sizes, including KDiCE = 5 and KNICE = 3, while avoiding overweighting a single discordant counterfactual.If either set is empty, the distance is undefined and that query is excluded from the across-query average.
- Counterfactual distance metrics: For each intervention pair, method, and seed, distances are averaged over queries with ˆy = 0 and then reported as seed means with 95% Student-t confidence intervals.Distances are computed at every eligible query before averaging across queries and seeds.
Consistency metrics: full definitions
The appendix formally defines explanation sets and pairwise consistency metrics for counterfactuals, including feature-set agreement, directional agreement, categorical-value agreement, and multi-counterfactual aggregation.
- Explanation set: An explanation set contains the features whose counterfactual values differ from the query, using 10^-9 absolute tolerance for continuous features and string comparison for categorical features.E(ˆx, x) = {j ∈ F : ˆx_j ≠ x_j}.
- Pairwise metrics: The symmetric Jaccard agreement compares the feature sets changed by two counterfactuals, equaling 1 for identical sets and 0 for disjoint sets.It captures agreement about which features changed, not necessarily how they changed.
- Pairwise metrics: Sign Agreement equals 1 when shared continuous features are changed in the same direction by both counterfactuals.The metric compares the signs of each counterfactual’s deviation from the query.
- Pairwise metrics: Value Agreement equals 1 when shared categorical features receive the same new value in both counterfactuals.It addresses cases where counterfactuals change the same categorical feature but choose different replacement values.
- Pairwise metrics: Both metrics are undefined when their corresponding continuous or categorical intersection is empty, so that query is excluded from the overall metric computation.This applies when no relevant feature was changed by both counterfactuals.
- Aggregation over multi-CE sets: Set-level consistency averages pairwise metrics across the Cartesian product of two counterfactual sets, retaining k = 4 for DiCE-kdtree and k = 2 for NICE.The appendix avoids reducing each set to one representative counterfactual, which would smooth differences.