Source-linked AI summary
If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques
Mark T Keane, Eoin M Kenny, Eoin Delaney, Barry Smyth
TL;DR
Counterfactual XAI methods are widely promoted as useful explanations, but their psychological and computational evaluation remains limited and insufficiently grounded in user evidence. The paper surveys 100 methods, quantifies five evaluation deficits, and proposes a standardized, psychologically grounded roadmap. The survey finds especially sparse direct user testing and argues for benchmark evaluations backed by user studies.
Problem
Counterfactual XAI methods make claims about plausible, relevant, feasible, or helpful explanations, but these claims are often supported by intuition-based proxy metrics rather than user tests.
Method
The paper critically reviews 100 CF XAI methods, quantifies psychological and computational evaluation deficits, and proposes standardized benchmark metrics grounded in user testing.
Results
21% of surveyed papers directly user-test a specific CF method, while 22% report coverage results and 40% report comparative testing.
Takeaways & Limitations
Future CF XAI evaluations should combine standardized computational metrics with broad and specific user testing across diverse domains.
Takeaways & Limitations
The paper does not address several wider requirements, including explanation fidelity, robustness, algorithmic speed and complexity, and ethical or stakeholder issues.
Abstract
from arXiv · showhide
In recent years, there has been an explosion of AI research on counterfactual explanations as a solution to the problem of eXplainable AI (XAI). These explanations seem to offer technical, psychological and legal benefits over other explanation techniques. We survey 100 distinct counterfactual explanation methods reported in the literature. This survey addresses the extent to which these methods have been adequately evaluated, both psychologically and computationally, and quantifies the shortfalls occurring. For instance, only 21% of these methods have been user tested. Five key deficits in the evaluation of these methods are detailed and a roadmap, with standardised benchmark evaluations, is proposed to resolve the issues arising; issues, that currently effectively block scientific progress in this field.
1 Introduction
Counterfactual explanations have attracted attention because they describe actions that could change an automated decision and are claimed to offer technical, psychological, and legal benefits. This paper surveys the field’s evaluation practices, identifies five deficits, and proposes standardized, psychologically grounded benchmarks.
- Why Counterfactuals?: Counterfactual explanations tell users what might be changed to obtain a different automated decision.The paper contrasts this action-oriented information with factual explanations that only describe similarities to previously rejected cases.
- Motivation & Novel Contributions: The proposed roadmap recommends standardized benchmark metrics grounded in psychological testing.The roadmap is intended to provide a firmer basis for evaluating counterfactual methods and addressing user-requirement issues.
- Why Counterfactuals?: Claims that counterfactual explanations are psychologically intuitive and legally compliant remain dependent on sufficiently demonstrated user comprehensibility.The paper notes that plausibility and comprehensibility claims were not always supported by solid evidence.
- Motivation & Novel Contributions: The survey updates earlier reviews by covering 100 CF XAI methods and focusing on psychological and computational evaluation deficits.Earlier reviews surveyed 52 techniques but emphasized technical properties rather than user studies and evaluation shortcomings.
- Motivation & Novel Contributions: The paper identifies five deficits involving user studies, plausibility, sparsity, coverage, and comparative testing.It frames these deficits as barriers to progress in counterfactual XAI evaluation.
2 Counterfactual Insights
The paper organizes counterfactual XAI around high-level insights that motivate its evaluation metrics. It highlights proximity, feature focus, distributional faithfulness, and instance-based generation, while noting that methods differ in which insights they adopt.
- Counterfactual Insights: The survey uses a discovery-based analysis of high-level insights rather than mainly cataloguing technical properties.This approach provides the basis for the paper’s proposed standardized evaluation metrics.
- Counterfactual Insights: Four recurring design ideas are proximity, feature focus, distributional faithfulness, and instance-based generation.Not all surveyed methods adopt every idea, but many treat them as high-level requirements.
- Counterfactual Insights: The paper’s next five sections critique psychological and computational evaluations before proposing ways to rectify the identified deficits.
3 Deficit #1: Neglecting Users
User testing is scarce in counterfactual XAI and often evaluates counterfactual explanations against no-explanation controls rather than testing specific methods. Although reported studies generally find counterfactuals useful, many have methodological weaknesses.
- Neglecting Users: 21% of surveyed papers directly user-test a specific CF method, compared with 31% performing any user study.The broader figure includes 36 of 117 papers, many of which conducted non-model tests.
- Neglecting Users: Many user studies use low sample sizes, inappropriate statistics, or unreproducible designs.These weaknesses limit the evidential value of the evaluation practices described in the survey.
- Neglecting Users: User studies often compare CF explanations with no-explanation controls instead of comparing specific CF methods.The paper therefore distinguishes evidence that CFs help users from evidence supporting one method over another.
- Neglecting Users: Most studies that report user tests find CF explanations useful and sometimes preferred by end users.Lim et al. found that What-if, Why-Not, How-to, and Why explanations all improved performance relative to no-explanation controls.
4 Deficit #2: What’s Plausible?
Counterfactual methods often treat plausibility as proximity, suitable features, distributional faithfulness, or training-data origin, but these claims are usually supported by intuition or proxy metrics rather than user testing. The EEG comparison illustrates the trade-off: the native counterfactual is within distribution but distant, while the synthetic one is close and sparse but out of distribution, and neither is likely plausible to users.
- What counts as plausible?: Most methods claim plausible explanations, but plausibility is variously defined by proximity, feature suitability, distributional faithfulness, or training-data origin.These claims are generally declared intuitively and supported by computational proxies that parallel the theoretical claim.
- Evaluation deficit: Evaluating plausibility through a proxy that directly mirrors the claim risks circularity.The paper explicitly sets aside this issue for later consideration.
- What counts as plausible?: Psychological evidence is missing for how actionable, mutable, feasible, and causal features affect perceived plausibility.The paper specifically leaves open how many suitable features are needed and how acceptability changes with users’ causal models.
- EEG example: L1-norm = 38.29 for the native CF, which is within distribution but distant; L1-norm = 3.88 for the synthetic CF, which is closer and sparser but out of distribution.The synthetic counterfactual is classified as out of distribution by the OCSVM metric because of a spike, and neither example is likely plausible to end-users.
5 Deficit #3: The Shape of Sparsity
Counterfactual sparsity is commonly justified as beneficial because explanations with fewer feature differences seem easier to process, but direct user evidence challenges the assumption that one difference is optimal. Studies found that users rated explanations with 2–3 feature differences better than one-difference explanations, while the best level may vary by data type.
- Intuition behind sparsity: Sparsity is usually treated as beneficial because explanations with few feature differences are intuitively easier to understand.This claim is sometimes linked to limits in working memory, visual memory, or human category learning.
- Evidence on the preferred level: 2–3 feature-difference CFs were rated much better than 1-difference CFs in direct user studies.The studies systematically varied the median number of feature differences shown to users.
- Scope of the finding: The optimal sparsity level may vary across tabular, text, and image data, especially when features are chunked.The paper therefore recommends a low-end benchmark score for distinguishing methods rather than claiming one universal optimum.
6 Deficit #4: Covering Coverage
Coverage asks whether a method reliably produces good counterfactuals across representative problems rather than occasionally generating implausible ones. The survey finds that coverage is rarely reported and proposes explanatory competence as a way to estimate how often users may encounter poor explanations, while noting that its psychological acceptance function needs stronger grounding.
- Coverage deficit: Only 22% of the 100 reviewed systems reported coverage results, and definitions of coverage differed across studies.Coverage may include measures tracking invalid or out-of-distribution counterfactuals.
- Coverage deficit: Out-of-distribution counterfactuals can reach 36% for some methods, and even close, low-sparsity counterfactuals can be out of distribution.The paper cites IM1, IM2, LOF, and OCSVM as measures used to track invalid counterfactuals.
- Explanatory competence: Explanatory competence measures the fraction of a dataset covered by counterfactuals that satisfy a psychologically acceptable explains function.The coverage set is defined over test instances and their counterfactual instances, with competence given as the covered fraction.
- Explanatory competence: Explanatory competence estimates the likelihood that users encounter a poor explanation, but its explains function requires more psychologically grounded definitions.A simple ≤2-feature-difference rule was used in one proposal, but the paper argues that more complex definitions are needed.
7 Deficit #5: Comparative Testing
Comparative testing remains uncommon despite the large and growing number of counterfactual methods. The survey reports that an agreed set of user-grounded evaluation measures could support stronger scientific advances.
- Comparative evidence: 40% of method papers—40 of 100—reported any form of comparative testing.The authors argue that the number of methods makes additional comparisons necessary.
- Comparative evidence: Recent papers are more likely to report comparative tests, and many research groups make their code publicly available.The paper presents these developments as favorable conditions for more systematic evaluation.
- Future evaluation: User-grounded evaluation measures are presented as a basis for solid scientific advances in the field.This conclusion is tied to the need for agreed evaluation measures rather than to any single method.
8 Roadmapping & Benchmarking
The survey recommends psychologically grounded user testing alongside standardized computational benchmarks to evaluate counterfactual explanation methods. It also identifies scope limits, including unstandardized featural constraints and wider requirements outside the survey’s focus.
- Roadmap for Psychological Grounding: The survey recommends broad user testing across diverse domains and specific studies linking computational metrics to human judgments of counterfactual quality.Specific tests should examine proximity, sparsity, and the cognitive appraisal of mutable, causal, and actionable feature differences.
- Benchmarking Evaluative Methods: Four proposed benchmark metrics are proximity, sparsity, coverage, and relative distance.The metrics are intended as standardized computational proxies for comparing the roughly 100 methods in the literature.
- Benchmarking Evaluative Methods: Relative distance compares test-to-counterfactual distances with distances to native counterfactuals to assess instance-guided insight.The proposed comparison assumes, subject to testing, that counterfactuals closer than native alternatives may be better.
- Caveats & Conclusions: The survey focuses on psychological and computational evaluation, while fidelity, robustness, complexity, ethics, stakeholder concerns, and featural constraints remain outside or incompletely standardized.The authors specifically note that mutability, actionability, and causal constraints require dataset-specific agreements and lack sufficient consensus for a broadly applicable metric.
Annotated Bibliography
The annotated bibliography records surveyed work across counterfactual explanation methods, user studies, model proposals, evaluation metrics, reviews, and related non-counterfactual research. Its entries include both foundational references and recent methods addressing explanation generation, evaluation, fairness, trust, and human-centered XAI.
- Reference annotations: The bibliography marks papers as reviews, user studies, models, evaluation works, or non-counterfactual research using explicit reference annotations.The legend distinguishes cited works, reviewed works, user studies, models, evaluation-metric papers, and non-CF papers.
- Reviews and surveys: The references include surveys and methodological reviews of explainable AI, counterfactual explanations, explainable agents, and interpretable machine learning.Examples include Adadi and Berrada, Doshi-Velez and Kim, and Anjomshoae et al.
- Human-centered evaluation: Several references examine human or organizational aspects of explanation, including fairness judgments, trust in automation, algorithm aversion, and the usefulness of case-based explanations.The bibliography also includes work on perceptions of justice and the effects of explanations on complementary team performance.
- Evaluation and metrics: Evaluation-focused entries address proxy tasks, subjective measures, counterfactual risk assessment, and density-based outlier detection.The cited works include Buçinca et al., Coston et al., Breunig et al., and related evaluation studies.
- Counterfactual methods: The bibliography lists counterfactual-generation methods for LVQ models, convex density constraints, multivariate time series, structured data, classification, and multiobjective explanations.These entries include Artelt and Hammer, Ates et al., Dandl et al., and related method papers.