PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications

Yu-Liang Chou, Catarina Moreira, Peter Bruza, Chun Ouyang, Joaquim Jorge

arXiv:2103.04244v2cs.AIcs.LG

TL;DR

The paper examines explainability challenges arising from the black-box problem and reviews model-agnostic counterfactual approaches for XAI. It proposes a taxonomy and concludes that these approaches rely on spurious correlations rather than cause-effect relationships, risking biased, sub-optimal, or erroneous explanations.

  • Problem

    The black-box problem leaves humans reliant on answers they cannot scrutinize from intelligent systems.

  • Method

    The paper extends the literature by proposing a new taxonomy for model-agnostic counterfactuals.

  • Results

    Current model-agnostic counterfactual explanation approaches are not grounded in formal causal theory and provide spurious correlations rather than cause-effect relationships.

  • Takeaways & Limitations

    The paper highlights new key points and directions for promoting causability in model-agnostic XAI systems.

  • Takeaways & Limitations

    Without a formal causal foundation, these systems cannot promote causal understanding to users and risk biased, sub-optimal, or erroneous explanations.

Abstract

from arXiv · show

There has been a growing interest in model-agnostic methods that can make deep learning models more transparent and explainable to a user. Some researchers recently argued that for a machine to achieve a certain degree of human-level explainability, this machine needs to provide human causally understandable explanations, also known as causability. A specific class of algorithms that have the potential to provide causability are counterfactuals. This paper presents an in-depth systematic review of the diverse existing body of literature on counterfactuals and causability for explainable artificial intelligence. We performed an LDA topic modelling analysis under a PRISMA framework to find the most relevant literature articles. This analysis resulted in a novel taxonomy that considers the grounding theories of the surveyed algorithms, together with their underlying properties and applications in real-world data. This research suggests that current model-agnostic counterfactual algorithms for explainable AI are not grounded on a causal theoretical formalism and, consequently, cannot promote causability to a human decision-maker. Our findings suggest that the explanations derived from major algorithms in the literature provide spurious correlations rather than cause/effects relationships, leading to sub-optimal, erroneous or even biased explanations. This paper also advances the literature with new directions and challenges on promoting causability in model-agnostic approaches for explainable artificial intelligence.

1. Introduction

XAI emerged to address opaque machine-learning decisions, especially as these systems entered high-stakes settings and exhibited bias. This paper systematically reviews counterfactual and causability approaches, proposing a taxonomy centered on theories, algorithms, and applications.

  • Black-box models make their internal mechanisms difficult for humans to scrutinize, forcing users to accept predictions without understanding how outcomes arise.
  • The need for understandable decisions has intensified because intelligent systems increasingly support high-stakes decisions and are subject to audit and verifiability requirements.
  • Reported applications reveal problematic model behavior, including reliance on metal tokens in X-ray images and gender, racial, and other discriminatory biases.
  • Interpretability extracts relevant information about model or data relationships, whereas explainability translates that information into human-understandable language.
  • Its analysis argues that model-agnostic counterfactual approaches cannot disentangle correlations from causation, limiting their ability to support causal understanding.
  • The review addresses an unclear state of the art by analyzing counterfactual and causability literature and introducing a taxonomy spanning theory, algorithms, and applications.

2. A General Overview of Current Model-Agnostic Approaches in XAI

Model-agnostic post-hoc methods explain black-box predictions through local approximations and are flexible across predictive models. The section surveys surrogate, LIME, Anchor, and SHAP-based approaches and their reported uses and limitations.

  • Model-agnostic post-hoc approaches extract explanations from opaque models and can be applied flexibly across pre-existing trained predictive models.
  • Partial dependency plots provide pairwise feature effects, while surrogate models train local interpretable models to approximate black-box predictions.
  • LIME: LIME perturbs samples near a local decision boundary and fits a locally weighted interpretable model whose feature weights represent local importance.
  • LIME: LIME explanations have been used for image saliency and debugging, but studies identify uncertainty from sampling randomness, proximity variation, and variation across data points.
  • Anchor: Anchor extends LIME with IF-THEN rules and coverage, indicating the boundaries within which an explanation is valid.
  • SHAP: SHAP maps feature values to players in a coalitional game and distributes prediction-related gains using Shapley values and fairness properties.

3. Systematic Literature Review Towards Counterfactuals and Causability in XAI

The review uses a PRISMA-guided search, LDA topic modelling, and keyword co-occurrence analysis to identify and organize research on counterfactuals, causality, and causability in XAI. The results indicate that relevant literature is recent and sparse, motivating further work on causally grounded approaches.

  • Scope and research questions: The review surveys theories, algorithms, applications, knowledge gaps, and challenges concerning counterfactuals, causality, and causability in XAI.
  • Review procedure: The systematic review follows PRISMA and combines database searching, inclusion and exclusion criteria, publication-data extraction, synthesis, and review-bias assessment.
  • Search process: The search covered Scopus, IEEE Xplore, and Web of Science using terms related to artificial intelligence, explainability, interpretability, causality, and counterfactuals.
  • Topic modelling: LDA identified four topics; the review selected publications classified under Topic 1, focused on explainability, causality, and artificial intelligence, or Topic 3, focused on human aspects.
  • Results and research landscape: After topic filtering and manual selection for causability, causal, and counterfactual terms, the final set contained 125 IEEE Xplore and 632 Scopus documents, while causal-relations literature remained scarce and recent.

4. Counterfactual Approaches Explainable AI: Properties for Good Counterfactuals

Counterfactual explanations seek changes that flip a model’s prediction while remaining close, sparse, feasible, plausible, and understandable. The review shows that distance functions shape these properties and involve trade-offs between sparsity and computational tractability.

  • Counterfactuals identify changes to an instance that can flip its prediction to a desired outcome.
  • Distance functions determine how close generated counterfactuals are to the original instance and produce candidates with different properties.
  • Distance functions: L0-norm methods count changed features and produce sparse candidates, but their nondifferentiability makes efficient solutions difficult.
  • Distance functions: L1-norm encourages sparse solutions, whereas L2-norm is differentiable and computationally efficient but does not produce sparse counterfactuals.
  • Distance functions: L∞-norm restricts the maximum feature change but generally yields less sparse solutions than L0- or L1-norms.
  • Good counterfactuals should satisfy properties including sparsity, proximity, plausibility, feasibility, diversity, and actionability.

5. Counterfactual Approaches in Explainable AI: The Theory

The review organizes model-agnostic counterfactual algorithms into categories based on their theoretical and optimization foundations. These categories differ in the properties they can satisfy and in their relationship to causal formalisms.

  • The proposed taxonomy classifies model-agnostic counterfactual approaches into seven categories representing their underlying theoretical algorithms.
  • Instance-Centric: Instance-centric approaches optimize counterfactuals directly but may fail plausibility, feasibility, and diversity properties.
  • Constraint-Centric: Constraint-centric approaches formulate counterfactual generation as constraint satisfaction and can satisfy properties such as diversity and plausibility.
  • Genetic-Centric: Genetic-centric approaches use feature-vector crossover and mutation and often satisfy diversity, plausibility, and feasibility.
  • Regression- and game-theory-centric approaches derive explanations from regression weights or Shapley values but face difficulties satisfying plausibility and diversity.
  • Case-based reasoning retrieves and adapts stored counterfactuals, while probabilistic graphical-model approaches have potential to satisfy Pearl’s causality framework.

6. Counterfactual Approaches to Explainable AI: The Algorithms

The review analyzed 18 model-agnostic counterfactual algorithms in depth, classifying them by properties, applications, supporting data structures, theoretical backgrounds, and public availability.

  • 18 model-agnostic XAI counterfactual algorithms were analyzed in depth.
  • The algorithms were classified according to counterfactual properties described in the review.
  • The review grouped applications into classification or regression problems and recorded their supporting data structures.
  • The classification also recorded whether each algorithm was publicly available and summarized theoretical backgrounds and applications.

master algorithm.

The reviewed algorithms use different objectives, distance functions, search procedures, and constraints to generate counterfactual explanations. Their designs trade off proximity, sparsity, interpretability, feasibility, diversity, and search efficiency.

  • Instance-Centric: WatcherCF and related instance-centric methods optimize prediction loss together with distance from the original instance.
  • WatcherCF: Median-scaled L1 distance is presented as a strong choice because it encourages sparse counterfactual candidates.
  • Prototype Counterfactuals: Prototype Counterfactuals add prototype, autoencoder, and regularization terms to improve interpretability and accelerate search.
  • Weighted Counterfactuals: Weighted counterfactuals use feature-importance or nearest-neighbour weighting, with feature-importance weights producing more compact explanations.
  • FACE: FACE uses density-weighted shortest paths and Dijkstra’s algorithm to generate feasible counterfactuals between data instances and candidate points.
  • FACE: FACE experiments found that the third weight equation combined with ϵ-graphs generated the most feasible counterfactuals.
  • DiCE: DiCE generates diverse counterfactual sets using a determinant point process, allowing users to choose more understandable alternatives.

6.2. Constraint-Centric Approaches

Constraint-centric approaches generate counterfactuals by encoding predictive, distance, and constraint requirements into optimization or satisfiability problems. MACE supports plausibility, diversity, flexible distance functions, and multiple constraints, while GIC compares perturbation and sensitivity-analysis methods.

  • MACE: MACE maps counterfactual search into satisfiability modulo theory problems combining a predictive model, distance function, constraints, and a threshold.Its constraints include plausibility and diversity, and it seeks explanations within a specified distance threshold.
  • MACE: MACE supports any Lp-norm distance function and any number of constraints.
  • MACE: MACE achieved high plausibility, termed coverage by its authors, and more favorable distances than existing optimization-based approaches.
  • Coherent Counterfactuals: Russell’s coherent counterfactuals use mixed-polytope constraints and integer programming to generate diverse, coherent counterfactuals efficiently.The approach first seeks minimum changes that alter the prediction, then applies mixed-polytope constraints.
  • Generalized Inverse Classification: LVP-FI outperformed the other tested GIC methods, while LVP-BI was comparable to GA and GA+LS; HC+LS performed worst.The methods were evaluated by the average likelihood that test instances conformed to a non-ideal class under varying budget constraints.

6.3. Genetic-Centric Approaches

Genetic-centric approaches use genetic algorithms to generate or optimize counterfactual explanations across mixed data, local surrogate models, and multiple objectives. The surveyed methods target feasibility, robustness, fairness, diversity, interpretability, and fidelity.

  • LORE: LORE generates a balanced neighborhood with a genetic algorithm, fits a decision tree, and extracts counterfactual rules as explanations.
  • CERTIFAI: CERTIFAI uses a custom genetic algorithm with distance functions and constraints to generate counterfactuals for black-box models.It evaluates robustness through CERScore and assesses fairness across linear and non-linear models and input types.
  • CERTIFAI: CERTIFAI supports mixed tabular and image inputs without approximations or assumptions about the model.
  • MOCE: MOCE formulates counterfactual search as multi-objective genetic optimization to produce diverse counterfactuals with trade-offs among objectives.Its objectives address prediction distance, instance similarity, sparse feature changes, and similarity to observed feature combinations.
  • CLEAR: CLEAR constructs balanced synthetic neighborhoods, fits constrained stepwise regressions, and iterates until the best explanation is found.In five case studies, regressions had significantly higher fidelity than LIME.

6.5. Game Theory Centric Approaches

This section surveys game-theoretic, case-based, probabilistic, and related approaches that construct counterfactual explanations through feature attribution, retrieved cases, probabilistic representations, or sampling.

  • SHAP Counterfactual: SHAP-C ranks features with Kernel SHAP, perturbs the most important features, and returns the minimum perturbation set that flips the prediction.
  • Contrastive and Counterfactual SHAP: SHAP-CC answers “Why P not Q?” by separating explanations into “why P?” and “why not Q?” using positive and negative Shapley values.
  • Case-Based Reasoning: Case-based reasoning retrieves, reuses, and revises nearby explanation cases whose counterfactuals are sparse, plausible, diverse, and feasible.The framework introduces explanatory coverage and counterfactual potential as properties supporting good counterfactuals.
  • PRINCE: PRINCE uses a heterogeneous knowledge network and a polynomial-time algorithm to compute recommendation-related PPR contributions for groups of actions.Experiments on Amazon and Goodreads found that PRINCE outperformed baselines in interpretability in user studies.
  • PRINCE: PRINCE outperformed baselines in interpretability in user studies, whereas simpler heuristics failed to find the best explanations.
  • C-CHVAE: C-CHVAE searches for faithful counterfactuals in dense data regions by perturbing low-dimensional representations and decoding them back into the original space.Generated candidates are passed through a pretrained black-box model to assess whether the prediction changes.

6.8. Summary

The review finds that most model-agnostic counterfactual approaches lack a formal causal foundation and rely on correlations rather than cause-effect relationships. It argues that causal grounding is relevant to humanly understandable explanations, although user benefits vary with prior knowledge.

  • Review scope: The review classifies collected model-agnostic counterfactual algorithms by underlying theories, properties, applications, and theoretical backgrounds.
  • User effects: Causal information did not improve decision quality for users with prior task knowledge but enabled better decisions among users without such knowledge.
  • Causal grounding: Most counterfactual generation approaches are not grounded in a formal and structured theory of causality such as Pearl’s framework.
  • Causal grounding: Current model-agnostic counterfactual explanations are based on spurious correlations rather than cause-effect relationships.
  • Implications: The inability to disentangle correlation from causation can produce sub-optimal, erroneous, or biased explanations for decision-makers.

7. Counterfactual Approaches to Explainable AI: Applications

Counterfactual explanations are applied across domains including medical decision support, finance, credit evaluation, and interactive machine-learning analysis. These applications aim to help users examine model decisions, data, and prediction changes.

  • Medical decision support: Medical decision support needs counterfactual explanations amid small labelled datasets and heterogeneous data integration challenges.Structured causal models could provide explanatory factors to support medical experts.
  • Finance and credit: Credit evaluation is a prominent application area, with model-agnostic explainers designed to generate compact and intelligible counterfactuals.Some interfaces use feature-importance weights to generate explanations for end users.
  • Interactive tools: What-IF lets practitioners probe and visualize machine-learning systems, investigate decision boundaries, and explore how data changes affect predictions.The application is open source and requires minimal coding.
  • Interactive tools: ViCE generates instance-centric counterfactuals to contextualize and evaluate model decisions in home-equity credit scenarios.Users can interactively explore both the data and the model.
  • Interactive tools: DECE generates instance- and subgroup-level counterfactuals that users can adjust to find more actionable alternatives.It showed effectiveness in supporting decision-exploration tasks and instance explanations.

8. Towards Causability: Opportunities for Research

The paper identifies properties for causability systems: formal causal grounding, counterfactual explanations, human-centred interaction, inference, and semantic context. It also highlights limited formal grounding and unresolved questions about how causal information affects decisions.

  • Foundations: Causability systems should ground explanations in a formal theory of causality rather than relying only on mathematical optimization.The review finds that existing literature is scarce in this respect.
  • Evidence and limitations: Evidence on causal information and decision quality is mixed: it did not improve decisions for users with prior task knowledge but enabled better decisions for users without it.The review notes that cognitive bias and interface presentation may contribute to these differing outcomes.
  • Counterfactual explanations: Counterfactual explanations are presented as necessary for causability and should incorporate diversity, feasibility, and plausibility.The paper links counterfactual reasoning to human understanding while acknowledging challenges to this view.
  • Human-centric design: Explanations should be adapted to users’ information needs, such as the differing interests of doctors and general users.Human-in-the-loop interaction can help users form mental representations of explanations.
  • Inference: Interactive causal inference should let users ask queries about how known information changes an explanation and its variable relationships.Such interaction is described as promoting transparency and human-centric understanding.
  • Semantic annotations: Semantic annotations, contextual knowledge, domain information, story models, and narratives are proposed to make explanations more human-understandable and believable.The paper treats these elements as currently missing ingredients in XAI.

9. Answers to Research Questions

The review classifies counterfactual algorithms by their theoretical origins and examines their properties, causal grounding, and research opportunities. It concludes that most model-agnostic approaches rely on correlations rather than causation, while identifying formal causal grounding and human-centred design as priorities.

  • Theory and algorithms: The review identifies seven algorithmic families: instance-, constraint-, genetic-, regression-, game-theory-, case-based-, and probabilistic-centric approaches.The taxonomy groups algorithms by their master theoretical approach and counterfactual-generation process.
  • Theory and algorithms: Constraint- and genetic-centric methods can satisfy properties such as diversity and plausibility, whereas regression- and game-theory-centric methods struggle with them.The latter families include approaches built on LIME and SHAP, respectively.
  • Causal grounding: Recourse is identified as the closest surveyed approach to causal counterfactuals, although extracting causal models from observational data remains difficult.Probabilistic approaches may use Pearl’s causality framework but require assumptions when learning from observational data.
  • Causal grounding: Most model-agnostic counterfactual algorithms lack a formal causal theoretical foundation and therefore may not promote causability to human decision-makers.The review reports that their explanations provide spurious correlations rather than cause/effect relationships, with risks of biased, sub-optimal, or erroneous explanations.
  • Causability properties: The paper proposes formal causal theories, counterfactual computation, human-centric explanations, user interaction, semantic annotations, and domain knowledge as causability properties.These properties are intended to connect explanations with users’ needs and understanding.
  • Research opportunities: Future work includes standardized evaluation protocols across objective, functional, and user levels, despite differences among algorithms and theoretical approaches.The review also identifies causal relationships from observational data as an open research question.

10. Conclusion

The conclusion presents a taxonomy and systematic review of model-agnostic counterfactual algorithms, finding that they generally lack formal causal grounding. It advocates causal theories, counterfactuals, probabilistic graphical models, and richer human-centred interfaces for future XAI.

  • Contributions: The review extends prior work with a taxonomy of six model-agnostic counterfactual approaches.The categories are instance-centric, constraint-centric, genetic-centric, regression-centric, game-theory-centric, and case-based reasoning centric.
  • Main finding: Model-agnostic counterfactuals are not based on a formal and structured theory of causality, so they cannot promote causal understanding without risks of biased, sub-optimal, or erroneous explanations.Current systems determine feature relationships through correlation rather than causation.
  • Future directions: The paper identifies formal causal theories, inference, counterfactuals, and probabilistic graphical models as directions for causability-oriented XAI.It describes causal models as a research area with challenges and opportunities.
  • Intelligent user interfaces: Causal models and fluid storytelling are positioned as foundations for next-generation intelligent user interfaces.The proposed vision combines persuasive argumentation with multiple interaction modalities.
Loading 2103.04244v2…