Source-linked AI summary
The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations
Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, Marcin Detyniecki
TL;DR
Post-hoc counterfactual explanations may reflect classifier artifacts rather than knowledge grounded in training data. The paper defines path-based justification, develops Local Risk Assessment to detect unjustified examples, and finds high risk across datasets plus vulnerability in state-of-the-art methods. It concludes that faithful post-hoc explanations tied to both classifier behavior and ground-truth data remain difficult.
Problem
Post-hoc counterfactual explanations lack guarantees of faithfulness to training data, raising whether generated instances are continuously connected to ground-truth examples.
Method
The paper defines justified counterfactuals through ε-chain connectedness and uses Local Risk Assessment to examine local neighborhoods for unjustified examples.
Results
The risk of generating unjustified counterfactuals is high across several datasets, and state-of-the-art approaches may generate them, with justified outputs falling as low as 63%.
Takeaways & Limitations
Counterfactual explanations should be connected to correctly predicted training data, while existing post-hoc approaches do not reliably ensure this property.
Takeaways & Limitations
The findings depend on classifier characteristics, data dimensionality and density, and label structure; the authors also report no satisfying current way to ensure faithfulness to both classifier and ground-truth data.
Abstract
from arXiv · showhide
Post-hoc interpretability approaches have been proven to be powerful tools to generate explanations for the predictions made by a trained black-box model. However, they create the risk of having explanations that are a result of some artifacts learned by the model instead of actual knowledge from the data. This paper focuses on the case of counterfactual explanations and asks whether the generated instances can be justified, i.e. continuously connected to some ground-truth data. We evaluate the risk of generating unjustified counterfactual examples by investigating the local neighborhoods of instances whose predictions are to be explained and show that this risk is quite high for several datasets. Furthermore, we show that most state of the art approaches do not differentiate justified from unjustified counterfactual examples, leading to less useful explanations.
1 Introduction
Post-hoc counterfactual explanations are useful but may be disconnected from training data and difficult to generate reliably. The paper proposes tests for this risk and reports that it is high and affects state-of-the-art approaches.
- Post-hoc methods generate explanations without access to the model’s training knowledge, so their faithfulness to the original data is not guaranteed.
- Counterfactual explanations identify the minimal perturbation that changes a classifier’s prediction, helping users understand locally influential features.
- The paper proposes a ground-truth-labelled-data desideratum for more relevant counterfactual explanations.
- The paper designs a test to identify the risk that undesirable counterfactual examples disturb explanation generation.
- The risk of generating undesirable counterfactual examples is high across several datasets and classifiers.
- State-of-the-art post-hoc counterfactual approaches may generate unjustified explanations.
2 Background
Post-hoc interpretability builds explanations from model-labelled samples or comparison instances, including actionable counterfactuals. Its flexibility and usefulness coexist with concerns about stability, misleading explanations, and disconnection from ground truth.
- Post-hoc systems sample and label instances with a black-box model to approximate its decision function and build understandable explanations.
- Counterfactual explanations provide actionable instances from another predicted class by finding minimal perturbations to the observation.
- Counterfactual formulations vary by distance, inverse-problem formulation, and classification-score target.
- Prior work identified risks including surrogate explanations that point in wrong directions and broader concerns about explanations disconnected from ground truth.
- Two classifiers with identical test accuracy, 0.78, can still make questionable decisions in different regions of feature space.
3 Ground-truth Justification
The paper defines justified counterfactuals through path connectedness to correctly predicted training instances and approximates this relation with ε-chains. This yields operational JCF/UCF labels for black-box settings.
- Ground-truth justification distinguishes explanations supported by training knowledge from counterfactuals caused by classifier artifacts.
- The distinction matters because explanations unsupported by existing medical cases could be conceptually useless or dangerous.
- A counterfactual is justified when it shares its prediction with a correctly predicted training instance and connects to it without crossing a decision boundary.
- An ε-chain replaces continuous connectedness with a finite sequence of points whose successive distances are below ε.
- An ε-justified counterfactual has an ε-chain to a correctly predicted instance, with every chain point retaining the counterfactual’s class.
- The paper labels counterfactuals satisfying Definition 2 as JCF and those failing it as UCF.
4 Assessment of the Local Risk of Generating Unjustified Counterfactuals
Local Risk Assessment estimates whether counterfactuals sampled near an observation are ε-connected to correctly predicted training instances. It expands the search through hyperspherical layers and flags residual unconnected regions as risky.
- Local Risk Assessment tests whether unjustified counterfactuals exist in a classifier’s local neighborhood around an observation to be explained.
- Definition Step: The procedure defines a neighborhood using the distance to the closest correctly predicted instance from another class.
- Initial Assessment: It samples many instances in that neighborhood, identifies those sharing the target counterfactual class, and clusters them with the reference instance using ε-connectivity.
- Iteration Step: Initial clusters may appear unconnected because their paths leave the explored neighborhood or lead to another correctly predicted training instance.
- Iteration Step: The procedure repeatedly samples hyperspherical layers around successive correctly predicted instances until clusters are justified or no longer updated.
- Output: If nU > 0, the studied area contains a risk of generating unjustified counterfactual examples.
4.2 Metrics
The paper measures local vulnerability to unjustified counterfactuals with risk and vulnerability scores computed from generated neighborhood regions, repeated across runs.
- Metrics: Sx labels whether instance x is vulnerable to the risk, meaning that its local neighborhood contains a nonzero amount of risk.The score is binary: Sx = 1 when the studied instance is vulnerable.
- Metrics: Rx measures the likelihood of generating an unjustified counterfactual example in the neighborhood explored around x.The score is based on the relative numbers of unjustified and justified regions identified by the procedure.
- Metrics: The Local Risk Assessment procedure tests whether neighborhood regions are connected to correctly predicted training instances and returns counts of justified and unjustified regions.The procedure uses sampled neighborhoods, clustering, and iterative updates before returning nJ and nU.
- Metrics: Average scores S̄ and R̄ are computed over 10 runs for each instance and then aggregated across multiple instances.Repeated runs account for the procedure’s random generation component.
4.3 Illustrative Results
On a deliberately simple half-moons example, Local Risk Assessment identifies an unjustified counterfactual pocket, demonstrating that the risk can occur even in a low-complexity setting.
- 4.3 Illustrative Results: A three-tree random forest trained on 70% of the half-moons data achieved 98% accuracy on the remaining data.The example explores the neighborhood bounded by the closest opposite-class instance.
- 4.3 Illustrative Results: A red pocket inside the explored neighborhood contained no red training instance, so the procedure classified it as unjustified with Sx = 1.The pocket was detected as a group of unjustified counterfactual examples.
- 4.3 Illustrative Results: The example demonstrates that unjustified counterfactual generation risk exists and motivates evaluating the procedure across multiple instances and datasets.Full Local Risk Assessment results are reported later in the paper.
4.4 LRA Parameters: n and ϵ
The parameters n and ϵ control how well the procedure approximates local topology, creating a tradeoff between detection accuracy and computation time.
- 4.4 LRA Parameters: n and ϵ: The choice of n and ϵ defines ϵ-justification and affects the average distance between generated instances.These parameters directly shape the notion of local connectedness used by the procedure.
- 4.4 LRA Parameters: n and ϵ: Higher n and smaller ϵ generally improve approximation of the local neighborhood’s topology.Too few samples can miss small unjustified pockets, while an overly large ϵ weakens the approximation of connected paths.
- 4.4 LRA Parameters: n and ϵ: Improving topological accuracy by increasing n or reducing ϵ increases computational cost.The procedure may need repeated runs to avoid pathological ϵ values, leaving n as the main setting problem.
- 4.4 LRA Parameters: n and ϵ: The procedure assumes that each observation has a sampling threshold beyond which increasing n has little effect on detected clusters and Rx.At that threshold, the local decision-boundary complexity is treated as sufficiently captured.
4.5 Quantitative Results
Across five datasets, Local Risk Assessment finds substantial variation and frequent exposure to unjustified counterfactuals, while the results depend on data and classifier characteristics.
- 4.5 Quantitative Results: The experiments use 70%-30% train-test splits, primarily random forests with 200 trees, and one Gaussian-kernel SVC comparison.The test set is used to run the procedure and calculate average R̄ and S̄ scores.
- 4.5 Quantitative Results: The evaluation covers half-moons, iris, Boston Housing, and ProPublica Recidivism datasets used in interpretability or fairness research.The passage describes two low-dimensional and two real structured datasets.
- 4.5 Quantitative Results: 29%–81% of studied instances across the considered datasets had unjustified counterfactual examples in their neighborhoods.This is reported as the proportion of instances with Sx = 1.
- 4.5 Quantitative Results: R̄ varies greatly across instances and datasets, which the authors relate to local complexity and to characteristics of the classifier, data, labels, and accuracy.The Gaussian-kernel SVC appears to achieve better results than the random forest in the discussed comparison.
- 4.5 Quantitative Results: The results confirm that unjustified-counterfactual risk is important, leaving post-hoc approaches vulnerable because they cannot distinguish ground-truth-backed decisions from classifier artifacts.The paper next evaluates the vulnerability of counterfactual methods to this risk.
5 Vulnerability of Post-hoc CF Approaches
The Vulnerability Evaluation procedure tests whether post-hoc counterfactual methods produce ground-truth-connected explanations in locally risky regions. On the considered approaches, justified counterfactual rates can fall to 63%, with substantial instance-level variability.
- 5.1 Vulnerability Evaluation Procedure: VE: The Vulnerability Evaluation procedure applies Local Risk Assessment to instances with Rx ≥0.25, then tests whether generated counterfactuals are justified.It examines local regions around each generated explanation and uses connectivity to correctly predicted same-class training instances.
- 5.2 Considered Approaches: The evaluation considers HCLS, Growing Spheres, and LORE as representative state-of-the-art post-hoc counterfactual approaches.
- 5.3 Illustrative Results: HCLS generates an unjustified counterfactual in the toy example despite the instance having a consequent local risk.The generated explanation lies in an unjustified area, yielding JE(x) = 0.
- 5.4 Quantitative Results: 63%: the proportion of justified counterfactuals can fall as low as 63% for approaches evaluated in vulnerable regions.The considered methods fail to consistently generate justified counterfactual explanations when Rx ≥0.25.
- 5.4 Quantitative Results: HCLS appears slightly better than Growing Spheres, while LORE shows a similar pattern; greater distance from x may favor connectedness over proximity.The paper relates this pattern to confidence maximization by HCLS and L0 minimization by LORE.
- 5.4 Quantitative Results: Justification scores vary substantially across instances, indicating dependence on the dataset and the classifier's connected predicted regions.
6 Conclusions
The paper defines justified counterfactual explanations through path connectedness, proposes a risk-assessment procedure, and finds post-hoc counterfactual methods vulnerable to unjustified examples. It concludes that satisfactory explanations faithful to both the classifier and ground-truth data remain unavailable, while further work is needed for higher-dimensional settings and classifier complexity.
- The paper defines counterfactual justification using an intuitive requirement based on path connectedness.
- It proposes a procedure to assess the risk of generating unjustified counterfactual examples and shows that post-hoc explanations are vulnerable to this risk.
- The authors argue that no existing post-hoc approach satisfactorily provides explanations faithful to both the classifier and ground-truth data.
- Using training instances seems necessary meanwhile, although this is not always possible in post-hoc use contexts.
- Future work includes adapting the procedure to high-dimensional data and studying how classifier complexity affects justification.
Aknowledgements
The authors acknowledge support from the Joint Research Initiative project funded by the AXA Research Fund and thank Arthur Guillon for discussions and comments.
- The work was conducted as part of the Joint Research Initiative project “Interpretability for human-friendly machine learning models,” funded by the AXA Research Fund.
- The authors thank Arthur Guillon for stimulating discussions and useful comments.