Source-linked AI summary
Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation
Raha Moraffah, Mansooreh Karami, Ruocheng Guo, Adrienne Raglin, Huan Liu
TL;DR
Machine learning models often make consequential decisions without explaining how or why, while existing interpretability surveys largely focus on traditional methods. This paper surveys causal interpretable models, their problems and methods, and evaluation metrics, concluding with an organized overview of traditional and causal approaches and scenario-specific evaluation considerations. It also notes that some interpretable models face accuracy gaps relative to complex models when feature interactions are omitted.
Problem
Black-box machine learning models obscure decision processes, while prior surveys do not comprehensively review interpretability models from a causal perspective.
Method
The survey defines interpretability, reviews traditional and causal methods, categorizes their subareas, and examines metrics for evaluating interpretability.
Results
The survey organizes existing work into traditional and causal interpretability categories and reviews evaluation metrics for different interpretability scenarios.
Takeaways & Limitations
Causal interpretability uses counterfactual analysis to examine how altering data features or model components changes predictions.
Takeaways & Limitations
Interpretable models that omit feature interactions can have a significant accuracy gap relative to complex models.
Abstract
from arXiv · showhide
Machine learning models have had discernible achievements in a myriad of applications. However, most of these models are black-boxes, and it is obscure how the decisions are made by them. This makes the models unreliable and untrustworthy. To provide insights into the decision making processes of these models, a variety of traditional interpretable models have been proposed. Moreover, to generate more human-friendly explanations, recent work on interpretability tries to answer questions related to causality such as "Why does this model makes such decisions?" or "Was it a specific feature that caused the decision made by the model?". In this work, models that aim to answer causal questions are referred to as causal interpretable models. The existing surveys have covered concepts and methodologies of traditional interpretability. In this work, we present a comprehensive survey on causal interpretable models from the aspects of the problems and methods. In addition, this survey provides in-depth insights into the existing evaluation metrics for measuring interpretability, which can help practitioners understand for what scenarios each evaluation metric is suitable.
1. INTRODUCTION
Machine learning systems increasingly make consequential decisions, yet their black-box behavior can obscure why decisions occur and allow bias or unexpected behavior to persist. The survey motivates causal interpretability as a way to analyze counterfactual changes to features or model components and organize existing approaches.
- Motivation: Black-box models in healthcare, law-making, and autonomous cars make decisions whose reasoning humans need to understand.The need is linked to reliability, especially when automated decisions affect people.
- Motivation: Model decisions can be biased or unexpected, including racial bias in COMPAS predictions and susceptibility to minimal image perturbations.These examples motivate explanations that address both social consequences and model robustness.
- Why Causality: Correlation-based interpretability cannot answer how changing a feature or model component would alter predictions or labels.Such questions require reasoning about imaginary scenarios that were not observed or cannot be observed.
- Why Causality: Counterfactual analysis estimates model outputs under imaginary scenarios and supports explanations from both data and model-component perspectives.The survey situates these approaches within causal inference and reviews their emerging use in interpretable machine learning.
2. AN OVERVIEW OF INTERPRETABILITY
Interpretability research spans definitions, inherently interpretable models, and post-hoc explanations. Traditional methods provide human-readable decision logic or feature-level insights, but may sacrifice accuracy or produce misleading visual explanations.
- Interpretability is defined variously as understanding decision causes, predicting model decisions, or explaining models intelligibly to humans.
- Existing reviews mainly address traditional interpretability, leaving causal and counterfactual questions without a comprehensive survey.
- Inherently interpretable models generate explanations during decision-making or training, whereas post-hoc methods explain already-trained models through auxiliary models.
- Interpretable Models: Decision trees explain predictions through root-to-leaf paths that correspond to if-then rules, while rule-based classifiers use potentially overlapping, non-exhaustive, or ordered rules.
- Post-hoc Interpretability: Post-hoc methods include LIME, saliency maps, example-based explanations, influence functions, activation maximization, and explanations using base interpretable models.
- Post-hoc Interpretability: Visual assessment of saliency explanations can be misleading, and activation maximization may return optical illusions because of complex input distributions.
- Interpretable Models: Models that omit feature interactions can show a significant accuracy gap from complex models, while GA2Ms add pairwise interactions and remain interpretable as heat maps.Two healthcare case studies used GA2Ms for pneumonia-risk prediction and uncovered patterns ignored by complex models while retaining their accuracy.
3. CAUSAL INFERENCE
The paper introduces causal-inference concepts used in causal interpretable models. These include structural causal models, causal Bayesian networks, and average causal effect defined through interventional distributions.
- The causal-inference review covers concepts and methods used for causal interpretable models.
- Structural Causal Models: A structural causal model consists of endogenous variables, exogenous variables, causal functions, and a probability distribution over exogenous variables.
- Causal Bayesian Networks: A causal Bayesian network represents a structural causal model with a directed graph whose edges encode parent-to-child causal mechanisms.
- Average Causal Effect: Average causal effect measures the effect of a binary treatment variable on an outcome variable.
- Average Causal Effect: The do(.) operator denotes the interventional distribution defined by the structural causal model or causal Bayesian network.
4. CAUSAL INTERPRETABLITY
Causal interpretability extends traditional interpretability by addressing interventional and counterfactual questions about model components, decisions, fairness, and data relationships. The survey organizes existing methods into four categories and reviews counterfactual generation and causal-effect estimation approaches.
- Causal interpretability addresses “What if” and “Why” questions, whereas traditional interpretability mainly focuses on statistical associations.
- The survey classifies existing work into model-based interpretations, counterfactual explanation generators, causal fairness, and causal interpretability for verifying causal relationships.
- Causal Interpretability and Fairness: Causal interpretability also supports fairness analysis and the verification of causal relationships discovered from data.
- Causal Inference and Model-based Interpretation: Model-based frameworks estimate causal effects of neural-network components by treating the network structure as a structural causal model and reasoning over interventions.
- Causal Inference and Example-based Interpretation: Counterfactual explanations modify selected features or components to answer why a decision occurred or what minimal changes would produce a desired outcome.
5. PERFORMANCE EVALUATION
The survey evaluates causal interpretability through both human-centered and non-human metrics, while emphasizing that evaluation is difficult because causal explanations lack ground-truth data. It also reviews benchmark datasets and their limitations for causal assessment.
- Datasets: Common benchmarks span image, text, and tabular data, including ImageNet, MNIST, PASCAL VOC, review datasets, and UCI datasets.
- Datasets: Existing datasets are generally not designed for causal interpretability and lack ground truth for counterfactual explanations or causal effects.
- Causal interpretability evaluation must assess both whether explanations are interpretable to humans and whether they are causal.
- Non-human Based Evaluation Metrics: Non-human evaluations examine properties such as recovery of important features without relying on human participants.
- Human Subject-Based Evaluation Metrics: Human-subject evaluations test model choice, forward prediction, trust, alignment with human intuition, and comparative explanation quality.
Evaluation Metrics
The survey evaluates causal interpretability through proxy metrics because ground truth for causal explanations is unavailable. Counterfactual explanations are assessed by their prediction validity, sparsity, proximity to data, speed, diversity, and visual-linguistic consistency, while model-based explanations remain difficult to evaluate.
- Evaluation Metrics: Because causal-explanation ground truth is unavailable, evaluation quantifies desired characteristics through predefined proxy metrics.The survey organizes these metrics for different categories of causal interpretability.
- Counterfactual Explanations Evaluation Metrics: Counterfactual predictions should approach the predefined target output, while perturbations should change only a small number of features.These criteria correspond to prediction validity and sparsity.
- Counterfactual Explanations Evaluation Metrics: Counterfactuals are considered more interpretable when they remain close to the training-data distribution and can be generated quickly enough for real-world use.Proximity can use normalized L1 distances for continuous features and mismatch-based distances for categorical features.
- Evaluation Metrics: Diversity metrics require counterfactuals for one input to differ from one another, while fidelity and stability assess local faithfulness and consistency across similar instances.Image occlusion can test whether identified important pixels affect classification, and instability may reflect variance or nondeterministic method components.
- Counterfactual Explanations Evaluation Metrics: Visual-linguistic counterfactuals should preserve class-specific predictive effects in visual regions and remain compatible with their corresponding linguistic explanations.The survey describes accuracy-based metrics using visual attributes and bounding-box intersection over union.
- Model-based Evaluation Metrics: Model-based causal explanations are evaluated by measuring components’ causal effects on outcomes, but evaluation remains an open problem because ground truth is lacking.Neuron-level causal attributions and datasets for causal-effect estimation are cited as evaluation approaches.
6. CONCLUSION
The survey reviews traditional and causal interpretability, emphasizing causal frameworks and the challenge of evaluating interpretable models. It categorizes causal approaches and organizes evaluation metrics according to the scenarios they address.
- 6. CONCLUSION: The survey covers traditional and causal interpretability, with primary focus on causal frameworks and differences between the two categories.Traditional methods are divided into inherently interpretable and post-hoc approaches.
- 6. CONCLUSION: Causal interpretability methods are organized into counterfactual examples, model-based interpretability, causal fairness, and causal-relationship verification.The categorization is intended to provide an overview of existing directions and approaches.
- 6. CONCLUSION: The survey details evaluation metrics for interpretable models and categorizes them by the scenarios for which they are designed.This addresses the challenge of evaluating interpretable models.