Source-linked AI summary
Feature relevance quantification in explainable AI: A causal problem
Dominik Janzing, Lenon Minorics, Patrick Blöbaum
TL;DR
The paper addresses confusion over which distribution should be used when Shapley-based feature attribution averages dropped features. Using Pearl’s causal distinction between observational and interventional probabilities, it argues that unconditional expectations are conceptually correct, while parts of SHAP remain unaffected because they use them as approximations.
Problem
Feature-attribution methods disagree about the appropriate probability distribution for dropped features, particularly whether observational conditional expectations provide the right basis.
Method
The paper clarifies the attribution problem using Pearl’s distinction between observational and interventional conditional distributions.
Results
The authors argue that unconditional expectations are the conceptually right way to average dropped features, contrary to SHAP’s theoretical justification.
Takeaways & Limitations
Parts of SHAP are unaffected because they approximate conditional expectations with unconditional expectations, while proposed improvements based on better conditional approximations are considered conceptually flawed.
Takeaways & Limitations
The sensitivity discussion depends on distinguishing the binary function used for Shapley values from the original function f.
Abstract
from arXiv · showhide
We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the confusion is based on not carefully distinguishing between observational and interventional conditional probabilities and try a clarification based on Pearl's seminal work on causality. We conclude that unconditional rather than conditional expectations provide the right notion of dropping features in contradiction to the theoretical justification of the software package SHAP. Parts of SHAP are unaffected because unconditional expectations (which we argue to be conceptually right) are used as approximation for the conditional ones, which encouraged others to `improve' SHAP in a way that we believe to be flawed.
1 Motivation
The paper motivates feature attribution as a way to understand why models produce particular outputs, especially when robustness, fairness, and baseline choice complicate interpretation. It focuses on attributing deviations from the average output by deciding how to average unused features.
- Why interpretability matters: Interpretability is motivated by failures under small adversarial image changes that humans may perceive as minor.Such failures make it difficult to determine which features were decisive for a classification.
- Why interpretability matters: Interpretability also supports scrutiny of algorithmic decisions involving fairness, legal, and ethical concerns.Understanding why an algorithm reached a conclusion is presented as relevant beyond robustness.
- Formal motivation: Feature attribution seeks to quantify how much each input feature contributes to a particular model output.The formal setup treats the model as a real-valued function of discrete or continuous input variables.
- Baseline and attribution target: The paper compares a specific input with the baseline E[f(X)] and attributes f(x) − E[f(X)] by averaging over unused features.The central question is which probability distribution should govern those averages.
- Paper focus: Its main contribution is a causal clarification of the distribution used for dropped features, including disagreement with a proposal by Lundberg and Lee.The discussion distinguishes interventional and observational conditional distributions.
2 Prior Work
Prior work established axiomatic foundations for feature attribution, including integrated gradients for differentiable models and Shapley values for set functions. The paper also situates its discussion within methods that attribute features relative to baselines or expectations.
- Integrated gradients: Integrated gradients attributes a feature relative to a given baseline under an almost-everywhere differentiability assumption.The method was introduced alongside five stated properties, including completeness, sensitivity, implementation invariance, linearity, and symmetry preservation.
- Integrated gradients: Theorem 1 states that attribution methods satisfying four axioms are convex combinations of path methods, while integrated gradients is the only symmetry-preserving path method.The theorem cites Friedman (2004) and Sundararajan et al. (2017).
- Integrated gradients: Convex combinations of path methods may still preserve symmetry without being integrated gradients.Thus the uniqueness claim applies specifically within the class of path methods.
- Shapley values: Shapley values average a feature’s contribution across subsets and can be applied to a set function formed from simplified inputs.The simplified function keeps selected features from x and takes the remaining features from the baseline x′.
- Shapley values: For the resulting binary function, Shapley values satisfy completeness, sensitivity, implementation invariance, and linearity, and preserve symmetry without coinciding with integrated gradients.This connects Shapley attribution to the same axiomatic framework while distinguishing the resulting method.
- Related attribution work: Earlier work investigated feature relevance relative to auxiliary baselines and expectations, including concerns that conditional-distribution Shapley values can assign nonzero attribution to unimportant features.The paper builds its clarification on this existing attribution literature.
3 How should we sample the dropped features?
The paper argues that dropping features should use unconditional, marginal expectations because interventions break their dependence on fixed inputs, unlike observational conditional expectations. A simple irrelevant-feature example and the distinction between the original function and its binary surrogate motivate this causal clarification.
- Marginal expectations: The proposed marginal expectation averages dropped features without conditioning on retained features, which the authors argue is conceptually correct for feature attribution.Lundberg and Lee’s conditional formulation is difficult to compute, while SHAP’s unconditional approximation is defended here as the right notion rather than merely a simplifying approximation.
- Interventional versus observational distributions: Observing X1 = x1 changes the distribution of other variables, so E[Y|X1 = x1] does not describe manually setting X1 to x1.The paper contrasts observational conditioning with Pearl’s interventional perspective and uses a confounded causal structure to illustrate the difference.
- Interventional versus observational distributions: Interventional expectations sample dropped features from their natural joint distribution without conditioning on the fixed inputs.The intervention breaks dependences between the adjusted feature and remaining inputs while preserving dependences among the remaining inputs.
- Irrelevant features: For f(x1, x2) = x1, the irrelevant feature X2 receives Shapley value φ2 = 0 under the marginal-expectation treatment.The example is used to contrast the behavior of conditional and marginal constructions when a feature does not affect the prediction function.
- Sensitivity and the surrogate function: Using conditional expectations can yield φi ≠ 0 even when f does not depend on xi, because the induced binary set function can depend on that feature.The paper resolves the apparent conflict with Shapley sensitivity by distinguishing sensitivity for the binary surrogate from sensitivity for the original function f.
- Scope of the causal interpretation: The causal interpretation concerns algorithm inputs causing the system output, not causal relations among the corresponding real-world features.The formal separation between true features and algorithm inputs permits agnosticism about real-world feature causality.
4 Numerical Evidence
The numerical experiments compare marginal and conditional Shapley expectations, using analytical linear-function attributions as ground truth and Gaussian and human-activity data settings. The authors evaluate KernelSHAP and SHAPR, including kernel estimation of conditional expectations, and report marginal expectations as the better choice.
- The experiments test whether marginal or conditional expectations better recover known feature attributions for linear functions.For linear models, the ground-truth attribution is α_j(x_j − E[X_j]).
- KernelSHAP approximates Shapley values through weighted least squares and subset sampling, while SHAPR estimates conditional expectations with kernels.KernelSHAP samples subsets according to Shapley-kernel weights; conditional estimation uses Mahalanobis distances, bandwidths, and weighted samples.
- Gaussian experiments: The Gaussian experiments vary feature dimension and compare errors from marginal-expectation and conditional-expectation Shapley values.They use three- and ten-dimensional multivariate Gaussian settings, with 1000 runs and samples; Figure 4 displays the resulting error histograms.
- Kernel estimates can approximate conditional expectations in low dimensions, but the paper uses the experiments to argue that marginal expectations yield more accurate attributions.The conditional distribution is sampled directly for Gaussian data, whereas the real-data comparison uses kernel estimation with SHAPR’s default σ² = 0.1.
- Human Activity Recognition experiment: For the human-activity dataset, the authors merge 7352 training and 2948 test samples, randomly select four features, and fit a three-feature linear model targeting the fourth.The experiment uses 10299 combined observations and evaluates attribution error with SHAP and SHAPR.
5 Conclusion
The conclusion frames feature attribution as a causal problem involving the distinction between observational and interventional distributions. It argues that SHAP’s implementation is partly unaffected because it approximates observational expectations with unconditional expectations, while criticizing proposed improvements based on the conditional formulation.
- The paper argues that feature attribution should use interventional rather than observational conditional distributions.This causal distinction is presented as the source of a misconception in recent feature-attribution proposals.
- The authors argue that parts of SHAP remain unaffected because its unconditional approximation is the conceptually appropriate expectation.They distinguish this implementation behavior from the corresponding theoretical justification, which they regard as problematic.
- The conclusion warns that attempts to improve SHAP using the conditional formulation are conceptually flawed according to the authors.The paper also revisits attribution properties, including a symmetry property under interventional probabilities.