Source-linked AI summary
True to the Model or True to the Data?
Hugh Chen, Joseph D. Janizek, Scott Lundberg, Su-In Lee
TL;DR
The paper examines controversy over whether feature attribution should use observational or interventional conditional expectations. Using linear models, a new observational-Shapley computation method, correlation analyses, and credit-risk and biological examples, it finds that the preferable approach depends on whether the goal is to be true to the model or true to the data.
Problem
The paper addresses how to connect a machine learning model to a coalitional game when local feature attribution uses conditional expectations, since observational and interventional formulations differ.
Method
The paper studies linear models, presents a novel algorithm for observational Shapley values, examines correlation effects on convergence, and evaluates credit-risk and biological data settings.
Results
The preferable value function is application dependent: interventional values better modify loan-model predictions, while observational values discover more true features in a sparse biological model.
Takeaways & Limitations
Feature attributions should be chosen according to whether the application prioritizes fidelity to the model or fidelity to correlations in the data.
Takeaways & Limitations
The paper's scientific-discovery conclusions are beyond linear models left for future work, and the RNA-seq alternatives have similar efficacy despite differing computational tractability.
Abstract
from arXiv · showhide
A variety of recent papers discuss the application of Shapley values, a concept for explaining coalitional games, for feature attribution in machine learning. However, the correct way to connect a machine learning model to a coalitional game has been a source of controversy. The two main approaches that have been proposed differ in the way that they condition on known features, using either (1) an interventional or (2) an observational conditional expectation. While previous work has argued that one of the two approaches is preferable in general, we argue that the choice is application dependent. Furthermore, we argue that the choice comes down to whether it is desirable to be true to the model or true to the data. We use linear models to investigate this choice. After deriving an efficient method for calculating observational conditional expectation Shapley values for linear models, we investigate how correlation in simulated data impacts the convergence of observational conditional expectation Shapley values. Finally, we present two real data examples that we consider to be representative of possible use cases for feature attribution -- (1) credit risk modeling and (2) biological discovery. We show how a different choice of value function performs better in each scenario, and how possible attributions are impacted by modeling choices.
1. Shapley values
Shapley values adapt a coalitional-game solution concept to local feature attribution, but the model-to-game mapping is non-trivial. Observational and interventional conditional expectations differ in how they handle feature dependence, creating a context-dependent choice between reflecting model behavior and data correlations.
- Shapley values: Shapley values allocate a coalitional game's surplus among players and are used to attribute a model output to features.For local attribution, known features serve as players and the game's reward is typically a conditional expectation of the model output.
- Conditional expectations: Observational Shapley values condition on known features while retaining their dependence with the remaining features.The input is treated as random, with features in S known at their observed values.
- Conditional expectations: Interventional Shapley values break dependence between known features and the remaining features when evaluating the model.The two approaches therefore encode different coalitional games for the same model.
- Trade-offs: Observational Shapley values can assign importance to model-irrelevant correlated features, whereas interventional values can evaluate points off the true data manifold.These concerns motivate treating each approach as meaningful in an appropriate context rather than as an irreconcilable flaw.
- Trade-offs: The preferred value function depends on whether attributions should reflect a particular model's behavior or correlations present in the data.This paper frames the choice as being true to the model versus true to the data.
2. Linear SHAP
The paper studies observational and interventional Shapley values for linear models, deriving an efficient method for computing observational values under a multivariate normal assumption. The method isolates the exponential computation in terms independent of the explained sample, enabling low-order polynomial-time explanations for multiple samples.
- The paper focuses on linear models and presents a novel algorithm for computing observational Shapley values.The model is represented as f(x) = βx + b.
- Estimating observational conditional expectations is difficult in general, so the method assumes multivariate normal inputs x ∼N(µ, Σ).
- The method computes conditional expectations for feature subsets and uses them to obtain observational conditional expectation Shapley values.For linear models, E[f(X)] = f(E[X]), allowing conditional expectations of inputs to be used in the model.
- The direct computation has complexity O(|N|^32|N|−1) for all feature attributions when terms are reused.
- The algorithm factors the computation into matrices independent of x, so one exponential computation supports explanations for all samples in low-order polynomial time.Computing T(µ) and T(x) remains exponential, but subsequent Shapley values can be computed quickly.
- Sampling the factored terms converges faster than traditional sampling because it avoids separate convergence for each input feature.
3. Effects of Correlation
Correlation changes both the allocation and convergence of observational Shapley values: correlated features share coefficient credit, and stronger correlation slows convergence. In the NHANES example, observational attributions also depend on which features are included in the explanation, allowing importance for an excluded but correlated feature to appear.
- 3.1. Impact of correlation on convergence: When features are independent, observational Shapley values equal interventional values, with estimates φ(xf, f) = [1, 2, 3] and immediate convergence.The simulated model is f(x) = 1 × x1 + 2 × x2 + 3 × x3 for xf = [1, 1, 1].
- 3.1. Impact of correlation on convergence: Correlation splits β as credit between correlated variables, and higher correlation leads to slower convergence of observational Shapley estimates.
- 3.2. Explaining a feature not used by the model: The NHANES mortality task uses 25,535 individuals and estimates observational values from the sampling covariance and mean.The task predicts 5-year mortality from 1999–2014 data.
- 3.2. Explaining a feature not used by the model: For interventional Shapley values, BMI has no importance because it is excluded from the linear model.The model uses Age, Income ratio, Systolic blood pressure, and Waist circumference.
- 3.2. Explaining a feature not used by the model: Observational attributions change when four versus five features are explained, and including BMI lowers the relative importance of the other features.BMI receives importance under the observational values because it correlates with features used by the model.
4. True to the Model or True to the Data
The paper argues that interventional and observational Shapley values suit different applications: model-faithful explanations favor intervention, while data-faithful biological discovery favors spreading credit across correlated features.
- 4.1. True to the Model: Credit-risk explanations should be true to the bank’s model, so the authors hypothesize that interventional conditional expectations are preferable.Interventional values satisfy the Dummy axiom by assigning importance only to features referenced by the model.
- 4.1. True to the Model: Interventional Shapley values produced significantly larger decreases in applicants’ predicted log odds of default than observational values.Applicants ranked features by Shapley value and mean-imputed them to measure changes in the model’s predicted risk.
- 4.1. True to the Model: For linear models, intervention prioritizes feature i when |βi(xi −µi)| is largest, matching the feature whose mean imputation most changes the model output.Observational values can spread importance to correlated features that the model does not explicitly use.
- 4.2. True to the Data: In biological discovery, observational Shapley values better reflect the data-generating process by spreading credit among correlated genes informative about the outcome.The experiment simulated drug response from 40 causal genes among 1000 genes, trained a Lasso model, and ranked genes by average-magnitude Shapley values.
- 4.2. True to the Data: With interventional Shapley values, recovering true features improves when the underlying model itself spreads credit across correlated groups, as Elastic Net does more than Lasso.This contrasts with explaining a sparse Lasso model, where observational values recover more true features by distributing credit among correlated features.
- 4.2. True to the Data: When seeking data-faithful attributions, observational Shapley values and Elastic Net offer two ways to spread credit across correlated features, with computation time affecting the choice.The paper presents these as alternative strategies for representing correlated information rather than treating one value function as universally correct.
5. Conclusion
The paper finds that the preferable Shapley value approach depends on whether attribution should be true to the model or true to the data. In RNA-seq, two similarly effective feature-identification solutions differ substantially in computational tractability.
- In loan modeling, interventional Shapley values modify model predictions more effectively when explanations should be true to the model.
- In biological discovery, observational Shapley values discover more true features for sparse Lasso models when explanations should be true to the data.
- Interventional Shapley values recover more true features with models that spread credit among correlated features than with sparse models.
- RNA-seq yielded two similarly effective solutions: Lasso with observational Shapley values and Elastic Net with interventional Shapley values.The approaches distribute correlation through attribution versus model estimation, respectively.
- The Elastic Net–interventional approach was far more computationally tractable, while its broader suitability beyond linear models remains future work.
- When perturbed features are independent, observational and interventional approaches yield the same attributions, potentially reducing the true-to-model versus true-to-data tradeoff.