Source-linked AI summary
An unexpected unity among methods for interpreting model predictions
Scott Lundberg, Su-In Lee
TL;DR
Interpreting predictions from complex models is difficult, despite the need to understand why a model made a particular prediction. The paper uses expectation Shapley values to unify feature-importance methods and shows that they are uniquely determined under stated assumptions, with visualizations illustrating their explanations.
Problem
Complex models can be difficult to interpret, while understanding why a particular prediction was made is important.
Source: Introduction · p. 1Method
The paper represents feature contributions with expectation Shapley values, treating interpretable inputs as binary indicators of known or missing features.
Source: Introduction · p. 1, Expectation Shapley values and LIME · p. 2Results
ES values connect several explanation methods and are the only values satisfying the stated assumptions and axioms.
Source: Introduction · pp. 1–2, Expectation Shapley values and LIME · p. 3Takeaways & Limitations
The paper uses ES values to visualize contributions to individual predictions and patterns across predictions, including an income-prediction example involving marriage status.
Source: Visualization of Expectation Shapley… · p. 4The uniqueness result depends on three game-theoretic assumptions that determine the loss, sample-weighting kernel, and regularization term.
Source: Expectation Shapley values and LIME · p. 3
Abstract
from arXiv · showhide
Understanding why a model made a certain prediction is crucial in many data science fields. Interpretable predictions engender appropriate trust and provide insight into how the model may be improved. However, with large modern datasets the best accuracy is often achieved by complex models even experts struggle to interpret, which creates a tension between accuracy and interpretability. Recently, several methods have been proposed for interpreting predictions from complex models by estimating the importance of input features. Here, we present how a model-agnostic additive representation of the importance of input features unifies current methods. This representation is optimal, in the sense that it is the only set of additive values that satisfies important properties. We show how we can leverage these properties to create novel visual explanations of model predictions. The thread of unity that this representation weaves through the literature indicates that there are common principles to be learned about the interpretation of model predictions that apply in many scenarios.
Introduction
As complex models increase the need for prediction-level explanations, the paper presents expectation Shapley (ES) values as a framework connecting several model-agnostic feature-importance methods. These connections relate the methods’ choices and approximations to ES values.
- Prediction-level explanations address the difficulty of interpreting complex models when their behavior cannot be succinctly summarized across all samples [3].Source: Introduction · p. 1
- ES values connect and motivate diverse explanation methods, extending a Shapley-value approach to distribute prediction credit among features.Source: Introduction · p. 1
- LIME’s local approximation fits the ES formalism, which justifies a specific sample-weighting kernel and frames its examples as ES approximations using a different locality kernel [3].Source: Introduction · p. 1
- DeepLIFT values are ES values for a linearized network, motivating DeepLIFT as a sampling-free approximation and uniquely justifying its linearization choices [5].Source: Introduction · p. 1
- Layer-wise relevance propagation is also an ES approximation; its primary difference from DeepLIFT is the reference input, and DeepLIFT’s improvement is described as better ES approximation [1, 5].Source: Introduction · pp. 1–2
- Shapley regression values can be viewed as brute-force ES computation, while ES values permit fast approximations when retraining on all feature subsets is intractable.Source: Introduction · p. 2
Expectation Shapley values and LIME
Expectation Shapley (ES) values express feature contributions using interpretable inputs that mark values or groups as known or missing. Under three game-theoretic assumptions, an additive explanation uniquely determines the local-fitting choices and yields Shapley values.
Expectation Shapley values and LIME
- Interpretable inputs x′ are binary indicators of whether input values or groups are known or missing.The mapping can represent word presence or absence, or whether groups of real-valued measurements were observed.Source: Expectation Shapley values and LIME · p. 2
- LIME’s local approximation fits a simple model g(x′) to the original model near one prediction, balancing fidelity with model complexity.The objective weights samples in interpretable-input space; its loss measures faithfulness and its regularizer penalizes complexity.Source: Expectation Shapley values and LIME · p. 2
- With an additive g, three game-theoretic assumptions uniquely determine the loss, sample-weighting kernel, and regularization, up to transformations that leave the solution unchanged.The stated assumptions include symmetry and monotonicity; the solution also enforces the prediction value and additive output constraints.Source: Expectation Shapley values and LIME · pp. 2–3
- The ES Shapley kernel is symmetric, unlike earlier heuristic local kernels.The paper motivates the weighting kernel by seeking a linear least-squares fit that recapitulates Shapley values.Source: Figure 2 · p. 3, Expectation Shapley values and LIME · p. 3
- ES values are the unique additive explanation satisfying the three stated assumptions, including across examples used to introduce LIME’s formalism [3].The paper identifies ES values as Shapley values of expected-value functions.Source: Expectation Shapley values and LIME · p. 3
Expectation Shapley values and DeepLIFT
DeepLIFT explains input impacts in compositional models using a reference input and recursive linearization. The paper connects its values to ES approximations and uses that connection to justify specific linearization choices.
Expectation Shapley values and DeepLIFT
- DeepLIFT values approximate ES values when its reference input estimates expected input values, and its recursive rules linearize nonlinear network components.The paper says ES values can uniquely justify component-specific linearizations; for the max function, the resulting assignment differs from DeepLIFT’s and may improve responsibility assignment.Source: Expectation Shapley values and DeepLIFT · p. 4
- DeepLIFT’s reference input is designed to represent typical input values.ES implementations approximate missing-data impact by taking expectations, so interpreting the reference as an estimate of E[x] yields an additive model of the same form.Source: Expectation Shapley values and DeepLIFT · p. 4
Visualization of Expectation Shapley values
The paper visualizes expectation Shapley values as additive forces for individual predictions and as stacked explanations for patterns across a dataset. These displays show how feature impacts combine and vary across inputs.
Visualization of Expectation Shapley values
- In the single-prediction display, red and blue segments push the output right or left from the base value in proportion to their ES values.The final position equals the model output f(x).Source: Visualization of Expectation Shapley… · p. 4
- Stacking rotated prediction displays and ordering them by explanation similarity reveals dataset-level patterns, including marriage status as the strongest income predictor in the UCI adult census example.The paper suggests this pattern indicates that many joint incomes were reported.Source: Visualization of Expectation Shapley… · p. 4
- Figure 4 compares ES, classical Shapley, and LIME estimates across increasing model evaluations for dense and sparse decision-tree examples.It shows the 10th and 90th percentiles from 200 replicate estimates at each sample size.Source: Figure 4 · p. 5
Sample Efficiency and the Importance of the Shapley Kernel
The section contrasts sampled classical Shapley estimation with penalized regression and emphasizes that the local weighting kernel changes feature attributions. The Shapley kernel improves estimation accuracy on the nonlinear tree examples, while alternative weighting can change impact magnitudes or directions.
Sample Efficiency and the Importance of the Shapley Kernel
- Connecting Shapley values to locally weighted linear models can make Shapley estimation more efficient and theoretically justify the models’ weighting kernel.The section illustrates both benefits.Source: Sample Efficiency and the Importance… · p. 5
- Penalized regression produces much more accurate Shapley estimates than the standard sampled-ordering approach on nonlinear dense 10-feature and sparse 3-of-100 decision trees.The standard approach averages impacts over a small sample of feature orderings.Source: Sample Efficiency and the Importance… · p. 5
- The Shapley kernel differs materially from LIME’s standard exponential kernel: impacts change magnitude for the sparse tree and reverse direction for the dense tree.The examples show that reasonable local weighting kernels need not produce similar attributions.Source: Sample Efficiency and the Importance… · p. 5