PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Approach to Interpreting Model Predictions

Scott Lundberg, Su-In Lee

arXiv:1705.07874v2cs.AIcs.LGstat.ML

TL;DR

How prediction-explanation methods relate and when to prefer each remain unclear amid the trade-off between accuracy and interpretability. SHAP unifies six methods within an additive attribution framework with a unique solution satisfying desirable properties. Its estimators improve alignment with human intuition and output-class discrimination, while Kernel SHAP provides more accurate estimates with fewer model evaluations than prior sampling methods.

  • Problem

    Relationships among prediction-explanation methods and when to prefer each remain unclear amid the trade-off between model accuracy and interpretability.

    Source: §1 · p. 1

  • Method

    SHAP treats explanations as models and unifies six attribution methods within a class with a unique feature-importance solution satisfying desirable properties.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Results

    SHAP estimators improve alignment with human intuition and output-class discrimination, while Kernel SHAP estimates feature importance more accurately with fewer model evaluations than prior sampling methods.

    Source: §1 · p. 2, §5.1 · p. 8

  • Takeaways & Limitations

    Common principles of model interpretation can inform the development of future explanation methods.

    Source: §6 · p. 9

    Local accuracy requires the explanation model’s output to equal the original model’s output when their inputs correspond under the input mapping.

    Source: Property 1 (Local accuracy) · p. 4

Abstract

from arXiv · show

Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications. However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability. In response, various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another. To address this problem, we present a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations). SHAP assigns each feature an importance value for a particular prediction. Its novel components include: (1) the identification of a new class of additive feature importance measures, and (2) theoretical results showing there is a unique solution in this class with a set of desirable properties. The new class unifies six existing methods, notable because several recent methods in the class lack the proposed desirable properties. Based on insights from this unification, we present new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.

1 Introduction

The paper unifies six existing interpretation methods within additive feature attribution and applies game-theoretic uniqueness results to propose SHAP feature importance values. New SHAP estimation methods better align with human intuition in user studies and distinguish model output classes more effectively than several existing methods.

1 Introduction

  • Interpretability supports appropriate trust, model improvement, and understanding, but growing benefits from complex models sharpen the accuracy–interpretability trade-off.Relationships among existing interpretation methods and when to prefer one over another remain unclear.Source: §1 · p. 1
  • Six existing methods are unified as additive feature attribution methods by treating an explanation of a prediction as a model itself.The paper calls this representation an explanation model.Source: §1 · p. 1
  • Game-theoretic uniqueness results apply across additive feature attribution methods, motivating SHAP values as a unified feature importance measure that various methods approximate.Source: §1 · p. 2
  • New SHAP estimation methods better match human intuition in user studies and discriminate model output classes more effectively than several existing methods.Source: §1 · p. 2

2 Additive Feature Attribution Methods

Six existing explanation methods share an additive explanation model: a linear function of binary simplified inputs that assigns feature effects whose sum approximates a prediction.

2 Additive Feature Attribution Methods

  • Additive feature attribution methods use linear explanation models over binary simplified inputs, assigning each feature an effect whose sum approximates the original model’s output.These interpretable approximations explain individual predictions through input-specific mappings from simplified inputs to original inputs. The simplified features are binary, and their attribution coefficients are real-valued.Source: §2 · p. 2

2.1 LIME

  • LIME [5] fits a local linear surrogate using penalized, kernel-weighted squared-error regression over binary interpretable inputs.For text, binary inputs retain original word counts or replace them with zero; for images, they retain superpixels or replace them with neighboring-pixel averages.Source: §2.1 · p. 2

2.2 DeepLIFT

  • DeepLIFT [8] [7] is additive: feature contributions sum to the prediction’s difference from a user-chosen reference, with the reference prediction serving as the baseline.Its recursive deep-network explanations map binary inputs to original or reference values, where the reference represents a typical uninformative background.Source: §2.2 · p. 3

2.3 Layer-Wise Relevance Propagation

  • Layer-wise relevance propagation shares the additive explanation model and is equivalent to DeepLIFT with every neuron’s reference activation fixed to zero.Its binary mapping selects either the original input value or zero.Source: §2.3 · p. 3

2.4 Classic Shapley Value Estimation

  • Shapley regression values [4] explain linear models with multicollinearity through a weighted average of prediction changes from adding each feature across all feature subsets.The method retrains models on every feature subset, comparing predictions with and without the feature. Its binary inputs indicate inclusion or exclusion, and the empty-feature model’s prediction supplies the additive baseline.Source: §2.4 · p. 3
  • Shapley sampling values [9] and Quantitative Input Influence approximate Shapley values through sampling, sharing the additive explanation model.Shapley sampling values [9] approximate feature removal by integrating over training-data samples, avoiding retraining and reducing the required prediction comparisons. Quantitative Input Influence independently proposes a nearly identical approximation within a broader framework.Source: §2.4 · p. 3

3 Simple Properties Uniquely Determine Additive Feature Attributions

For a fixed simplified-input mapping, local accuracy, missingness, and consistency uniquely determine additive feature attributions as Shapley values [6].

Property 1 (Local accuracy)

  • Local accuracy requires the explanation model g(x′) to match the original model f(x) when x = hx(x′).The simplified input x′ corresponds to the original input x through the mapping hx.Source: Property 1 (Local accuracy) · p. 4, §3 · p. 4

Property 2 (Missingness)

  • Missingness requires features with x′_i = 0 to receive no attributed impact when simplified inputs represent feature presence.All methods described in Section 2 satisfy this property.Source: Property 2 (Missingness) · p. 4, Property 1 (Local accuracy) · p. 4
  • Consistency requires a feature’s attribution not to decrease when its contribution increases or stays unchanged regardless of the other inputs.This requirement compares attributions across changes to the model.Source: Property 2 (Missingness) · p. 4

3 Simple Properties Uniquely Determine Additive Feature Attributions

  • For a fixed simplified-input mapping, Shapley values [6] uniquely satisfy local accuracy, missingness, and consistency within additive feature attribution methods.Methods not based on Shapley values therefore violate local accuracy and/or consistency, since the reviewed methods already satisfy missingness.Source: Property 2 (Missingness) · p. 4, §3 · p. 4

Property 2 (Missingness)

  • Additive feature attribution methods have a unique solution satisfying three desirable properties; the uniqueness proof combines cooperative game theory results and requires Property 2 to adapt Shapley proofs to this class.Young (1985) established Shapley-value uniqueness using axioms similar to Property 1 and Property 3 plus another axiom shown redundant in this setting. The three properties were familiar in classical Shapley value estimation but previously unknown for other additive feature attribution methods.Source: Property 2 (Missingness) · p. 4, §3 · p. 4

4 SHAP (SHapley Additive exPlanation) Values

SHAP defines feature importance through Shapley values of conditional expectations, uniquely satisfying local accuracy, missingness, and consistency within this additive framework. Model-agnostic and model-specific approximations make their computation more tractable.

4 SHAP (SHapley Additive exPlanation) Values

  • SHAP values are Shapley values of conditional expectations, uniquely satisfying local accuracy, missingness, and consistency among additive explanations using this input mapping.Conditional expectations handle missing features. Attributions average changes across all feature orderings, explaining the transition from the expected prediction E[f(z)] to the current output f(x).Source: §4 · pp. 4–5, Figure 1 · p. 5, Kernel SHAP (Linear LIME + Shapley… · p. 5

4.1 Model-Agnostic Approximations

  • Under feature independence, Shapley sampling values [9] and equivalent Quantitative Input Influence estimate each attribution separately through sampled permutations.Kernel SHAP requires fewer model evaluations for similar approximation accuracy.Source: §4.1 · p. 5

Kernel SHAP (Linear LIME + Shapley values)

  • Kernel SHAP recovers Shapley values through weighted linear regression, jointly estimating feature attributions more sample-efficiently than direct Shapley equations.Theorem 2 specifies the loss, weighting kernel, and regularization that recover local accuracy, missingness, and consistency. Linear LIME’s heuristic choices do not recover Shapley values, violating local accuracy and/or consistency.Source: Kernel SHAP (Linear LIME + Shapley… · pp. 5–6

4.2 Model-Specific Approximations

  • Model-specific approximations exploit model structure for speed; the framework describes four such methods, including novel Max SHAP and Deep SHAP.The two model-agnostic alternatives are existing Shapley sampling values and novel Kernel SHAP. Feature independence and model linearity are optional assumptions simplifying expected-value computations.Source: §4 · p. 5, §4.2 · p. 6

Linear SHAP

  • For linear models with independent input features, Linear SHAP approximates feature attributions directly from model weight coefficients.This connection was previously noted by Štrumbelj and Kononenko.Source: Linear SHAP · p. 6

Low-Order SHAP

  • Linear regression using Theorem 2 has complexity O(2^M + M^3) and is efficient for small values of M when conditional expectations are approximated using Equation 11 or 12.Source: Low-Order SHAP · p. 6

Max SHAP

  • Sorting inputs and calculating each input’s probability of increasing the maximum computes Shapley values for an M-input max function in O(M^2) time instead of O(M * 2^M).The method uses a permutation formulation of Shapley values.Source: Max SHAP · p. 7

Deep SHAP (DeepLIFT + Shapley values)

  • Deep SHAP adapts DeepLIFT [8], replacing heuristic component linearizations with SHAP-based backward propagation to approximate whole-network attributions.Interpreting DeepLIFT’s [8] reference as E[x] connects it to SHAP under feature independence and model linearity. Deep SHAP recursively combines analytic attributions for linear, max-pooling, and single-input activation components.Source: Deep SHAP (DeepLIFT + Shapley values) · p. 7

5 Computational and User Study Experiments

Kernel SHAP improved computational efficiency, while SHAP attributions agreed more strongly with human explanations than LIME and DeepLIFT. MNIST experiments also supported attribution estimates closer to SHAP values.

5.1 Computational Efficiency

  • Kernel SHAP estimated feature importance more accurately with fewer model evaluations than Shapley sampling, particularly with regularization.Weighted linear regression provides the computational connection. Dense and sparse decision-tree comparisons also showed that LIME attributions can differ significantly from SHAP values satisfying local accuracy and consistency.Source: §5.1 · p. 8

5.2 Consistency with Human Intuition

  • SHAP agreed much more strongly with human credit assignments than LIME and DeepLIFT in symptom-scoring and maximum-based profit-allocation tasks.Human estimates were the most common explanations among 30 and 52 participants, respectively. SHAP’s performance on maximum functions addresses the max-pooling problem in DeepLIFT [7].Source: §5.2 · p. 8, Figure 4 · p. 9

5.3 Explaining Class Differences

  • DeepLIFT’s compositional approach motivates Deep SHAP, while an updated DeepLIFT version better approximates Shapley values [7].Both DeepLIFT versions explained a normalized linear layer; Kernel SHAP and LIME explained model outputs using 50k samples each. LIME used single-pixel segmentation over digit pixels.Source: §5.3 · pp. 8–9
  • On MNIST, estimates closer to SHAP values improved pixel-masking performance for changing the predicted class from 8 to 3.The test masked 20% of pixels selected by each attribution method; changes in log odds were evaluated over 20 random images.Source: Figure 5 · p. 9, §5.3 · p. 9

6 Conclusion

SHAP unifies six previous methods within an additive feature importance class and identifies a unique solution satisfying desirable properties. The conclusion highlights estimation methods and directions for further development.

6 Conclusion

  • Six previous methods belong to SHAP’s additive feature importance class, which has a unique solution adhering to desirable properties.This unification suggests that common principles of model interpretation can inform future methods.Source: §6 · p. 9
  • Several SHAP-value estimation methods are presented, alongside proofs and experiments supporting the desirability of SHAP values.Source: §6 · p. 9
  • Future directions include faster model-type-specific estimation with fewer assumptions, incorporating game-theoretic interaction-effect estimation, and defining new explanation model classes.Source: §6 · p. 9

Related papers

Loading 1705.07874v2…