Source-linked AI summary
A Unified Approach to Interpreting Model Predictions
Scott Lundberg, Su-In Lee
TL;DR
Interpreting complex model predictions is difficult, and existing explanation methods have unclear relationships and applicability. SHAP unifies additive feature attributions, proves a unique desirable solution, and yields estimates that better align with human explanations and improve computational efficiency.
Problem
Existing methods for interpreting complex model predictions have unclear relationships and when one method is preferable to another.
Method
SHAP defines additive feature attribution methods, unifies six existing methods, and proves a unique solution satisfying desirable properties.
Results
SHAP estimation methods better align with human intuition and more effectively discriminate among model output classes than several existing methods.
Takeaways & Limitations
SHAP shows that common principles for model interpretation can unify existing methods and inform development of future methods.
Takeaways & Limitations
Computing SHAP expected values can be simplified using optional feature-independence or model-linearity assumptions.
Abstract
from arXiv · showhide
Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications. However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability. In response, various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another. To address this problem, we present a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations). SHAP assigns each feature an importance value for a particular prediction. Its novel components include: (1) the identification of a new class of additive feature importance measures, and (2) theoretical results showing there is a unique solution in this class with a set of desirable properties. The new class unifies six existing methods, notable because several recent methods in the class lack the proposed desirable properties. Based on insights from this unification, we present new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.
1 Introduction
The paper introduces SHAP, a unified framework that treats explanations as models and unifies additive feature-attribution methods. It establishes SHAP values as a uniquely justified feature-importance measure and proposes estimation methods aligned with human intuition and model-output discrimination.
- Motivation: Interpreting model predictions supports user trust, model improvement, and understanding the modeled process, but complex models create a tension between accuracy and interpretability.Simple models may be preferred for interpretability even when less accurate, while big data increases the benefits of complex models.
- Unified framework: Viewing each prediction explanation as an explanation model defines additive feature-attribution methods and unifies six current methods.This perspective forms the basis for the additive feature-attribution class developed in Section 2.
- SHAP values: Game-theory results guarantee a unique solution across additive feature-attribution methods, motivating SHAP values as a unified feature-importance measure that existing methods approximate.The framework applies these uniqueness results to the entire additive class.
- New estimation methods: New SHAP estimation methods are better aligned with human intuition in user studies and more effectively discriminate among model output classes than several existing methods.These methods are introduced and evaluated in Section 5.
2 Additive Feature Attribution Methods
Additive feature attribution methods explain complex-model predictions with an interpretable linear model over binary simplified features. Each feature receives an effect value whose sum approximates the original model output, unifying several existing approaches.
- Local explanation setup: Local methods explain f(x) from one input by using simplified inputs mapped to the original space and fitting g to resemble f near that input.The mapping can preserve the current input even when the simplified representation contains less information.
- Definition: An additive feature attribution explanation is a linear function of binary simplified-input variables, with one effect value assigned to each feature.The simplified input has M binary features, and each attribution φ_i is real-valued.
- Definition: Summing feature effects approximates the original model output, making the explanation model an interpretable approximation for complex models.The framework is motivated by the difficulty of using ensemble or deep models themselves as understandable explanations.
- LIME: LIME fits the additive form locally by minimizing a kernel-weighted loss while penalizing explanation complexity, solvable as penalized linear regression.Its mapping converts binary interpretable inputs into original word-count or image-superpixel representations.
- Existing methods: DeepLIFT, layer-wise relevance propagation, Shapley regression values, Shapley sampling values, and Quantitative Input Influence also match the additive explanation form.Shapley regression requires retraining on feature subsets, whereas Shapley sampling uses sampling approximations and training-data integration to avoid retraining.
3 Simple Properties Uniquely Determine Additive Feature Attributions
Within additive feature attribution methods, three properties—local accuracy, missingness, and consistency—uniquely determine a single explanation model. This theorem identifies Shapley values as the sole attribution method satisfying all three properties for a given simplified-input mapping.
- Three desirable properties: The additive attribution class has a unique solution satisfying local accuracy, missingness, and consistency.These properties were familiar from classical Shapley value methods but previously unknown for other additive attribution methods.
- Three desirable properties: Local accuracy requires the explanation model to match the original model’s output for the corresponding simplified input.Formally, g(x′) matches f(x) when x = h_x(x′).
- Three desirable properties: Missingness requires features absent from the original input to have no attributed impact, and all Section 2 methods satisfy it.For feature representations, any feature with x′_i = 0 contributes nothing.
- Three desirable properties: Consistency requires an attribution not to decrease when its simplified-input contribution increases or stays constant across model changes.The condition must hold regardless of the other inputs.
- Uniqueness theorem: Under Properties 1–3, only one additive feature attribution method exists for a given simplified-input mapping, and it uses Shapley values.Methods not based on Shapley values therefore violate local accuracy and/or consistency; Section 4 addresses these violations.
4 SHAP (SHapley Additive exPlanation) Values
SHAP values provide a unified, uniquely justified additive measure of feature importance based on conditional expectations. The section develops model-agnostic and model-specific approximations, including Kernel SHAP and Deep SHAP, to make computation practical.
- SHAP definition: SHAP values are the unique additive feature importance measure satisfying Properties 1–3 while using conditional expectations and the mapping h_x(z′) = z_S.Missing features are represented by omitted values in z_S, although most models cannot directly process arbitrary missing-input patterns.
- SHAP definition: SHAP values assign each feature the change in expected model prediction when conditioning on that feature, explaining the transition from E[f(z)] to f(x).The contribution depends on feature ordering when models are nonlinear or features are dependent, so SHAP values aggregate this attribution across orderings.
- Approximation methods: Exact SHAP computation is challenging, but Shapley sampling values and Kernel SHAP provide model-agnostic approximations under optional feature-independence or model-linearity assumptions.Feature independence yields an unconditional expectation approximation, while model linearity approximates missing features with their expected values.
- Kernel SHAP: Linear LIME recovers SHAP values when its loss, weighting kernel, and regularization are chosen according to the Shapley kernel, enabling weighted linear-regression estimation.Theorem 2 identifies the specific forms that make Equation 2 consistent with Properties 1 through 3.
- Deep SHAP: Deep SHAP combines analytically computed SHAP values for simple network components through recursive back-propagation, providing a fast approximation for compositional deep models.The method leverages connections to DeepLIFT and avoids heuristically selecting component linearizations.
5 Computational and User Study Experiments
The experiments evaluate SHAP approximations for computational efficiency, accuracy, and agreement with human feature-importance judgments. They compare Kernel SHAP with LIME and Shapley sampling, and compare SHAP with LIME and DeepLIFT in user and neural-network studies.
- Computational experiments: The study compares Kernel SHAP, LIME, and Shapley sampling on dense and sparse decision-tree models to assess computational efficiency and accuracy.These comparisons illustrate Kernel SHAP’s improved sample efficiency.
- Computational experiments: Kernel SHAP uses weighted linear regression to estimate Shapley values more accurately with fewer model evaluations than sampling-based estimates.Regularization further improves the linear model’s estimates, as shown in Figure 3.
- User studies: User studies compare LIME, DeepLIFT, and SHAP explanations with human explanations for a sickness-score task and a max-allocation task.Participants assigned credit for the model output or money won in the two settings.
- Deep-network experiments: Deep SHAP extends DeepLIFT’s compositional approach to better approximate SHAP values, and updated DeepLIFT estimates improve performance in a convolutional MNIST example.The example uses a 10-way softmax network and evaluates changes in log odds when masking pixels over 20 random images.
6 Conclusion
The conclusion presents SHAP as a unified framework that identifies additive feature-importance methods and uniquely satisfies desirable properties, while outlining estimation methods and future directions.
- 6 Conclusion: SHAP identifies a class of additive feature-importance methods, includes six previous methods, and has a unique solution satisfying desirable properties.The framework is presented as evidence of common principles for interpreting model predictions.
- 6 Conclusion: The paper presents multiple SHAP-value estimation methods supported by proofs and experiments showing that the values are desirable.
- 6 Conclusion: Future work includes faster model-specific estimators with fewer assumptions, interaction-effect estimation from game theory, and new explanation-model classes.