PaperlayerView exact revision on arXiv

Source-linked AI summary

The many Shapley values for model explanation

Mukund Sundararajan, Amir Najmi

arXiv:1908.08474v2cs.AIcs.LGecon.TH

TL;DR

Shapley-based model explanations have multiple operationalizations whose differing references to models, training data, and context can produce very different attributions. The paper compares these approaches axiomatically, proposes Baseline Shapley, and establishes proper uniqueness results for Baseline Shapley and Integrated Gradients.

  • Problem

    Different Shapley operationalizations for model explanation can produce very different attributions, undermining the relevance of the standard uniqueness justification.

  • Method

    The paper uses axiomatic analysis to compare Shapley operationalizations and proposes Baseline Shapley as a baseline-based attribution method.

  • Results

    The paper provides uniqueness results for Baseline Shapley and Integrated Gradients, while finding that Shapley with Conditional Expectations is highly sensitive to data sparsity and can produce counterintuitive attributions.

  • Takeaways & Limitations

    Baseline Shapley and Integrated Gradients receive distinct proper axiomatizations for continuous model explanation.

  • Takeaways & Limitations

    The standard four Shapley axioms do not yield uniqueness in continuous settings, so additional axioms are required.

Abstract

from arXiv · show

The Shapley value has become a popular method to attribute the prediction of a machine-learning model on an input to its base features. The use of the Shapley value is justified by citing [16] showing that it is the \emph{unique} method that satisfies certain good properties (\emph{axioms}). There are, however, a multiplicity of ways in which the Shapley value is operationalized in the attribution problem. These differ in how they reference the model, the training data, and the explanation context. These give very different results, rendering the uniqueness result meaningless. Furthermore, we find that previously proposed approaches can produce counterintuitive attributions in theory and in practice---for instance, they can assign non-zero attributions to features that are not even referenced by the model. In this paper, we use the axiomatic approach to study the differences between some of the many operationalizations of the Shapley value for attribution, and propose a technique called Baseline Shapley (BShap) that is backed by a proper uniqueness result. We also contrast BShap with Integrated Gradients, another extension of Shapley value to the continuous setting.

1 Motivation and Related Work

Model explanation uses Shapley values to attribute a specific prediction to base features, but multiple operationalizations reference the model, training data, and context differently. These choices can substantially change attribution quality and even produce counterintuitive results.

  • Attributions distribute a model’s prediction for a specific input across base features, indicating each feature’s influence on that decision.The paper frames this as explanatory information for decisions such as loan approvals.
  • The Shapley value is motivated by a uniqueness theorem stating that it is the unique method satisfying certain desirable properties.This theorem is presented as a strong justification for using Shapley values.
  • Different Shapley operationalizations reference the model, training data, and explanation context differently, making the uniqueness result non-diagnostic for model explanation.The approaches include retraining models, conditional expectations, and alternative feature distributions.
  • Previously proposed approaches can yield counterintuitive attributions, while baseline methods turn features off by replacing their values with those from an auxiliary baseline.Integrated Gradients instead follows a continuous path and requires well-defined gradients, excluding models such as tree ensembles.
  • Conditional-expectation approaches can change attributions substantially because the selected feature distribution affects both their quantity and quality.The literature uses assumptions ranging from independent uniform features to products of marginals and other distributional approximations.

2 Preliminaries

The paper formalizes model explanation as attributing a real-valued model output to features and compares several Shapley-based extensions. It defines their inputs and axioms, then motivates empirical examination of the resulting attribution behavior.

  • The model is represented as a real-valued function f over feature vectors, with x denoting the input to be explained and x_S its restriction to feature set S.For classification, f represents a class score.
  • The Shapley value assigns each feature an attribution whose total equals the value of the set function, equivalently averaging marginal contributions over random feature orderings.The permutation view adds features one at a time and assigns expected marginal contributions.
  • Conditional Expectations Shapley takes an explicand, function, and feature distribution, defining coalition values through the expected model output conditional on the explicand’s coalition features.With the empirical training distribution, this approach is denoted CES(D̂) and is associated with SHAP.
  • Baseline Shapley takes an explicand, function, and baseline, representing a feature’s absence by replacing its value with the corresponding baseline value.Random Baseline Shapley averages these baseline attributions over baselines drawn from a distribution.
  • Integrated Gradients accumulates gradients along the straight-line path from a baseline x′ to the explicand x, and is an analog of the Aumann-Shapley method.Its attribution requires gradients of the prediction score with respect to the features.
  • The comparison framework includes Dummy, Efficiency, Linearity, Symmetry, Affine Scale Invariance, Demand Monotonicity, and Proportionality.Dummy requires zero attribution for features not referenced by the model, while Efficiency allocates the prediction difference from the baseline.
  • The paper also replicates theoretical observations on a diabetes prediction task using 442 patients, ten base features, and a Lasso model trained with a 75%-25% split.The model explains 35% of variance and includes reported coefficients for BMI, blood pressure, and serum measurement s5.

3 An Analysis of CES

The analysis shows that CES(ˆD), which uses conditional expectations over the training distribution, is highly sensitive to sparsity and can violate intuitive attribution properties. These effects produce unstable and counterintuitive attributions, including non-zero scores for dummy features and failures of linearity.

  • CES(ˆD) procedure: CES(ˆD) defines conditional-expectation values over training examples agreeing with the explicand on selected features, then averages marginal contributions across feature permutations.The procedure uses the subset TS of training examples matching the explicand on S and exploits the Downward Closure property to update these sets efficiently.
  • Effect of sparsity: CES(ˆD) is extremely sensitive to sparsity because exact feature-value matches become unlikely for continuous variables.The analysis identifies sparsity as a natural consequence of continuous features and motivates an explicit procedural treatment of CES(ˆD).
  • Effect of sparsity: When every explicand feature value is unique in the training data, the first feature in each permutation receives f(x) − E_D[f(x′)] while all others receive zero.Averaging over permutations therefore gives equal attributions to all variables, even when the function is not symmetric in them.
  • Effect of sparsity: Adding tiny noise can make all features receive identical attributions, while unperturbed CES(ˆD) attributions vary across features such as BMI and Sex.Across 20 explicands, the reported variation differs by feature; smoothing averages predictions over nearby training points using thresholds 0.1 and 0.2.
  • Axiomatic analysis: Because CES’s set function depends on both f and D, axioms applied to that set function need not express natural properties of the model function, producing counterintuitive attributions and large linearity failures.The paper reports large linearity failures for a two-tree, depth-two ensemble on the diabetes prediction task and notes that CES may fail Dummy, Linearity, and Demand Monotonicity when D is dependent.
  • Axiomatic analysis: CES can assign substantial attribution to a dummy feature, including 22.5/2 for each variable in the paper’s example and non-zero scores for seven dummy variables in the diabetes model.BShap assigns zero attributions to the dummy features in the diabetes example, whereas CES can suggest that an irrelevant variable influences the prediction.

4 Baseline Shapley and its Properties

Baseline Shapley (BShap) treats a baseline as the reference for turning features off and receives a proper uniqueness result under cost-sharing axioms. The paper contrasts it with Integrated Gradients (IG), shows their attributions can differ, and relates BShap to conditional-expectation methods.

  • Baseline Shapley: BShap models a feature’s absence by replacing its value with the corresponding value in an auxiliary baseline.It takes an explicand, function, and baseline as inputs.
  • Baseline Shapley versus Integrated Gradients: BShap and IG are continuous extensions of distinct cost-sharing methods: Shapley-Shubik and Aumann-Shapley, respectively.Setting the baseline to zero recovers the corresponding classic cost-sharing methods.
  • Baseline Shapley versus Integrated Gradients: IG follows a straight-line path and aggregates gradients, whereas Shapley applied to micro-features averages over random monotone walks that converge to this path for smooth functions.The convergence argument connects IG to a limiting construction based on discrete Shapley values.
  • Axiomatization: BShap is uniquely characterized by Linearity, Dummy, Affine Scale Invariance, Demand Monotonicity, and Symmetry, while IG has a parallel uniqueness result using Proportionality instead.Both results include minor technical conditions and apply to all attribution problems.
  • Baseline Shapley versus Integrated Gradients: BShap and IG can produce different attributions because they use different paths; in one min-function example, IG assigns 180 and 36, while BShap assigns 170 and 46.The example uses explicand values x1 = 5 and x2 = 1 with a zero baseline.
  • BShap and CES: BShap can coincide with conditional-expectation Shapley under a suitable distribution, isolating differences between the methods to the distribution choice.The baseline parameter adds decision-relevant flexibility but also creates cognitive load and may lack a compelling single choice.

5 Conclusions

The paper finds that Shapley with Conditional Expectations is sensitive to data sparsity and can yield counterintuitive attributions. It responds with uniqueness results for Baseline Shapley and Integrated Gradients, including an additional Strong Monotonicity axiom in appendix results.

  • Conclusions: Shapley with Conditional Expectations is highly sensitive to data sparsity and can produce counterintuitive attributions.The conclusion identifies this as a central finding about the previously used approach.
  • Conclusions: The paper provides proper axiomatizations, or uniqueness results, for Baseline Shapley and Integrated Gradients.These results address the multiplicity of operationalizations emphasized in the paper.
  • Conclusions: The paper introduces Strong Monotonicity as an additional axiom used in results presented in the appendix.Strong Monotonicity compares attribution magnitudes when one function’s partial derivative for a feature is everywhere nonnegative.

B Issues with prior Axiomatic Results

The paper identifies problems in prior axiomatic claims for Shapley-based explanations. One claim that Symmetry is redundant is refuted, while layer-wise application in Deep Shap makes attributions depend on network implementation.

  • Counterexample to prior axiomatization: A proposed redundancy claim is false: Missingness, Local Accuracy, and Consistency can hold while attributions differ from Shapley values.For x1*x2*x3 with all explicand values 1, the alternative permutation method assigns 1 to x3 and 0 to the others, whereas Shapley assigns 1/3 to each.
  • Counterexample to prior axiomatization: The counterexample shows that Symmetry cannot be omitted from the earlier axiomatization.The alternative method satisfies the three claimed axioms but yields unequal attributions for variables treated symmetrically by Shapley.
  • Deep Shap: Deep Shap applies the Shapley value layer by layer rather than to the network as a whole.This makes the resulting attributions sensitive to parameter arrangement instead of only the function computed by the network.
  • Deep Shap: For x1*x2*x3, two algebraically equivalent implementations produce different Deep Shap attributions, violating Symmetry.The implementation ((x1*x2)*x3) yields 1/4, 1/4, and 1/2, while x1*(x2*x3) yields 1/2, 1/4, and 1/4 for x1, x2, and x3, respectively.

C How CES( ˆD) fails other Axioms

CES based on the empirical distribution can violate Demand Monotonicity, Symmetry, and Strong Monotonicity. These failures can make explanations suggest non-monotonicity or asymmetry that the underlying function does not have.

  • Demand Monotonicity: Increasing y can reduce its CES attribution even when the function is monotone in y.The example compares explicands differing only in y and demonstrates a Demand Monotonicity failure.
  • Symmetry: CES can assign unequal attributions to symmetric variables under an independent distribution.For T = 2 and B = 2, the attributions are 1 − p and 1 − q, which differ when p ≠ q.
  • Symmetry: The symmetry failure can lead readers to believe that two variables are non-symmetric when the function is actually symmetric.The paper states this implication for consumers of the explanation.
  • Strong Monotonicity: Strengthening a variable’s influence can reduce its CES attribution even when the feature distribution is independent.In the Strong Monotonicity example, f2 has larger partial derivatives for x than f1, yet x’s attribution falls from approximately 0.032 to 0.
  • Strong Monotonicity: The Strong Monotonicity failure occurs because the transformation affects background-distribution points more than the explicand.Consequently, CES attribution need not increase with the variable’s influence.
  • Empirical scope: The empirical study found no significant Demand Monotonicity failures, and its asymmetric models could not test Symmetry failures.This limits what the diabetes prediction experiment establishes about those axioms.

D.1 Proof of Lemma 3.1

The proof of Lemma 3.1 uses set inclusion: examples agreeing with the explicand on S′ also agree on every subset S of S′.

  • D.1 Proof of Lemma 3.1: If xt belongs to TS′, then it also belongs to TS whenever S is a subset of S′.Agreement with the explicand on the larger feature set S′ implies agreement on each smaller subset.
  • D.1 Proof of Lemma 3.1: The lemma follows from the fact that agreement on more features entails agreement on fewer features.The proof applies this inclusion relation between the corresponding sets of examples.
  • D.1 Proof of Lemma 3.1: The proof establishes the required relationship between TS′ and TS by directly comparing feature agreement conditions.No distributional or model-specific assumption appears in the supplied proof steps.

D.2 Proof of Lemma 4.6

The proof of Lemma 4.6 compares CES with BShap by constructing a distribution concentrated near the baseline. As ε approaches zero, the CES set function converges to the BShap set function.

  • D.2 Proof of Lemma 4.6: The proof constructs a CES feature distribution with two support points, the explicand value xi and baseline value x′i.Each feature’s distribution assigns probability 1 − ε to one support point, as used in the subsequent limiting argument.
  • D.2 Proof of Lemma 4.6: The CES and BShap set functions become arbitrarily close as ε approaches zero.The proof compares their set-function values rather than comparing attributions directly at the outset.
  • D.2 Proof of Lemma 4.6: As ε approaches zero, the CES numerator is dominated by the term with empty S′.That numerator tends to f(xS; x′N\S) multiplied by (1 − ε)^|N\S|.
  • D.2 Proof of Lemma 4.6: The denominator tends to (1 − ε)^|N\S|, so dividing numerator by denominator yields the BShap set function.This limiting ratio is the final bridge from CES to BShap in the proof.

D.3 Proof of Lemma 4.8

The proof shows that, under feature independence, RBShap and CES induce the same set function v(S) for every feature subset S.

  • Under independent feature distributions, RBShap and CES have identical set functions v(S) for all sets S.The proof fixes an arbitrary set S and analyzes the RBShap set function.
  • The derivation simplifies because the relevant expression is dummy in x′.
  • Feature independence yields the intermediate equality, and the final expression is CES’s set function.

D.4 Proof of Corollary 4.3

The proof establishes BShap’s uniqueness by combining its satisfaction of the axioms with uniqueness of the underlying cost shares.

  • BShap satisfies all the stated axioms, and no other method can satisfy them.The reverse direction uses Theorems 4.1 and 4.2 to establish uniqueness.
Loading 1908.08474v2…