Source-linked AI summary
Problems with Shapley-value-based explanations as feature importance measures
I. Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, Sorelle Friedler
TL;DR
Shapley-value explanations are widely used to assign feature importance, but the paper asks whether their mathematical formulation and explanatory uses are actually aligned. It develops a two-part critique and concludes that these methods generally fail their intended purpose, warranting use only where their interpretation is clear.
Problem
Shapley-value feature explanations require choices about feature coalitions and value functions, raising concerns about whether their mathematical properties and explanatory outputs fit the goals of explainability.
Method
The paper combines a mathematical critique of Shapley-based feature importance with a human-centric evaluation against established frameworks of explanation.
Results
The paper finds that Shapley-value-based explanations generally fail to serve their desired purpose because their game-theoretic formulation does not match proposed explanatory use cases.
Takeaways & Limitations
Shapley values should not be treated as a general solution for feature importance; focused, human-accessible approaches should be developed for specific use cases and models.
Takeaways & Limitations
Conditional and interventional formulations each introduce serious issues, respectively requiring feature-dependence modeling or risking out-of-distribution inputs.
Abstract
from arXiv · showhide
Game-theoretic formulations of feature importance have become popular as a way to "explain" machine learning models. These methods define a cooperative game between the features of a model and distribute influence among these input elements using some form of the game's unique Shapley values. Justification for these methods rests on two pillars: their desirable mathematical properties, and their applicability to specific motivations for explanations. We show that mathematical problems arise when Shapley values are used for feature importance and that the solutions to mitigate these necessarily induce further complexity, such as the need for causal reasoning. We also draw on additional literature to argue that Shapley values do not provide explanations which suit human-centric goals of explainability.
1. Introduction
Shapley-value methods have become prominent tools for attributing model predictions to features, motivated by concerns about inscrutable decisions and hidden undesirable signals. This paper argues that these explanations generally fail their intended purpose because their mathematical formulation and human-centered use cases do not align.
- Shapley-value methods treat model features as players and the model prediction as the game's payoff.
- The methods seek to explain complicated model outputs in terms of their inputs, partly to expose illegal bias or task-irrelevant artifacts.
- The paper's critique has two parts: mathematical properties may conflict with explanation expectations, and human-centric frameworks find the explanations wanting.
- The authors caution against general use except in narrowly constrained settings where the resulting attributions have a clear interpretation.
2. Background
Shapley values allocate a cooperative game's total payoff among players by averaging each player's marginal contribution across coalitions or permutations. Feature-importance methods instantiate this framework through value functions for feature subsets, but differ in how those functions are estimated.
- 2.1. Classical Shapley values: In cooperative game theory, a characteristic function assigns each player subset a collective payoff, and Shapley values allocate the grand coalition's value among players.
- 2.1. Classical Shapley values: A player's marginal contribution is the additional value generated when that player joins a coalition.
- 2.1. Classical Shapley values: The Shapley value is a weighted average of a player's marginal contributions across every possible subset, equivalently represented through feature permutations.
- 2.2. Feature importance: Feature-importance methods treat model features as players and define a value function for each feature subset, with the resulting Shapley value interpreted as feature influence.
- 2.2.1. Value functions: Local methods define instance-dependent value functions whose grand-coalition value is the model prediction at that instance, while subset values require an additional specification.
- 2.2.2. Algorithms: Methods differ in whether they estimate conditional or interventional value functions, with these choices relying on different distributional assumptions and approximation procedures.
3. Mathematical issues
The paper identifies mathematical problems in both conditional and interventional Shapley-based feature-importance methods, including dependence on modeling choices, out-of-distribution behavior, and the feature set being explained.
- Conditional distributions: Conditional value functions require modeling feature relationships and produce explanations that can change with the chosen feature set.The paper describes both the need to approximate many multivariate distributions and the ambiguity of which statistically related features count as separate players.
- Conditional distributions: Statistically related features can make two fairness audits of the same function yield quantitatively different results.The choice of whether correlated features are separate players changes the output of additive explanation models.
- Conditional distributions: Causal knowledge can mitigate redundancy problems by assigning zero attribution to fully redundant causal descendants, but this makes explanations sensitive to prior agency assumptions.Asymmetric Shapley Values restrict permutations using causal structure, and the paper treats sensitivity to that knowledge as a significant limitation.
- Interventional distributions: Interventional methods evaluate models on out-of-distribution samples, so explanations can depend on model extrapolation in regions irrelevant to learned in-distribution behavior.Engineered-feature examples require evaluating combinations that violate the data-generating relationship, while such edge-case behavior can be manipulated to distort explanations.
- Interventional distributions: Intervening on feature subsets before applying a model trained on a particular distribution is inherently misleading.The paper presents this as a general challenge for interventional interpretations of feature influence.
- Additivity constraints: Shapley axioms can yield uninformative attributions: multiplicative functions over independent, zero-centered variables receive equal influence regardless of individual feature values.For the described value function, all subsets except the grand coalition evaluate to zero, producing equal Shapley values even when variable magnitudes differ.
4. Human-centric issues
The paper argues that Shapley-value feature importance is poorly matched to human-centered explanation goals, including autonomy, actionability, and normative evaluation. Human explanations are contrastive and selective, whereas Shapley-based explanations often rely on implicit or averaged foils that may not correspond to meaningful alternatives.
- Human-centered explanation goals include autonomy and dignity, educating decision subjects, and enabling debate about whether model rules are justifiable.
- Shapley-value properties are not naturally well-suited to any of these three human-centered objectives.
- Explanations as contrastive statements: Human explanations are typically contrastive, answering why an event occurred rather than a counterfactual event that did not occur.
- Explanations as contrastive statements: Shapley explanations may implicitly contrast f(x) with E(f(x)), even though no data-space scenario may produce that expected outcome.
- Marginal contributions as contrastive statements: Averaging marginal contributions across all feature orderings may obscure the selective contrasts humans find meaningful, such as necessary versus sufficient causes.
- Using Shapley methods to enable action: Shapley influence does not directly guide desirable behavior, and changing a highly influential feature may not improve the outcome.
- Normative evaluation: For normative evaluation, users may lack a clear mental model of Shapley analysis, rely on confirmation bias, and still lack direct answers to task-specific questions.
5. Conclusion
The conclusion argues that mathematically satisfying game-theoretic properties do not make Shapley values a suitable general solution for feature importance. It instead calls for focused approaches grounded in specific use cases and models and designed for human accessibility.
- Shapley values have mathematically satisfying theoretical properties as solutions to game-theory problems, but applying them does not automatically solve feature importance.
- The paper concludes that Shapley-based feature importance is ill-suited as a general solution and recommends focused, use-case-specific approaches developed with human accessibility in mind.