Source-linked AI summary
Attention Flows are Shapley Value Explanations
Kawin Ethayarajh, Dan Jurafsky
TL;DR
NLP explanations still often rely on leave-one-out values and attention weights, raising the question of how these methods relate to Shapley Values. The paper proves that attention weights and leave-one-out values generally are not Shapley Values, while attention flow can be Shapley-valued at the layerwise level under a specific payoff.
Problem
Although Shapley-value explanations are widely adopted in machine learning, leave-one-out and attention-based explanations still predominate in NLP, leaving their relationship unresolved.
Source: §1 · p. 1Method
The paper formally tests whether attention weights, attention flows, and leave-one-out values can arise as Shapley Values for a player set and payoff function.
Source: §1 · p. 1Results
Except in the degenerate case, attention weights and leave-one-out values cannot be Shapley Values, whereas attention flows are Shapley Values for same-layer players under an appropriate payoff.
Source: §1 · p. 1Takeaways & Limitations
The authors argue that NLP practitioners should use attention-flow explanations alongside more traditional explanations when possible.
Source: §1 · p. 2The equivalence holds only for the payoff function defined as total flow through a layer, which reflects model confidence but not prediction correctness.
Source: §7 · p. 5
Abstract
from arXiv · showhide
Shapley Values, a solution to the credit assignment problem in cooperative game theory, are a popular type of explanation in machine learning, having been used to explain the importance of features, embeddings, and even neurons. In NLP, however, leave-one-out and attention-based explanations still predominate. Can we draw a connection between these different methods? We formally prove that -- save for the degenerate case -- attention weights and leave-one-out values cannot be Shapley Values. $\textit{Attention flow}$ is a post-processed variant of attention weights obtained by running the max-flow algorithm on the attention graph. Perhaps surprisingly, we prove that attention flows are indeed Shapley Values, at least at the layerwise level. Given the many desirable theoretical qualities of Shapley Values -- which has driven their adoption among the ML community -- we argue that NLP practitioners should, when possible, adopt attention flow explanations alongside more traditional ones.
1 Introduction
Although Shapley-value explanations are widely adopted in machine learning, leave-one-out and attention-based explanations still predominate in NLP. This paper examines their relationship and proves that attention flow, unlike attention weights and leave-one-out values, can have Shapley-value properties under specific conditions.
1 Introduction
- NLP still predominantly uses leave-one-out and attention-based explanations despite the wider adoption of Shapley-value explanations in machine learning.The paper asks whether these explanation paradigms are connected.Source: §1 · p. 1
- A motivation is that Shapley Values guarantee zero credit to a token that never changes the payoff, regardless of which other tokens are masked.The paper frames this as a null-player property.Source: §1 · p. 1
- Except for the degenerate case, attention weights and leave-one-out values cannot be Shapley Values for any player set and payoff function.The paper frames players as explanation subjects and the payoff as prediction quality.Source: §1 · p. 1
- Attention flow is a max-flow post-processing of attention weights and is Shapley-valued when players come from the same layer, supporting its use alongside traditional explanations when possible.The paper argues this can extend Shapley guarantees to attention flow and increase confidence in black-box NLP interpretations.Source: §1 · pp. 1–2
2 Model Interpretation as a Game
The paper frames model interpretation as a cooperative game in which explanation subjects share credit for prediction quality. It presents equitable allocation criteria and the Shapley Value as a way to assign expected marginal contribution.
2 Model Interpretation as a Game
- Model interpretation is framed as a transferable-utility game in which explanation subjects cooperate to achieve a prediction payoff.Players can include characters, tokens, embeddings, or neurons; the payoff can reflect prediction correctness, confidence, or entropy.Source: §2 · p. 2
2.1 Equitable Allocation
- Equitable allocation is characterized by null-player, symmetry, and additivity conditions for players' values.A player inducing no payoff change gets zero; players inducing equal changes receive equal values; values add across games.Source: §2.1 · p. 2
2.2 The Shapley Value
- The Shapley Value uniquely satisfies the equitable-allocation criteria and assigns each player its expected incremental contribution across coalitions.It can also be expressed as a sum over possible coalitions.Source: §2.2 · p. 2
- Exact Shapley-value calculation is often impractical because of its combinatorial form, so sampled random permutations are used for estimates.The paper asks whether attention weights, attention flows, or leave-one-out values are Shapley Values for some game.Source: §2.2 · pp. 2–3
3 Attention Weights
Except in the degenerate case, attention weights cannot be Shapley Values. Efficiency forces a candidate payoff to be the sum of attention weights, but the proof shows this assignment contradicts the Shapley Value definition for both attending and non-attending players.
3 Attention Weights
- Attention weights are not Shapley Values when some player receives more attention than another.The section states this result outside the degenerate case.Source: §3 · p. 3
- Efficiency would require the payoff to be the sum of attention weights, making each player's value its total incoming attention.Each player has one Shapley Value for a given payoff function.Source: §3 · p. 3
- For an attending player, the proof's payoff assignment implies a Shapley Value of 1, contradicting unequal attention.The argument uses that the attention weights a player assigns sum to 1.Source: §3 · p. 3
- For a non-attending player, redistribution when it is absent implies a Shapley Value of 0, contradicting its nonzero incoming attention.The proof notes that all input embeddings fall into this case and at least one receives attention.Source: §3 · p. 3
4 Attention Flows
Attention flows are Shapley Values when players are restricted to one layer and the payoff is total flow. The result depends on players being disjoint; the paper also shows that attention rollout cannot be a Shapley Value.
4 Attention Flows
- With players from one layer and total flow as payoff, each player's total outflow is its Shapley Value.The flow is computed by max-flow on the self-attention graph, using attention weights as capacities.Source: §4 · p. 3
- Restricting players to one layer makes them disjoint, so blocking one player's flow does not change another player's outflow.The argument would fail across layers, where upstream flow changes can affect downstream flow.Source: §4 · p. 3
- With total flow as payoff, players from different layers violate Shapley efficiency, so the result is limited to the same-layer setting.Under the stated conditions, Shapley Value theoretical properties extend to attention flows.Source: §4 · p. 4
- Attention rollout multiplies attention matrices across layers, but its values cannot be Shapley Values.The paper says this follows by a trivial extension of the proof for attention weights.Source: §4 · p. 4
5 Leave-One-Out
Leave-one-out values are not Shapley Values except in the degenerate case, and the paper’s condition for non-equivalence is said to hold almost always in practice. Leave-one-out can also miss the importance of redundant contributors, particularly with discrete payoffs.
5 Leave-One-Out
- Leave-one-out removes only the entity being evaluated and requires two forward passes, versus O(2^n) passes for Shapley Values.It is a simpler subset of erasure-based interpretability methods.Source: §5 · p. 4
- The section concludes that leave-one-out values are not Shapley Values except in the degenerate case.Source: §5 · p. 4
- Leave-one-out values are not Shapley Values when any player is non-null even after excluding the coalition of all other players.The paper says this condition is almost always satisfied in practice.Source: §5 · p. 4
- For discrete payoffs, leave-one-out can assign zero to two equally important, redundant representations when only one is necessary.With correctness as a binary payoff, the Shapley Values of both players would be non-zero and identical.Source: §5 · pp. 4–5
6 Applications
The paper describes applications of Shapley Values and attention flow to interpreting token behavior, analyzing groups of tokens, and assessing training examples. These applications rely on Shapley properties or on treating groups or examples as players.
6 Applications
- A token with zero attention flow in layer k but non-zero flow in layer k−1 suggests its label-related information was extracted before layer k.This interpretation follows from the Shapley null-player property; zero leave-one-out value does not support the same conclusion.Source: §6 · p. 5
- Shapley Values allow a group of tokens to be defined as one player, with all group members included or excluded together.The paper notes that most interpretability methods lack a canonical way to aggregate scores across units.Source: §6 · p. 5
- Data Shapley can estimate each training example’s contribution to a model’s decision boundary; the paper notes prior use in computer vision but not NLP.The examples contrast two sentences for fine-tuning BERT on sentiment classification.Source: §6 · p. 5
7 Limitations and Future Work
The paper encourages using attention-flow explanations alongside traditional methods, while noting that their Shapley equivalence is tied to a particular payoff. It also discusses computational barriers to general Shapley-value calculation and a trade-off in faster approximations.
7 Limitations and Future Work
- Attention flow is equivalent to Shapley Values only for the payoff of total flow through a layer, which reflects model confidence rather than prediction correctness.Source: §7 · p. 5
- Exact Shapley-value calculation is computationally intractable in most cases, and Monte Carlo estimates can have loose bounds.Finding games with exact polynomial-time Shapley calculation, as with attention flow, is identified as future work.Source: §7 · p. 5
- SHAP is faster because it assumes linear contributions, but that assumption restricts the payoff functions it can represent.Source: §7 · p. 5