Source-linked AI summary
Is Attention Interpretable?
Sofia Serrano, Noah A. Smith
TL;DR
Attention is often assumed to reveal which inputs matter to a model, but its faithfulness has been insufficiently tested. The paper manipulates attention in trained text classifiers and finds that attention weights only noisily predict importance, with gradient-based rankings often performing better. Thus, attention should not be treated as a fail-safe explanation of model decisions.
Problem
The paper asks whether attention weights faithfully identify the relative importance of inputs, rather than merely providing a plausible visualization of decisions.
Method
The authors erase selected intermediate representations by zeroing and renormalizing attention weights, then compare the resulting output distributions and decisions.
Results
Attention weights are only noisy predictors of intermediate importance, while gradient-based rankings often outperform attention-based orderings in finding decision-flipping item sets.
Takeaways & Limitations
Attention can correlate with input impact, but its magnitude is not a fail-safe indicator of what drives a model’s decision.
Takeaways & Limitations
The analysis focuses on five- and ten-class text classification and uses decision flips as an importance signal, which may be too coarse for tasks with much larger output spaces.
Abstract
from arXiv · showhide
Attention mechanisms have recently boosted performance on a range of NLP tasks. Because attention layers explicitly weight input components' representations, it is also often assumed that attention can be used to identify information that models found important (e.g., specific contextualized word tokens). We test whether that assumption holds by manipulating attention weights in already-trained text classification models and analyzing the resulting differences in their predictions. While we observe some ways in which higher attention weights correlate with greater impact on model predictions, we also find many ways in which this does not hold, i.e., where gradient-based rankings of attention weights better predict their effects than their magnitudes. We conclude that while attention noisily predicts input components' overall importance to a model, it is by no means a fail-safe indicator.
1 Introduction
Attention improves many NLP tasks and is often visualized as an explanation of model decisions. This paper tests whether attention weights faithfully identify important inputs and finds that they are only noisy indicators.
- Interpretability matters because complex data-trained models require understandable reasons for individual decisions.
- Attention mechanisms assign normalized weights to input representations before combining them into a fixed-length representation.The weights are nonnegative, sum to 1, and scale the corresponding representations.
- Because attention produces an input distribution, prior work has used its weights to interpret decisions and characterize datasets.
- The paper tests whether attention rankings faithfully describe input importance to the attention layer using intermediate representation erasure.
- Attention weights are only noisy predictors of intermediate components’ importance and should not alone justify a model decision.
2 Testing for Informative Interpretability
The paper treats attention visualization as an importance ranking and tests it by erasing selected attention inputs before classification. The evaluation deliberately targets attention’s low-level faithfulness to intermediate representations.
- Higher attention is treated as implying greater importance of the corresponding input representation.
- The method compares model outputs before and after zeroing selected attention weights and renormalizing the remaining distribution.Renormalization avoids artificially shrinking the document representation toward regions not encountered during training.
- The tests assess intermediate representation importance rather than whether attention provides a holistic explanation of the full model decision.
- Importance is measured using Jensen-Shannon divergence between output distributions and whether their argmax decisions differ.
3 Data and Models
The experiments use attention-based text classifiers across topic and review-rating datasets, varying attention depth and encoder contextualization. Models include hierarchical and flat attention architectures with recurrent, convolutional, or absent contextualizing encoders.
- Experiments cover Yahoo Answers topic classification and IMDB, Yelp 2017, and Amazon review-rating datasets.
- The architectures are inspired by hierarchical attention networks with word-level and sentence-level attention followed by linear softmax classification.
- The tested attention uses the additive softmax formulation, computing weights from encoded representations and a learned context vector.
- The study compares hierarchical models with two attention layers against flat models using one attention layer over contextualized word tokens.
- Encoder contextualization varies from bidirectional recurrent encoders to local convolutional encoders and models without contextualization.
4 Single Attention Weights’ Importance
Single-weight tests provide a tentatively positive but limited picture of attention interpretability: the highest-attended item usually has greater impact, yet impact differences are often small and erasing it usually leaves the decision unchanged.
- 4.1 JS Divergence of Model Output Distributions: The tests compare removing the highest-attended item i∗ with removing a uniformly random attended item r, using output-distribution changes and decision flips.Attention weights are renormalized after the selected item is zeroed out.
- 4.1 JS Divergence of Model Output Distributions: Most of the time, i∗ produces a larger output change than r, and negative ∆JS cases tend to occur when their attention weights are nearly tied.∆JS compares the Jensen–Shannon divergences after removing i∗ versus r.
- 4.1 JS Divergence of Model Output Distributions: Even when attention differs by 0.4, many positive ∆JS values remain close to zero; larger gaps eventually produce an upward swing when one weight dominates.The magnitude of the impact difference therefore remains difficult to predict from attention magnitude alone.
- 4.2 Decision Flips Caused by Zeroing Attention: Decision-flip comparisons expect i∗-only flips to exceed r-only flips, but some datasets show non-negligible reversals and most erasures of i∗ do not change the decision.The authors suggest distributed document-level signal partly explains the many unchanged decisions.
- 4.2 Decision Flips Caused by Zeroing Attention: Overall, single-weight results are tentatively favorable but motivate broader tests because impact differences are often nearly identical or neither erasure flips the decision.These cases limit how confidently attention rankings can explain model behavior.
5 Importance of Sets of Attention Weights
The paper evaluates whether attention-weight rankings identify concise sets of representations whose removal changes model decisions. Across encoder models, gradient-based rankings generally find smaller decision-flipping sets than attention alone, while contextualization and sequence length affect how late flips occur.
- Importance of Sets of Attention Weights: The authors test collective importance by erasing representations from each ranking’s top downward and comparing how many removals are needed to flip the decision.Alternative rankings provide evidence about whether attention places decision-relevant items near the top.
- Attention Does Not Optimally Describe Model Decisions: Descending attention weights often beat random ranking but frequently trail gradient and gradient-attention-product rankings in decision-flipping efficiency.This pattern indicates that attention magnitude is informative in some cases but not consistently optimal.
- Attention Does Not Optimally Describe Model Decisions: For all encoder models, gradient ranking produced faster flips on at least 1.6 times as many test instances as attention ranking than vice versa.Across 16 encoder models, gradient ranking beat attention on over 50% of instances for ten models.
- Decision Flips Often Occur Late: Large fractions of items often had to be removed before a decision flipped, especially in FLAN models attending over sequences hundreds of tokens long.Softmax distributions can concentrate weight on a few items even when decision-relevant information lies among many small-weight components.
- Effects of Contextualization Scope on Attention’s Interpretability: Contextualization scope helps explain architectural differences: encoderless models flip after fewer removals, whereas recurrent encoders preserve decisions longer than local-context or encoderless models.The authors distinguish attention over contextualized intermediate representations from the model’s use of original input tokens.
- Effects of Contextualization Scope on Attention’s Interpretability: The results continue to show that attention rankings are not optimal, with non-random alternatives often finding fewer removals sufficient for a decision flip.The authors therefore caution against treating groups of attention magnitudes as reliable importance indicators.
6 Limitations
The analysis is limited by its focus on text classification, a strict argmax-based importance definition, and coverage of only selected attention architectures and final attention layers.
- Text classification provides a useful decision-flip signal, but such flips may be too coarse for tasks with much larger output spaces.Extending the analysis requires an informative threshold for changes in model outputs beyond decision flips.
- Many tests define importance strictly through the output’s argmax rather than influence toward any output class.A relaxed criterion would require evaluating an attended item’s importance across multiple output classes and identifying what counts as most influential without decision flips.
- Other attention functions, including multi-headed and scaled dot-product attention, may have different interpretability properties.The study also does not examine cases where one attention layer produces multiple attended representations, such as self-attention.
- The evaluation covers only final attention layers, leaving lower-level layers in larger models outside its scope.
7 Related and Future Work
The paper situates its erasure-based analysis among alternative interpretability approaches and proposes turning importance-ranking comparisons into more informative ranking methods.
- The paper’s erasure-based probing is one of several approaches to studying interpretability, alongside training-instance attribution, human-interpretable features, and explanation-behavior comparisons.
- Human feedback could refine what counts as a high-quality explanation derived from attention.
- Future work could transform the importance-ranking comparisons used for evaluation into methods that derive better, more informative rankings.Such rankings could support development of more interpretable models.
8 Conclusion
Attention weights sometimes correlate with impact, but frequently fail to identify the representations most important to a model’s decision. Their usefulness also depends on contextualization scope and remains insufficient for reliably concise explanations.
- Attention does not necessarily correspond to importance, despite often being treated as a model-interpretation tool.
- Higher attention weights often have greater impact than lower weights, but many highest-weight items have little impact.
- Attention-based rankings often fail to identify the sets of representations most important to the model’s final decision.
- Even when attention rankings flip decisions faster than alternatives, they often require zeroing too many attended items to provide a helpful explanation.
- Attention magnitudes appear more helpful without contextualization, while their weaker retrieval of decision rationales elsewhere remains concerning.
A.1 Model Hyperparameters and Performance
The appendix reports model training configurations and test-set classification performance, while noting that some dataset results are not directly comparable or did not match prior performance.
- The models use dataset- and architecture-specific dropout, learning-rate, and patience settings reported for FLAN and HAN variants.
- Table 3 reports classification accuracy for the different trained models on their respective test sets.
- IMDB and Yelp use review datasets different from those in Yang et al. (2016), so their reported performances are not directly comparable.
- The study did not reach comparable performance on Amazon, and Yelp performance was also not comparable because its dataset differed.The authors attribute this partly to insufficient word2vec pretraining and hardware-driven batch-size reductions.
- For datasets with accuracies that passed or approached the original HAN results, the analysis patterns matched those observed elsewhere.
A.2 Full Sets of Plots
The full plot sets examine how attention-based and alternative rankings affect decision flips across architectures and datasets. Results show contextualization and architecture shape when rankings are effective.
- Most ΔJS values remain above 0 and close to 0 when comparing the highest-attention item with a random attended item.
- Negative ΔJS values cluster when the highest and randomly selected attention weights are similar, with notable fat tails for Yahoo Answers HAN models.The Yahoo Answers HAN models also struggle to flip decisions quickly under highest-attention ranking.
- Encoderless models generally have more instances where at least one of the highest-attention or random items flips the decision than other architectures.Yahoo Answers FLAN is the stated exception.
- The appendix includes complete fraction-removed distributions for the first decision flips under each ranking scheme.
A.3 Additional Tests
Additional tests compare attention magnitudes with gradient-based and product-based rankings using decision-flip analyses. Alternative rankings often flip decisions with less removed attention mass, while gradient-based ranking has a more mixed single-item pattern.
- Alternative rankings often require attention probability-mass distributions shifted downward by around 0.25 or more relative to attention-based ranking.The gradient-based and product-based rankings usually flip decisions faster.
- The additional experiments rerun single-weight decision-flip tests using the top items selected by gradient-based and attention-gradient product rankings.
- Product-based ordering yields higher first-item decision-flip lower bounds than attention-based ordering across all contingency tables.The sum of the “yes” rows lower-bounds cases where a ranking flips a decision after removing the first item.
- Gradient-based ordering has a lower first-item-flip sum than attention-based ranking in 14 of 24 models.Its more efficient overall flip distributions therefore arise from faster flips in cases where both rankings require multiple removals.
- Figures 7 and 8 relate attention-weight differences to ΔJS values and group negative ΔJS counts by corresponding weight differences.
- The tests exclude instances with final sequence length 1 because the randomly selected comparison item must differ from the top-ranked item.