Source-linked AI summary
The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?
Jasmijn Bastings, Katja Filippova
TL;DR
The paper addresses whether attention should explain model predictions when explanations often aim to identify important input tokens for model developers. It compares that goal with input saliency methods and argues that saliency is better suited, while noting task-scope and expressiveness limitations. The paper concludes that attention remains a valid object of study, but attention-as-explanation should receive less focus for this goal and user.
Problem
It is unclear whether attention qualifies as explanation and what goal and user attention-based explanations are intended to serve.
Method
The position paper reviews the attention debate and surveys gradient-, propagation-, and occlusion-based input saliency methods as alternatives.
Results
For identifying relevant inputs for model developers, the paper argues that input saliency methods are better suited than attention.
Takeaways & Limitations
Authors should clearly state the goal and user of an explanation, and attention should not be preferred merely because it supplies one weight per input token.
Takeaways & Limitations
Input saliency methods have limited expressiveness for fully explaining predictions of deep nonlinear models, and some methods have reliability or evaluation problems.
Abstract
from arXiv · showhide
There is a recent surge of interest in using attention as explanation of model predictions, with mixed evidence on whether attention can be used as such. While attention conveniently gives us one weight per input token and is easily extracted, it is often unclear toward what goal it is used as explanation. We find that often that goal, whether explicitly stated or not, is to find out what input tokens are the most relevant to a prediction, and that the implied user for the explanation is a model developer. For this goal and user, we argue that input saliency methods are better suited, and that there are no compelling reasons to use attention, despite the coincidence that it provides a weight for each input. With this position paper, we hope to shift some of the recent focus on attention to saliency methods, and for authors to clearly state the goal and user for their explanations.
1 Introduction
Attention offers an accessible view of model operation, but its use as explanation is debated because the goal and intended user are often unclear. For identifying important input tokens for model developers, the paper argues that saliency methods are better suited than attention.
- Motivation: Attention has improved NLP performance and can visualize which source tokens target tokens attend to, sometimes aligning translations.The paper cites machine translation, generation, and inference as application areas.
- Motivation: The central debate is whether attention’s view into model operation qualifies as an explanation.The paper notes that explanation goals are often insufficiently defined in explainable AI research.
- Explanation goal: Key studies treat explanation as faithfully identifying the input tokens most responsible for a prediction and summarizing the model’s reasoning.This operational goal corresponds to selecting the most important input tokens.
- Explanation goal: That explanation goal coincides with what input saliency methods are designed to produce, while the implied user is often a model developer who values faithfulness.The paper frames this overlap as the reason to compare saliency methods directly with attention.
- Position: The paper argues for shifting attention from attention-as-explanation toward input saliency methods, while retaining attention as a valid object of study and considering methods beyond saliency.The proposal is scoped to the identified goal and user rather than to every possible use of attention.
2 The Attention Debate
The attention-as-explanation debate has focused largely on single-sequence BiLSTM classifiers, where evidence questions whether attention tracks importance or uniquely determines predictions. The paper also notes that these findings may not generalize to tasks where attention is more central, while saliency methods can perform better on machine-translation alignments.
- Debate setup: The debate mostly examines simple BiLSTM text classifiers with one typically MLP-based attention mechanism.For a single input text, the query may be absent or represented by a trained parameter.
- Faithfulness debate: Jain and Wallace report weak correspondence between attention and gradient-based importance, while Serrano and Smith find attention often misses representations most important to predictions.Wiegreffe and Pinter challenge whether these findings disprove attention as explanation altogether.
- Task scope: Findings from single-sequence text classification may not generalize to NLI or machine translation, where attention can be more important to model performance.Uniform attention substantially degrades NMT, whereas random attention minimally affects text-classification performance.
- Task scope: Even in machine translation, saliency methods have been reported to produce better word alignments than attention.Machine translation was an early setting in which attention was visualized for model inspection.
- Causal account: A causal account of explanation raises a further objection because attention cannot be surgically varied while all other deep-network variables remain invariant.This argument treats intervention as a prerequisite for causal explanation.
- Responses: Attempts to improve attention faithfulness modify hidden representations or replace attention with latent alignment models.These approaches respond to concerns about faithfulness or plausibility rather than resolving the broader explanation-goal question.
3 Saliency Methods
Input saliency methods estimate how individual inputs affect a model prediction through gradients, propagation, or feature removal. The section distinguishes sensitivity from saliency and surveys methods that trace or measure input contributions.
- 3 Saliency Methods: Saliency methods for NLP include gradient-based, propagation-based, and occlusion-based approaches applicable to neural models with accessible internals.The overview excludes surrogate-model methods such as LIME and does not endorse a specific saliency technique.
- 3 Saliency Methods: Gradient methods quantify output sensitivity to an input embedding, while gradient×input converts that signal into a scalar saliency for the input.For target class c, the model output is f_c(x_1:n), and the gradient is taken with respect to the input embedding at time step i.
- 3 Saliency Methods: Integrated gradients addresses gradient saturation by averaging gradients along interpolations between a baseline and the original input.The method requires a baseline, such as zero vectors or repeated [MASK] vectors, and averages over m gradients.
- 3 Saliency Methods: Sensitivity measures how input changes affect output, whereas saliency measures each input word’s marginal effect on the prediction.Gradients measure sensitivity; gradient×input and integrated gradients measure saliency because their values depend on the actual input vector.
- 3 Saliency Methods: Layer-wise Relevance Propagation redistributes the output relevance backward through each layer until it reaches the input layers.LRP uses layer-specific propagation rules and allows precise control to preserve relevance, but requires a custom backward pass.
- 3 Saliency Methods: Occlusion-based saliency erases an input feature and measures the resulting change in model output, treating larger changes as evidence of greater importance.For a single example, the method zeroes an input word embedding while leaving other inputs unmodified; this requires n + 1 forward passes.
4 Saliency vs. Attention
The paper argues that explanations are often intended to identify important input words for model developers, for whom faithfulness matters most. For that goal, it prefers saliency methods because they address token relevance directly and can account for the full computation path, while attention is not a strong efficiency advantage.
- 4 Saliency vs. Attention: The explanation goal in many cited studies is to reveal which input words are most important for the final prediction.The paper notes that this framing may follow from attention assigning one weight per input.
- 4 Saliency vs. Attention: The implied explanation user is typically a model developer, for whom faithfulness matters more than agreement with human intuition.Developers aim to understand and possibly improve models, which need not align with human intuition.
- 4 Saliency vs. Attention: Input saliency methods directly estimate each input word’s relevance to a particular prediction and typically account for the computation path from embeddings to output.Attention weights reflect processing at one point, while their representations may already contain information from other inputs.
- 4 Saliency vs. Attention: The paper argues that attention has no compelling advantage as explanation when the goal is faithful input-token importance for model developers.Attention-based explanations are sometimes evaluated against gradient-based measures, which raises why those measures would not be used directly.
- 4 Saliency vs. Attention: Attention’s easy extraction and forward-pass efficiency are not strong arguments against saliency methods, because gradients are simple to compute and many methods need only one backward pass.Attention requires one forward pass, while many alternatives require at most a forward and backward pass.
5 Attention is not not interesting
The paper criticizes attention as input-saliency evidence for model developers while preserving attention analysis as a legitimate research goal and noting that usefulness depends on the explanatory user and goal.
- The paper’s criticism targets using attention to assess input saliency for model developers, not attention research generally.Attention-head analyses and linguistic biases remain valid research uses.
- The authors distinguish studying attention’s role in a model from using attention weights as explanations of input importance.They cite analyses identifying distinct Transformer-head functions and training heads for dependency parsing.
- Attention may be useful when explanations serve a different user and goal than identifying important inputs for model developers.Supervised attention has helped humans complete a task faster than random or unsupervised attention.
6 Is Saliency the Ultimate Answer?
Saliency methods fit the paper’s token-importance goal, but they have evaluation and expressiveness limitations and cannot fully expose why deep nonlinear models predict as they do.
- Beyond saliency: Counterfactual analysis and visualization can serve explanatory goals that differ from assigning flat saliency weights to input tokens.DiffMask additionally indicates where in a model important information is stored and at what layer it becomes known.
- Limitations of saliency: Occlusion-based saliency and erasure evaluations can mistake out-of-distribution corruption for evidence that a feature was important.A probability drop may result because the corrupted input falls off the training-data manifold.
- Limitations of saliency: Per-token saliency is expressive only in a narrow sense and cannot fully explain predictions from deep nonlinear models.Feature-interaction extensions can represent dependencies between important features, but input tokens alone remain insufficient for full understanding.
7 Conclusion
The paper concludes that explanations often seek the inputs most relevant to predictions for a model developer, and argues that saliency methods better fit that stated goal than attention.
- The debate concerns whether attention is explanation, with the recurring explanatory goal of identifying inputs most relevant to a prediction.
- For model developers seeking faithful input-importance explanations, the authors argue that saliency methods are better suited than attention.
- The authors hope researchers will state explanatory goals and users clearly, potentially shifting attention toward saliency or other methods.