Source-linked AI summary
Attention Interpretability Across NLP Tasks
Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, Manaal Faruqui
TL;DR
The paper investigates why attention weights appear interpretable in some studies but not others. It analyzes diverse NLP tasks and model types, finding that interpretability depends on whether attention is essential to prediction rather than reducible to a gating unit. Experiments and manual evaluation support this task- and architecture-sensitive account.
Problem
Existing studies provide conflicting evidence about whether attention weights explain predictions, and earlier negative conclusions focused mainly on text classification.
Method
The paper compares attention across diverse NLP task categories and models, evaluates effects of altering weights, and uses manual evaluation to assess interpretability.
Results
Attention weights are interpretable and correlate with feature importance when essential for prediction, while single-sequence attention can behave as a gating unit.
Takeaways & Limitations
Interpretability claims about attention should be conditioned on the task and on whether the attention layer materially affects the model’s prediction.
Abstract
from arXiv · showhide
The attention layer in a neural network model provides insights into the model's reasoning behind its prediction, which are usually criticized for being opaque. Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.
1 INTRODUCTION
The paper addresses conflicting findings about attention interpretability by testing diverse NLP tasks and proposing conditions that reconcile when attention is and is not interpretable.
- Prior studies disagree on whether attention weights explain predictions or capture linguistic structure.Some work finds that changing weights leaves predictions largely unchanged, while other work links attention to syntax and coreference.
- Earlier conclusions were based mainly on text classification, so their generality across NLP tasks was uncertain.
- The study analyzes text classification, pairwise text classification, and text generation, including self-attention models and manual evaluation.
- Attention weights are claimed to be interpretable when computed using two input-dependent vectors, but not when they function as a gating unit for a single sequence.
2 TASKS AND DATASETS
The experiments cover three task categories—single-sequence, pair-sequence, and generation tasks—using datasets spanning classification, inference, question answering, and translation.
- Single Sequence tasks: Single-sequence tasks classify one text sequence, using SST, IMDB, Yelp 2017, and AG News datasets.
- Pair Sequence tasks: Pair-sequence tasks take two text sequences as input, including NLI and question answering.The experiments use SNLI, MultiNLI, CNN News Articles, and three bAbI settings with one, two, or three supporting statements.
- Generation tasks: Generation tasks generate a sequence from an input sequence; the paper studies NMT using three English-German datasets.The datasets are Multi30k, En-De News Commentary v11, and the full En-De WMT13 dataset.
3 NEURAL ATTENTION MODELS
The paper uses attention-based recurrent and self-attention architectures tailored to single-sequence, pair-sequence, and generation tasks, producing weighted representations or decoder contexts for prediction.
- Single Sequence Models: Single-sequence models encode token embeddings with a Bi-RNN, compute additive attention weights, and feed a weighted representation to a classifier.The weighted representation is hα = Σ_i α_i h_i, followed by a dense layer and softmax prediction.
- Single Sequence Models: Hierarchical attention additionally attends over sentences after attending over tokens to form the instance representation.
- Pair Sequence Models: Pair-sequence models separately encode two inputs and compute attention over tokens of one sequence before classification.For NLI, the inputs are a premise and hypothesis; for question answering, they are a question and paragraph.
- Pair Sequence Models: A conditional pair-sequence variant initializes the second encoder with the final state of the first encoder.This produces a conditional encoding of the second sequence given the first.
- Generation Task Models: NMT models use a recurrent encoder for source-token representations and a decoder that computes attention weights at each target-generation step.The paper evaluates Bahdanau and Luong attention formulations.
- Self-Attention Models: Self-attention models are also examined using fine-tuned BERT for classification tasks and a Transformer for neural machine translation.
4 IS ATTENTION AN EXPLANATION?
The paper evaluates whether attention explains predictions across single-sequence, pair-sequence, and generation tasks. It finds that attention is less consequential and less explanatory for single-sequence tasks, whereas altering it substantially affects other tasks.
- Interpretability conditions: Attention weights in single-sequence models can be reduced to a gating mechanism rather than interpreted as explanations.The reduction uses an identity transformation with softmax gating; the same reduction does not hold when attention also depends on another sequence or decoder state.
- Performance effects: On single-sequence tasks, permuting attention reduces performance by around 4.2 points across datasets and models, while altered weights generally have limited effect on output.Uniform and Random changes produce smaller average decreases when applied during both training and inference than during inference alone.
- Performance effects: 0.1% average relative performance decrease occurs for Uniform (Train+Infer) on single-sequence tasks, compared with 49.5% for pair-sequence and 51.2% for generation tasks.These results support the proposition that altering attention matters more when attention cannot be modeled as a gating unit.
- Feature importance: For single-sequence tasks, large attention-weight differences yield small changes in JS divergence, unlike pair-sequence and generation tasks.The analysis compares removing the highest-weight input with removing a randomly selected input.
- Self-attention layers: In self-attention models, altering the first decoder step or encoder-decoder attention has the largest effect because it nearly stops information flow from encoder to decoder.The comparison covers cumulative permutation across encoder, decoder, and encoder-decoder attention layers.
- Human interpretability: Manual evaluation finds that original attention weights are generally meaningful in both single- and pair-sequence tasks, but meaningless single-sequence weights often preserve correct predictions.This contrast supports attention reflecting model reasoning in pairwise tasks but not necessarily in single-sequence tasks.
5 CONCLUSION
The paper reconciles opposing views on attention explainability by showing that interpretability depends on whether attention is essential to prediction rather than reducible to gating. Experiments across diverse NLP tasks and manual evaluation support this conditional account.
- Attention weights are interpretable and correlate with feature importance when they are essential for the model’s prediction.