PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention is not Explanation

Sarthak Jain, Byron C. Wallace

arXiv:1902.10186v3cs.CLcs.AI

TL;DR

It is unclear whether attention weights faithfully indicate which inputs drive neural NLP predictions. Across NLP tasks, the paper empirically tests this relationship and finds that attention is often weakly related to feature importance, while substantially different attention distributions can preserve the same predictions.

  • Problem

    The paper asks whether attention weights indicate why a neural NLP model made a prediction, an assumption that had not been formally evaluated.

  • Method

    The paper runs experiments comparing attention weights with feature-importance measures and tests scrambled or adversarially different attention distributions for prediction changes.

  • Results

    Across tasks, attention weights show weak, inconsistent agreement with feature-importance measures, while markedly different distributions often produce effectively equivalent predictions.

  • Takeaways & Limitations

    Standard attention modules, especially with complex recurrent encoders, should not be treated as meaningful explanations of model predictions.

  • Takeaways & Limitations

    The study does not establish that gradient- or erasure-based feature-importance measures are ideal ground truth for explanation reliability.

Abstract

from arXiv · show

Attention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work, we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful `explanations' for predictions. We find that they largely do not. For example, learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and one can identify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention modules do not provide meaningful explanations and should not be treated as though they do. Code for all experiments is available at https://github.com/successar/AttentionExplanation.

1 Introduction and Motivation

The paper tests whether attention weights faithfully explain neural NLP predictions by measuring their alignment with feature importance and the effect of alternative attention distributions. Across recurrent BiLSTM encoders, attention is only weakly and inconsistently aligned with these criteria, while substantially different attention patterns can preserve predictions.

  • Research questions and contributions: Randomly permuting attention weights often causes only minimal output changes, further challenging the interpretation of attended tokens as responsible for predictions.One example reports a median output difference of 0.006 after permutation.
  • Research questions and contributions: The study evaluates whether attention weights explain predictions by correlating them with gradient- and leave-one-out feature-importance measures and testing alternative attention configurations.These tests target whether highly attended inputs are actually responsible for model outputs.
  • Research questions and contributions: Attention weights in recurrent BiLSTM encoders correlate only weakly and inconsistently with feature-importance measures.In one example, attention correlates weakly with gradient-based importance (τg = 0.29).
  • Research questions and contributions: Adversarial attention distributions can attend to entirely different input features while yielding effectively equivalent predictions to the original attention.Figure 1 reports the same prediction, 0.01, for observed and adversarially constructed attention distributions.
  • Research questions and contributions: By contrast, simple feedforward weighted-average encoders exhibit better behavior on the feature-importance and alternative-attention criteria.The contrast is stated for the criteria examined in the recurrent encoder experiments.

2 Preliminaries and Assumptions

The paper evaluates attention on classification, natural language inference, and question answering under a shared neural modeling framework. Inputs are embedded, encoded into token-level hidden states, scored against a query to produce attention weights, and decoded into predictions from a weighted representation.

  • The study considers attention-based models for classification, natural language inference, and question answering.
  • Inputs are one-hot word sequences transformed by an embedding matrix into d-dimensional token representations.
  • An encoder processes embedded tokens into T m-dimensional hidden states, predominantly using a Bi-RNN and contrasting unordered average-embedding variants.
  • Attention applies a similarity function between hidden states and a query, then uses softmax scores to form the attention distribution.The similarity functions include additive and scaled dot-product forms.
  • A dense decoder predicts from the attention-weighted sum of hidden states, with σ as the output activation and |Y| as the label-set size.

3 Datasets and Tasks

The study evaluates attention mechanisms across binary text classification, question answering, and natural language inference datasets. It uses comparatively simple attention mechanisms and excludes BiAttentive architectures that condition attention on the other input.

  • Binary text classification: Binary classification covers sentiment, adverse drug reaction detection, and topic, diabetes, and anemia classification across six datasets.The datasets include SST, IMDB, Twitter adverse drug reactions, 20 Newsgroups, AG News, and two MIMIC ICD9 tasks.
  • Binary text classification: SST contains 10,662 sentences, filtered to positive-versus-negative sentiment after removing neutral instances.Positive labels are 4 and 5, while negative labels are 1 and 2.
  • Binary text classification: IMDB provides 50,000 polarized movie reviews split evenly between training and testing.The task is binary sentiment classification.
  • Question Answering: Question answering uses CNN cloze questions and three bAbI tasks requiring one, two, or three supporting statements.CNN instances contain paragraph-question-answer triplets whose answers are anonymized paragraph entities.
  • Natural Language Inference: Natural language inference uses 570k labeled SNLI sentence pairs with neutral, contradiction, and entailment classes.The model generates attention over premise words conditioned on the hidden representation of the hypothesis.
  • Scope and limitations: The experiments restrict attention to comparatively simple mechanisms and exclude BiAttentive architectures that attend across respective inputs.The excluded architectures condition attention on the other input.

4 Experiments

The experiments test whether learned attention weights agree with gradient- and erasure-based feature-importance measures and whether substantially different attention distributions change model predictions. Across tasks, attention generally shows weak correspondence to feature importance and often permits very different configurations with effectively unchanged outputs, with simple averaging encoders and some Diabetes examples as exceptions.

  • Experimental questions: The experiments evaluate attention interpretability through feature-importance agreement and counterfactual changes in model outputs.They compare learned attention with gradient and leave-one-out measures, then examine scrambled and adversarial attention distributions.
  • Feature-importance agreement: Observed attention correlations are modest for BiLSTM models, generally hovering around or below 0.5, despite more consistent significance on MIMIC and weakerly on QA datasets.The passage defines 0 as no correspondence and 1 as perfect concordance; significance does not imply strong correlation.
  • Feature-importance agreement: Attention generally agrees weakly or inconsistently with standard feature-importance scores, whereas simple averaging encoders show stronger correspondence.Projection-based models also show stronger average attention–LOO correlations than BiLSTM models, while irrelevant features may add noise to Kendall τ.
  • Counterfactual attention: Counterfactual analyses scramble observed weights or seek maximally distinct adversarial weights that remain within ϵ of the original prediction.Adversarial attention is designed to differ from observed attention while preserving the model output, using divergence-based separation and output constraints.
  • Scrambled attention: Many instances have large attention weights but small prediction changes after scrambling, indicating that apparently focused attention need not determine the output.This pattern occurs when heatmaps suggest a small explanatory feature set, yet perturbing attention makes little difference.
  • Adversarial attention: Adversarial searches frequently find attention distributions near the JSD upper bound of 0.69 while barely changing outputs, although Diabetes positive examples are an exception.For Diabetes positive cases, perturbing attention causes large output changes; across datasets, the relationship between attention magnitude and output sensitivity remains weak.

5 Related Work

Related work includes gradient- and erasure-based importance measures, causal explanations, and attention variants designed explicitly for interpretability. The paper also motivates further attention methods that combine predictive performance with insight into model predictions.

  • Alternative explanation methods: Alternative approaches assess neural NLP models using gradient-based feature importance, representation erasure, or causal explanations of black-box predictions.These methods are presented as strategies unrelated to standard attention mechanisms for providing model insights.
  • Alternative explanation methods: Comparing importance measures with attention scores may provide additional insights into how attention-based models work.
  • Interpretability-focused attention: Interpretability-focused attention variants impose hard, sparse attention and explicitly select modest input subsets that are responsible for predictions by construction.Structured attention models are also noted as a generalized framework for describing and fitting attention.
  • Interpretability-focused attention: The work calls for further development of attention variants that improve predictive performance while providing insights into model predictions.

6 Discussion and Conclusions

The discussion concludes that attention weights offer weak evidence of feature importance and can support misleading explanations, particularly with recurrent encoders. It also identifies limitations concerning alternative importance measures, attention variants, adversarial counterfactuals, and the restricted task scope.

  • Conclusions: Attention weights correlate weakly with gradient- and feature-erasure-based importance measures for recurrent encoders, while counterfactual attention distributions often have modest effects on outputs.These findings come from Sections 4.1 and 4.2.
  • Conclusions: Attention improves NLP performance but provides questionable transparency, especially with complex encoders whose hidden representations may entangle inputs.Attention heatmaps can therefore suggest an unclear or misleading story about how predictions were produced.
  • Limitations: The reported correlations do not establish that gradients or other alternative importance measures are ideal or constitute ground truth.Such measures have intrinsic model semantics, but their interpretation in nonlinear neural models remains limited.
  • Limitations: Attention over non-recurrent encoders correlates more strongly with leave-one-out importance than attention over BiLSTM outputs, despite possible noise depressing Kendall τ.Leave-one-out and gradient-based measures also correlate more strongly with one another than recurrent attention measures do.
  • Limitations: The study examined only a handful of attention variants and unstructured-output tasks, excluding sequence-to-sequence settings such as translation.The authors suggest that feed-forward and possibly convolutional encoders may show stronger fidelity when designed carefully.
  • Limitations: Counterfactual experiments show that alternative attention heatmaps can yield equivalent predictions, so a prediction cannot be attributed to one specific attention pattern.Adversarial weights may nevertheless be unlikely under the attention module parameters, and multiple plausible explanations may exist.

Attention is not Explanation: Appendix … B Further details regarding attentional module of gradient

The appendix specifies dataset preprocessing, embeddings, and architecture configurations for the evaluated models. Gradient experiments cut off computation at the attention module, treating attention as an input independent of the original input for feature-importance calculation.

  • A Model details: All datasets use spaCy tokenization, map out-of-vocabulary words to <unk>, and map words containing numeric characters to ‘qqq’.Vocabulary words are initialized with pretrained embeddings, except for MIMIC, where embeddings are learned with Gensim over discharge summaries.
  • A Model details: General-domain datasets use either FastText embeddings trained on Simple English Wikipedia or GloVe 840B embeddings, depending on the dataset.FastText is used for SST, IMDB, 20News, and CNN; GloVe 840B is used for AG-News and SNLI.
  • A.1 BiLSTM: BiLSTM models use embedding size 300 and hidden size 128, except bAbI, which uses sizes 50 and 30, respectively.All models use ℓ2 regularization with λ = 10−5, task-dependent sigmoid or softmax outputs, maximum likelihood loss, and Adam with default PyTorch parameters.
  • A.2 CNN: CNN models use embedding size 300 and four kernels of sizes [1, 3, 5, 7], each with 64 filters, producing hidden size 256.For bAbI, the corresponding sizes are 50 and 8, with the same kernel sizes; filter outputs use ReLU.
  • A.3 Average: Average models use embedding size 300 and projection size 256, with ReLU applied to the projection output.All other configurations remain the same as for the BiLSTM.
  • B Further details regarding attentional module of gradient: Gradient experiments cut off computation at the attention module so gradients through attention do not contribute to feature-importance scores.This treats attention as a separate network input independent of the original input, measuring output change when particular input features are perturbed.

D Adversarial Heatmaps

The section presents an adversarial movie-review input whose prediction changes by only ∆ˆy: 0.004. The review combines film-noir praise with plot and cast details.

  • D Adversarial Heatmaps: The adversarial input describes the film as “fantastic” and “one of the best film noir movies ever made.”It also praises the cast and encourages the reader to enjoy the movie.
  • D Adversarial Heatmaps: The review lists bad guys, bad girls, a jewel heist, twisted morality, and a kidnapping.These details appear as part of the adversarial review text.
  • D Adversarial Heatmaps: ∆ˆy: 0.004 accompanies the adversarial review.The passage reports this prediction-change value directly.

20 News Group - Sports

The sports example argues that baserunning is not the main driver of runs-created accuracy and reports an adversarial output change of ∆ŷ 0.001.

  • The original passage argues that baserunning cannot be very important because runs created is usually accurate at the team level.
  • The passage attributes the remaining percentage to factors beyond baserunning.
  • ∆ŷ: 0.001 is reported for the adversarial version, which preserves the original sports passage.

AG News

In the AG News example, the adversarial attention distribution produced only a small prediction change, with ∆ˆy: 0.006.

  • AG News: The adversarial attention example changed the prediction by only ∆ˆy: 0.006.The example concerns a report that General Motors and DaimlerChrysler were teaming up to develop hybrid technology.

Babi Task 1

Babi Task 1 presents original and adversarial attention distributions for question-answering examples. In the shown instance, changing attention produces only a small prediction difference, with ∆ˆy = 0.003.

  • Babi Task 1: ∆ˆy: 0.003 for the question “Where is Sandra?”, despite different original and adversarial attention distributions.Both distributions are shown over the same two-sentence context about John and Sandra travelling to the garden.
  • Babi Task 1: The task includes a question about where a federal education minister visited and what she saw.The associated context describes a minister visiting a store and seeing cameras.
  • Babi Task 1: An adversarial version of the news context retains the same reported security-camera incident while altering the displayed passage continuation.The original and adversarial passages both describe four employees arrested after a minister discovered a surveillance camera overlooking a changing room.
Loading 1902.10186v3…