PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention is not not Explanation

Sarah Wiegreffe, Yuval Pinter

arXiv:1908.04626v2cs.CL

TL;DR

The paper addresses whether attention weights provide explanations for model predictions and argues that the answer depends on the definition of explanation and on testing the complete model. It introduces four alternative, model-aware tests and finds that reliably constructed adversarial distributions perform poorly in a diagnostic MLP, so prior work does not disprove attention’s usefulness for explainability.

  • Problem

    Attention is widely used in NLP, but evidence remains contested about whether its weights explain model predictions, while prior tests may leave excessive freedom in the experimental setup.

  • Method

    The paper evaluates attention with uniform-weight baselines, random-seed variance calibration, a frozen-weight MLP diagnostic, and end-to-end model-consistent adversarial training.

  • Results

    Adversarial attention distributions can be found, but they perform poorly in the diagnostic MLP, so prior work does not disprove attention’s usefulness for explainability.

  • Takeaways & Limitations

    Meaningful interpretation of attention requires tests that account for the model components producing the attention distributions.

  • Takeaways & Limitations

    The paper’s critique is constrained by the claim that explanation depends on definitions and by its focus on attention distributions within the examined model settings.

Abstract

from arXiv · show

Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.

1 Introduction

The paper challenges whether attention weights can explain model predictions, arguing that this question requires model-consistent tests and depends on the definition of explanation. It proposes four experiments and finds that adversarial attention distributions can be found but perform poorly in a diagnostic model.

  • Motivation: Attention is widely used in NLP, making its suitability for explaining model predictions an important research question.The paper links this interest to understanding the internals of neural models that use attention.
  • Problem: The authors argue that prior experimental assumptions leave too much freedom to measure attention distributions’ utility in specific settings.They specifically challenge the treatment of attention as detached from the model that computes it.
  • Approach: The paper tests attention with a uniform-weight baseline, random-seed variance calibration, a frozen-weight MLP diagnostic, and model-consistent adversarial training.These experiments manipulate different components of a standard RNN-with-attention architecture.
  • Findings: Adversarial attention distributions can be found, but they are not as extreme as inconsistent alternatives and perform poorly in the diagnostic MLP.This result calls into question whether they are equally powerful for explainability.
  • Interpretation: The paper also provides a theoretical discussion of interpretability and explainability to situate its experimental findings.The discussion grounds the results in accepted definitions of these concepts.

2 Attention Might be Explanation

Jain and Wallace test whether alternative attention distributions can preserve predictions, but this paper argues their setup gives adversaries too much freedom by detaching attention from the model and optimizing per instance. The authors therefore question whether those alternatives establish that attention cannot explain model predictions.

  • Prior experimental design: Jain and Wallace evaluate eight classification datasets and two question answering tasks when testing whether attention can explain model predictions.Their analysis includes correlations between attention and gradient-based or leave-one-out importance measures, followed by adversarial searches for alternative attention distributions.
  • Prior experimental design: The correlation experiments find that attention is not strongly correlated with gradient-based and leave-one-out feature-importance metrics.The paper accepts this part of Jain and Wallace’s analysis and does not focus its critique there.
  • Critique of the adversarial setup: Jain and Wallace detach attention distributions and output layers from the parameters that compute them, treating attention scores as standalone units.They also compute an independent adversarial distribution for each instance.
  • Critique of the adversarial setup: Detaching attention removes the learned linkage between the attention mechanism and the rest of the model, so alternative distributions do not demonstrate an adversarial model with the same trained structure.The paper argues that attention scores are computed by an integral component whose parameters were trained alongside the other layers.
  • Critique of the adversarial setup: Because binary classification aggregates many token-level values into one scalar, independently searching each instance creates substantial freedom to find alternative distributions.The paper notes that an average IMDb instance may contain 179 tokens, producing 179 scalars to aggregate.
  • Critique of the adversarial setup: The authors conclude that per-instance demonstrations without directly learning or manipulating model parameters do not establish adversarial models or equally plausible explanations.They also identify the absence of a baseline for expected variation in learned attention distributions as a further interpretive gap.

3 Examining Attention Distributions

The section evaluates attention distributions through controlled baselines, seed variation, diagnostic models, and adversarial comparisons. It finds that attention is sometimes unnecessary, but pretrained attention weights can remain useful for simpler models.

  • 3.1 Experimental Setup: The experiments focus on binary classification with single-layer bidirectional LSTMs, additive attention, and softmax prediction across several sentiment, news, and clinical datasets.The analysis follows Jain and Wallace’s task focus and reports positive-class F1 together with TVD and JSD comparisons.
  • 3.2 Uniform as the Adversary: Freezing attention to uniform weights shows little or no performance loss on three tasks, making AG NEWS and 20 NEWSGROUPS unsuitable tests of attention explainability.SST is retained as a borderline case, while the two news datasets are subsequently ignored.
  • 3.3 Variance within a Model: Eight random-seed models provide a baseline for normal attention variance before adversarial distributions are judged unusually different.SST attention distributions are robust to seed changes, whereas Diabetes already shows substantial negative-class arbitrariness under different seeds.
  • 3.4 Diagnosing Attention Distributions by Guiding Simpler Models: The diagnostic model replaces contextual LSTM attention with a token-level MLP whose outputs are guided by fixed distributions during training and testing.The tested guides include uniform weights, learned MLP weights, base-LSTM attention weights, and adversarial distributions.
  • 3.4 Diagnosing Attention Distributions by Guiding Simpler Models: Across datasets, pretrained LSTM attention guides outperform learned MLP weights and the unweighted baseline, and also outperform LSTMs trained with uniform attention.These results suggest that attention weights encode token-importance information that remains useful without contextual access.

4 Training an Adversary

The paper introduces a model-consistent adversarial training protocol that preserves base-model predictions while distancing attention distributions, then evaluates the resulting tradeoffs and diagnostic usefulness.

  • Adversarial model: The adversarial model matches each base-model prediction while learning attention distributions that diverge from the base model.Predictions and attention distributions are defined per instance, and the model is trained with stochastic gradient updates.
  • TVD/JSD tradeoff: λ controls the tradeoff between low prediction distance and high attention divergence, with convex curves supporting manipulability and concave curves opposing it.The relevant axes are prediction variance measured by TVD and attention distance measured by JSD.
  • Prediction performance: The adversarial models retain test-set F1 scores on par with original models while producing attention distributions with average JSD above 0.4.The reported models are selected using the JSD threshold and corresponding λ and comparison metrics.
  • Adversarial weights as guides: Adversarial attention distributions usually perform poorly in the diagnostic MLP despite imitating the base model’s local decisions.The authors interpret this as evidence that adversarial distributions remove information useful for non-contextual prediction guides.
  • Dataset variation: The extent to which adversarial distributions can be found varies across datasets, with data imbalance and indicative tokens making some distributions harder to replace.The Diabetes positive class and the Anemia dataset are identified as settings where the data distribution affects adversarial search.
  • TVD/JSD tradeoff: The adversarial models achieve attention-distance increases on all four datasets, but Jain and Wallace’s setup exaggerates the extent of this effect.Their results appear below the parameterized model curve in Figure 5; SST is an outlier where JSD can increase with little prediction-variance cost.

5 Defining Explanation

The paper distinguishes transparency, explainability, and interpretability, arguing that attention’s value depends on which kind of explanation is sought. It proposes functionally grounded proxy evaluations rather than treating one definition as universally applicable.

  • Definitions: Explainable AI includes transparency, explainability, and interpretability, which describe different relationships between models, decisions, and human understanding.Transparency concerns correspondence between a model component and a human-understandable construct, while explainability may reconstruct decisions plausibly without being faithful.
  • Transparency: Attention scores can provide partial transparency because they expose an understandable weighting of hidden states within the model.This addresses model components rather than necessarily reconstructing the entire decision process.
  • Multiple explanations: The requirement that only one or a few correct explanations exist is not necessary under definitions that allow multiple plausible explanations.The paper notes that another explanation does not by itself make a provided explanation false or meaningless.
  • Responsibility: Jain and Wallace’s responsibility-based framing aligns more closely with transparency or interpretability than with every possible notion of explanation.Their question concerns whether highly attended input elements correspond to prediction responsibility.
  • Evaluation: The proposed methods use functionally grounded proxy tasks to evaluate attention without human evaluation, tailored to the explanation type being tested.The paper presents these proxies as ground-up tests of attention’s validity as explanation.

6 Attention is All you Need it to Be

The paper concludes that attention is not uniformly explanation: its status depends on whether plausible or faithful explanations are required. Although adversarial distributions can sometimes be found, they perform poorly in the paper’s diagnostic, and the proposed experiments support informed evaluation.

  • Conclusion: Whether attention is explanation depends on whether the desired explanation is plausible, faithful, or both.The paper does not treat prior work on plausible rationales as invalidated by its findings.
  • Adversarial distributions: Adversarial attention distributions can be found for LSTM models in some classification tasks, confirming the possibility originally hypothesized by Jain and Wallace.The authors present this result as a reason for caution when seeking one true, faithful interpretation.
  • Practical use: The paper provides a suite of experiments for making informed decisions about attention mechanisms used to explain model predictions.The proposed evaluation is intended to assess attention quality in specific model settings.
  • Diagnostic evidence: Adversarial distributions perform poorly relative to traditional attention in the diagnostic MLP, despite matching predictions in their original setting.The results suggest that trained RNN attention captures information about token–prediction relationships that is not easily reproduced adversarially.
  • Future work: Future work should examine when adversarial distributions can be found, other tasks and languages, and human agreement with the proposed measures.The authors specifically mention sequence modeling, multi-document analysis, non-English languages, and human evaluation.

A Compute and Environment

The study executed roughly 250 model-training commands, generally using a single local NVIDIA Tesla K40 GPU, with runs lasting 10–120 minutes.

  • Compute: Roughly 250 model-training commands were executed, with individual runs lasting 10–120 minutes.Nearly all experiments ran on a single local NVIDIA Tesla K40 GPU.
  • Environment: Nearly all experiments used a single local NVIDIA Tesla K40 GPU.The paper also reports the electricity-generation mix of Georgia as implementation context.

B All Results on Adversarial Setup

Table 5 reports all tested λ settings for the trained-adversary experiment, using test-set results without post-selection.

  • B All Results on Adversarial Setup: All tested λ settings are included in the trained-adversary experiment’s test-set results.The results were not post-selected.

C TVD/JSD Tradeoff by Class

The class-specific analysis highlights a near-concave positive-class pattern in the Diabetes dataset, where adversarial distributions are expected to be hardest to find. Figure 6 distinguishes negative and positive instances and compares random-seed, uniform, and adversarial setups using JSD and TVD.

  • C TVD/JSD Tradeoff by Class: The Diabetes dataset’s positive class has a unique, near-concave shape in the scatterplot breakdown.This detection-type dataset is biased toward the negative class.
  • C TVD/JSD Tradeoff by Class: Adversarial distributions are expected to be most difficult to find in the Diabetes positive-class setting.The paper states that the observed curve confirms this expectation.
  • C TVD/JSD Tradeoff by Class: Figure 6 reports per-instance test-set JSD and TVD separately for negative and positive instances.Negative instances appear on top and positive instances on the bottom.
  • C TVD/JSD Tradeoff by Class: The figure compares random-seed, uniform-weight, and two adversarial setups.The authors’ adversarial setup is shown with a dotted line, while Jain and Wallace’s is marked with plus signs.
  • C TVD/JSD Tradeoff by Class: Table 5 defines λ = 0 as a model seeking only minimum TVD.SST models were trained for 80 epochs, while other models were trained for 40 epochs.
Loading 1908.04626v2…