PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer Interpretability Beyond Attention Visualization

Hila Chefer, Shir Gur, Lior Wolf

arXiv:2012.09838v2cs.CV

TL;DR

Transformer explanations often rely on attention maps or heuristic propagation, while attention-based explanations are fragmented and non-competitive in tested ViT and fine-tuned BERT settings. This paper propagates LRP-based relevance through attention graphs and reports state-of-the-art results against Transformer-literature methods, LRP, and Grad-CAM.

  • Problem

    Existing Transformer visualizations rely on attention scores or heuristic propagation, while attention explanations are fragmented and non-competitive in tested ViT and fine-tuned BERT settings.

    Source: §1 · p. 1, §5 · p. 8

  • Method

    The method uses LRP-based relevance and integrates relevance with gradient information across the attention graph to produce class-specific visualizations.

    Source: §3 · p. 3

  • Results

    The method achieves state-of-the-art results against Transformer-literature methods, LRP, and Grad-CAM, and outperforms all baselines on ImageNet segmentation and both ImageNet perturbation tests.

    Source: §4.1 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    Attention values alone are insufficient explanations for the tested ViT and fine-tuned BERT models.

    Source: §5 · p. 8

    The paper notes that the property discussed in its non-parametric relevance propagation does not hold for matrix multiplication.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.

1. Introduction

The paper introduces relevance propagation for Transformer visualization to address challenges from attention and skip connections, and presents it as class-specific by design. It evaluates the method on computer-vision and NLP explainability benchmarks.

  • Single-layer attention visualizations and attention averaging have limitations, while rollout [1] can highlight irrelevant tokens because it assumes attention combines linearly into later contexts.The passage says rollout seems to improve on using a single attention layer, but reports that its simplistic assumptions can highlight irrelevant tokens.Source: §1 · p. 1
  • Prior Transformer relevance propagation [42] was partial and did not propagate attention through all layers, unlike the relevance-preserving line of work.Source: §1 · p. 1
  • The method propagates relevancy through attention and skip connections while maintaining total relevancy, with rules for positive and negative attributions and normalization for non-parametric layers.The normalization applies to layers such as skip-connection addition and matrix multiplication; attention relevancy scores are integrated across multiple attention blocks.Source: §1 · p. 1
  • The method provides class-based separation by design and is presented as the only Transformer visualization method with this property, as far as the authors could ascertain.The paper notes that salient image regions can blur class-specific signals in methods that propagate to the input.Source: §1 · p. 2
  • The evaluation covers ImageNet-subset segmentation, positive and negative perturbations on ImageNet validation, and an NLP benchmark [10] that identifies human-marked decision excerpts.The paper frames its goal as improving performance on accepted explainability benchmarks across diverse, complementary computer-vision approaches.Source: §1 · p. 2

2. Related Work

Related work divides image explainability into gradient-based and attribution-propagation methods, while Transformer visualization often relies on attention scores. The paper positions its approach as propagating relevance through all layers, with class-specific behavior by construction.

Explainability in computer vision

  • Gradient methods, including Gradient*Input [34], Integrated Gradients [39], SmoothGrad [36], and FullGrad [37], are class-agnostic, whereas GradCAM [32] is class-specific but produces coarse maps.GradCAM combines layer features and gradients and is used in weakly-supervised semantic segmentation [21], but relies on deep, low-resolution layers.Source: §2 · p. 2
  • LRP applies Deep Taylor Decomposition recursively but assumes ReLU; because Transformers use other nonlinearities, this method adapts DTD and is class-specific by construction.CLRP [15] and SGLRP [20] obtain class specificity by contrasting relevance for the target class with other classes.Source: §2 · p. 2
  • Other approaches include saliency, Activation Maximization, Excitation Backprop, perturbation methods, and Shapley-value methods [24].Perturbation methods [13] [14] work with black-box models but are computationally expensive and difficult to apply to discrete text tokens; Shapley methods are also costly and often less accurate than other methods.Source: §2 · p. 2

Explainability for Transformers

  • Attention-score visualizations omit much of self-attention computation and other layers, whereas this method propagates relevance from the decision through all layers to the input.A self-attention head also computes queries, keys, and values, not just attention scores.Source: §2 · pp. 2–3
  • Prior Transformer LRP [42] assessed relative attention-head importance without propagating relevance to the input or directly evaluating relevance scores.The scores were used to visualize relative importance and prune less relevant heads.Source: §2 · p. 3
  • Rollout [1] assumes linear attention composition and can emphasize irrelevant tokens or mix positive and negative contributions; attention flow [1] may correlate better but is slower.Attention flow was not evaluated for computational reasons; the concurrent work [1] did not benchmark either method independently of BERT [9] or against methods beyond raw attention scores.Source: §2 · p. 3

3. Method

The method uses LRP-based relevance and gradient information to compute class-specific visualizations for self-attention models, iteratively removing negative contributions [41].

3.1. Relevance and gradients

  • For target class t, not necessarily the predicted class, it initializes one-hot relevance and propagates gradients and Deep Taylor relevance, retaining positive weighted relevance for GELU activations.The relevance rule considers only elements with positive weighted relevance.Source: §3.1 · pp. 3–4

3.2. Non parametric relevance propagation:

  • Addition preserves relevance conservation, but matrix multiplication does not generally; normalization addresses attention nonconservation and skip-connection instability while maintaining conservation and bounding each tensor’s relevance sum.Skip-connection relevance can attain large absolute values and cause numerical instability even when the sum is conserved.Source: §3.2 · p. 4

3.3. Relevance and gradient diffusion

  • The method multiplies weighted attention-relevance matrices across blocks, using positive gradient–relevance products averaged across heads and adding identity to account for skip connections.The attention relevance and gradients are computed with respect to a target class.Source: §3.3 · p. 5

3.4. Obtaining the image relevance map

  • For classification, the [CLS] row provides token influence scores; for vision models, input patch-token scores are reshaped to the patch grid and bilinearly upsampled to the original image size.Special tokens such as [CLS] and separators are excluded from the image relevance map.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate Transformer explanations on visual and linguistic classification tasks using perturbation, segmentation, and rationale-extraction benchmarks. Across these tests, the proposed method consistently outperforms the reported baselines, with ablations examining the roles of rollout and attention-gradient integration.

4. Experiments

  • Experiments use BERT-base for linguistic classification and pretrained ViT-base for visual classification, with [CLS] feeding each classification head.The visual task uses 16×16 image patches; linguistic inputs are limited to 512 tokens. Visual evaluation uses ImageNet validation and ImageNet-Segmentation, while linguistic evaluation uses binary-sentiment Movie Reviews.Source: §4 · pp. 5–6
  • Visual evaluation measures positive and negative perturbation AUC and segmentation pixel-accuracy, mIoU, and mAP; rationale extraction uses token-F1 across top-k selections.Perturbation AUC is measured while erasing 10%–90% of pixels; segmentation thresholds visualizations at their mean for pixel-accuracy, while mAP uses soft scores. Rationale top-k ranges from 10 to 80 tokens.Source: §4 · p. 6

4.1. Results

  • In two-object images, our method produces different, accurate class-specific visualizations, whereas the other methods produce similar maps for each class.The qualitative comparison also reports that baseline visualizations are inconsistent, while our method produces clearer, more consistent visualizations.Source: §4.1 · p. 6, Figure 3 · p. 7, Figure 2 · p. 7
  • Our method outperforms baselines on ImageNet perturbation and segmentation tests and consistently leads on Movie Reviews rationale extraction.The reported comparisons describe large-margin performance on perturbation tests and significant-margin performance on segmentation; no table or figure values are provided in the passages.Source: §4.1 · p. 6, Table 1 · p. 8, Table 2 · p. 8
  • Ablations show a moderate performance drop without rollout; among single-block variants, the output-nearest block is more informative than the input-nearest block.The combined attention-gradient and relevance variant at the output-nearest block outperforms raw attention, partial LRP, and GradCAM, indicating the benefit stems mostly from combining relevance with attention-map gradients.Source: §4.1 · pp. 6, 8, Table 3 · p. 8

5. Conclusions

Attention values capture only one aspect of Transformer behavior and produce fragmented, non-competitive explanations, while Transformer interpretability methods face challenges that limit the application of approaches developed for other neural networks. The proposed method addresses these challenges and achieves state-of-the-art results against Transformer-literature methods, LRP, and GradCam.

  • Attention values reflect only one aspect of a Transformer or self-attention head, and yield fragmented, non-competitive explanations for fine-tuned BERT and ViT.Source: §5 · p. 8
  • Non-positive activations, frequent skip connections, and self-attention matrix multiplication challenge the application of methods developed for other neural networks.Source: §5 · p. 8
  • The proposed method addresses Transformer interpretability challenges and obtains state-of-the-art results against Transformer-literature methods, LRP, and GradCam.Source: §5 · p. 8

A. Details of the various Baselines

The baselines visualize the final attention layer or propagate relevance using LRP rules, with one method inspecting an intermediate relevance map and rollout following Eq. 16.

GradCAM

  • GradCAM uses the last attention layer’s [CLS] row, excluding its [CLS] column, reshaped to the patch grid as a feature map.The last layer is used because of the sparse-gradients issue.Source: A. Details of the various Baselines · p. 11

raw-attention

LRP

partial-LRP

rollout

B. Proofs for Lemmas

The proofs show that addition preserves relevance conservation while matrix multiplication does not in general. They motivate a normalization that restores conservation and addresses relevance growth and numerical instability in skip connections.

B. Proofs for Lemmas

  • For binary operators on two tensors, addition preserves the conservation rule, whereas matrix multiplication does not do so in general.The lemma contrasts adding two tensors with multiplying u ∈ R^(k×m) and v ∈ R^(m×l).Source: B. Proofs for Lemmas · p. 12
  • Although adding tensors preserves conservation, their values may explode and cause numerical instability.The method’s normalization is presented as addressing numerical issues in skip connections.Source: B. Proofs for Lemmas · pp. 12–13
  • Normalization maintains the conservation rule and bounds each tensor’s relevance sum by weighting it according to its relative absolute-value contribution.The proofs apply this normalization to address both multiplication in attention and numerical issues in skip connections.Source: B. Proofs for Lemmas · pp. 13–14

C. Visualizations - Multiple-class Images

For multiple-class images, only the proposed method and GradCAM produce class-specific visualizations; the proposed method shows fewer artifacts and captures objects more completely.

C. Visualizations - Multiple-class Images

  • The proposed method and GradCAM are the only methods shown with class-specific visualizations, and the proposed method has fewer artifacts and captures objects more completely.Figure 1 compares two different classes for each input image.Source: Figure 1 · p. 15

D. Visualizations - Single-class Images

The section presents sample images from the ImageNet validation set.

D. Visualizations - Single-class Images

E. Visualizations - Text

Text visualizations show that the proposed method can present support for both sentiment sides. They also illustrate limitations involving class specificity, evidence direction, rollout focus, and tokenization.

E. Visualizations - Text

  • Visualizations use TAHV to show relevance alongside human-marked excerpts; class-dependent methods are shown for ground-truth and counter-factual classes, unlike class-agnostic rollout and raw attention.Source: E. Visualizations - Text · p. 27, Figure 13 · p. 28
  • The proposed method alone presents support for both sentiment sides; GradCAM can reverse evidence, with its counter-factual explanation supporting negative ground-truth sentiment instead of the positive sentiment.The cited example is Fig. 13(g).Source: E. Visualizations - Text · p. 27, Figure 13 · p. 28
  • Partial LRP is not class-specific in practice, highlighting words with positive and negative connotations together, which better matches annotators marking complete sentences.Source: E. Visualizations - Text · p. 27
  • Rollout often focuses mostly on the [SEP] token and fails to generate meaningful visualizations, corresponding to the quantitative results.Source: E. Visualizations - Text · p. 27
  • BERT tokenization can produce unintuitive explanations: “joyless” is split into “joy” and “less,” each supporting a different side of the decision.Source: E. Visualizations - Text · p. 27

Related papers

Loading 2012.09838v2…