Source-linked AI summary
Transformer Interpretability Beyond Attention Visualization
Hila Chefer, Shir Gur, Lior Wolf
TL;DR
Transformer explanations often rely on attention maps or heuristic propagation, but these can overlook other computations and highlight irrelevant or fragmented evidence. The paper proposes Deep-Taylor-based relevance propagation through attention and skip connections, with a formulation that maintains relevance across layers. Across vision and NLP benchmarks, the method reports stronger performance than existing Transformer explainability approaches and comparisons, while attention-based explanations are described as fragmented and non-competitive.
Problem
Existing Transformer visualizations often rely on attention or heuristic propagation, overlooking other computations and producing irrelevant or fragmented explanations.
Method
The method assigns relevance using Deep Taylor Decomposition and propagates it through attention, skip connections, and other Transformer layers with normalization and relevance-gradient integration.
Results
The method achieves state-of-the-art results against Transformer-literature, LRP, and GradCAM methods across visual and NLP explainability evaluations.
Takeaways & Limitations
The approach provides class-specific Transformer visualizations and outperforms existing baselines on perturbation and ImageNet segmentation benchmarks.
Takeaways & Limitations
Attention values capture only one aspect of a Transformer or self-attention head, and attention-based explanations can be fragmented and non-competitive.
Abstract
from arXiv · showhide
Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.
1. Introduction
Transformers have become central across language and vision, creating a need for methods that reveal their decision processes. The paper addresses limitations of attention-based visualization by propagating relevance through Transformer components while preserving total relevance.
- Transformers are state-of-the-art across NLP benchmarks and increasingly used for object detection and image recognition.
- Attention visualization can blur signals across layers, while rollout’s simplifying assumptions may highlight irrelevant tokens.
- Skip connections, mixed positive and negative features, and self-attention make naive relevance propagation unstable or non-conservative.
- The method introduces propagation for positive and negative attributions, normalization for non-parametric layers, and integration of attention with relevance across blocks.
- The approach is designed to provide class-separated visualizations, addressing methods that often produce similar heatmaps across classes.
- The evaluation covers ImageNet segmentation, ImageNet perturbation tests, and an NLP benchmark identifying human-marked decision excerpts.
2. Related Work
Prior Transformer explanations often use attention scores or heuristic graph propagation, while broader interpretability methods include gradients, perturbations, and attribution propagation. These approaches differ in class specificity, computational cost, spatial resolution, and how completely they propagate information to the input.
- Gradient methods propagate input-layer gradients, often multiplied by activations, but are observed to produce similar outputs across classes.
- GradCAM is class-specific and useful for downstream segmentation, but its reliance on deep low-resolution layers produces coarse visualizations.
- Attribution propagation methods use Deep Taylor Decomposition to recursively assign network decisions to earlier layers and inputs.
- Perturbation methods work with black-box models and are intuitive, but generating their heatmaps is computationally expensive.
- Transformer visualization often ignores value computations and other layers, while earlier LRP work propagated only partial information rather than relevance to the input.
- Attention rollout assumes linear combination of attentions and can emphasize irrelevant tokens while failing to distinguish positive from negative contributions.
- Attention flow can correlate better with masking- or gradient-derived relevance than rollout in some cases, but is slower and was not evaluated here.
3. Method
The method propagates class-specific relevance and gradients through Transformer layers, addressing the conservation and numerical challenges introduced by attention, matrix multiplication, and skip connections. It integrates normalized attention relevance across blocks to produce token- or patch-level relevance maps.
- Relevance and gradients: The method computes class-specific LRP-based relevance for each Transformer attention head and integrates it with gradient information across the attention graph.Relevance is initialized for a target class, which need not be the predicted class.
- Relevance and gradients: Deep Taylor relevance propagation recursively decomposes the network decision while preserving a layer-wise conservation rule.The generic rule propagates relevance from layer outputs to inputs; the paper modifies LRP to handle positive and negative activations from non-ReLU nonlinearities.
- Non-parametric relevance propagation: Skip connections and matrix multiplication require relevance propagation through both operands, but conservation holds for addition and not generally for matrix multiplication.These binary operators can produce positive and negative values, and their inputs may have different shapes in matrix multiplication.
- Non-parametric relevance propagation: Skip-connection propagation can produce numerically unstable relevance values with large absolute magnitudes even when their total relevance remains constant.The instability arises from the relevance computation used for addition operators.
- Non-parametric relevance propagation: The method normalizes relevance for non-parametric operators to address attention non-conservation and skip-connection numerical issues while maintaining conserved relevance and bounded tensor relevance sums.The normalization is applied to relevance associated with operators such as addition and matrix multiplication.
- Relevance and gradient diffusion: For each attention block, positive gradient–relevance products are averaged across heads, combined with an identity matrix, and multiplied across blocks to form weighted attention relevance.The identity term accounts for skip connections and avoids self-inhibition for each token.
- Relevance and gradient diffusion: Unlike rollout, which is input-fixed and uses only pairwise attention scores, the proposed visualization incorporates target-class relevance and gradient signals.Rollout is represented as a product of attention matrices across Transformer blocks.
- Obtaining the image relevance map: The resulting matrix provides a relevance map for each token, while the classification explanation uses the [CLS] row and reshapes vision-token relevance into an upsampled image patch map.Special tokens are excluded from the final content-token map, which is resized using bilinear interpolation.
4. Experiments
Experiments evaluate the method on visual Transformer classification and segmentation, perturbation tests, and Movie Reviews rationale extraction, alongside several baseline families. Across these settings, the method yields clearer visualizations, stronger class-specific separation, better perturbation and segmentation performance, and consistently higher token-F1 scores.
- Evaluation setup: Experiments use ViT-base for visual classification and BERT-base for linguistic classification, with attention-map, relevance-propagation, and gradient-based baselines.The visual evaluation uses ImageNet and ImageNet-Segmentation; the linguistic evaluation uses the Movie Reviews dataset.
- Qualitative evaluation: Qualitatively, the method produces clearer and more consistent visualizations and provides different accurate visualizations for different target classes.Other methods generally produce similar maps across classes, while the method separates the visualizations for two objects from different classes.
- Perturbation tests: Perturbation tests show that our method achieves better AUC performance by a large margin for both positive and negative perturbations across predicted and target classes.Positive perturbation masks pixels from highest to lowest relevance, whereas negative perturbation reverses that order.
- Segmentation: Our method outperforms all baselines by a significant margin on pixel-accuracy, mAP, and mIoU segmentation metrics.The segmentation evaluation compares visualizations with ImageNet-Segmentation ground-truth masks.
- Language reasoning: On Movie Reviews rationale extraction, our method consistently outperforms baselines as the number of selected tokens increases from 10 to 80.Performance is measured with token-F1 for top-k tokens.
- Ablation study: Removing rollout causes a moderate performance drop, while combining relevance with attention-map gradients accounts for most of the method’s advantage.The single-block variant closest to the output outperforms the compared raw-attention, partial-LRP, and GradCAM methods.
5. Conclusions
Attention values alone produce fragmented, non-competitive explanations, while the proposed method addresses Transformer-specific challenges and achieves state-of-the-art results against several baselines.
- Attention values reflect only one aspect of Transformer computation and produce fragmented, non-competitive explanations for BERT and ViT models.
- Transformer interpretability is constrained by non-positive activations, skip connections, and matrix multiplication in self-attention.
- The method provides specific solutions to these challenges and obtains state-of-the-art results against Transformer-literature methods, LRP, and GradCAM.
A. Details of the various Baselines
The evaluated baselines use distinct visualization strategies, ranging from a final attention map to partial or full LRP propagation and attention rollout.
- GradCAM extracts the [CLS] token’s row from the last attention layer and reshapes it into a patch-grid feature map.
- Raw-attention visualizes the final output of the last attention layer.
- LRP propagates relevance to the input image using standard LRP rules rather than the proposed modified rules and normalizations.
- Partial-LRP visualizes relevance from the last attention map instead of propagating relevance to the input.
- Rollout follows the paper’s stated Eq. 16.
B. Proofs for Lemmas
The proofs distinguish addition from matrix multiplication: addition preserves relevance conservation, whereas multiplication generally does not, motivating normalization that also controls numerical magnitude.
- For binary operators, relevance propagation is computed for two input tensors, u and v.
- Addition preserves the conservation rule, but matrix multiplication does not generally preserve it.
- Although addition preserves conservation, relevance values can explode because their absolute magnitudes become large.
- The resulting large absolute values can cause numerical instability during skip-connection propagation.
- The proposed normalization addresses missing conservation in attention multiplication and numerical issues in skip connections.
- The normalization maintains conservation and bounds each tensor’s relevance sum by weighting tensors according to relative absolute-value contributions.
C. Visualizations - Multiple-class Images
The multiple-class visualizations compare input images and class-specific maps across raw attention, GradCAM, LRP variants, rollout, and the proposed method.
- The visualization panel includes input images, raw-attention, GradCAM, LRP, partial LRP, and the proposed method.
- For each input image, the figure visualizes two different classes.
- Only the proposed method and GradCAM produce class-specific visualizations; the proposed method has fewer artifacts and captures objects more completely.
D. Visualizations - Single-class Images
The section presents visualizations for single-class ImageNet validation images, comparing input images with several attribution methods. The displayed methods include rollout, raw attention, GradCAM, LRP variants, and the proposed method.
- The comparison includes input images, rollout, raw attention, GradCAM, LRP, partial LRP, and Ours.
- The visualizations use sample images from the ImageNet validation set.
E. Visualizations - Text
The text visualizations compare relevancy explanations across methods for ground-truth and counter-factual sentiment classes. The proposed method supports both sides, while other methods show class-specificity, tokenization, or visualization issues.
- The visualizations use TAHV to display relevancy scores and human-marked excerpts for the compared methods.Class-dependent methods are shown for both ground-truth and counter-factual classes.
- The proposed method is the only method shown supporting both sentiment sides in the visualizations.Ground-truth and counter-factual attributions are presented for class-dependent methods.
- GradCAM can reverse the evidence direction, as its counter-factual explanation may support the ground-truth sentiment.The passage gives Fig. 13(g) as an example of this sign reversal.
- Partial LRP highlights positive and negative connotations within the same sentence rather than behaving class-specifically in practice.This matches human annotators who mark complete sentences in the quantitative experiments.
- Rollout often focuses on the separation token [SEP] and fails to produce meaningful visualizations.The passage connects this observation to the quantitative-experiment results.
- BERT tokenization can split one word into tokens supporting different sides of the decision.The example “joyless” is split into “joy” and “less”.