Source-linked AI summary
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Yaru Hao, Li Dong, Furu Wei, Ke Xu
TL;DR
Prior saliency methods attribute predictions to individual input features but do not explain how features interact. The paper introduces ATTATTR, an integrated-gradient-based method for self-attention interactions, and applies it to BERT for head pruning, interaction-tree analysis, and adversarial pattern discovery. The studies show that attribution identifies decision-relevant interactions, supports competitive head pruning, and exposes patterns usable in attacks.
Problem
Prior methods attribute model decisions to individual input features but fail to explain how those features interact to produce predictions.
Method
ATTATTR applies integrated gradients to self-attention relations in Transformer and is studied extensively on BERT.
Results
ATTATTR identifies important heads, supports interaction-tree construction, and uses salient interaction patterns as adversarial triggers for BERT-based models.
Takeaways & Limitations
Self-attention attribution makes Transformer information interactions more interpretable and can reveal anomalous patterns in data.
Abstract
from arXiv · showhide
The great success of Transformer-based models benefits from the powerful multi-head self-attention mechanism, which learns token dependencies and encodes contextual information from the input. Prior work strives to attribute model decisions to individual input features with different saliency measures, but they fail to explain how these input features interact with each other to reach predictions. In this paper, we propose a self-attention attribution method to interpret the information interactions inside Transformer. We take BERT as an example to conduct extensive studies. Firstly, we apply self-attention attribution to identify the important attention heads, while others can be pruned with marginal performance degradation. Furthermore, we extract the most salient dependencies in each layer to construct an attribution tree, which reveals the hierarchical interactions inside Transformer. Finally, we show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.
1 Introduction
The paper proposes ATTATTR, an integrated-gradient-based self-attention attribution method for interpreting information interactions inside Transformer. Studies on BERT use it to identify important heads, visualize information flow, and expose adversarial interaction patterns.
- ATTATTR attributes model decisions to interactions between input tokens rather than only to individual input features.
- Attention weights do not always correlate well with the contributions of attention connections to model predictions.
- Attribution scores support interaction-tree construction that visualizes compositional structures and information flow inside Transformer.
- ATTATTR identifies important attention heads for pruning, with important heads roughly consistent across homogeneous tasks and datasets.
- On MNLI, adding one adversarial pattern to the premise reduces entailment accuracy from 82.87% to 0.8%.
- The authors conclude that ATTATTR can interpret model decisions and find anomalous patterns in data.
2 Background
Transformer represents token sequences through stacked layers whose core computation is multi-head self-attention. BERT applies this architecture bidirectionally and adds task-specific classification through the final-layer [CLS] representation.
- Transformer packs input word embeddings into X0 and computes successive representations through L stacked Transformer layers.
- The core component of each Transformer block is multi-head self-attention.
- Each attention head computes queries, keys, values, attention weights, and a weighted output, while multi-head attention concatenates head outputs.
- BERT is a bidirectional Transformer pretrained for downstream fine-tuning and can be adapted to classification tasks with a softmax classifier over [CLS].
3 Methods: Self-Attention Attribution
ATTATTR estimates how each self-attention relation contributes to a specified model prediction by integrating gradients along a path from zero attention to the original attention matrix. Compared with raw attention scores, it highlights sparser and more decision-relevant interactions.
- ATTATTR manipulates the internal attention matrix and observes model dynamics to inspect the contribution of token interactions.
- For each attention head, ATTATTR computes an attribution score for every token-to-token attention connection.
- The attribution integrates gradients from a zero-attention baseline to the original attention matrix, combining attention scores with prediction sensitivity.
- The Riemann approximation uses gradients sampled along the straight-line path, with m set to 20 in the experiments.
- In the illustrated BERT example, large attention scores do not necessarily indicate contribution, while attribution emphasizes the “don’t”–“I know” interaction for contradiction.
4 Experiments
Experiments evaluate ATTATTR for identifying important attention heads, visualizing hierarchical information flow, and extracting adversarial triggers in BERT. The method supports effective pruning, reveals task- and layer-dependent interaction patterns, and exposes vulnerabilities to trigger-based attacks.
- Effectiveness Analysis: ATTATTR identifies attention heads whose pruning produces more salient performance changes than pruning by average attention scores.Pruning the two highest-attribution heads in every layer sharply decreases accuracy, whereas retaining them preserves nearly 97% accuracy.
- Attention Head Pruning: ATTATTR outperforms the accuracy-difference baseline and remains competitive with Taylor expansion for attention-head pruning.On MNLI, retaining only 10% of attention heads yields approximately 60% accuracy with ATTATTR versus about 40% with Taylor expansion.
- Attention Head Pruning: Important BERT attention heads are roughly consistent across datasets with homogeneous tasks but differ between entailment and sentiment classification.RTE, MRPC, and MNLI share similar important heads, whereas SST-2 identifies a different subset.
- Visualizing Information Flow Inside Transformer: Attribution trees expose hierarchical word interactions and aggregate information toward tokens that help explain the model’s final prediction.Examples show sentence-local aggregation followed by cross-sentence combination in MNLI, and aggregation toward emotionally informative tokens before the final decision in SST-2.
- Visualizing Information Flow Inside Transformer: The effective receptive field is generally narrower in earlier layers and broader in later layers, with pairwise tasks first encoding locally before matching sentences.MNLI shows relatively local interactions in early and middle layers and broader dependencies in the top layers, while SST-2 increases monotonically across layers.
5 Related Work
Prior interpretability work studies feature or word importance and hierarchical explanations, but attention-based explanations remain contested and self-attention distributions are not directly interpretable.
- Saliency methods attribute predictions to individual input features, while contextual decomposition captures contributions from word combinations.
- Prior work develops hierarchical explanations to reveal how features combine, extending interpretability beyond isolated word importance.
- Attention weights intuitively indicate word dependencies, but studies disagree about whether they provide meaningful explanations for model predictions.
- For BERT, attention visualization, probing classifiers, and latent syntactic-tree methods analyze model behavior or hidden representations, while self-attention distributions remain not directly interpretable.
6 Conclusion
The paper proposes self-attention attribution to interpret information interactions inside Transformer and applies it to BERT through attribution-based analyses and applications.
- ATTATTR interprets information interactions inside Transformer and makes self-attention more explainable.
- The method identifies important attention heads for pruning, derives interaction trees that visualize information flow, and analyzes Transformer receptive fields.
- ATTATTR also constructs adversarial triggers for non-targeted attacks on BERT-based models.