Measuring the Mixing of Contextual Information in the Transformer

1 Introduction

• Attention weights provide a layer-wise distribution over input units, often presented as their relative importance, but their faithfulness has produced contradictory findings.The cited faithfulness studies reach differing conclusions.Source: §1 · p. 1
• Attention weights alone do not describe the Transformer’s information flow fully, and layer-wise attention distributions can mismatch global input attributions because intermediate layers attend to token mixtures.Prior work also found that residual connections can reduce the influence of multi-head attention; rollout assumes equal influence for the skip connection and attention weights.Source: §1 · pp. 1–2
• ALTI scores higher than gradient-based methods and similar approaches on two faithfulness metrics and shows greater robustness in Text Classification and Subject-Verb Agreement.The paper reports these comparisons across the two tasks.Source: §1 · p. 2
• ALTI measures token-to-token contributions across layers and aggregates them to produce input attributions for model predictions.It refines attention-block contribution measurements and tracks information mixing in token representations.Source: §1 · p. 2

2.1 Attention Block Decomposition

• The attention block comprises multi-head attention, the residual connection, and layer normalization, with each output token expressed through transformed input-token representations.The decomposition makes token-to-token contributions within a layer interpretable.Source: §2.1 · pp. 2–3
• Kobayashi et al. (2021) estimate each input token’s contribution using the Euclidean norm of its transformed vector.This metric is introduced as a way to quantify how much each transformed vector affects the layer output.Source: §2.1 · p. 3

2.2 Attention Rollout

• Attention rollout augments each attention matrix with the identity matrix, then multiplies matrices across layers so path contributions sum into an input-relevance matrix.The augmentation assigns equal influence to the skip connection and attention weights.Source: §2.2 · p. 3

3 Proposed Approach

• The prior ℓ2-norm estimate may be inaccurate because contextual representations are anisotropic, transformed representations can be more isotropic, and outlier dimensions dominate squared-component metrics.The authors argue that these properties make vector orientation and robustness to large component values relevant to contribution measurement.Source: §3 · pp. 3–4
• ALTI measures a transformed input vector’s contribution by its proximity to the attention-block output, using negative Manhattan distance and excluding vectors beyond the output’s ℓ1 length.The decomposition represents the block output as a sum of transformed input vectors; the closest vector is presented as contributing most.Source: §3 · p. 4, Figure 4 · p. 4
• Computing contributions for every output and input token yields matrix C, which records the layer’s token-to-token interactions.The distance between output yi and transformed input Ti(xj) is defined as di,j = ∥yi − Ti(xj)∥1.Source: §3 · p. 4
• ALTI aggregates layer-wise contribution matrices to track contextual information from input tokens and produce input attribution scores.The resulting relevance can be read for each final-layer token representation or for [CLS].Source: §3 · p. 4, Figure 5 · p. 4

4 Experimental Setup

• The study evaluates text classification on SST-2, IMDB, and Yelp, and subject–verb agreement on sentences with a masked present-tense verb.The classification datasets include short sentences and longer reviews; the agreement task predicts whether the masked verb is singular or plural.Source: §4 · pp. 4–5

4.1 Models

• Experiments use BERT, DistilBERT, and RoBERTa; classification models are fine-tuned models, while BERT is fine-tuned for the agreement task.For robustness analysis, the authors fine-tune 10 pretrained BERT models on SST-2.Source: §4.1 · p. 5

4.2 Faithfulness Metrics

• Faithfulness is evaluated with comprehensiveness and sufficiency, erasure-based metrics that assess prediction changes after removing important tokens or retaining only them.Comprehensiveness is more faithful when the predicted-class probability drops more; sufficiency is more faithful when the prediction changes less.Source: §4.2 · p. 5

4.3 Input Attribution Methods

• Input attribution methods rank tokens by their impact on model predictions and fall into gradient-based, perturbation-based, and attention-based categories; Attention Rollout and Globenc are attention-based examples.Gradient × input scores are aggregated using either the ℓ2 norm or the mean.Source: §4.3 · pp. 5–6

5.1 Faithfulness Results

• Across three models and four datasets, ALTI beats other attribution methods, exceeding IGℓ2 by 58% on average in comprehensiveness and 38% in sufficiency.ALTI repeatedly performs best across different tasks and models.Source: §5.1 · pp. 6–7, Table 2 · p. 7
• Gradient methods often miss cues for negative predictions and become less accurate on longer sequences, while ALTI highlights sentiment-bearing spans.Examples include negative-connotation phrases in Yelp and restaurant reviews.Source: §5.1 · p. 7

5.2 Robustness Analysis

• Across 10 BERT random-initialization models, ALTI yields more homogeneous interpretations by Jaccard-25% similarity and Spearman rank correlation than gradient-based methods.The comparison uses 10 pre-trained MultiBERTs models that differ only in random weight initialization.Source: Figure 7 · p. 7, Figure 8 · p. 8, §5.2 · p. 8

5.3 Ablation Study

• Using ℓ2 to isolate norm choice, ALTI’s layer-wise contribution measure largely improves on Kobayashi et al. (2021)’s Norms approach under Rollout.The comparison aggregates each type of contribution with the Rollout method.Source: Table 4 · p. 8, §5.3 · p. 8
• ℓ1 outperforms ℓ2 in almost every setting; its advantage is less noticeable on BERT, which the authors hypothesize reflects reduced representation anisotropy.The norm comparison reports faithfulness results across BERT, RoBERTa, and DistilBERT.Source: Table 5 · p. 8, §5.3 · pp. 8–9

5.4 Addition of Layer Norm 2

• Adding LN2 has negligible influence on SST-2 BERT probability-drop results across 10 seeds, with similar patterns across models and datasets.The comparison concerns removing important tokens and observing probability drop.Source: Figure 9 · p. 9, §5.4 · p. 9

• ALTI outperforms every tested input attribution method on two common faithfulness metrics and shows greater robustness.Source: §6 · p. 9
• ALTI quantifies Transformer information mixing through accurate layer-wise token-to-token contribution measurements based on ℓ1 metrics.Source: §6 · p. 9
• The interpretable attention-block decomposition is powerful when combined with the rollout method.Source: §6 · p. 9

• ALTI traces each input token’s influence to the last Transformer layer representation but does not include the classifier above it.Source: Limitations · p. 9
• Consequently, ALTI does not provide explanations for individual output classes, unlike gradient-based methods.Source: Limitations · p. 9
• Faithfulness in this work is evaluated using sufficiency and comprehensiveness metrics.Source: Limitations · p. 9

• The authors report no major ethical issue with ALTI, which does not take subjective priors into account.Source: Ethical Considerations · p. 9
• The method’s usefulness is evaluated on text-classification and subject-verb-agreement benchmarks.Source: Ethical Considerations · p. 9
• The authors note that these benchmarks have previously been used without major ethical concerns.Source: Ethical Considerations · p. 9

• Layer normalization transforms u by subtracting its mean, scaling by its standard deviation, and applying elementwise weights γ and bias β.Source: A Layer Normalization decomposition · p. 13
• The decomposition represents layer normalization as a scaled linear transformation Lu plus bias β.Source: A Layer Normalization decomposition · p. 13
• The displayed diagonal matrix has entries γ1, γ2, …, γn along its diagonal; the passage does not identify them as layer-normalization weights.Source: A Layer Normalization decomposition · p. 13
• The linear map subtracts the input vector’s mean, while the other map applies the layer-normalization weights elementwise.Source: A Layer Normalization decomposition · p. 13

• Table 6 reports faithfulness results for interpretability methods on DistilBERT across IMDB, Yelp, and SVA.Source: Table 6 · p. 13
• The tables mark metrics with ↑ when higher values are better and ↓ when lower values are better.Source: Table 6 · p. 13, Table 7 · p. 13
• Table 7 reports faithfulness results for interpretability methods on RoBERTa across IMDB, Yelp, and SVA.Source: Table 7 · p. 13

• Table 8 presents BERT saliency maps from three gradient methods and ALTI for a negative Yelp sentiment prediction.Source: Table 8 · p. 13
• The review text describes a low-budget horror movie, praising performances and the beginning and end while criticizing the slow middle.Source: C Qualitative Examples · p. 14
• Table 9 presents BERT saliency maps from three gradient methods and ALTI for a positive IMDB sentiment prediction.Source: Table 9 · p. 14
• Figures 10–12 show BERT relevancies across layers for Attn Rollout, Globenc, and ALTI, respectively.Source: Figure 10 · p. 15, Figure 11 · p. 16, Figure 12 · p. 17