Source-linked AI summary
Why do LLMs attend to the first token?
Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, Razvan Pascanu
TL;DR
The paper addresses why LLMs learn attention sinks and how those patterns are used. It develops theoretical connections between sinks and over-mixing, then tests predictions across pretrained and newly trained models. The results support sinks as a mechanism for reducing perturbation sensitivity, with stronger sinks under larger models or longer contexts and construction affected by pre-training choices.
Problem
Although attention sinks are widespread and their occurrence has been studied, why LLMs learn them and how they are useful remains shallowly understood.
Method
The paper combines mathematical analysis of mixing, collapse, and perturbation sensitivity with measurements in frontier LLMs and experiments on models trained from scratch.
Results
Attention sinks help control over-mixing and perturbation sensitivity; stronger sinks occur with larger models or longer contexts, while pre-training choices affect their construction.
Takeaways & Limitations
Attention sinks provide a learned mechanism for keeping representations separated and making Transformers less sensitive to token perturbations.
Takeaways & Limitations
The analyzed Gemma 7B sink head has two operating modes at least for in-distribution examples, and these mechanisms eventually disperse with increasing context length.
Abstract
from arXiv · showhide
Large Language Models (LLMs) tend to attend heavily to the first token in the sequence -- creating a so-called attention sink. Many works have studied this phenomenon in detail, proposing various ways to either leverage or alleviate it. Attention sinks have been connected to quantisation difficulties, security issues, and streaming attention. Yet, while many works have provided conditions in which they occur or not, a critical question remains shallowly answered: Why do LLMs learn such patterns and how are they being used? In this work, we argue theoretically and empirically that this mechanism provides a method for LLMs to avoid over-mixing, connecting this to existing lines of work that study mathematically how information propagates in Transformers. We conduct experiments to validate our theoretical intuitions and show how choices such as context length, depth, and data packing influence the sink behaviour. We hope that this study provides a new practical perspective on why attention sinks are useful in LLMs, leading to a better understanding of the attention patterns that form during training.
1 Introduction
Attention sinks, often concentrated on the first token, are widespread learned patterns that this work studies as a useful mechanism for controlling information mixing in Transformers. The paper combines theoretical analysis with measurements and experiments to explain when and why sinks help.
- Attention sinks are widespread patterns in which seemingly meaningless tokens, often the first token, capture most attention.
- The paper asks why attention sinks are useful, since they arise as a byproduct of gradient descent rather than explicit priors.
- The paper supports this perspective through theory, measurements in frontier LLMs, and models trained from scratch.
- The first-position sink commonly coincides with the beginning-of-sequence token, which receives almost 80% of attention in a typical Llama 405B prompt.
- The central contribution is that attention sinks help deep Transformers avoid over-mixing and the resulting representational collapse.
2 Background
The paper places first-token attention sinks in the context of causal information propagation, collapse, and over-mixing in decoder-only Transformers. It defines a sink-rate measure and uses theoretical predictions plus experiments to study how sinks form and why they matter.
- Decoder-only Transformers use causal masking, so each token attends only to itself and earlier tokens.
- Attention sinks are tokens with limited semantic meaning that attract a large share of attention within an attention head.
- The paper focuses on sinks at the first token and measures sink rate as the proportion of model heads attending to the sink with coefficient at least ϵ = 0.3.
- Theoretical bounds predict stronger sinks with greater depth, model size, or context length, because sinks control perturbation sensitivity.
- Repeated attention mixing can cause rank collapse, over-smoothing, and representational collapse, while longer contexts can force global attention toward pure mixing.
- The paper connects these phenomena to attention sinks, arguing that sinks counter representation collapse by keeping token representations meaningfully separated.
3 Transformers blocks need to avoid over-mixing
Deep Transformers can over-mix token information, causing representations to collapse or become overly similar. The paper argues that attention sinks, especially at ⟨bos⟩, help control this mixing and limit perturbation spread.
- 3.1 Collapse and over-mixing: Rank collapse implies representational collapse, but representational collapse can occur without rank collapse.Rank collapse is stronger and primarily concerns depth, whereas representational collapse can also be studied in nonlinear systems over long sequences.
- 3.1 Collapse and over-mixing: Excessive depth or context length causes catastrophic over-mixing, motivating learned mechanisms that preserve separation between token representations.The paper connects this problem to rank collapse, representational collapse, over-smoothing, and over-squashing.
- 3.1 Collapse and over-mixing: The Jacobian measures how sensitive token j’s layer-L representation is to perturbations of token i, providing a measure of information mixing.The analysis extends over-squashing bounds to multi-head attention and relates sensitivity to attention paths.
- 3.1 Collapse and over-mixing: Attention sinks control perturbation sensitivity through weighted attention paths, with stronger sinks expected as depth, head count, or context length increases.The bound motivates predictions that larger models and longer-context training produce stronger sinks.
- 3.2 How sinks help prevent mixing in Gemma 7B: Keeping ⟨bos⟩ attenuates perturbation spread in Gemma 7B, whereas removing it makes representations change substantially more.Removing ⟨bos⟩ also smooths attention maps and increases the mixing-related quantity J_ij.
- 3.2 How sinks help prevent mixing in Gemma 7B: A Gemma 7B apostrophe head attends sharply when its activation condition is met and otherwise attends to ⟨bos⟩, implementing an approximate if-else no-op.The ⟨bos⟩ value has the smallest norm, while the apostrophe value has the largest, allowing default updates to remain small.
4 How does over-squashing predict attention sinks?
The paper tests whether over-squashing predicts stronger attention sinks in larger models and models trained with longer contexts. Experiments support both predictions, with sink formation increasing with context length and model size.
- Models trained on different context lengths process the same total of 5B tokens, isolating context length as the varied pre-training condition.
- Longer-context training produces more prevalent attention sinks, while very short-context training yields nearly none.This pattern accumulates during training, with sink development generally increasing with context length until saturation.
- Larger LLaMA 3.1 models exhibit stronger sinks, while the smallest 8B model is significantly more active across heads.Middle layers are more active, suggesting the sink metric may also proxy layer activity.
- The results support the theoretical prediction that larger models and longer training contexts require stronger sinks to control information mixing.
5 Is ⟨bos⟩in any way special?
The sink does not require the ⟨bos⟩ token specifically: training determines whether ⟨bos⟩ or another first token becomes the sink. Removing the sink at inference reduces downstream performance, especially for long-context tasks.
- When ⟨bos⟩ is fixed first during training, removing it at inference removes the sink and greatly reduces performance.
- Without ⟨bos⟩ during training, a slightly weaker sink forms at the first token regardless of the token identity.
- Sink formation remains consistent across causal and intra-document masking, although pre-training packing choices affect how sinks are constructed.
- Removing attention sinks at inference consistently lowers downstream performance, with the largest drop on the long-context ruler benchmark.The reported table caption states that performance can drop to 0 on the ruler task.
6 Conclusion
The paper presents attention sinks as a response to over-squashing and over-mixing. It concludes that sink attention reduces sensitivity to token perturbations, while scale and context length strengthen the vulnerability and the sink.
- Attention sinks emerge as a natural response to over-squashing and over-mixing in Transformer architectures.
- Directing substantial attention to ⟨bos⟩ helps models become less sensitive to token perturbations.
- Larger models and longer training contexts increase vulnerability to perturbations and produce stronger sinks.
- Data packing during pre-training can heavily influence the attention patterns that emerge.
A Experimental Details and Additional Results
This section provides additional details about the experiments reported in the main section.
- The appendix supplies additional experimental details for the main section.
A.1 Pre-training Experimental Details
The synthetic pre-training experiments use LLaMa2-style language models with roughly 120M parameters, varying training tokens by ablation.
- Packing experiments train on 30B tokens, while context-length ablations train on 5B tokens.The setup follows Gu et al. (2025) and adapts an MIT-licensed codebase.
- A single 5B-token training run takes up to 24 hours on one NVIDIA H100, depending on the experimental setup.
A.2 Additional Context Length Experimental Results
Context length strongly affects attention-sink formation: short-context models show little or no sink formation, whereas longer-context models develop much stronger sinks despite similar loss curves.
- Essentially no sink forms at context length 128, while longer contexts produce much stronger sinks.Models trained at different context lengths have similar training-loss curve patterns.
- Attention-sink development increases with context length during training, with sinks nearly absent in very short-context-trained models.The experiments keep tokens processed per training step equal and train each model on 5B total tokens.
- Data packing concatenates documents into fixed-length sequences and inserts ⟨bos⟩ and ⟨eos⟩ markers at document boundaries.
- Fixing ⟨bos⟩ in the first context position modifies attention masking so every token can attend to it, including across intra-document boundaries.
- The packing experiments evaluate 120M-parameter LLaMA2-style models trained on 30B tokens with context length 2048.They measure autoregressive loss and the first-position attention-sink metric on 100 validation sequences.
B How does BoS help avoid over-mixing?
The paper argues that ⟨bos⟩ attention helps prevent over-mixing by routing attention toward a low-norm token, allowing information to pass mainly through residual pathways.
- Causal attention mixes token representations across layers, but its spectral properties can drive rank collapse and oversmoothing.
- The experiments compare representations with and without ⟨bos⟩ using progressively longer prompts that ask the model to sum numbers.
- Including ⟨bos⟩ preserves greater representational dispersion as prompts lengthen, whereas omitting it increases divergence between representations.The comparison uses progressively longer number-summation prompts and Gemma 7B representations.
- The ⟨bos⟩ token’s disproportionately low value norm means attending to it effectively nullifies the mixed information, leaving the residual pathway to carry information forward.
- This mechanism counteracts over-mixing and rank collapse, resembling mixture-of-depths behavior without its attention-gating efficiency gains.
- The LLaMA sink evaluation uses 170 diverse texts and measures the sink metric over the first 64 tokens.
D Mathematical Results
The mathematical results analyze how token-to-token sensitivity propagates through deep, multi-head Transformers under causal attention and relate this propagation to over-squashing.
- The analysis assumes queries and keys are independent of values to simplify multi-head over-squashing bounds.
- The derivation applies to sub-multiplicative matrix norms, including induced operator norms associated with common ℓ1 and ℓ2 norms.
- Theorem 3.2 bounds sensitivity using the greatest layer Lipschitz constant, the number of attention heads, and causal paths between tokens.
- The proof decomposes the target partial derivative, bounds its components, and combines the bounds into a final upper bound.
- A causal path is a length-L walk from token i to token j that respects the causal attention structure.