Source-linked AI summary
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Raito Kiya, Satoki Ohashi, Kosuke Sato, Go Kamoda, Ryosuke Takahashi, Yuji Yamamoto, Daiki Shiono, Keisuke Sakaguchi, Goro Kobayashi
TL;DR
The paper asks why Attention Sink and Massive Activations emerge at sequence-initial positions regardless of the occupying token, phenomena that matter for understanding attention and quantization. It disentangles BOS identity, positional encoding, and causal-mask self-concentration through interventions and repeated-token experiments. The results implicate self-concentration and resulting Value-non-mixing, with repeated-token evidence supporting this mechanism in three of five models.
Problem
The properties specific to sequence-initial positions that contribute to Attention Sink and Massive Activations remain unclear, although Massive Activations challenge low-precision quantization.
Method
The study intervenes on BOS placement, RoPE indices, and attention self-concentration, then tests Value-non-mixing using repeated-token sequences.
Results
Forcing self-concentration induces Attention Sink and Massive Activations across all five models, while repeated-token experiments support Value-non-mixing in three of five models.
Takeaways & Limitations
The findings support a mechanistic link between causal-mask self-concentration, Value-non-mixing, Attention Sink, and Massive Activations, informing understanding of attention-layer dynamics and future quantization strategies.
Takeaways & Limitations
The evaluated models span four families and use English datasets, so extension to other architectures, languages, and domains remains open.
Abstract
from arXiv · showhide
Large Language Models (LLMs) often exhibit "Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it. Our experiments suggest that self-concentration of attention, resulting from the causal mask, and the subsequent Value-non-mixing in attention outputs contribute to AS and MAs. These findings provide new empirical evidence on the internal dynamics of LLMs, offering insights that may inform future quantization strategies and advance our understanding of the internal mechanisms of attention layers.
1 Introduction
LLMs exhibit Attention Sink and Massive Activations at sequence-initial positions, but the properties driving these phenomena remain unclear. The study decomposes initial-position effects and identifies causal-mask self-concentration and resulting Value-non-mixing as contributing factors.
- Attention Sink concentrates attention on specific tokens, especially the first token, while Massive Activations create challenges for low-precision quantization.These phenomena frequently co-occur at initial positions.
- Prior accounts invoke Softmax no-op behavior, first-token key bias, and causal-mask asymmetry, but initial-position-specific contributions remain unclear.
- Value-non-mixing contributes to Attention Sink and Massive Activations at the initial position regardless of the occupying token.
- The study disentangles the BOS token, first positional encoding, and forced self-concentration under causal masking.
- Experiments provide evidence that self-concentration produces Value-non-mixing, in which the attention output carries a single Value vector rather than a mixture.
2 Background: Attention Mechanism and Attention Sink Metric
The attention mechanism computes each token output from Value vectors using learned Query, Key, Value, and Output projections with positional rotation. Attention Sink strength is quantified by the fraction of layer-head pairs exceeding a threshold at a target position.
- Attention computes each token output yi from the input sequence using Query, Key, Value, and Output projection matrices.
- RoPE contributes positional information through a rotation matrix RΘ,i in the attention computation.
- Attention Sink denotes disproportionately large attention weights αi,j assigned to a target position j across many heads.
- Sinkϵj measures Attention Sink strength using the proportion of layer-head pairs whose attention exceeds threshold ϵ.The metric averages an indicator over L layers and H heads.
3 Preliminaries: Influence of BOS Token
BOS relocation separates token identity from sequence position. Attention Sink remains associated with both the relocated BOS token and the initial position, while similar behavior appears for Massive Activations.
- The BOS token is swapped with the token at position t = 16 to disentangle BOS identity from initial-position effects.The experiment uses WikiText and models whose tokenizers prepend BOS.
- ϵ = 0.3 is used for the Sinkϵ metric, with additional analyses confirming findings for ϵ ∈ {0.2, 0.3, 0.4, 0.5}.
- Table 1 reports Sinkϵj when BOS is moved to position 16 using T = 64 and N = 100 WikiText sequences.
- Attention Sink still occurs at the BOS token after relocation and consistently emerges at the initial position even when that token is not BOS.
- Massive Activations show a similar trend, indicating that both BOS identity and initial position contribute to AS and MAs.
4 Mechanistic Analysis of the Initial-Position Effect on AS and MAs
The analysis separates BOS-token, positional-encoding, and causal self-concentration effects on the initial-position phenomena. Results indicate that self-concentration and the resulting Value-non-mixing state contribute to AS and MAs, while RoPE has limited influence under the tested intervention.
- Initial-position factors: Swapping the BOS token with a middle-position token leaves AS and MAs at the initial position, indicating an effect independent of token type.AS can also remain associated with the relocated BOS token, so both BOS identity and initial position contribute.
- Influence of Positional Encoding: RoPE intervention does not substantially reduce Sinkϵ1 in any of the five models.The intervention replaces the initial token’s Key-side RoPE index with a randomly sampled non-initial index in sequences without BOS.
- Influence of Self-Concentration of Attention: Forcing self-concentration at a non-initial position induces AS and MAs across all five models, although both effects are weaker in Mistral-7B-v0.3.The intervention sets the target position’s self-attention weight to 1.0 and all other attention weights to 0.0.
- Value-non-mixing as a Driver: Value-non-mixing occurs when the attention output uses a single Value vector rather than mixing distinct context-token Values.Repeated-token sequences reproduce this state without forcing self-concentration because identical tokens share the same Value vector.
- Value-non-mixing as a Driver: In repeated-token sequences, AS and MAs occur at positions that remain Value-non-mixing and subside after a distinct token interrupts that pattern.Uniform sequences show MAs across positions despite distributed attention, while inserted distinct tokens delimit the affected positions.
- Value-non-mixing as a Driver: The repeated-token effect is consistent in three models but weaker and more input-dependent in Mistral-7B-v0.3 and pythia-1b.The source leaves the cause of this input dependence, such as similarity between repeated and inserted-token Values, for future work.
5 Conclusion
The paper concludes that causal masking enforces self-concentration at the sequence-initial position, while experiments show that imposed self-concentration induces AS and MAs across five models. It further investigates Value-non-mixing as a potential mechanism underlying these phenomena.
- Causal masking inherently enforces self-concentration at the sequence-initial position, with α1,1 = 1.0.
- Forcing self-concentration at a non-initial position induces AS and MAs across all five evaluated models.
- The study investigates the resulting Value-non-mixing state as a potential mechanism through which self-concentration produces AS and MAs.
Limitations
The study’s scope and interventions leave several questions unresolved, including generalization beyond the evaluated models and English datasets, naturally occurring intermediate sinks, causal pathways, and quantization effects.
- Whether the findings extend to other architectures, languages, and domains beyond the evaluated English datasets remains open.The five evaluated models span four families, while Mixture-of-Experts models and other settings were not tested.
- The study does not establish whether naturally occurring intermediate Attention Sinks can be explained by the same Value-non-mixing framework.The intervention induces sinks at non-initial positions, but naturally occurring intermediate sinks remain outside the demonstrated explanation.
- The interventions cannot exclude other emergent factors, including query-side effects, query–key interactions, and head-level causal pathways.The RoPE manipulation changes only the initial token’s key-side positional index, and head-selective analysis remains future work.
- Value-non-mixing is defined as a discrete state, and a continuous predictive Value Mixing Index remains to be developed and validated across models and natural inputs.Such validation would need to test prediction of both Sinkϵ_j and massive-activation magnitude.
- Fixed-threshold Sinkϵ_j comparisons are sensitive to sequence length, motivating length-aware metrics.Attention weights average 1/i over accessible keys, making fixed thresholds easier to exceed in short sequences.
- The study does not evaluate whether Massive Activations or Value-non-mixing affect quantization error, perplexity, or downstream accuracy.Establishing that connection requires dedicated experiments.
A Related Work
Prior work studies the roles and effects of existing Attention Sinks, while this study examines how Attention Sinks and Massive Activations emerge at the sequence-initial position. It introduces Value-non-mixing as a distinct within-output concept and controls it experimentally.
- Attention Sinks aggregate attention on specific tokens, enabling KV-cache reuse and compression, while Massive Activations challenge effective low-precision quantization.The paper distinguishes Massive Activations from the outlier features discussed in related quantization work.
- Earlier studies describe delimiter-focused attention as a near-no-op because large attention weights multiply small value vectors, leaving the residual stream almost unchanged.This account concerns BERT attention heads and was related to broader activation outliers in trained transformers.
- This study differs by investigating how Attention Sinks and Massive Activations emerge at the sequence-initial position rather than primarily studying existing sinks’ effects or functional roles.Its direction of causality is therefore distinct from prior accounts.
- Value-non-mixing denotes the composition of Value vectors within one attention output, unlike prior work’s mixing of contextual information across positions and layers.The study controls this state using interventions and repeated-token inputs with an inserted distinct token.
B Experimental Setup
The experiments evaluate the proposed mechanism across multiple open-weight LLMs, datasets, and architectural configurations, reporting model-specific layers where Attention Sinks and Massive Activations emerge for relocated BOS tokens.
- The main experiments use WikiText and evaluate the hypothesis across prominent open-weight LLMs, with GSM8K and SlimPajama added for robustness analyses.The evaluated models include Llama-3.2-3B, Llama-2-7b-hf, Mistral-7B-v0.3, and pythia-1b, among others listed in the setup.
- The model set spans standard and grouped-query attention, LayerNorm and RMSNorm, and differing positional-encoding and projection-bias configurations.These architectural variations are used to assess the mechanism across non-identical attention and normalization designs.
- Llama-3.2-3B shows Attention Sinks and Massive Activations for the relocated BOS token from Layer 1 onward.
- Mistral-7B-v0.3 shows Attention Sinks from Layer 1 and Massive Activations from Layer 2, reversing the emergence order observed in Llama-2-7b-hf.
- Llama-2-7b-hf shows Massive Activations from Layer 2 and Attention Sinks from Layer 3, later than in the other two reported models.
D Results for Repeated Token Sequences across Models
Repeated-token experiments support the Value-non-mixing hypothesis across most evaluated models, while the effect is weaker in pythia-1b and Mistral-7B-v0.3. Inserting one distinct token localizes Attention Sinks and Massive Activations to preceding identical-token positions.
- Figures support the Value-non-mixing hypothesis in Llama-3.2-3B, Llama-2-7b-hf, and Qwen2-7B, with a weaker trend in Mistral-7B-v0.3.
- The analysis uses short fixed-token sequences and layer-wise hidden-state norm trajectories to examine the positional pattern across repeated-token conditions.The robustness analyses vary datasets, sequence lengths, repeat lengths, intervened layers and positions, and the Sinkϵ threshold.
- Inserting one distinct token raises Sinkϵ_1 from 0.000 to 0.68–0.80 for Llama-2-7b-hf and 0.69–0.80 for Llama-3.2-3B across Patterns 1–3.Qwen2-7B rises to 0.42–0.55, while Mistral-7B-v0.3 rises more weakly to 0.07–0.16.
- pythia-1b shows a much smaller effect: Sinkϵ_1 is 0.006 for Pattern 1 and rounds to 0.000 for Patterns 2–3.At ϵ = 0.2, Pattern 1 reaches 0.030, while Sinkϵ_1 remains at or below 0.03 across tested thresholds.
E.1 Varying Dataset and Input Sequence Length
Across datasets, sequence lengths, and repeat lengths, the reported AS patterns persist, with a short-length threshold exception caused by higher baseline attention weights.
- Dataset and input length: At ϵ = 0.3, self-concentration produces nonzero Sinkϵ16 for every model across WikiText, GSM8K, SlimPajama, and tested sequence lengths.The magnitude varies substantially across models and settings, particularly for Mistral-7B-v0.3.
- Dataset and input length: Large first-token hidden-state norms in Llama-3.2-3B also appear across WikiText, GSM8K, and SlimPajama.The baseline comparison uses input length 64 and confirms the pattern is not specific to WikiText.
- Repeat length: The gap between Uniform and Distinct-token patterns persists across repeat lengths 16–256, except at length 16 with ϵ = 0.2.At that shortest length and lowest threshold, even uniform sequences yield high Sinkϵ1 values because baseline attention weights are higher.
- Repeat length: At repeat length 128, Llama-2-7b-hf shows the same layer-wise hidden-state norm trend previously reported at length 64.Figure 21 extends the repeated-token analysis to the longer sequence.
E.3 Restricting the Self-Concentration Intervention to Layer Subsets
Restricting self-concentration to early layers largely preserves the induced attention sink, whereas middle or late layers do not. The effect also extends beyond position 16 and remains robust across threshold choices.
- Early, middle, and late layers: Early-layer intervention reaches 90–93% of the all-layer Sink0.3 intervention across all five models.The comparison uses the three early layers and evaluates Sink0.3 at the intervened position.
- Early, middle, and late layers: Middle- or late-layer intervention leaves Sinkϵ16 near zero or at its vanilla level in all five models.Layer 1 alone nearly reproduces the all-layer effect in Mistral-7B-v0.3 and pythia-1b, but has minimal impact in Llama-2-7b-hf and Qwen2-7B.
- Other intervened positions: Self-concentration increases Sinkϵt over vanilla at both t = 8 and t = 32 in all five models, although the increase is markedly smaller for Mistral-7B-v0.3.For Mistral-7B-v0.3, the values rise from 0.0660 to 0.1347 at t = 8 and from 0.0000 to 0.0866 at t = 32.
- Threshold robustness: The overall Sinkϵj trends remain consistent across thresholds ϵ ∈ {0.2, 0.3, 0.4, 0.5} for both RoPE ablation and self-concentration intervention.The comparisons are reported in Tables 12 and 13.