What are you sinking? A geometric approach on attention sink

1 Introduction

• Beginning-of-sequence tokens receive 30–40% of attention regardless of semantic relevance, and removing the pattern degrades model performance.The paper argues these allocations serve an essential function beyond content processing.Source: §1 · p. 1
• The paper proposes that attention sinks act as reference frames, anchoring representational spaces through stable angular relationships.These anchors help tokens establish relative positions in high-dimensional spaces.Source: §1 · p. 1
• The paper identifies centralized, distributed, and bidirectional reference frames as optimal solutions to constraints imposed by softmax on the probability simplex.This geometric interpretation unifies attention-sink patterns across architectures.Source: §1 · p. 1

2 Related works

• Xiao et al. [2023] identified disproportionate attention to the beginning-of-sequence token, while mechanistic interpretability work provides broader context.The beginning token receives this attention consistently, according to the cited account.Source: §2 · p. 1
• Attention sinks can occur beyond sequence beginnings [Yu et al., 2024; Cancedda, 2024], while high query-key cosine similarity and low key norm are reported for first-token sinks [Gu et al., 2024].The paper interprets the former as distributed reference frames and the latter as a distinguished point in the representation manifold.Source: §2 · p. 2
• Prior studies associate positional-encoding choices with attention patterns, including RoPE’s first-token bias [Su et al., 2024] and absolute embeddings’ topological complexity [Darcet et al., 2024].Other cited work reports wavelet-like head patterns [Ruscio and Silvestri, 2024], effects of scaled RoPE [Kazemnejad et al., 2023], and mitigation by explicit bias parameters [Sun et al., 2024].Source: §2 · p. 2

3 Reference Frames

• Reference frames are modeled as distinguished points on a representation manifold with a mapping that supplies coordinate charts; architecture-specific inductive biases shape their self-organization during training.The paper formalizes an attention sink as a token position whose attention weight exceeds a threshold across many source tokens.Source: §3 · p. 2

3.1 Vector geometry of reference frame types

• Centralized frames use one dominant anchor, distributed frames use multiple weaker anchors, and bidirectional frames shift reference between sequence endpoints across layers.The paper associates these patterns with decoder-only models using standard RoPE, modified positional encodings, and encoder models with absolute position embeddings, respectively.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2 Information Geometry and the Probability Simplex

• Softmax constrains attention to the probability simplex, whose bounded, curved geometry and zero-sum allocation favor sparse distributions concentrated on a minimal set of tokens.The paper describes attention distributions as inducing a Fisher-information metric on the representation space.Source: §3.2 · p. 4

3.3 The Emergence of Attention Sinks

• With standard RoPE, the first token’s identity rotation and high angular alignment can make a small-norm key an effective attention sink.The paper presents the balance of small key norm and high cosine similarity as a geometric trade-off that supports consistent relationships without dominating output magnitudes.Source: §3.3 · pp. 4–5

3.4 Position Encoding and Reference Frame Formation

• Position encodings guide distinct reference-frame geometries: standard RoPE favors a centralized origin, scaled RoPE supports distributed anchors, and absolute embeddings support bidirectional anchors.The paper associates these with star-like attention graphs for standard RoPE, reduced first-token positional bias under scaled RoPE, and layer-wise shifts between sequence endpoints with absolute embeddings.Source: §3.4 · pp. 5–6

4 Methodology

• Persistent homology tracks connected components and cycles across layers, while thresholded attention graphs are assessed through connectivity, star-likeness, attention inequality, and degree centralization.The spectral measures are evaluated at seven attention thresholds, from 0.001 to 0.2.Source: §4 · p. 6
• Information-theoretic and value-space analyses measure sink-removal effects, reference-token magnitude and direction, attention entropy, transformation magnitude, and geometric-semantic alignment.The study also computes layer-wise Fisher norms and uses early Pythia checkpoints for random-matrix analysis of reference-frame emergence.Source: §4 · pp. 6–7
• The study analyzes decoder-only and encoder-only models using STEM-focused Wikipedia sentences, with 500 samples for topology, spectral, and Fisher analyses, 50 for KL analysis, and 100 checkpoint samples for temporal RMT analysis.The model set includes multiple sizes and variants of LLaMA, Phi, Qwen, Mistral, Gemma, Pythia, BERT, and XLM-RoBERTa.Source: §4 · p. 7

5 Analysis Results

• Each reference-frame type exhibits characteristic mathematical signatures across the analyses while establishing stable coordinate systems for representation learning.Source: §5 · p. 7

5.1 Centralized Reference Frames

• Centralized frames in decoder-only models form star-like attention structures, with negative connectivity-centralization correlation and about 60% of Fisher information concentrated in early layers.Removing attention sinks yields consistently negative KL reduction values in the analyzed models.Source: §5.1 · pp. 7–8

5.2 Distributed Reference Frames

• Distributed frames in modified-position-encoding models show threshold-dependent correlation sign flips, three-phase KL reduction, and multiple Fisher-information peaks across network depth.Source: §5.2 · p. 8

5.3 Bidirectional Reference Frames

• Bidirectional frames in encoder models show high early-layer topological complexity, rapid connectivity reorganization, a U-shaped KL profile, and middle-layer Fisher-information peaks.Source: §5.3 · p. 8

5.4 Value space analysis

• Value-space analysis distinguishes centralized single-origin frames, distributed local reference points, and bidirectional frames that shift from semantic to geometric organization with depth.Centralized frames prioritize geometric organization over semantics; distributed frames trade computational efficiency for adaptability to semantic structure.Source: §5.4 · p. 9

6 Limitations and Conclusion

• Topological and spectral analyses examine attention patterns at selected network snapshots, not their continuous evolution throughout training.Source: §6 · p. 9

A Geometric Effects of the Softmax Operation

• Softmax maps unbounded logits to a curved probability simplex, whose geometry creates pressure toward sparse attention distributions.Source: A Geometric Effects of the Softmax… · p. 11
• Because attention weights sum to one, tokens compete for a finite attention resource, and concentrating weight on reference tokens is presented as an optimal allocation.Source: A Geometric Effects of the Softmax… · p. 11
• The paper argues that softmax geometry and its constraints make reference frames optimal solutions for stable coordinates, with distinct types balancing efficiency and flexibility.Source: A Geometric Effects of the Softmax… · p. 11

B Random Matrix Theory Analysis of Attention Evolution

• Random Matrix Theory tracks attention-matrix eigenvalue metrics across Pythia training checkpoints to identify when reference frames form and how they develop.The analysis compares observed eigenvalue distributions with the Marchenko–Pastur distribution; deviations indicate non-random structure.Source: B Random Matrix Theory Analysis of… · pp. 11–12

B.1 Temporal Emergence of Reference Frames

• Across Pythia model scales, spectral-gap changes are nonmonotonic, while participation-ratio changes suggest a geometric phase transition between 6.9B and 12B models.Source: §B.1 · p. 12
• As models scale, attention generally becomes more uniform across tokens while larger models develop more pronounced sinks and increasingly specialized layer patterns.Source: §B.1 · p. 12

C.1 Reference frames statistical analysis

• Statistical analyses distinguish the frame types through topology, spectral correlations, KL reduction, and Fisher information, with centralized, distributed, and bidirectional models showing different patterns.The reported examples include consistently negative KL reductions for centralized frames, three-phase reductions for distributed frames, and U-shaped reductions for bidirectional frames.Source: C Statistical Analysis · p. 12, §C.1 · p. 13

C.2 Value space statistical analysis

• Centralized frames maintain about one dominant reference token, whereas distributed and bidirectional frames use more reference points and show distinct magnitude-alignment relationships.The reported mean reference counts are 1.0006 for centralized, 1.23 for distributed, and 1.58 for bidirectional frames.Source: §C.2 · pp. 13–14
• Across architectures, attention entropy decreases, value-transformation magnitude increases, and geometric-semantic alignment becomes more negative with depth.The reported magnitude increases are 81.61-fold for centralized, 206.33-fold for distributed, and 4.08-fold for bidirectional frames; all are reported with p < 0.001.Source: §C.2 · p. 14
• Layerwise correlations vary by frame type, with distributed correlations progressing from strongly negative toward near zero and bidirectional correlations ranging from negative to positive.Source: §C.2 · p. 15

• Betti0 counts remain stable across layers while persistence increases toward later layers, with reference-point concentration reaching 96+% in layer 25 for three models.Source: E Llama Family · pp. 15–16
• All Llama variants show 100% specialization on the beginning-of-sequence token, across parameter counts and fine-tuning status.Source: Table 9 · p. 15, E Llama Family · pp. 15, 17
• Algebraic-connectivity peaks are 1.01×–1.61× higher in middle layers, with a 1.61× peak in 1B versus 1.01× in 8B.Source: E Llama Family · p. 16, Table 10 · p. 16
• Meaningful graph structures emerge at thresholds ≤0.05, while Fiedler-value correlations sign-flip near 0.2 across model scales.Source: E Llama Family · p. 16, Table 11 · p. 17
• Removing sinks reduces attention-distribution uniformity, with negative KL reduction averaging -0.07 to -0.13 across network regions except layer 0.Source: E Llama Family · p. 16, Table 12 · p. 17
• Sink concentration remains 77–89% across network depths, exceeding Qwen's early-layer concentrations of 39–48%.Source: E Llama Family · p. 16

• Qwen models show moderate specialization (36.6%–65.4%) on common linguistic elements rather than BOS, unlike Llama's 100% BOS specialization.Source: F Qwen Family · p. 18, Table 15 · p. 18
• All Qwen models show decreasing persistence through depth, and the instruction-tuned model has greater component integration (-9.27) than its base counterpart (-8.19).Source: F Qwen Family · p. 18
• Star-likeness peaks in middle layers across three Qwen models, with ratios 1.00–1.05× higher there across scales and training objectives.Source: F Qwen Family · p. 18, Table 16 · p. 19
• Algebraic-connectivity peaks occur only at thresholds 0.1–0.2 in both 7B models, while the 3B model reaches 1.08–1.26× versus approximately 1.00× in 7B models.Source: F Qwen Family · p. 18
• Fiedler-value correlations sign-flip around threshold 0.05 in all Qwen models, with the 3B model remaining effective up to 0.1 versus 0.05 for 7B models.Source: F Qwen Family · p. 19, Table 17 · p. 20
• Degree centralization peaks in middle layers for all three models, with the 3B peaks mainly at thresholds 0.05–0.2 and 7B peaks across multiple thresholds.Source: F Qwen Family · p. 19
• KL reduction follows a three-phase pattern: early-layer values are +0.078 to +0.109, middle-layer values -0.026 to -0.051, and late-layer values -0.008 to -0.018.Source: F Qwen Family · p. 19, Table 18 · p. 20
• Maximum sink concentration occurs in deep layers 23–25, reaching 84–92%, while the most negative KL reduction occurs in layer 5 across models.Source: F Qwen Family · p. 19

G Gemma and Mistral

• Persistence increases by +0.0796 in Mistral and +0.3163 in Gemma, while both models retain Betti1 = 0.00 at every layer.Gemma’s persistence rises from 0.2015 to 0.5179; both models’ Betti0 counts remain stable from early to late layers.Source: G Gemma and Mistral · p. 21
• 100% specialization on <s> in Mistral and <bos> in Gemma marks a single beginning-token reference point in both models.The passage identifies this as the hallmark of centralized reference frames.Source: G Gemma and Mistral · p. 20, Table 21 · p. 21
• Mistral’s algebraic connectivity peaks in middle layers at 13.3343, whereas Gemma’s decreases from 13.0545 early to 9.9455 late.The passage associates these patterns with different reference-frame strategies.Source: G Gemma and Mistral · p. 22, Table 22 · p. 22
• Mistral’s algebraic connectivity and degree centralization correlate at -0.9405, while its Fiedler values and degree variance correlate at 0.9481.Both models also maintain star-likeness above 0.53 across layers.Source: G Gemma and Mistral · p. 22, Table 24 · p. 23
• Removing sinks yields negative KL reduction through middle layers (-0.0557 for Mistral; -0.0590 for Gemma), but positive values in first and final layers.The passage describes this as a U-shaped pattern across network depth.Source: G Gemma and Mistral · p. 22, Table 23 · p. 23
• Gemma’s sink concentration rises from 73.59% at t=0.8 in early layers to 80.28% in middle layers and 80.75% in late layers, while Mistral shifts from 89.62% to 81.55% to 84.90%.The first-layer concentration difference is 25.49 percentage points; the passage associates it with the models’ different position encodings.Source: G Gemma and Mistral · pp. 22–23
• 95.44% for Mistral and 91.57% for Gemma are their maximum sink concentrations, both at layer 21.The passage interprets the similarly high peaks as evidence that strong reference points are mathematically necessary.Source: G Gemma and Mistral · p. 23

H Encoder only (BERT and Roberta)

• Initial Betti1 values are 15.40 in BERT and 19.69 in RoBERTa, compared with 0.00 in decoder models.The passage links the encoders’ loops to richer interconnected attention structures.Source: H Encoder only (BERT and Roberta) · p. 24
• Betti1 decreases by 12.61 in BERT and 16.58 in RoBERTa as bidirectional frames simplify through the network.RoBERTa’s Betti1 falls from 22.68 to 1.69.Source: H Encoder only (BERT and Roberta) · pp. 24–25
• Early layers show 100% specialization on beginning tokens, while middle and late layers show 100% specialization on ending tokens in both encoders.This layer-specific pattern is described as a bipolar reference structure.Source: H Encoder only (BERT and Roberta) · pp. 23–24, Table 27 · p. 25
• BERT’s U-shaped algebraic connectivity profile and lower baseline star-likeness of 0.33–0.39 distinguish encoder structures from decoder reference frames.The passage also reports peak middle-layer degree centralization of 0.1579 for BERT and 0.1274 for XLM-RoBERTa.Source: H Encoder only (BERT and Roberta) · p. 25
• KL reduction is positive in first layers for BERT (0.0392) and XLM-RoBERTa (0.0635), with XLM-RoBERTa reaching 0.0774 in layer 23.The passage describes a U-shaped profile and reports a deeper peak negative layer in the larger model: layer 9 versus layer 5 in BERT.Source: H Encoder only (BERT and Roberta) · pp. 25–26
• Connectivity peaks after the initial layer in BERT and XLM-RoBERTa, then drops from 46.69 to 23.79 in BERT and 38.27 to 31.25 in XLM-RoBERTa from early to middle layers.The passage contrasts these sharp drops with gentler declines or increases in decoder models.Source: H Encoder only (BERT and Roberta) · p. 25, Table 28 · p. 26
• Early-layer KL reduction drops by 45.2% for BERT and 58.1% for XLM-RoBERTa from t=0.8 to t=0.95, while first-layer sink concentrations are 32.07% and 28.19%.The passage contrasts these lower concentrations with the 80–90% typically seen in decoder models and relates encoder patterns to absolute position embeddings.Source: H Encoder only (BERT and Roberta) · p. 26

I Phi

• KL reduction shifts from +0.1416 in layer 0 to -0.1525 to -0.1995 in middle layers, reaching -0.2787 in layer 25.Sink concentration rises from 35.23% in layer 0 to 97.19% in layer 25.Source: Table 30 · p. 27, I Phi · p. 28
• Phi-2 allocates 37.4% of Fisher information to attention and 32.7% to MLP components, with a triple-peaked layer distribution.The peaks include layers 0–1, around layer 26, and layers 29–31, which contain 49.6% of total Fisher norm.Source: I Phi · pp. 26–27, Table 31 · p. 27
• Fiedler-value and centralization correlations flip from -0.4518 at threshold 0.001 to +0.6009 at threshold 0.1.The passage says this pattern mirrors Qwen models using similar NTK-aware scaled rotary position embeddings.Source: I Phi · p. 28
• Betti0 rises from 0.0 in layer 0 to 25.99 in layer 31, indicating progressive fragmentation of attention structure.The passage contrasts this evolution with the stable topology of centralized reference-frame models.Source: I Phi · p. 28

• Persistence changes become more negative from -0.1065 in 2.8B to -0.1671 in 12B; late persistence reaches 0.0000 in 6.9B and 12B, versus 0.0760 in 2.8B.Source: J Pythia Family · pp. 29–30
• Larger Pythia models show stronger reference points alongside more nuanced relationships between those points and contextual tokens.Source: J Pythia Family · pp. 29–30
• All three models maintain Betti0 values near 25.7 and Betti1 values of 0.00 across layers, consistent with tree-like rather than loop-forming organization.Source: J Pythia Family · p. 30
• Maximum attention standard deviation stays within 0.1342–0.1471, while its peak shifts from layer 16 in 2.8B to layers 8–9 in larger models.Source: J Pythia Family · p. 30
• Token specialization on “ ˙Gthe” rises from 31.0% to 42.2% as Pythia scales from 2.8B to 12B, while specialized heads increase from 1 to 4.Table 35 reports the centralized reference-frame signatures across model sizes.Source: J Pythia Family · pp. 29–30, Table 35 · p. 31