Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation

1.1. Background

• AS concentrates disproportionate attention on a few uninformative tokens, shaping Transformer training and inference, complicating interpretability, and exacerbating hallucinations and robustness challenges.Other Transformer limitations include quadratic self-attention cost, large historical KV caches, limited extremely long-context capacity, and limited interpretability [17].Source: §1.1 · p. 4
• Prior work exploits AS for efficient inference through sink-token preservation, studies its outlier-circuit mechanisms, and uses gated attention to mitigate it.The introduction identifies open questions about AS utilization, its emergence and functions, and strategies and trade-offs for designing Transformers that operate independently of AS; it says the literature remains fragmented.Source: §1.1 · p. 4

1.2. Position and Contributions

• The paper presents the first comprehensive survey of over 210 AS studies, organizing the literature into utilization, mechanistic interpretation, and mitigation.It synthesizes methodologies across Transformer models and offers a taxonomy of the field.Source: §1.2 · pp. 6–7
• AS research has progressed from empirical utilization, to mechanistic investigation, and then to structural mitigation.The survey reviews over 210 studies and describes this cumulative progression as the field’s focus broadens.Source: §1.2 · p. 6, Figure 3 · p. 6
• The survey’s contributions include methodological synthesis, future research directions, and guidance across nine application scenarios.It also identifies opportunities such as lightweight AS handling, adaptation for pretrained models, and standardized benchmarks.Source: §1.2 · p. 7

2.1. Preliminaries on Transformers

• A standard Transformer block combines multi-head self-attention and a position-wise feed-forward network, with residual connections and LayerNorm; domain adaptations retain Softmax attention as a core primitive [177].Self-attention captures long-range dependencies, while the feed-forward network acts independently at each position.Source: §2.1 · p. 8

2.2.1. Conceptual Background

• Softmax’s sum-to-one constraint can force attention onto specific tokens when no semantically relevant key is available, making them numerical reservoirs for excess attention.The section identifies this as a widely discussed explanation and defers its detailed analysis to Section 4.1.Source: §2.2.1 · p. 9

2.2.2. Attention Sink: Extreme Attention Concentration on Uninformative Tokens

• Across Transformer architectures, attention sinks pair disproportionately high attention with low-information content; in LLaMA, the first token receives maximum attention in 98% of heads.A prior identification criterion classifies tokens whose cumulative attention deviates substantially from the global average as sinks.Source: §2.2.2 · p. 10

2.3.1. Classical Language Models

• In BERT’s deeper layers, substantial attention consistently concentrates on [SEP] and [CLS] regardless of their semantic relevance to the query; these fixed-position sinks form vertically persistent high-attention bands in attention maps.Source: §2.3.1 · p. 11

2.3.2. Large Language Models

• In decoder-only LLMs, causal masking makes initial tokens visible to the whole sequence, while sink intensity increases in intermediate and deep layers and can also concentrate on delimiters or weak-semantic tokens.Sinks emerge during pre-training as optimization converges; higher learning rates and substantial weight decay strengthen them, whereas lower learning rates or minimal weight decay weaken or delay them. Research also treats initial anchors as useful in pruning and streaming, and studies redistribution, outlier circuits, attention bias, gating, and modified Softmax [25, 27, 30, 43, 44, 45, 54, 64, 75, 88, 93, 94, 146, 151, 156, 157, 175, 183, 184, 185].Source: §2.3.2 · pp. 12–13

2.3.3. Mixture-of-Experts Large Language Models

• In MoE LLMs, sink formation is tied to a sparse set of Super Experts: pruning three of 6,144 experts in Qwen3-30B-A3B causes catastrophic degradation, and removing these experts collapses sinks and harms coherence, reasoning, and output quality.Sink tokens score highly on Super Experts despite auxiliary balancing losses; the section also connects MoE sinks to massive activations and discusses gating, learnable attention bias, and pre-training prevention [54].Source: §2.3.3 · pp. 14–15, Figure 9 · p. 15

2.3.4. Multi-Modal Large Language Models

• Multimodal LLM sinks affect both inherited initial text tokens and visual background or non-semantic patches, especially in shallow vision-encoder and early fusion layers.Visual sinks divert attention from relevant image regions and can contribute to hallucinations; proposed directions include redistribution, interpreting cross-modal outliers, and decorrelation losses during pre-training.Source: §2.3.4 · p. 16

2.4. Vision Transformers

• In ViTs, high-norm sinks concentrate attention on stable, often background or boundary patches; register tokens redirect sink mass, while gated or modified softmax methods target outlier formation.These patches have comparatively stable activation magnitudes and can act as implicit bias terms; register tokens reallocate rather than eliminate the sink effect.Source: Figure 11 · p. 17, §2.4 · pp. 17–18

2.5. Diffusion Transformers

• DiT sinks usually emerge in central layers and carry global scene context, while outlier tokens reflect corrupted local patch semantics; suppressing sinks can shift perception without affecting text-image alignment [123–135].In some architectures, ablating sinks has minimal effect on generation quality; causal interventions found a sink-specific perceptual shift roughly six times larger than random masking.Source: §2.5 · p. 18
• DiT applications preserve early-frame KV states as video anchors, use register tokens to absorb outlier attention, and repurpose sinks for audio-visual stability or text-region localization.Rolling Forcing uses initial-frame KV states for long-horizon consistency; Deep Sink dedicates half its sliding window to persistent sinks, and MotionStream supports infinite-length video generation at constant computational cost.Source: Figure 12 · p. 19, §2.5 · p. 19

2.6. Attention Sink in Other Transformers

• In diffusion language models, sinks move during generation and their removal causes only minor degradation; sink-aware pruning and cross-layer consistency can reduce inference computation [137] [189] [190] [191] [192].Shallow layers use floating sinks to build global structure while deep layers capture semantics; an added self-attending sink token can stabilize the position-independent structural sink.Source: §2.6 · pp. 19–20
• Other architectures use sinks as learnable meta tokens or recycled registers in linear-time models, shared output biases in omni-modal LLMs, and persistent structural context for streaming video [193] [194].OutRo enhances reasoning with 1.1× decoding overhead; Mamba-R inserts and recycles registers to eliminate feature-map artifacts, and GFSSM borrows sinks to improve long-sequence stability.Source: §2.6 · p. 20
• RetoVLA repurposes discarded vision-encoder register tokens for action planning, recovering spatial awareness without added parameters and improving real-world robotic manipulation by 17.1%.The registers encode dense global spatial context.Source: §2.6 · p. 20
• FLEX enables 6× extrapolation (30 seconds) and matches long-video fine-tuned baselines at 12× scale, while LongStream achieves kilometer-long stable metric-scale reconstruction at 18 FPS.FLEX uses an inference-only sink with frequency-aware RoPE modulation and antiphase noise sampling; LongStream discards the first-frame anchor and uses cache-consistent training with periodic refresh.Source: §2.6 · p. 20

3.1. Sink Token Preservation

• Preserving sink tokens anchors attention under aggressive compression, but static-position assumptions can miss shifting sinks; dynamic detection must remain efficient and kernel-compatible.The approach is structurally simple and broadly applicable, but identifying non-initial sinks—including in ViTs and MLLMs—remains an open challenge.Source: §3.1 · p. 21, §3.1.3 · p. 24
• Retaining initial and recent tokens supports streaming without fine-tuning, while heavy-hitter selection, sparse attention, and sink protection enable efficient inference and 2-bit KV-cache quantization.MInference reports up to 10× faster pre-filling without accuracy loss; sink-token protection also enables 2-bit KV-cache quantization with minimal performance loss.Source: §3.1.2 · pp. 22–23
• Sink preservation also extends to video and multimodal models, using initial frames as global anchors and preserving visual attention sinks for cross-modal fusion.Video approaches enable stable generation of multi-minute videos without fine-tuning; PEVLM and SparseVILA preserve visual tokens that consistently receive high attention.Source: §3.1.2 · p. 24

3.2. Attention Redistribution

• Attention redistribution actively reduces sink influence by reallocating attention mass to semantically relevant targets, either through predefined rules or input-adaptive calibration.Explicit methods attenuate sink scores and redistribute the freed mass while preserving total attention mass; calibration dynamically responds to sink behavior.Source: §3.2.1 · pp. 25–26
• Explicit methods fully redirect attention in applications including visual grounding, hallucination mitigation, adversarial robustness, and sink-head pattern broadcasting.VAR redirects attention from background patches to foreground objects, AttnReal recycles output-token attention to visual tokens, GasEraser reallocates attention away from misleading text, and EVAS broadcasts a shallow sink head’s pattern.Source: §3.2.2 · pp. 26–27
• Adaptive calibration suppresses harmful sinks based on input context: ACT adjusts redistribution, ZeroTuning biases the initial token, and A2SF decays historical attention scores.ACT identifies harmful sinks, including non-initial ones; ZeroTuning uses a bias on the first token, while A2SF’s forgetting factor reduces older tokens’ cumulative importance.Source: §3.2.2 · p. 27
• Other calibration methods transfer positional attention patterns for long-context reasoning, align text-to-image cross-attention with syntax, or reduce sink reliance in continual learning.Pos2Distill transfers attention from advantageous start positions to middle positions; T-SAM aligns cross-attention with text self-attention; RoBERTa continual learning scales non-sink attention before fine-tuning.Source: §3.2.2 · p. 28
• Redistribution flexibly improves visual grounding and reduces hallucinations, but sink identification and score reallocation add overhead and can conflict with optimized attention kernels.Future work should identify sinks accurately with minimal overhead and preserve attention mass while integrating redistribution with high-performance kernels.Source: §3.2.2 · p. 28, §3.2.3 · pp. 28–29

3.3. Learnable Prefix Tokens

• Learnable Prefix Tokens are optimized explicit sinks that provide controllable behavior across streaming stability, vision artifacts, quantization, and task-relevant information aggregation.They require additional training and empirical tuning, unlike approaches that rely on naturally emerging sinks.Source: §3.3 · p. 29, §3.3.1 · p. 29, §3.3.3 · p. 33

3.3.1. Core Methodology

• Inserted at the sequence beginning, trainable prefixes absorb redundant or shared attention; because they remain fixed at inference, they act as stable anchors, as in ViT registers.Every sequence token can attend to the prefixes, and the registers absorb attention otherwise drawn to low-information background patches, cleaning attention maps and improving dense prediction.Source: §3.3.1 · p. 30

3.3.2. Practical Implementations

• Learnable placeholders ensure stable attention over arbitrarily long sequences; prefixes buffer outlier activations for per-tensor quantization without significant accuracy loss, while vision registers absorb attention artifacts.A [SINK] token absorbs attention that would collapse onto the class token with less than 1% parameter overhead; post-hoc registers can be guided by a frozen teacher.Source: §3.3.2 · pp. 30–31

3.3.3. Discussion and Insights

• Learnable prefixes cost additional training or fine-tuning, require empirical choices of token number and position, and may not generalize across architectures; future work could adapt their allocation and explain their learned representations.Effectiveness also depends on base-model capacity and training data.Source: §3.3.3 · p. 33

3.4. Sink Token Repurposing

• Sink Token Repurposing exploits existing sinks’ stable attention, fixed positions, numerical outliers, or geometry as computational primitives without adding tokens or altering attention distributions.The surveyed paradigms cover offensive, defensive, and efficiency-oriented uses.Source: §3.4 · p. 33, §3.4.1 · p. 33

3.4.1. Core Methodology

• Offensive methods exploit sink positions for attacks: trigger placement and attention alignment can strengthen backdoor persistence, while sink behavior can drive MLLM hallucinations.The backdoor attack uses training objectives rather than direct attention perturbation.Source: §3.4.1 · p. 33, §3.4.2 · p. 34

3.4.2. Practical Approaches

• Practical approaches use sinks offensively, defensively, and for efficiency, including sink-divergence regularization [174], register-based robust features, and sink-informed token or query pruning and dialogue compression.KeyDiff identifies and preserves critical tokens using their low cosine similarity to the mean key vector; OmniSparse uses sinks as memory anchors, while StreamingDialogue uses end-of-utterance sinks to compress histories.Source: §3.4.2 · p. 34

3.4.3. Discussion and Insights

• Repurposing offers a high-leverage way to shift model behavior, but sink identity and function are dynamic and context-dependent, and theory is lacking; future work calls for adaptive control and generalizable defenses.Open challenges include quantifying sink capacity, utility trade-offs, and downstream effects; proposed controllers could fortify, attenuate, prune, or ignore sinks per layer and input.Source: §3.4.3 · p. 35

4.1. Softmax Limitations and No-Op Theory

• Softmax’s sum-to-one constraint forces attention onto uninformative sink tokens when no key is relevant; negligible sink values make this a no-op with minimal residual updates.Quantizable Transformers motivates this account; extreme logits and growing activation outliers help sustain the behavior during training.Source: §4.1 · p. 36, §4.1.1 · pp. 36–37

4.1.2. Supporting Evidence

• Softpick achieves a 0% sink rate and eliminates massive activations; Softmax-1 lowers first-token attention from 65% to 3.3%, while post-Softmax gating mitigates AS.Source: §4.1.2 · p. 39

4.1.3. Discussion and Insights

• The no-op theory unifies high sink attention, small sink-value norms, and activation outliers, but sink formation during training and the causes of value suppression remain unclear.Future work should examine training dynamics, value-norm reduction, and mitigation beyond gating and modified Softmax.Source: §4.1.3 · pp. 39–40

4.2.1. Core Concepts

• Outlier Circuits link weight and activation outliers to attention outliers: aligned features amplify activations, increase query-key alignment, and focus attention on sink tokens with small values.The framework treats these interconnected outliers as numerical infrastructure sustaining AS, but causal validation of component interactions and training dynamics remains incomplete.Source: §4.2 · p. 40, §4.2.1 · pp. 41–42

4.2.2. Supporting Evidence

• Pruning only three of 6,144 Super Experts collapses AS and produces repetitive, uninformative outputs, while other interventions show that modulating outliers can control AS.Outlier–AS links are observed across language, vision, multimodal, and audio-visual speech models; activation outliers also follow a lifecycle from early emergence through middle-layer stabilization to late-layer decline.Source: §4.2.2 · p. 43, §4.2.3 · p. 44

4.2.3. Discussion and Insights

• Outlier Circuits provide a cross-architecture account of AS, but component interactions and the training dynamics governing circuit emergence and evolution remain underexplored.The section calls for systematic causal interventions and longitudinal analyses across training.Source: §4.2.3 · pp. 44–45

4.3.1. Core Concepts

• Implicit Attention Bias treats AS as a fixed, input-independent bias in the attention output; explicit attention biases and learnable key biases can modulate it.Training dynamics and the fragmented characterization of bias types remain open questions.Source: §4.3 · p. 45, §4.3.1 · p. 45
• Sink-token value updates are nearly identical across queries and inputs, acting as a constant bias; adding learnable key and value biases eliminates Massive Activations and attention sinks.This supports the interpretation that attention sinks are an implicit bias learned to cope with the Softmax sum-to-one constraint.Source: §4.3.1 · pp. 45–46

4.3.2. Supporting Evidence

• Observational and causal interventions support attention sinks as implicit biases, including shifting sinks to learnable bias positions and eliminating them with context-aware scaling.KVSink finds value updates remain consistent across tokens when sinks emerge; Massive Activations visually demonstrates implicit attention biases.Source: §4.3.2 · p. 46

4.3.3. Discussion and Insights

• The implicit-bias framework explains attention sinks through Softmax constraints and applies across LLMs, ViTs, and multimodal tasks.Training dynamics and relationships among bias forms remain unclear; future work should formalize emergence, unify bias mechanisms, and explore inference-efficiency and interpretability uses.Source: §4.3.3 · pp. 46–47

4.4. Geometric Anchoring

• Geometric Anchoring treats sink tokens as stable reference points that structure high-dimensional representations and guide other tokens’ attention and computations.The framework also motivates interpretability and control, but its evidence is mainly correlational, and the formation of specific anchors remains unexplained.Source: §4.4 · p. 47

4.4.1. Core Concepts

• Geometric formulations identify sink positional vectors as anchors [170], use sink-relative orthogonality to assess token importance, and characterize sink keys as outliers.Near-static sink keys can support KV-cache management and selective attention; other work uses structural anchors in code, dedicated sink tokens, user sequences, and multimodal memory, [192],,,.Source: §4.4.1 · p. 48

4.4.2. Supporting Evidence

• Approximately 0.94 average Spearman correlation links key distinctiveness with higher attention scores, while sink keys have near-zero cosine similarity to the mean key.Other tokens geometrically move toward the nearly unchanged sink representation; positional-vector distinctness correlates with sink attention, and out-of-distribution positions beyond the training window make sinks vanish as perplexity rises [170].Source: §4.4.2 · p. 49

4.4.3. Discussion and Insights

• Geometric structure supports positional-vector replacement for context extension, orthogonality-based pruning, and key-similarity eviction, but causal validation and anchor-formation mechanisms remain limited.Future work should formalize anchor emergence, lower detection and use costs, and integrate anchors into optimization, pruning, KV-cache management, or context extension.Source: §4.4.3 · pp. 49–50

4.5. Other Mechanistic Interpretations

• Complementary theories attribute attention sinks to structural biases, anti-overmixing, spectral dynamics, specialized heads, compression phases, and outlier-driven rescaling.Causal masking and RoPE can bias attention toward early or nearby tokens,; sinks may prevent representational collapse, arise from low-rank dynamics or reinforced active heads, emerge during middle-layer compression, and stabilize training through normalization-related rescaling [199]. The synthesis groups explanations across mathematical, training, numerical, geometric, and functional levels.Source: §4.5 · pp. 50–51

5.1.2. Practical Approaches

• Output gating after SDPA and value-state gating are two approaches; head-wise scalar gates offer an effectiveness-efficiency trade-off, while VGA gates values before attention weighting.Studies report that SDPA gating improves performance and stability, while VGA mitigates AS, stabilizes value-state norms, and improves downstream performance and quantization fidelity.Source: §5.1.2 · pp. 54–56

5.1. Gated Attention Mechanisms

• Gated attention decouples no-op behavior from extreme Softmax logits, directly suppressing attention outputs and mitigating AS, activation outliers, and training instability.The mechanism enables robust low-bit quantization and can suppress loss spikes, with minimal computational overhead.Source: §5.1 · p. 53, §5.1.1 · p. 54, §5.1.3 · p. 56

5.1.3. Discussion and Insights

• Gated attention generally requires training from scratch and can add parameters, while its training dynamics and evaluation standards remain underdeveloped.Future work includes lightweight post-hoc injection, parameter-efficient gates, studying gate evolution, and standardized benchmarks.Source: §5.1.3 · pp. 56–57

5.2. Modified Softmax Functions

• Modified Softmax functions intervene in normalization to prevent extreme logits and forced attention allocation without adding parameters; clipped Softmax enables full INT8 activation quantization while maintaining task performance.The section notes risks from excessive distribution flattening, retraining requirements, and incompatibility with optimized attention kernels.Source: §5.2 · p. 57, §5.2.1 · p. 58, §5.2.3 · p. 60

5.2.2. Practical Approaches

• Softmax-1 reduces first-token attention from 65% to 3.3% and activation kurtosis from 1657 to 3.1; Elastic-Softmax reports 59.58% attention sparsity.Softmax-1 enables robust 4-bit quantization, while Elastic-Softmax flattens attention to suppress forced attention on irrelevant tokens [165, 55].Source: §5.2.2 · p. 59
• Softpick [75] achieves a 0% sink rate and reduces activation kurtosis from 33,510 to 340, while SWAT makes AS impossible by removing cross-token normalization.Integral Transformer [65] smooths logits, mitigating AS and rank collapse while outperforming baselines on knowledge and reasoning benchmarks.Source: §5.2.2 · pp. 59–60

5.3. Learnable Attention Bias

• Learnable attention bias replaces AS’s implicit bias with a controllable explicit mechanism; proposed forms include key-value concatenation, key, scaling-factor, and denominator biases.Key-value bias training eliminates massive activations and AS [94]; key bias absorbs the sink at the bias position, and denominator bias creates a virtual sink.Source: §5.3 · p. 61, §5.3.2 · pp. 61–62

5.3.3. Discussion and Insights

• Learnable attention bias improves interpretability and control but requires training from scratch, can add parameters, and lacks standardized evaluation and a complete account of its training dynamics.Future directions include post-hoc or lightweight injection, common evaluation benchmarks, and further study of interactions with attention distributions and training dynamics.Source: §5.3.3 · p. 62

5.4.1. Core Mechanism

• Pre-training interventions proactively reduce AS by adjusting optimization dynamics rather than architecture, including optimizer, loss, normalization, and regularization choices.Adaptive optimizers such as Adam can favor privileged weight-matrix bases, causing activation spikes aligned with AS; gradient noise and parameter updates can also exacerbate outliers.Source: §5.4.1 · p. 63

5.4.2. Practical Approaches

• Activation outliers fell from over 10000 to below 20 with TWEO, which matched BF16-baseline performance under otherwise catastrophic standard FP8 training and raised throughput 36%.TWEO also enables W8A8 per-tensor static quantization at state-of-the-art quality. Sink-Aware Training [200] promotes uniform attention-head utilization and improves performance across vanilla, sink, and gated attention; decorrelation loss reduces intermediate sinks and improves word error rates under high feature downsampling while remaining stable at lower rates.Source: §5.4.2 · pp. 63–64
• Activation kurtosis drops from 1657 to 3.1 and the 4-bit weight-quantization perplexity penalty from 3565 to 0.3 with OrthoAdam.Muon removes Adam-style privileged bases in weight matrices, preventing extreme activation values without sacrificing convergence speed.Source: §5.4.2 · p. 65
• OSP combines Muon, single-scale RMSNorm, and learnable embedding projection to prevent privileged-basis amplification, channel-wise outlier amplification, and extreme embedding values.Training a 1.4B-parameter model on 1 trillion tokens produced the first production-scale LLM without extreme activation outliers; OSP combines strategies to achieve outlier-free pre-training without sacrificing efficiency or performance.Source: §5.4.2 · p. 65

5.4.3. Discussion and Insights

• Pre-training interventions address AS at its source and can make models robust to low-bit quantization, but require training from scratch and often add computational overhead.Future work should develop lightweight post-hoc methods for pretrained models and lower-overhead pre-training interventions, such as parameter-free regularization.Source: §5.4.3 · pp. 65–66

5.5. Other Mitigation Techniques

• Other mitigation methods include modified norm layers, gated rescaling, Value-path gradient changes, auxiliary losses, and architectural isolation for ViTs.The survey groups strategies into providing explicit, controllable alternatives to implicit AS and cutting the causal chain that generates it; examples include ME-layer masking, head-wise RMSNorm, semantic reconstruction and sink-divergence losses, and EDIT’s encoder-decoder design.Source: §5.5 · pp. 66–67

6.1. Model Pre-training

• For ViT pre-training, learnable prefixes can stabilize optimization [192], while LLM pre-training can preserve early sinks as anchors and use architectural or optimization interventions to reduce AS-related outliers.Suggested interventions include learnable attention bias [54], gated attention, modified softmax, auxiliary losses, outlier-safe optimizers, and gated normalization [199].Source: §6.1 · p. 68

6.3. Efficient Inference

• For efficient inference, preserve initial AS tokens as anchors for KV-cache compression and sparse attention, use learnable prefixes for low-bit quantization, or repurpose sinks for token selection.Sink Token Preservation supports cache eviction or quantization without performance collapse, while Sink Token Repurposing uses AS as geometric anchors.Source: §6.3 · p. 68

6.4. Mechanism Interpretability

• AS can be interpreted through softmax limitations and the no-op hypothesis [55] [200], outlier circuits and massive activations biasing attention logits, or implicit bias from model parameters or attention dynamics.Source: §6.4 · p. 68

6.5. Reducing Hallucination

• For reducing MLLM visual hallucinations, redistribute attention from AS tokens to informative visual tokens or fixed-vocabulary sinks, and preserve beneficial shallow-layer visual sink heads.For long-context LLMs in multimodal tasks, preserving initial AS tokens as context anchors can stabilize attention and reduce hallucination.Source: §6.5 · p. 69

6.6. Safety & Robustness

• For safety and robustness, account for attacks that exploit initial AS tokens as backdoor gateways or induce additional MLLM sinks with adversarial visual inputs, and use register embeddings or robustness tokens as defenses.Source: §6.6 · p. 69

6.7. General Capability Enhancement

• For general capability enhancement, redistribute attention to harness hidden sinks or reduce position bias through inter-position distillation, and use learnable prefixes for local-context aggregation.The section also recommends repurposing first-token masking when converting decoders to text encoders and modified softmax to denoise attention on low-semantic tokens.Source: §6.7 · p. 69

6.8. Long-Context Enhancement

• For long-context tasks, preserve initial AS tokens in the KV cache for streaming inputs, block-wise attention, and streaming-head compression, while retaining deep AS tokens as video-generation anchors.The section also recommends keeping AS on GPU for exact retrieval computation and redistributing attention in continual learning.Source: §6.8 · p. 69

6.9. Multi-Modal Enhancement

• For multimodal enhancement, redistribute attention away from visual or OCR sinks to informative regions, use registers as training-time or test-time sink solutions, and repurpose them as spatial memory or features [138].Preserving ViT sinks can provide semantic anchors and support pruning non-essential tokens during efficient inference.Source: §6.9 · p. 70

7.1. Challenges

• Dynamic sink detection and attention redistribution add computational cost, while post-Softmax operations, gated-attention latency, and kernel-incompatible softmax hinder efficient deployment.Element-wise gates add non-negligible latency, and modified softmax functions may lack implementations compatible with optimized attention kernels.Source: §7.1 · p. 71
• Gated attention, modified softmax, learnable biases, and prefix tokens often require retraining, while lightweight adaptation for pretrained models remains underexplored.Full retraining is often prohibitively costly for large models; adapters and continual pre-training are proposed as directions for closing this deployment gap.Source: §7.1 · p. 71
• The Softmax Limitations and No-Op Theory does not explain training dynamics fully; reinforcement between attention scores and value states, outlier alignment [198], and implicit-bias formation remain unclear.These gaps leave the emergence, stability, and evolution of AS during pre-training incompletely formalized, affecting mechanistic interpretation and mitigation.Source: §7.1 · p. 71

7.2. Future Directions

• Future work should make dynamic-sink detection, redistribution, gated attention, geometric measures, and modified softmax efficient, especially for ViTs and MLLMs with background-patch sinks.The goal is practical deployment in large models without sacrificing speed or scalability.Source: §7.2 · p. 71
• Parameter-efficient adaptation with adapters, LoRA, or continual pre-training could insert AS-aware gates, modified softmax, biases, or prefix tokens into pretrained models without full retraining.The aim is to preserve original functionality while suppressing AS and minimizing activation outliers, broadening access to mitigation across pretrained models.Source: §7.2 · p. 72
• A unified theory should formalize AS emergence and evolution, including score–value reinforcement and Outlier Circuit stabilization, and integrate existing perspectives.Connecting Softmax constraints, optimization, and implicit bias could guide interventions, predict AS behavior, and support mechanistic interpretation across Transformer architectures.Source: §7.2 · p. 72
• Research should test AS in hybrid linear attention [21] and 3D Transformers [208], and evaluate cross-architecture and cross-modal transfer beyond gated attention’s limited vision and multimodal study.These investigations could reveal which methods generalize and which require adaptation, with implications for efficiency, interpretability, and task-specific performance.Source: §7.2 · p. 72
• Standardized benchmarks should compare mitigation efficacy, computational overhead, and parameter costs, while coordinated complementary methods may improve efficiency, robustness, and generalizability.Fair, reproducible evaluation and systematic study of within-category synergies could guide robust solutions and hybrid approaches beyond standalone methods.Source: §7.2 · p. 72

8. Conclusion

• The survey organizes more than 180 AS studies into Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation, establishing a foundation for understanding AS in Transformers.It presents utilization strategies as ways to leverage AS for performance, mechanistic studies as explanations of its causes and roles, and mitigation as a means to control or suppress it for robustness and low-bit deployment.Source: §8 · p. 73
• AS influences training dynamics, model interpretability, and inference efficiency across diverse Transformer architectures.The survey also notes that mitigation techniques can support enhanced robustness and low-bit deployment.Source: §8 · p. 73
• Remaining challenges include computational efficiency, the need for training from scratch, and incomplete understanding of training dynamics; future directions include lightweight AS handling, emerging architectures, and standardized mitigation benchmarks.Source: §8 · p. 73

Limitations

• Coverage centers on established CLMs, LLMs, MLLMs, MoE LLMs, and ViTs; hybrid-linear attention models [21] [209] and VGGT [208] [210] are not comprehensively covered.The authors consider their insights and methods broadly applicable and plan to incorporate relevant studies as research on novel architectures expands.Source: §9 · p. 73

A. Comprehensive Overview of Surveyed Papers

• Table 1 tags surveyed papers by their attention-sink contributions to Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation.A dash marks a dimension that a study does not address.Source: Table 1 · p. 75

• The section lists [212] Attention Redistribution.Source: Large Language Models · p. 74
• One entry is labeled “Mechanism Interpretability ICML 2026 Link.”Source: Large Language Models · p. 74
• Another entry is labeled “Mechanism Interpretability General Capability.”Source: Large Language Models · p. 74

• The related-work list includes Learnable Attention Bias [54], Learnable Attention Bias Model Pre-training [213], and Outlier Circuits [43].Source: Mixture-of-Experts Large Language Models · p. 80

Vision Transformers

• [112] is listed as a 2026 arXiv work on learnable prefix tokens for multimodal enhancement.Source: Vision Transformers · p. 83
• The section lists “Efficient Inference” and “General Capability” as topic labels.Source: Vision Transformers · p. 83

Diffusion Language Models

• Sink-token repurposing is listed alongside efficient inference [190], multimodal enhancement [138], and a further repurposing entry [194].The multimodal-enhancement entry is labeled ArXiv 2025.Source: Diffusion Language Models · pp. 84–85
• Sink-token preservation is listed for efficient inference [191, 193] and long-context enhancement [195, 196].The entries include ArXiv 2026 listings for efficient inference and long-context enhancement.Source: Diffusion Language Models · pp. 84–85
• Learnable prefix tokens appear in four entries: [192], [139], [140], and [11].The supplied titles provide no further findings for these entries.Source: Diffusion Language Models · pp. 84–85
• Mechanism-interpretability entries are listed as [189] and [137], while other labels include No-Op Theory, General Capability Enhancement, and Efficient Inference.The mechanism-interpretability entries are labeled ArXiv 2026 and ArXiv 2025, respectively.Source: Diffusion Language Models · p. 84
• The surrounding entries name linear and hybrid linear attention, vision-language-action models, autoregressive video diffusion, omni-modal LLMs, and implicit attention bias.The implicit-attention-bias entry is labeled as multimodal enhancement and ArXiv 2026.Source: Diffusion Language Models · pp. 84–85