PaperlayerView exact revision on arXiv

Source-linked AI summary

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu

arXiv:2603.05498v1cs.AIcs.CL

TL;DR

The paper asks why massive activations and attention sinks co-occur and what functional roles they serve. Through systematic experiments and architectural analysis, it shows that their overlap is largely an artifact of pre-norm Transformer design, while the phenomena have distinct global and local functions. The findings support independently mitigating either phenomenon without sacrificing language-modeling performance.

  • Problem

    Massive activations and attention sinks frequently co-occur in Transformer language models, but their functional roles and causal relationship remain unclear.

  • Method

    The paper uses systematic experiments and architectural analysis to relate normalization, residual accumulation, massive activations, and attention sinks.

  • Results

    The co-occurrence is an architectural artifact of pre-norm Transformers: massive activations act globally as implicit parameters, while attention sinks locally modulate heads toward short-range dependencies.

  • Takeaways & Limitations

    The two phenomena can be independently mitigated through alternative architectural choices without sacrificing language-modeling performance, enabling more targeted optimization for inference efficiency.

  • Takeaways & Limitations

    The architectural description focuses on the Llama family, chosen as a widely used open-weight model whose design influenced later open models.

Abstract

from arXiv · show

We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.

1. Introduction

The paper examines massive activations and attention sinks, two phenomena that often co-occur in pre-norm decoder-only Transformers and matter for efficient inference. It argues that their overlap reflects architectural and training choices rather than an inherent Transformer property.

  • Phenomena: Massive activations are extreme outliers in a few hidden channels for a handful of tokens, while attention sinks attract disproportionate attention across many heads and layers.The paper calls the associated tokens and channels spike tokens, spike channels, sink tokens, and sink heads.
  • Motivation: Both phenomena have practical implications for quantization, pruning, KV-cache management, and long-context inference.
  • Contribution: Prior explanations emphasize overlapping tokens, but the paper develops a mechanistic account of how this overlap emerges in pretrained LLMs.
  • Contribution: The paper’s core finding is that the co-occurrence is a predictable consequence of specific architectural and training choices, not an inherent Transformer property.
  • Central claims: Normalization bridges the phenomena: it can suppress massive activations while preserving attention sinks, whose overlap can therefore be decoupled by design choices.
  • Central claims: The paper identifies separate roles: massive activations provide near-constant representations, whereas attention sinks modulate attention outputs across heads and bias short-range dependencies.

2. Preliminaries

The preliminaries describe autoregressive next-token prediction and the decoder-only Transformer architecture used in the paper. They focus on token embeddings, pre-norm residual blocks, RMSNorm, attention, feed-forward transformations, and final prediction logits.

  • 2.1. Next-Token Prediction: Next-token prediction trains a language model to model each token conditional on its preceding prefix, using sequential structure as supervision.Autoregressive factorization turns joint-distribution modeling into conditional probabilities over the vocabulary.
  • 2.1. Next-Token Prediction: During training, teacher forcing supplies the ground-truth prefix at every position so all conditionals can be produced in parallel.
  • 2.2. Transformer Architecture: The described Llama-style model tokenizes text, maps tokens to dmodel-dimensional embeddings, and transforms them through L Transformer layers.Each layer contains an attention block and a feed-forward block, yielding 2L blocks overall.
  • 2.2. Transformer Architecture: Every block uses a pre-norm residual configuration, with RMSNorm applied row-wise before the block transformation.The block transformation is an attention block at odd indices and a feed-forward block at even indices.
  • 2.2. Transformer Architecture: Multi-head attention projects normalized inputs into head-specific representations, applies causal masking and row-wise softmax, then concatenates and projects the head outputs.The causal mask prevents each position from attending to future tokens.
  • 2.2. Transformer Architecture: The feed-forward block operates independently at each position and typically uses SwiGLU, while a final RMSNorm and linear projection produce next-token logits.Its intermediate dimension dffn is typically three or four times dmodel.

3. From Spikes to Sinks

Massive activations emerge when early blocks amplify a shared direction in a few channels, then persist through residual accumulation until late blocks cancel them. Their formation is linked to self-sinking tokens, whose early attention dynamics establish the trajectory needed for amplification.

  • Observed spike properties: Massive activations exhibit synchronized, nearly fixed-ratio spikes in a small number of channels and tokens.These properties define the sparse structure traced through the section.
  • The life cycle of massive activations: Early step-up blocks inject extreme values, which residual connections propagate through intermediate layers until late step-down blocks cancel them.The resulting rise–plateau–fall trajectory confines massive activations to intermediate depth.
  • Feed-forward block as directional quadratic amplifier: SwiGLU feed-forward blocks generate spikes through a directional quadratic amplifier operating in a near-identity SiLU regime.The high-gain quadratic forms are concentrated in spike channels and occur in step-up and step-down blocks.
  • Feed-forward block as directional quadratic amplifier: A dominant eigenvalue and shared principal eigenvector amplify aligned inputs across multiple spike channels simultaneously.The shared direction explains synchronized activation and nearly invariant inter-channel ratios.
  • What makes a token a spike token: Initial-position tokens become spike tokens across models largely independent of semantic identity, with over 98% of vocabulary items spiking at position 0.A static attention transformation steers first-token representations toward the trigger direction; delimiter tokens can follow a similar self-sinking trajectory.

3.2. The Emergence of Attention Sinks

Normalization transforms spike tokens into sparse, bounded, nearly constant vectors, causing their attention keys to occupy a low-dimensional subspace. Attention sinks then emerge when query subspaces align more strongly with these fixed sink-key subspaces than with non-sink keys.

  • Bounded Range: Normalization maps spike-token representations to bounded values, suppressing extreme magnitudes even when pre-norm inputs reach thousands.The resulting block output remains moderate and numerically stable.
  • Sparsification: Normalization suppresses non-spike channels, producing sparse representations concentrated in a low-dimensional subspace.The normalized state is approximately a multi-hot vector over spike channels.
  • Near-constant vector: Post-step-up spike-token representations collapse to nearly identical directions, with cosine similarities approaching 1.0.This empirically supports the near-constant approximation across spike tokens.
  • Key-space restriction: Spike-token keys are confined to the span of only a few weight-matrix rows, often collapsing to one or two dimensions instead of the full head dimension.This dimensionality restriction follows from sparse normalized representations.
  • Geometric alignment: Sink heads arise when query subspaces lie closer to fixed sink keys than to non-sink keys, creating consistent logit gaps and privileged attention positions.Non-sink heads instead align more closely with non-sink keys, preserving semantically distributed attention.
  • Mechanism: Sparsity and near-constancy jointly separate sink keys from non-sink keys into distinct subspaces, producing the logit gaps characteristic of attention sinks.These properties make sink behavior reliable across prompts.

3.3. Summary of Findings

The paper links massive activations and attention sinks through an architecture-driven pathway in pre-norm Transformers. Early feed-forward amplification creates persistent spikes, while normalization converts them into sparse, nearly constant vectors that support sink formation.

  • Spike formation: Early step-up feed-forward blocks create massive activations, which persist through the additive residual stream.The amplified representations then undergo normalization before contributing to sink formation.
  • Sink formation: Normalization transforms spike-token representations into sparse, nearly constant vectors whose keys occupy distinct low-dimensional subspaces.Attention sinks emerge when query subspaces align more strongly with these fixed sink-key subspaces than with non-sink keys.

4. Anatomy of Spikes and Sinks

Ablations show that spikes and sinks respond differently to architectural, optimization, context-length, and routing changes. Their frequent co-occurrence is therefore attributed to incidental interactions in standard Transformer designs rather than necessary functional coupling.

  • Feed-forward design: Massive activations and attention sinks emerge across all tested feed-forward designs, while SwiGLU and GeLU produce higher spikes than linear or attention-only blocks.Linear and attention-only blocks require gradual accumulation across layers rather than one-step amplification.
  • Normalization configuration: Normalization can suppress spikes while preserving sinks: Sandwich normalization reduces spikes, QKNorm nearly eliminates them, and DynamicTanh yields high sink ratios with low spike magnitudes.DynamicTanh still designates the first token as a stable reference point through alternative strategies.
  • Attention geometry: Increasing head dimension from 8 to 128 monotonically increases both sink ratio and spike magnitude, supporting head dimension as the dominant architectural factor for sink emergence.Larger attention subspaces separate sink and non-sink keys more cleanly.
  • Attention geometry: With fixed total attention capacity, fewer larger heads strengthen sink behavior and improve perplexity, whereas adding heads at fixed head dimension yields only marginal sink-ratio gains.Sink formation saturates once sufficient per-head capacity is available.
  • Gated attention: Representation-conditioned per-channel and per-head gating eliminates attention sinks and spikes with minimal perplexity impact, while static or unconditional gating preserves them.A single per-token gate leaves elevated sink ratios, consistent with sink formation being head-level.
  • Context-length induction: Removing short contexts causes the sink ratio to collapse, showing that sinks are induced by training regimes requiring local prediction within global attention.Short sequences make the first token a cheap, universally available reference for reducing far-context influence.
  • Summary: Spikes and sinks can be independently suppressed without measurable language-modeling degradation, so their overlap is best understood as a byproduct of default normalization and training choices.Conditional gating further supports sinks as an implicit input-dependent routing mechanism rather than an unavoidable structural feature.

5. Related Work

Prior work documents attention sinks and massive activations across Transformer and multimodal models, their practical uses and costs, and several mitigation strategies. Existing accounts connect them to token frequency, normalization, optimization, routing, and compression, motivating a more unified analysis.

  • Attention sinks: Attention sinks have been observed across LLMs, vision-language models, and multimodal models, and have been linked to token frequency in training data.Related work also describes sink-dominated heads as dormant or garbage heads.
  • Functional roles: Sink tokens support streaming heads, adaptive KV-cache eviction, layer-condensed caches, and hybrid sparse attention, but boundary sinks can reduce information retrieval from the middle of long contexts.This literature frames sinks as both an efficiency opportunity and a retrieval limitation.
  • Massive activations: Massive activations are channel-concentrated outliers that grow with model scale, behave like implicit bias terms, and co-locate with attention sink tokens.Their magnitudes are reported as stable across inputs and coupled with massive weights.
  • Practical implications: Massive activations degrade low-precision serving and training, motivating per-token scaling, mixed precision, and outlier-migration techniques.The practical issue is the adverse effect of outlier channels on quantization.
  • Mitigations and theories: Prior mitigation approaches modify attention, normalization, routing, spectral constraints, or weight scaling to reduce sinks and outliers.Related theories also attribute joint emergence to compression, spectral dominance, or adaptive optimization dynamics.

6. Conclusion

The study finds that massive activations and attention sinks often co-occur as architectural artifacts rather than inherently linked phenomena. They have distinct roles: massive activations act globally as implicit parameters, while attention sinks locally modulate attention heads.

  • Massive activations function as global implicit parameters, whereas attention sinks serve as local modulators for attention heads.The distinction follows from the roles of normalization and residual accumulation identified in the study.
  • Alternative architectural choices can independently mitigate both phenomena without sacrificing language-modeling performance.Modifying the normalization configuration is given as an example of such a choice.
  • Separating the phenomena's functional roles provides a path toward optimizing quantization, pruning, and long-context inference.

A. Experimental Settings

The experiments use a shared DCLM-based pretraining setup and evaluate attention sinks on sampled C4 text across fixed sequence lengths. Sink ratio measures the fraction of heads containing an early position with sufficiently high average received attention.

  • Models in Section 4 use the DCLM dataset, a shared codebase, and a common baseline recipe, varying only the factors under study in each ablation.The default recipe follows the Llama-style pretraining setup, with additional implementation details from torchtitan and Olmo codebases.
  • Evaluation samples C4 text and partitions it into chunks of 64, 256, 1024, 2048, or 4096 tokens to match each model's context window.The total evaluation budget is up to 1024 × 4096 tokens.
  • A head exhibits an attention sink when an early sequence position receives more than threshold ϵ average attention.The criterion searches positions in the first half of the sequence.
  • The model-level sink ratio averages the fraction of qualifying heads across evaluation sequences, using ϵ = 0.3 and T = 64.

B. Theorems and Derivations

The derivations express attention-block outputs through head-wise value transformations and the output projection. Partitioning the projection by head enables an additive decomposition of the final output.

  • Each attention head produces O(h) := A(h)V(h), where V(h) is obtained by applying the value projection to pre-attention hidden representations.
  • Partitioning the output projection matrix by head provides the structure used to analyze how individual head outputs contribute to the attention-block output.

Then the attention block output

The derivations decompose attention and feed-forward outputs into head-wise or coordinate-wise contributions. They establish additive projection structure, quadratic-form representations, and coordinate bounds under RMS normalization.

  • Then the attention block output: Multiplying the concatenated head outputs by the head-wise blocks of the output projection decomposes the result additively across heads.
  • Then the attention block output: Under the SiLU approximation for spike-token representations, a feed-forward output coordinate is approximated by a quadratic form.
  • Then the attention block output: The final projected output coordinate is represented in quadratic form through a weighted sum of rank-1 components.
  • Then the attention block output: The attention-output derivation concludes by establishing the stated form and completing the proof.
  • Then the attention block output: RMS normalization bounds every coordinate of the normalized hidden vector in magnitude by √dmodel.

C. Additional Empirical Results

Across diverse open-source Transformer families and sizes, massive activations and attention sinks recur consistently. The activations follow step-up/step-down dynamics, arise in specific feed-forward blocks, and align with high-norm quadratic directions.

  • Universality: Massive activations and attention sinks appear consistently across Llama 2, Llama 3, Qwen2.5, and Qwen3 models ranging from 7B to 14B parameters.The evaluated models span 28 to 48 layers and multiple architectural families.
  • Step-Up/Step-Down Dynamics: Every evaluated model exhibits activation spikes orders of magnitude above baseline variance, with abrupt step-up blocks followed by neutralizing step-down blocks.Comparing block outputs with post-residual representations indicates that the spikes originate in specific feed-forward blocks rather than accumulated residual error.
  • Quadratic Amplification: Massive-activation channels correspond to quadratic-form matrices Uk with exceptionally large Frobenius norms.These high-norm coordinates occur in the step-up and step-down blocks.
  • Quadratic Amplification: In Llama 3 8B, channels 788, 1384, and 4062 show distinct spikes, supporting high-gain quadratic directions as a structural invariant across Llama families.The channels align with the massive activation spikes observed in the corresponding activation analysis.
Loading 2603.05498v1…