PaperlayerView exact revision on arXiv

Source-linked AI summary

When Attention Sink Emerges in Language Models: An Empirical View

Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, Min Lin

arXiv:2410.10781v2cs.CLcs.AIcs.LG

TL;DR

Attention sinks are widely used, but their origins and mechanics remain insufficiently understood. This paper studies their emergence during language-model pre-training across optimization, data, loss, and architecture, finding that sinks act as key biases and disappear under unnormalized sigmoid attention in models up to 1B parameters.

  • Problem

    The paper addresses the limited understanding of why language models allocate substantial attention to semantically unimportant initial tokens.

  • Method

    The paper empirically investigates attention-sink emergence in autoregressive language models by varying optimization, data distribution, loss function, and model architecture during pre-training.

  • Results

    Attention sinks emerge after effective optimization on sufficient data, depend on loss and data distribution, act as key biases, and do not emerge with unnormalized sigmoid attention in models up to 1B parameters.

  • Takeaways & Limitations

    Attention sinks store extra attention without contributing to value computation, and relaxing softmax-induced score dependence removes them in the tested models.

  • Takeaways & Limitations

    The study focuses on first-position sink tokens, leaving word-token sinks and their relation to pre-training for future work; downstream benefits also remain unclear.

Abstract

from arXiv · show

Language Models (LMs) assign significant attention to the first token, even if it is not semantically important, which is known as attention sink. This phenomenon has been widely adopted in applications such as streaming/long context generation, KV cache optimization, inference acceleration, model quantization, and others. Despite its widespread use, a deep understanding of attention sink in LMs is still lacking. In this work, we first demonstrate that attention sinks exist universally in LMs with various inputs, even in small models. Furthermore, attention sink is observed to emerge during the LM pre-training, motivating us to investigate how optimization, data distribution, loss function, and model architecture in LM pre-training influence its emergence. We highlight that attention sink emerges after effective optimization on sufficient training data. The sink position is highly correlated with the loss function and data distribution. Most importantly, we find that attention sink acts more like key biases, storing extra attention scores, which could be non-informative and not contribute to the value computation. We also observe that this phenomenon (at least partially) stems from tokens' inner dependence on attention scores as a result of softmax normalization. After relaxing such dependence by replacing softmax attention with other attention operations, such as sigmoid attention without normalization, attention sinks do not emerge in LMs up to 1B parameters. The code is available at https://github.com/sail-sg/Attention-Sink.

1 INTRODUCTION

The paper examines why attention sinks arise in language models, showing that they are widespread and emerge during pre-training. It relates their emergence to optimization, data, loss, and architecture, and interprets sinks as key biases rather than value-carrying content.

  • Attention sink assigns substantial attention to initial tokens regardless of semantic relevance and supports streaming, cache optimization, efficient inference, and quantization applications.
  • The phenomenon appears universally across inputs, including random sequences and small models, and emerges during language-model pre-training before instruction tuning.
  • The paper investigates optimization, data distribution, loss function, and model architecture as factors influencing attention-sink emergence.
  • Effective optimization on sufficient data promotes attention-sink emergence, whereas small learning rates make it less obvious and weight decay encourages it.
  • The sink position depends strongly on the loss function and data distribution and can shift away from the first token.
  • Attention sinks behave like key biases that store extra attention without contributing to value computation.

2 PRELIMINARIES ON LMS AND ATTENTION SINK

This section defines the autoregressive language-model and transformer attention setup used to study attention sinks. It covers token packing, positional embeddings, normalization structure, causal attention, and threshold-based sink measurement.

  • An autoregressive language model maps token inputs to next-token prediction logits using transformer decoder blocks.
  • Transformer blocks combine multi-head self-attention and feed-forward networks, with pre-norm or post-norm structures determined by layer-normalization placement.
  • Causal attention masks future positions, and the final hidden states are mapped through an unembedding layer to produce predictions.
  • Positional embeddings may be absent, absolute, learnable, relative, ALiBi, or Rotary, with some modifying query-key dot products directly.
  • Pre-training concatenates documents with optional BOS and EOS boundaries, then chunks them into sequences of context length C.
  • Attention sink denotes significant attention allocated to specific positions, such as the first token, producing vertical attention patterns.
  • The Sinkϵ 1 metric decreases as token length T increases, especially under stricter definitions using larger ϵ.

3 PROPERTIES OF ATTENTION SINK

Attention sink appears broadly across models and inputs, while its position and strength depend on token structure, positional design, and model training. Mechanistically, the first token behaves like a key bias that attracts attention without necessarily contributing to value computation.

  • The first token acts as biases: The first token acts as a key bias by reducing query-key angles, producing attention sink despite having relatively small key and value norms.
  • Measuring attention sink: The Sinkϵ metric identifies attention sink using a threshold on each head’s first-token importance score, with ϵ = 0.3 selected empirically.
  • Attention sink under different inputs: Random token sequences retain attention sinks, whereas repeated tokens eliminate them in Mistral and LLaMA models by equalizing hidden states.
  • Attention sink under different inputs: Instruction tuning has an insignificant impact on attention sink, producing comparable metrics between chat and base models.
  • Attention sink emerges even in small LMs, including Pythia-14M, and input domains have negligible effects on the Sinkϵ metric.

4 EFFECTS OF OPTIMIZATION ON ATTENTION SINK.

Attention sink develops during effective optimization and sufficient training, with its visibility shaped by learning rate and weight decay rather than batch size alone.

  • Weight decay: Larger weight-decay ratios induce more attention-sink heads, but excessive weight decay harms optimization and can eliminate the sink.
  • Training data amount: With less training data, attention sink disappears while trained LMs can exhibit overfitting behaviors.
  • Optimization steps: Attention sink emerges between 1k and 2k optimization steps under the default setup and becomes more obvious as pre-training progresses.
  • Learning rate: Smaller learning rates delay attention-sink emergence and produce less obvious sinks even when additional training steps compensate for slower optimization.
  • Batch size: Batch-size changes alone have no effect on attention sink.

5 EFFECTS OF DATA DISTRIBUTION pDATA ON ATTENTION SINK

Attention sink depends on how training data and attention context are structured: changing token placement can shift the sink, while insufficient data or local windows can suppress it.

  • Training data amount: With less training data, attention sink disappears, and further evidence indicates this effect is not related to overfitting.
  • Randomness in data distribution: Randomizing the first token yields Sinkϵ 1 = 27.03%, while randomizing the first two tokens shifts the sink to the second token.
  • Randomness in data distribution: When the first two tokens are randomized, second-token sink is Sinkϵ 2 = 14.08% and first-token sink is Sinkϵ 1 = 1.98%.
  • Randomness in data distribution: Increasing the number of random tokens during pre-training tends to make attention sink disappear.
  • Fixing token position: Fixing a token in the second or third position causes attention sink to appear at that fixed position instead of the first.
  • Window attention: In shifted-window attention, the global first token retains the sink, but corresponding window-start tokens do not, and smaller windows prevent emergence.

6 EFFECTS OF LOSS FUNCTION L ON ATTENTION SINK

The loss function influences both whether attention sink emerges and where it appears. Effective training and weight decay encourage sinks, while prefix and shifted-window objectives alter their positions or suppress them.

  • Attention sink emerges after effective training on sufficient data, but remains less obvious with small learning rates.
  • Weight decay: Weight decay generally encourages attention sink, although excessive weight decay can impair optimization and weaken or eliminate it.Even γ = 0 permits emergence; larger γ initially increases the number of sink heads.
  • Prefix language modeling: With prefix language modeling, the sink shifts among prefix tokens rather than remaining exclusively on the first token.The corresponding sink token also exhibits massive activations.
  • Shifted window attention: With shifted-window attention, tokens within the window can retain a sink on the absolute first token, whereas later tokens’ relative first token typically does not.Smaller window sizes prevent attention-sink emergence.

7 EFFECTS OF MODEL ARCHITECTURE pθ ON ATTENTION SINK

Attention sink is robust to many architectural choices but is strongly tied to attention biases and score normalization. Key biases can absorb the sink, while removing normalization prevents its emergence in the tested models.

  • Layer normalization: Post-norm LMs retain attention sink, with massive activations occurring before LayerNorm rather than in the normalized hidden states.The reported Sinkϵ value is 13.54%.
  • Attention biases: Key biases shift attention sink from the first token to the bias position, whereas value biases alone do not affect the sink.These setups have comparable model performance.
  • Attention biases: Attention sink behaves like a key bias that stores extra attention without contributing to value computation.Removing introduced sink or bias parameters eliminates the first-position sink but substantially reduces model performance.
  • Attention biases: Increasing the fixed value-bias norm shifts attention from the key bias toward the first token.The authors attribute this shift to the difficulty of removing larger value-bias effects from predictions.
  • Attention operation: Without normalization, sigmoid attention yields comparable validation loss but no attention sink, whereas normalization restores sink emergence.MLP-kernel attention shows no sink with or without normalization.
  • Attention operation: Relaxing tokens’ dependence on normalized attention scores removes attention sink and massive activations in models tested up to 1B parameters.Sigmoid attention without normalization remains free of massive activations at 1B parameters.
  • Architecture-wide effects: Positional embedding, FFN design, LayerNorm location, and multi-head design do not eliminate attention sink.Models with NoPE and several alternative positional embeddings still exhibit the phenomenon.

8 FUTURE WORK

The paper identifies scope boundaries and open questions around attention sink, especially beyond the first-position sink and its downstream effects. It also situates the phenomenon within broader Transformer literature and positional-embedding formulations.

  • Future work: The study focuses on sink tokens in the first position, while word-token sinks such as periods and newlines may vary across models and lack fixed positions.Future work will examine how these sink words relate to pre-training.
  • Future work: Whether attention sink benefits downstream language-model performance remains unclear.
  • Related phenomena: Related work connects attention sink with attention entropy collapse, attention-logit growth, strong outliers, and distinct representational spaces across Transformer models.The phenomenon has been reported in ViTs, encoder-only LMs, and autoregressive LMs.
  • Activation outliers: Token-wise activation outliers differ from channel-wise outliers and require different mitigation strategies.
  • Positional embeddings: Positional embeddings may be added to initial hidden states or incorporated by modifying query–key dot products.NoPE sets P = 0; absolute and learnable embeddings belong to the first category, while relative, ALiBi, and rotary embeddings belong to the second.

C.1 HOW POSITIONAL EMBEDDING RELATES TO ATTENTION SINK

Positional embeddings help determine whether repeated-token inputs develop an attention sink and where that sink appears. NoPE, relative PE, and ALiBi avoid a first-token sink in repeated sequences, while GPT2’s learned positional embedding attaches the sink to p1.

  • GPT2-XL retains significant first-token attention on repeated tokens, unlike the no-sink behavior established for several rotary models.
  • For NoPE, repeated tokens have identical initial states and uniformly distributed attention scores, so no attention sink appears.
  • Relative PE and ALiBi produce no attention sink for repeated tokens because their position-dependent scores favor nearer tokens rather than the first token.
  • Rotary PE also avoids a sink for sufficiently large repeated-token sequences when query-key norm products are bounded by ξ.
  • Replacing GPT2-XL’s first positional vector reduces the sink, while swapping it makes another position the sink, showing strong attachment to p1.
  • Across domains, attention-sink metrics are similar at ϵ = 0.2 and ϵ = 0.3, with only small fluctuations at ϵ = 0.4.

D.2 DATA DISTRIBUTION

Training-data quantity, rather than overfitting alone, influences attention-sink emergence, while fixing a token position relocates the sink to that token. Batch size and FFN activation choices do not affect emergence.

  • With 50M or 100M training data, models overfit early while Sinkϵ remains below 1%; with 5B data, Sinkϵ continues increasing after a later step.
  • Less training data makes attention sink disappear, and Figure 28 indicates this effect is not explained by overfitting.
  • Fixing a token in the first, second, or third position causes the attention sink to appear at the fixed token instead of the first token.
  • Batch size has no effect on attention-sink emergence.
  • Changing FFN activation functions does not affect whether attention sink emerges.

D.4 ATTENTION DESIGN

Attention architecture changes reveal that multi-head design and FFN-independent choices do not prevent attention sinks, whereas bias sharing and normalization alter their location or amplitude. Unnormalized sigmoid attention avoids sink emergence in the tested models.

  • Multi-head design does not affect the emergence of attention sink.
  • Sharing KV biases across heads shifts the sink from K biases back to the first token, while K biases are less affected.
  • Even one learnable dimension in K biases receives significant attention, and more learnable dimensions make the sink more obvious.
  • Reducing softmax normalization below one mitigates attention sink in fewer heads but does not prevent its emergence.
  • P-normalized softmax attention diverges for p = 2, 3, or 4, while smaller p values converge but mitigate massive activations less effectively than unnormalized sigmoid attention.
  • P-normalized sigmoid attention retains strong massive activations, whereas unnormalized sigmoid attention produces no attention sink across tested learning rates and weight-decay ratios.

E MORE EXPERIMENTS IN LM AFTER PRE-TRAINING

The paper further evaluates how attention design affects training after pre-training by comparing supervised fine-tuning behavior for softmax and unnormalized sigmoid attention.

  • Supervised fine-tuning is conducted on pre-trained 1B models using softmax attention and sigmoid attention without normalization.
Loading 2410.10781v2…