PaperlayerView exact revision on arXiv

Source-linked AI summary

Massive Activations in Large Language Models

Mingjie Sun, Xinlei Chen, J. Zico Kolter, Zhuang Liu

arXiv:2402.17762v2cs.CLcs.LG

TL;DR

Research on LLMs’ internal workings remains relatively limited. This paper studies rare, unusually large activations across LLMs and Vision Transformers, finding that in LLMs they are input-agnostic, crucial to performance, and implement implicit attention biases.

  • Problem

    Research on LLMs’ internal workings remains relatively limited despite their growing use, leaving their internal mechanisms less studied than their external behavior.

    Source: §1 · p. 1

  • Method

    The authors examine hidden-state activations across LLMs and Vision Transformers, characterizing their locations, input dependence, functional roles, and relation to attention.

    Source: §1 · pp. 1–2

  • Results

    Massive activations are input-agnostic yet crucial for LLM performance, and LLMs use them to implement implicit biases in attention computation.

    Source: §1 · p. 2, §7 · p. 13

  • Takeaways & Limitations

    Massive activations offer an alternative interpretation of ViT register tokens as input-independent fixed biases and connect them to attention concentration in LLMs.

    Source: §1 · p. 2

    Potential interpretability and downstream benefits of the proposed attention formulation are presented as possibilities, not established outcomes.

    Source: §7 · p. 13

Abstract

from arXiv · show

We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers. Code is available at https://github.com/locuslab/massive-activations.

1 Introduction

Because research on LLM internal mechanisms remains relatively limited, this work identifies rare, unusually large hidden-state activations and examines their roles as input-independent biases in LLMs, self-attention, and Vision Transformers.

1 Introduction

  • Massive activations can exceed the median by more than 4 orders of magnitude and 15,000 in LLaMA2-70B, yet number fewer than 10 among tens of millions of activations; they occur across LLM families and sizes.The paper names these unusually large activations “massive activations.”Source: §1 · pp. 1–2
  • Massive activations emerge abruptly after a layer and diminish near the end; they occupy a few input-agnostic feature dimensions, often in starting-word or delimiter tokens, and differ from outlier features (Dettmers et al., 2022).Source: §1 · p. 2
  • Zeroing just four massive activations in LLaMA2-7B causes catastrophic performance collapse, while replacing them with their mean values does not hurt performance; associated tokens store these fixed biases after initial layers.Source: §1 · p. 2
  • Massive activations attract attention to associated tokens beyond the first-token “attention sink” phenomenon (Xiao et al., 2023b); training with explicit bias key/value embeddings eliminates the need to learn massive activations.The paper proposes that the activations encode implicit self-attention bias components learned during pretraining.Source: §1 · p. 2
  • In examined Vision Transformers, massive activations are less frequent than in LLMs but widespread, occur at fixed feature dimensions and varying patch tokens, and act as fixed biases; this offers an alternative to the register-token interpretation (Darcet et al., 2023).Both massive activations and register tokens learn values independent of input images, whereas the original work hypothesized that register tokens aggregate global image information.Source: §1 · p. 2

2 Massive Activations

Massive activations are exceptionally few but can exceed typical hidden-state magnitudes by orders of magnitude, and their values persist through most middle layers. They occupy a small number of feature dimensions at model-dependent token positions and differ from outlier features.

2 Massive Activations

  • In LLaMA2-7B, at most four of roughly 40,000 activations are massive, and the largest is about 10,000 times the median.A loose criterion is magnitude above 100 and roughly 1,000 times the hidden-state median; in Mixtral-8x7B, the largest reaches 7,000, around four orders of magnitude above the median.Source: §2 · p. 3

2.1 Which Layers?

  • Massive activations emerge rapidly, remain nearly constant through most intermediate layers, then diminish in the last few layers.In LLaMA2-7B they first appear at layer 2 and remain nearly constant through layer 30; emergence is also rapid in LLaMA2-13B, at layer 4.Source: §2.1 · p. 4

2.2 Which Feature and Sequence Dimensions?

  • Across LLMs, massive activations occupy a few fixed feature dimensions, while sequence locations range from starting tokens alone to delimiters and weak-semantic words.For example, Mixtral-8x7B activations occur at the starting token, delimiters, and words such as “and” and “of”; the broader location categories also include starting-token-only and starting-token-plus-strong-delimiter patterns.Source: §2.2 · pp. 4–5

2.3 Difference from Outlier Features

  • A massive activation is a scalar tied jointly to one token and feature, unlike an outlier feature vector, and the two phenomena do not overlap in LLaMA2-7B or 13B.The study identified 10 and 25 outlier features in those models, respectively, but none matched the massive-activation feature dimensions.Source: §2.3 · p. 5

3 Massive Activations Act as Biases in LLMs

Interventions show that massive activations are important to LLM computation, while their values are nearly input-invariant and function like bias terms. The paper also offers possible explanations for why they occur at particular tokens and layers.

Variance across input sequences

  • Across 100 sequences, massive activations have considerably smaller variance relative to their mean than activations at other magnitude-based hidden-state positions.The comparison includes the two largest, top 1% and 10%, and median-magnitude positions in LLaMA2-7B and 13B.Source: §3 · p. 5, Table 2 · p. 6

Setting massive activations to zero

  • Zeroing massive activations significantly degrades performance, while zeroing an equal number of median-magnitude activations causes no performance drop.The zeroing intervention is applied where massive activations first appear.Source: Table 3 · p. 6, §3 · p. 6

Setting massive activations to mean values

  • Setting massive activations to their empirical means produces negligible changes in perplexity and zero-shot accuracy, indicating that they function like input-agnostic bias terms.The empirical means are computed over 100 RedPajama sequences.Source: Table 3 · p. 6, §3 · p. 6

Why these layers and tokens?

  • The starting token may host massive activations because it occurs in every autoregressive training instance and is used in every forward pass.The paper proposes this as a possible explanation for their location.Source: §3 · p. 6
  • Delimiter tokens may offer low-semantic-cost locations for biases, while their emergence after initial layers may allow token meaning to be processed and transferred elsewhere first.The paper suggests self-attention can transfer these tokens’ semantics to other positions, where they are then preserved.Source: §3 · p. 6

4 Effects on Attention

Massive activations shape self-attention by drawing attention to a few tokens and using their value updates as implicit bias terms. Adding explicit attention biases to GPT-2 eliminates massive activations while preserving performance at convergence.

4.1 Attention is Concentrated on Massive Activations

  • From layer 3 onward in LLaMA2-7B, attention concentrates mostly on the two massive-activation tokens, a pattern also observed in LLaMA2-13B and Phi-2.Most other attention logits are negative, while logits for keys associated with massive activations are slightly positive, so softmax assigns them most of the probability; unlike a starting-token-only pattern, substantial attention can go to other tokens too.Source: §4.1 · pp. 7–8, Figure 6 · p. 7

4.2 Massive Activations Impose Implicit Attention Biases

  • After attention LayerNorm, the two massive-activation tokens have sparse embeddings with two distinct nonzero elements, while their subsequent QKV states vary considerably less within each embedding.The authors hypothesize that attention LayerNorm may play a pivotal role in this process.Source: §4.2 · p. 8
  • Value updates from massive-activation tokens are nearly identical across query tokens and inputs, acting as implicit additive bias terms in attention outputs.The analysis isolates value updates from the attention-concentrated tokens and separates them from updates aggregated from other tokens.Source: §4.2 · pp. 8–9, Figure 8 · p. 9

4.3 Explicit Attention Biases Eliminate Massive Activations

  • GPT-2 with explicit attention biases eliminates massive activations, while matching standard GPT-2 and sink-token GPT-2 performance at convergence.The added learnable key and value vectors are concatenated with the key and value matrices, making the formulation a drop-in replacement for standard attention; top activation magnitudes then increase gradually across layers.Source: §4.3 · p. 9, Figure 9 · p. 10, Figure 10 · p. 10

5 Massive Activations in Vision Transformers

ViTs exhibit massive activations selectively: in some standard models they act as input-stable biases, while DINOv2-reg localizes them to registers that function as learned attention biases.

Massive activations in ViTs

Massive activations are biases in ViTs

  • Zeroing the two largest CLIP ViT-L activations at layer 13 significantly reduces ImageNet zero-shot accuracy, while replacing them with means causes negligible loss, supporting a fixed-bias function.Source: Table 4 · p. 11, §5 · p. 11
  • In CLIP and DINOv2 ViT-L, massive activations persist across images with values near layerwise means and arise only in later stages.Source: §5 · p. 10, Figure 13 · p. 11

Registers are biases in ViTs

6 Related Work

Prior work identified unusual activations, outlier features, attention concentration, and self-attention biases; this work relates massive activations to these phenomena and analyzes their role in producing attention patterns and implicit biases.

Intriguing properties of autoregressive Transformers

Outlier features

Attention concentration patterns

Biases in self-attention

7 Conclusion and Discussion … A.3 BOS Token <s>

The paper identifies rare, input-agnostic massive activations as important to LLM performance and attention, and reports that the phenomenon also extends to Vision Transformers. Its discussion highlights implications for compression and interpretability while noting limits on the attention-formulation experiments.

7 Conclusion and Discussion

  • Massive activations are widespread in LLMs, remain largely input-agnostic yet crucial for performance, and act as implicit biases in attention; the findings also generalize to Vision Transformers.The paper describes the activations as extremely rare and establishes a close connection to self-attention.Source: §7 · p. 13
  • The paper identifies a new type of LLM outlier activation that may inform quantization and compression research, where large-magnitude activations are a known challenge.The discussion relates this challenge to Dettmers et al., 2022 and Xiao et al., 2023a.Source: §7 · p. 13
  • Excessive attention to a few fixed tokens may hinder mechanistic interpretability; the proposed attention formulation could improve interpretability, but its effects were tested only on a small GPT-2 model.The authors identify testing at larger scales and effects on training stability as future directions; the GPT-2 experiments were limited by computational resources. They cite Olsson et al., 2022, Darcet et al., 2023, and Wortsman et al., 2023.Source: §7 · p. 13

A Additional Results on Massive Activations in LLMs

A.1 Pretrained LLMs

  • Massive activations are consistently present in these models; LLaMA2-70B exhibits them in some but not all numerical tokens, while Mistral-7B and Mixtral-8x7B share feature dimension 2070.Source: §A.1 · p. 17

A.2 Fine-tuned LLMs

  • The appendix demonstrates massive activations in instruction-fine-tuned LLaMA2 and Mistral-family models.The evaluated models include LLaMA2-7B-Chat, LLaMA2-13B-Chat, Mistral-7B-Instruct, and Mixtral-8x7B-Instruct.Source: §A.2 · p. 19, Figure 24 · p. 19, Figure 25 · p. 19

A.3 BOS Token <s>

  • The main experiments disabled the optional BOS token; additional figures show massive activations in LLaMA2-7B, LLaMA2-13B, and Mixtral-8x7B when inputs are prepended with <s>.In some tokenizers, including LLaMA2, <s> is the beginning-of-sequence token.Source: §A.3 · p. 20, Figure 28 · p. 20, Figure 29 · p. 21

A.4 Layer-Level Analysis … B.2 Attention LayerNorm

Across the examined LLMs, massive activations persist through most intermediate layers, emerging early and disappearing in the last layer. In self-attention, they concentrate attention on corresponding tokens, while attention normalization preserves outliers and makes some normalized token features extremely similar.

A.4 Layer-Level Analysis

  • Massive activations follow the same general pattern across LLMs: they remain nearly constant through most intermediate layers, emerge early, and disappear in the last layer.The layer-level analysis covers LLaMA2, Mistral, MPT, Falcon, OPT, and GPT-2 model families.Source: §A.4 · p. 21, Figure 31 · p. 22

B Additional Results on Self-Attention

  • Appendix B extends self-attention results with broader-model concentration analysis and LayerNorm findings, and also covers implicit attention biases and explicit-bias GPT-2 training.The supplied passages give further detail on the concentration and LayerNorm analyses, but only identify the implicit-bias and GPT-2 studies as additional topics.Source: B Additional Results on Self-Attention · p. 23, §B.1 · p. 23, §B.2 · p. 25

B.1 Attention Concentration on Massive Activations

  • Massive activations form in the output feature of layer 9 in LLaMA2-70B and layer 2 in Mistral-7B.These examples extend the attention-concentration analysis to additional LLMs.Source: §B.1 · p. 23, Figure 32 · p. 23, Figure 33 · p. 23
  • Attention is predominantly focused on massive-activation sequence positions; in LLaMA2-70B, the starting word and token 2 receive substantial logits and much higher attention probabilities.The attention softmax is computed along each row, allocating higher probability to these special tokens.Source: §B.1 · p. 23, Figure 32 · p. 23, Figure 34 · p. 24

B.2 Attention LayerNorm

  • Tokens associated with massive activations have drastically different key and value states, and the analysis investigates how attention layernorm plays a crucial role in this process.Source: §B.2 · p. 24
  • In both LayerNorm and RMSNorm, massive activations dominate the normalization denominator, pushing normal values toward zero while preserving outliers.The resulting normalized features at tokens Who and newline become extremely similar.Source: §B.2 · p. 25, Figure 35 · p. 25

B.3 Implicit Attention Biases

Across four prompts, LLaMA2-7B shows remarkably similar value updates within the same layer, extending the finding across additional inputs and layers.

B.3 Implicit Attention Biases

  • The analysis extends the earlier value-update finding to four input prompts and layers 3, 15, and 30 of LLaMA2-7B.The prompts include conversational and quoted-text examples.Source: §B.3 · pp. 25–26
  • The comparison focuses on the latter half of each input sequence, following the two tokens associated with massive activations.Source: §B.3 · p. 26
  • Within each layer, value updates P_i v_i remain remarkably similar across four input sequences in LLaMA2-7B.The comparison covers layers 3, 15, and 30, focusing on the latter half of each sequence after the two tokens associated with massive activations.Source: §B.3 · pp. 25–26

B.4 Explicit Attention Biases … C.2 Registers are Biases in Masked Autoencoders

Explicit attention-bias vectors attract most attention, yet alternative bias formulations do not eliminate massive activations. In ViTs, massive activations vary by model, and fixed MAE register features leave performance unchanged, supporting the view that registers function as biases.

B.4 Explicit Attention Biases

  • The GPT-2 experiments train each model for 50,000 iterations on approximately 2B tokens using the NanoGPT reproduction and its recommended settings.For the sink-token model, sequences are prepended with a learnable [SINK] token, whose cross-entropy loss is excluded; explicit-bias k′ and v′ are initialized from N(0, 0.02I).Source: §B.4 · pp. 26–27
  • All three GPT-2 models reached validation perplexity 3.04 after 50,000 training iterations.The validation split was constructed from OpenWebText2.Source: §B.4 · p. 27
  • With explicit attention biases, most attention probability goes to the inserted k′ and v′ vectors, including in early layers such as layer 1.The attention distributions average probabilities over 50 sentences of 30 tokens each; the default GPT-2 also shows the observed relationship between massive activations and attention concentration.Source: §B.4 · p. 27, Figure 39 · p. 27
  • The three tested bias formulations do not eliminate massive activations during GPT-2 pretraining.The variants include zero k′ and v′ vectors, extra learnable query and key feature dimensions, and an extra value parameter.Source: §B.4 · p. 27, Figure 40 · p. 28

C Additional Results on Vision Transformer

  • The additional ViT results cover massive activations across images, MAE register tokens, and layer-level analyses.The appendix introduces image illustrations in C.1, register-token analysis in C.2, and further layer-level results in C.3.Source: C Additional Results on Vision… · p. 28

C.1 Massive Activations in ViTs

  • In standard CLIP ViT-L and DINOv2 ViT-L, massive activations occur in varying patch-token positions across images, while DINOv2-reg ViT-G has them in fixed register 3.The examples cover four images.Source: §C.1 · p. 28, Figure 42 · p. 29, Figure 43 · p. 29

C.2 Registers are Biases in Masked Autoencoders

  • Fixing MAE register features to their average values over 10k ImageNet training images does not affect model performance, supporting registers as ViT biases.The analysis treats the MAE dummy token as a register token and keeps its features constant across output features of all ViT layers.Source: §C.2 · p. 29, Table 6 · p. 30

C.3 Layer-Level Analysis

ViTs differ from LLMs in whether they exhibit massive activations and in how those activations vary across layers. In DINOv2 ViT-L, they occur in later stages but are absent from the final-layer output.

C.3 Layer-Level Analysis

  • Unlike LLMs, some ViTs lack massive activations, and observed ViTs show different layer trends; DINOv2 ViT-L activations occur in later stages but not the final-layer output.The analysis covers MAE and CLIP (Figure 45), DINOv2 (Figure 46), and DINOv2-reg (Figure 47).Source: §C.3 · p. 30, Figure 45 · p. 30, Figure 46 · p. 30
  • MAE ViT-B/L and DINOv2 ViT-S lack massive activations, whereas CLIP ViT-L and DINOv2 ViT-L exhibit them.Source: §C.3 · p. 30

D Models and Datasets

The section identifies the LLM and ViT models used and lists the datasets used in the work with their license information.

D Models and Datasets

Related papers

Loading 2402.17762v2…