Source-linked AI summary
Massive Activations in Large Language Models
Mingjie Sun, Xinlei Chen, J. Zico Kolter, Zhuang Liu
TL;DR
Research on LLMs has emphasized external behavior, while their internal mechanisms remain relatively limited in study. This paper characterizes massive activations across LLMs and ViTs, analyzes their bias-like role and connection to self-attention, and finds that explicit attention biases eliminate massive activations. The findings identify rare, input-agnostic activations as crucial components of model computation and attention.
Problem
Internal mechanisms of increasingly deployed LLMs remain relatively limited in study compared with their external behavior.
Method
The paper examines hidden-state activations across LLMs and ViTs, characterizes their locations and input dependence, and tests their roles in model performance and self-attention.
Results
Massive activations are widespread, input agnostic, crucial for LLM performance, and connected to implicit self-attention biases; explicit attention biases remove them.
Takeaways & Limitations
Massive activations provide a mechanism by which LLMs implement fixed biases in representations and self-attention, while analogous ViT activations offer an alternative interpretation of register tokens.
Takeaways & Limitations
The paper notes that the tendency for massive activations to appear at starting tokens may be attributable to autoregressive training, where every sequence contains an initial token.
Abstract
from arXiv · showhide
We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers. Code is available at https://github.com/locuslab/massive-activations.
1 Introduction
The paper identifies massive activations as rare, unusually large internal values that occur across LLMs, occupy characteristic locations, and function as input-independent biases linked to attention. It also reports analogous activations in Vision Transformers, where they act similarly as fixed biases.
- Phenomenon: Massive activations can exceed the median by more than four orders of magnitude and exceed 15,000 in LLaMA2-70B, while numbering fewer than 10 among tens of millions of activations.The phenomenon persists despite normalization layers and is illustrated in LLaMA2-7B.
- Locations: They appear across LLM families and sizes, with abrupt emergence across depth, concentration in a few input-agnostic feature dimensions, and frequent occurrence at starting or delimiter tokens.They emerge after a single layer and diminish near the final layers.
- Bias function: Massive activations act as fixed but crucial bias terms: zeroing only four in LLaMA2-7B catastrophically collapses performance, whereas replacing them with mean values does not hurt the model.The associated tokens are repurposed after initial layers to store these biases.
- Self-attention: They attract attention to their associated tokens, helping LLMs implement implicit attention biases during pretraining.Explicitly adding bias-like key and value embeddings eliminates the need for models to learn massive activations.
- Vision Transformers: Vision Transformers also contain massive activations, which occur less frequently, at fixed feature dimensions but varying patch tokens, and function as fixed biases.The paper connects this behavior to register tokens and offers an alternative to interpreting them as global-image-information aggregators.
2 Massive Activations
Massive activations are exceptionally large and rare scalar values found across many LLMs, typically concentrated in a few fixed feature dimensions and specific token positions. They emerge abruptly in early layers, remain nearly constant through intermediate layers, and diminish near the end.
- Existence and properties: Massive activations can exceed 10,000 times the median magnitude and occur only a few times among tens of thousands of hidden-state activations.In LLaMA2-7B, the largest activation is about 2,000 versus a median of about 0.2, with at most four massive activations in each presented hidden state.
- Existence and properties: 7,000 is the largest reported Mixtral-8x7B activation magnitude, around 4 orders of magnitude above its median feature magnitude of around 0.3.
- Layer locations: Massive activations emerge rapidly after an initial layer, remain nearly constant through most intermediate layers, and diminish in the final few layers.They first appear at layer 2 in LLaMA2-7B and layer 4 in LLaMA2-13B rather than accumulating gradually across many layers.
- Model-specific locations: In LLaMA2-7B, massive activations appear in feature dimensions 1415 and 2533 at the starting token and the first period or newline token.Inputs containing a period or newline show four massive activations; otherwise, two are associated with the initial token.
- Locations: Across LLMs, massive activations occupy very few fixed feature dimensions and follow model-specific token-location patterns.They may occur only at the starting token, at delimiters, or at certain weak-semantic word and numeric tokens.
- Difference from outlier features: Massive activations are scalar values tied jointly to token and feature dimensions, unlike outlier features, which are vectors spanning all tokens.The paper reports that massive activations do not overlap with outlier feature dimensions.
3 Massive Activations Act as Biases in LLMs
The paper tests whether massive activations are functionally important and finds that they behave as input-agnostic constant biases. Removing them causes severe performance degradation, while replacing them with empirical means has negligible effects.
- Input dependence: Massive activations have much smaller relative variance across inputs than other activation positions, supporting their interpretation as input-agnostic values.The comparison uses positions ranked by average magnitude, including the top 1%, top 10%, and median.
- Functional importance: Zeroing massive activations causes significant performance degradation, including exploding perplexity, whereas zeroing an equal number of median-magnitude activations causes no performance drop.
- Interpretation: Massive activations act as fixed but important biases in LLMs, analogous to the constant bias term b in y = Wx + b.Setting only four activations to zero in LLaMA2-7B produces catastrophic performance collapse, while setting them to mean values does not hurt performance.
- Possible placement rationale: The proposed explanation is that starting tokens are present in every autoregressive training instance, while delimiters offer relatively low-semantic-value locations for storing biases.The paper presents these as possible reasons for the observed token locations.
4 Effects on Attention
Massive activations concentrate self-attention on a few associated tokens and thereby create input-independent bias-like contributions to attention outputs. Adding explicit attention-bias parameters removes the need for these activations in GPT-2.
- 4.1 Attention is Concentrated on Massive Activations: After massive activations emerge, attention concentrates mainly on the associated tokens across deeper layers and across multiple LLMs.In LLaMA2-7B, massive activations first appear after layer 2; concentration is observed from layer 3 onward and also in LLaMA2-13B and Phi-2.
- 4.1 Attention is Concentrated on Massive Activations: Slightly positive logits for massive-activation tokens attract most softmax probability, while many logits for other tokens are negative.The relevant logits arise from query-key inner products involving tokens without massive activations versus keys associated with them.
- 4.2 Massive Activations Impose Implicit Attention Biases: Massive activations produce distinctive normalized representations and relatively stable QKV states for their associated tokens.After normalization, the embeddings appear sparse with two distinct non-zero elements, while subsequent QKV states vary little within each embedding.
- 4.2 Massive Activations Impose Implicit Attention Biases: Value updates from tokens receiving concentrated attention are nearly identical across output tokens, functioning as additive bias terms across inputs.The decomposition separates updates from concentrated tokens from updates aggregated from other tokens.
- 4.3 Explicit Attention Biases Eliminate Massive Activations: Explicit attention biases add learnable k′ and v′ vectors per head, concatenated with K and V as a drop-in attention replacement.This formulation leaves positional embeddings, MLP blocks, and other Transformer components unchanged.
- 4.3 Explicit Attention Biases Eliminate Massive Activations: GPT-2 models with explicit attention biases retain converged performance while massive activations disappear and top magnitudes increase gradually with depth.The comparison includes standard GPT-2, a sink-token variant, and GPT-2 with explicit attention biases.
5 Massive Activations in Vision Transformers
Vision Transformers exhibit massive activations in some model families, where they behave as input-stable biases, while register tokens provide an alternative bias mechanism.
- Massive activations in ViTs: Massive activations occur in CLIP and DINOv2 ViT-L but not MAE ViT-L.In CLIP and DINOv2, fewer than four unusually large activations occur in specific feature dimensions and random patch tokens; MAE instead has one uniformly large feature dimension.
- Massive activations are biases in ViTs: Across 1k images, CLIP and DINOv2 massive activations remain consistently present with values largely near their mean values.Unlike in LLMs, they first appear only in later ViT stages.
- Massive activations are biases in ViTs: Setting CLIP ViT-L massive activations to zero significantly reduces ImageNet zero-shot accuracy, whereas replacing them with means has negligible impact.The intervention targets the two largest massive activations at layer 13, where they first appear.
- Registers are biases in ViTs: DINOv2-reg ViT-G massive activations occur exclusively in fixed register tokens rather than patch tokens.The register-token pattern differs from standard ViTs, where massive activations appear in patch tokens.
- Registers are biases in ViTs: Fixing register features to their means preserves comparable accuracy, indicating that registers act as learned biases in ViTs.Their constant key and value states effectively introduce explicit attention-bias terms.
6 Related Work
The paper connects massive activations to prior findings on outlier features and attention concentration, while distinguishing its mechanism from those earlier observations.
- Intriguing properties of autoregressive Transformers: Prior work examined unusually large feature dimensions, growing initial-token norms, and related autoregressive Transformer activation patterns.These studies motivate comparison with massive activations in GPT-2 and other LLMs.
- Outlier features: Massive activations resemble quantization-related outlier features but differ fundamentally because outlier features are large across most sequence dimensions.The paper presents massive activations as a distinct type of outlier activation.
- Attention concentration patterns: Earlier studies reported attention focusing on special tokens such as [SEP] or the starting token, and sparse ViT activation patterns attracting attention.The paper relates these patterns to massive activations and analyzes why they emerge.
- Biases in self-attention: Existing self-attention bias mechanisms include additive QKV biases, positional biases, and manually designed softmax variants.The paper studies implicit attention biases arising within standard self-attention.
7 Conclusion and Discussion
The paper concludes that massive activations are widespread, input-agnostic, and crucial to LLM performance, with related phenomena extending to ViTs. It also identifies compression and interpretability implications while limiting explicit-bias experiments to small GPT-2 models.
- Conclusion: Massive activations are widespread in LLMs, input agnostic, extremely rare, and crucial for model performance.The conclusion connects them to implicit bias terms in self-attention and reports generalization to ViTs.
- Practical implications and future directions: The findings may inform LLM compression and mechanistic interpretability because massive activations create large values and concentrate attention on fixed tokens.The proposed attention formulation is suggested as potentially improving interpretability, while compression relevance is framed as a practical implication.
- Practical implications and future directions: Explicit-attention-bias experiments were limited to a small GPT-2 model, leaving larger-scale generalization and training stability for future work.The limitation is attributed to computational resource constraints.
- Additional LLM results: In LLaMA2-70B, massive activations occur in some numerical-token representations but not all numerical tokens; Mistral-7B and Mixtral-8x7B share feature dimension 2070.These observations extend the reported locations and cross-model similarities.
- Fine-tuned models and BOS token: Fine-tuned LLaMA2 and Mistral models also exhibit massive activations, with additional experiments varying whether sequences begin with the BOS token.The appendix reports figures for chat and instruction-tuned variants and BOS-prepended inputs.
A.4 Layer-Level Analysis
Across examined LLMs, massive activations generally emerge abruptly in early layers, remain nearly constant through intermediate layers, and disappear near the output. Attention LayerNorm can preserve these activations while suppressing ordinary values.
- All examined LLM families show the same broad layer-level trend of massive activations.The analysis covers LLaMA2, Mistral, MPT, Falcon, OPT, and GPT-2 families.
- Massive activations remain constant through most intermediate layers, emerge early, and disappear in the last layer.
- In LLaMA2-70B and Mistral-7B, massive activations form in the outputs of layers 9 and 2, respectively.Figures 32 and 33 analyze these models' attention logits after those formations.
- Massive-activation tokens receive predominantly positive attention logits, so softmax assigns them much higher attention probabilities.In LLaMA2-70B, the starting-word token and token 2 are highlighted as examples.
- Attention LayerNorm makes massive-activation tokens distinct by driving ordinary normalized values toward zero while preserving the massive activations.The normalization denominator is largely determined by the few massive activations.
B.3 Implicit Attention Biases
Across four prompts and multiple layers of LLaMA2-7B, value updates from tokens associated with massive activations remain remarkably similar. This supports their input-independent behavior in self-attention.
- The analysis tests this cross-prompt similarity in layers 3, 15, and 30 of LLaMA2-7B.
- Value updates from massive-activation tokens display remarkable similarity across four different input sequences.The comparison examines LLaMA2-7B layers 3, 15, and 30.
B.4 Explicit Attention Biases
Explicit attention-bias formulations are evaluated as alternatives to massive activations in GPT-2. They reproduce attention concentration and validation perplexity, but the tested alternatives do not eliminate massive activations during pretraining.
- The experiments include a learnable sink token and explicit bias parameters initialized with N(0, 0.02I).The sink token is prepended to each training sequence, while its cross-entropy loss is excluded.
- After 50,000 training iterations, all three GPT-2 models achieve validation perplexity 3.04 on OpenWebText2.
- Explicit attention biases assign most attention probability to the inserted k′ and v′ vectors.The attention distribution is averaged over 50 sentences of 30 tokens each.
- The tested bias formulations in Equations 6, 7, and 8 do not eliminate massive activations during pretraining.
- In ViTs, fixed register features preserve model performance and support interpreting registers as biases.Fix-Reg-Mean uses register features averaged over 10k ImageNet training images.
C.3 Layer-Level Analysis
Massive activations are not universal across Vision Transformers, and their layer-wise patterns differ from those observed in LLMs. When present, they can arise in later layers rather than following the LLM pattern.
- Some ViTs, including MAE ViT-B/L and DINOv2 ViT-S, do not exhibit massive activations.
- In DINOv2 ViT-L, massive activations appear in later stages but are absent from the output layer.
- ViT layer-level behavior differs from the LLM trend across the examined models.
D Models and Datasets
The paper reports model information for its LLM and ViT experiments and lists the datasets used with their license statuses.
- Models: Tables 7 and 8 provide relevant information about the LLM and ViT models used in the experiments.
- Datasets: The datasets include RedPajama, OpenWebText2, C4, PG-19, WikiText, MMLU, BoolQ, PIQA, WinoGrande, ARC, and ImageNet.
- Dataset licenses: License information is provided for each listed dataset, including Apache, MIT, Creative Commons, and Open Data Commons licenses.
- Dataset licenses: The license status is unclear for PIQA and ImageNet.