PaperlayerView exact revision on arXiv

Source-linked AI summary

House of Cards: Massive Weights in LLMs

Jaehoon Oh, Seungjun Shin, Dokwan Oh

arXiv:2410.01866v2cs.LGcs.AIcs.CL

TL;DR

The paper traces massive activations to early-layer feed-forward network intermediate states and identifies weights tied to those activations. It introduces MacDrop for parameter-efficient fine-tuning; the supplied findings show that top-five massive weights are a tiny share of parameters yet critical for generation.

  • Problem

    The paper investigates where massive activations originate and which weights produce them, since attacking massive weights can impair LLM generation.

    Source: §1 · p. 2

  • Method

    MacDrop applies dropout to pre-trained massive weights during parameter-efficient fine-tuning, beginning with a high rate and progressively reducing it.

    Source: §1 · p. 2, §3 · p. 6

  • Results

    0.0005% of Llama-3-8B parameters are top-5 massive weights; zeroing them removes instruction-tuned models’ generation ability, while retaining only them preserves it.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Massive weights are predominantly learned during pre-training and are highly related to LLM performance, motivating MacDrop’s effort to reduce reliance on them during fine-tuning.

    Source: §1 · p. 2, §3 · p. 6

    LLMs that are not sensitive to massive phenomena may not perform effectively with algorithms based on those phenomena.

    Source: §2.3 · p. 6

Abstract

from arXiv · show

Massive activations, which manifest in specific feature dimensions of hidden states, introduce a significant bias in large language models (LLMs), leading to an overemphasis on the corresponding token. In this paper, we identify that massive activations originate not from the hidden state but from the intermediate state of a feed-forward network module in an early layer. Expanding on the previous observation that massive activations occur only in specific feature dimensions, we dive deep into the weights that cause massive activations. Specifically, we define top-$k$ massive weights as the weights that contribute to the dimensions with the top-$k$ magnitudes in the intermediate state. When these massive weights are set to zero, the functionality of LLMs is entirely disrupted. However, when all weights except for massive weights are set to zero, it results in a relatively minor performance drop, even though a much larger number of weights are set to zero. This implies that during the pre-training process, learning is dominantly focused on massive weights. Building on this observation, we propose a simple plug-and-play method called MacDrop (massive weights curriculum dropout), to rely less on massive weights during parameter-efficient fine-tuning. This method applies dropout to the pre-trained massive weights, starting with a high dropout probability and gradually decreasing it as fine-tuning progresses. Through various experiments, including zero-shot downstream tasks, long-context tasks, and ablation studies, we demonstrate that \texttt{MacDrop} generally improves performance and strengthens robustness.

1. Introduction

The paper traces massive activations to an early-layer feed-forward network’s intermediate state, defines the weights that produce them, and proposes MacDrop to reduce reliance on those weights during fine-tuning.

1. Introduction

  • Prior work linked attention sinks and massive activations to initial or weak-semantic tokens, with massive activations concentrated in a few fixed feature dimensions and associated with implicit attention bias.The paper uses this fixed-dimension observation to focus its analysis on the bos token.Source: §1 · p. 1
  • In Llama-3-8B, top-5 massive weights are 0.0005% of total parameters; zeroing them disrupts generation, while zeroing other weights preserves some capability despite affecting many more weights.The comparison concerns weights in Wgate and Wup at a specific layer.Source: §1 · p. 2
  • The bos token has massive activations in consistent feature dimensions; these originate in an early-layer intermediate state and propagate through skip connections into the hidden state.This points to the early-layer feed-forward network as important to the phenomenon.Source: §1 · p. 2
  • Top-k massive weights are the Wup and Wgate rows in layer l that produce the top-k magnitudes of the intermediate state.The definition uses the consistent feature dimensions where massive activations appear.Source: §1 · p. 2
  • MacDrop applies progressively decreasing dropout to pre-trained massive weights during parameter-efficient fine-tuning; the paper reports generally improved performance and robustness on zero-shot and long-context tasks.The method starts with a high dropout rate and reduces it as fine-tuning progresses.Source: §1 · p. 2

2. Massive Weights

The section revisits massive activations and traces them to early-layer FFN intermediate states, then defines the corresponding weights and tests their importance through opposing attacks. The attacks reveal cross-family differences, while zeroing massive weights is more damaging than retaining only those weights.

2.1. Prerequisite: Massive Activations

  • Prior work describes massive activations in a few fixed feature dimensions at starting or delimiter tokens; the section relates them to attention sinks and uses a starting-position BOS token for continued analysis.In Mistral-7B, inserting BOS at the start triggers massive activations there and removes them from the first delimiter token.Source: §2.1 · p. 3, §2.2 · p. 3

2.2. Further Analysis on Massive Acitvations

  • Massive activations first appear in the layer-2 FFN intermediate state, then pass directly into the next hidden state and propagate through residual connections.The analysis decomposes the FFN intermediate state to show its connection to Wup and Wgate.Source: §2.2 · pp. 3–4, Figure 3 · p. 4

2.3. Massive Weights

  • Top-k massive weights are rows of Wup and Wgate corresponding to the top-k intermediate-state magnitudes in one layer, and form only a tiny fraction of total parameters.For Llama-3-8B, top-5 massive weights account for approximately 0.0005% of total parameters.Source: §2.3 · p. 4
  • Top-k zeroing is a much stronger attack than top-k retaining, despite retaining zeroing several thousand times more weights; the findings indicate massive weights are predominantly learned during pre-training.The attacks affect Wup and Wgate in the layer containing massive weights.Source: Figure 4 · p. 5, §2.3 · p. 5
  • Llama models show performance drops with zeroing as few as three weights; retaining one row nearly restores smaller models, while larger models require 30, and Mixtral declines considerably when 50 are zeroed.Mistral is also significantly disrupted when k is 5; Mixtral’s targeted expert is expert 4.Source: §2.3 · p. 5, Figure 5 · p. 5
  • Phi-3 and Gemma-2 differ from the Llama-family pattern: Phi-3 robustness varies with model size, while Gemma-2 remains resilient even when Wup and Wgate are eliminated.The authors caution that models insensitive to massive phenomena may not perform effectively with algorithms based on them.Source: §2.3 · p. 6

3. Massive Weights Curriculum Dropout

MacDrop applies curriculum dropout to pre-trained massive weights during parameter-efficient fine-tuning, reducing reliance on them early and progressively restoring their use.

3. Massive Weights Curriculum Dropout

  • MacDrop applies a gradually decreasing dropout probability to pre-trained massive weights during parameter-efficient fine-tuning, without modifying trainable adapter parameters.The dropout is applied to frozen weights and can operate alongside adapter training.Source: §3 · p. 6
  • Early dropout is intended to reduce reliance on massive weights predominantly learned during pre-training, while the curriculum restores their use as fine-tuning ends.The fine-tuned model ultimately uses the undamaged pre-trained model.Source: §3 · p. 6
  • MacDrop identifies massive indices before fine-tuning, masks the corresponding pre-trained weights during adapter training, and progressively enables their unmasked use.The authors report approximately 0.35 seconds of additional computation per step for Llama-3-8B with LoRA on 8xA100 GPUs.Source: §3 · p. 6

4. Experiments

Experiments report gains from MacDrop on zero-shot downstream and long-context tasks, and ablations examine dropout scope, probability, and curriculum schedules. The ablations favor applying dropout only to massive weights and gradually reducing it toward the end of fine-tuning.

4.1. Zero-shot Downstream Task

  • MacDrop consistently improves zero-shot task performance across both evaluated models and both PEFT methods, especially on ARC-Easy and ARC-Challenge.The authors do not claim MacDrop is superior to LoRA or DoRA; they report that each performs better with MacDrop than without it.Source: §4.1 · p. 7

4.2. Long Context Task

  • MacDrop increases performance on LongBench tasks requiring long-context understanding across five subcategories and 16 English datasets.The evaluation sets the models’ maximum length to 7,500.Source: §4.2 · p. 7

4.3. Ablation Study

  • The ablation study evaluates MacDrop variations using Llama-3-8B, with the variations serving as baselines.The experiments vary dropout scope, dropout probability, and curriculum method.Source: §4.3 · p. 8, §4.3.1 · p. 8

4.3.1. DROPOUT SCOPE AND PROBABILITY

  • Only dropout applied solely to massive weights surpasses the 76.0 no-dropout mean zero-shot accuracy; sustained strong dropout on those weights degrades performance.The authors note that the pre-trained weights should be safeguarded in the final fine-tuning stages.Source: §4.3.1 · p. 8, Figure 6 · p. 8

4.3.2. CURRICULUM METHODS AND INITIAL DROPOUT PROBABILITY

  • Step-based schedules generally outperform epoch-based schedules, but low initial dropout can yield no gain, and Step or Epoch(before) with p0=1.0 can cause a substantial drop.The authors recommend Step or exponential schedules with a smaller α, beginning at a moderately high p0; a smaller α improves the exponential method’s performance.Source: §4.3.2 · p. 8, Table 5 · p. 8

5. Conclusion

The paper identifies massive weights in an early-layer FFN as predominantly pre-trained and influential to LLM performance, and proposes MacDrop for parameter-efficient fine-tuning.

5. Conclusion

  • The paper identifies influential, predominantly pre-trained massive weights in an early-layer FFN and proposes MacDrop, which drops those weights rather than adapter parameters during parameter-efficient fine-tuning.The authors present MacDrop as a plug-and-play method.Source: §5 · p. 8

Impact Statement

The authors report that access to model weights can allow massive-weight zeroing to disrupt models, and that MacDrop improves robustness to such attacks.

Impact Statement

  • The authors state that massive weights do not inherently present a risk when model weights remain undisclosed.They emphasize developers’ responsibility to understand these mechanisms and ensure robustness.Source: Impact Statement · p. 9

A. Implementation Details

The appendix specifies implementation resources and summarizes where massive weights occur, then compares BOS-token activations and attention behavior across model families. The BOS token’s effects vary by family and position.

A. Implementation Details

  • The experiments use 8xA100-80GB GPUs, Llama-Factory for parameter-efficient fine-tuning, and separate tools for perplexity, zero-shot accuracy, and generation evaluation.Table 6 gives LoRA and DoRA configurations.Source: A. Implementation Details · p. 13, Table 6 · p. 13

B. Position of Massive Weights

C.1. Llama-2 family

  • Llama-2-7B has massive activations at the starting or first delimiter token, whereas Llama-2-13B has them only at the starting token.In both models, inserting BOS produces similar activation tendencies.Source: §C.1 · p. 14, Figure 8 · p. 14, Figure 9 · p. 14

C. bos Token Analysis for Various LLMs

  • Llama-3 and Mistral/Mixtral show BOS-triggered massive activations across positions, while Phi-3 resembles Llama-2-13B with activations concentrated at the starting token.Llama-3 BOS tokens also exhibit attention sinks; Mixtral activations are approximately ten times higher than Mistral’s.Source: §C.1 · p. 14, §C.2 · p. 15, Figure 10 · p. 15

C.2. Llama-3 family

C.3. Mistral and Mixtral family

  • Mistral lacks starting-position massive activations and an attention sink, while Mixtral shows the same behavior with activations about ten times higher.With BOS in the middle or end, both models show massive activations at BOS and the first delimiter token.Source: §C.3 · p. 16, Figure 12 · p. 16, Figure 13 · p. 16

C.4. Phi-3 family

  • Phi-3-mini and Phi-3-medium resemble Llama-2-13B in activation placement, but show weaker attention to the first token, largely redistributed to recent tokens.Their massive activations appear only at the starting token, with similar responses when BOS is inserted.Source: §C.4 · p. 17, Figure 14 · p. 17, Figure 15 · p. 17

C.5. Gemma-2 family

Gemma-2’s activation magnitudes and attention differ from other model families. BOS placement affects smaller and larger Gemma-2 models differently.

C.5. Gemma-2 family

  • The Gemma-2 family’s activation magnitudes and attention scores differ significantly from other families, whether BOS is absent or present.Source: §C.5 · pp. 18–19
  • Gemma-2-2B and Gemma-2-9B lack noticeably large activations or attention sinks without BOS; with BOS, massive activations appear in certain dimensions regardless of its position.Even then, these models allocate more attention to recent tokens, especially their own tokens, than models with attention sinks.Source: §C.5 · p. 18, Figure 16 · p. 18, Figure 17 · p. 18
  • Gemma-2-27B has large feature-dimension values across tokens without BOS, but BOS at the starting position triggers activations and attention sinks exceeding other tokens’ magnitudes by more than tenfold.BOS in the middle or ending position does not trigger massive activations.Source: §C.5 · p. 19, Figure 18 · p. 19

D. Futher Analysis for Various LLMs

Comparing pre-trained and instruction-tuned models reveals nearly identical graph shapes, consistent with massive weights forming during pre-training. Most families show early-layer amplification and sustained hidden-state activations, unlike Gemma-2.

D. Futher Analysis for Various LLMs

D.1. Llama-2 family

D.2. Llama-3 family

D.3. Mistral and Mixtral family

D.4. Phi-3 family

D.5. Gemma-2 family

  • Gemma-2 instead shows hidden-state values that continuously rise and then fall, with a considerably lower intermediate-state explosion than other families.These characteristics suggest different internal dynamics that may influence its performance and stability.Source: D. Futher Analysis for Various LLMs · p. 20, Figure 23 · p. 23

E. Attention Sinks

Early-layer attention sinks occur in several model families after the massive-weights layer, while Gemma-2 instead focuses attention locally. The reported attention averages heads.

E. Attention Sinks

  • Across layers 1–8, attention sinks appear after the massive-weights layer in Llama-2, Mistral, Mixtral, Llama-3, and Phi-3, but not in Gemma-2.Llama-2, Mistral, and Mixtral sink on the initial and first delimiter tokens; Llama-3 and Phi-3 sink only on the initial token.Source: E. Attention Sinks · p. 24, Figure 24 · p. 24
  • Before sinks occur in five examined models, significant attention commonly concentrates on non-semantic delimiter tokens; Gemma-2 instead assigns attention primarily to local tokens.Source: E. Attention Sinks · p. 24
  • Figure 24 reports attention averaged across heads, so individual heads may not fully exhibit the described sinking behavior.Source: E. Attention Sinks · p. 24, Figure 24 · p. 24

F. Zero-shot Downstream Task across Various LLMs

MacDrop is not effective for LLMs that are not sensitive to massive weights, including Phi-3-medium and the Gemma-2 family. The paper reports feasibility for models exhibiting massive phenomena and uses its broad model evaluation to reveal this limitation.

F. Zero-shot Downstream Task across Various LLMs

  • MacDrop is not effective for LLMs that are not sensitive to massive weights, including Phi-3-medium and the Gemma-2 family.Table 8 reports zero-shot downstream-task performance across different LLMs.Source: F. Zero-shot Downstream Task across… · p. 25, Table 8 · p. 25
  • MacDrop demonstrates sufficient feasibility for LLMs exhibiting massive phenomena.The authors frame evaluation across a wide variety of recent LLMs as a way to reveal algorithmic limitations.Source: F. Zero-shot Downstream Task across… · p. 25
  • The authors note that previous and ongoing studies of massive phenomena excluded the Gemma-2 family from their algorithms and experimental results.Their broad evaluation includes models that reveal limitations of methods based on massive phenomena.Source: F. Zero-shot Downstream Task across… · p. 25

G. Generation Tasks

On Spec-Bench generation tasks, MacDrop shows limited performance improvements. The related-work discussion distinguishes this paper's weight-space analysis from prior work on attention sinks, weight importance, and super weights.

G. Generation Tasks

  • MacDrop shows limited performance improvements on Spec-Bench generation tasks.The benchmark covers six subtasks, with 80 instances per task, and uses Prometheus-2-7B direct assessment on a 5-point Likert scale.Source: G. Generation Tasks · p. 26, Table 9 · p. 26
  • The generation evaluation covers multi-turn conversation, translation, summarization, question answering, mathematical reasoning, and retrieval-augmented generation.Table 10 provides an example from translation; the supplied passages do not state specific judgments or table values.Source: G. Generation Tasks · p. 26, Table 10 · p. 26

H. Related work

  • Prior work uses attention sinks in key-value-cache compression and token eviction or merging, while this paper explores the phenomenon in weight space.The related-work discussion also notes prior research on global attention tokens and massive activations associated with sink tokens.Source: H. Related work · p. 27
  • Unlike Wanda and AWQ, which identify important weights in every linear layer, this paper defines activation-aware massive weights within a single layer using the bos token.The massive weights are included among weights selected through Wanda or AWQ.Source: H. Related work · p. 27
  • Yu et al. (2024a) define a super weight using Wdown connections to super activations in both the intermediate and hidden states; this paper focuses on the intermediate state from Wup's output perspective.The paper defines its massive weights as corresponding rows of Wup.Source: H. Related work · p. 27

Related papers

Loading 2410.01866v2…