The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

1 INTRODUCTION

• Massive activations act as implicit attention biases, and removing them can degrade model quality; they also obstruct low-precision training and inference.Prior work reports that outlier dimensions depend on pretraining optimization conditions, and one study trained a 1.4B model on 1T tokens without usual activation outliers using a different optimizer.Source: §1 · p. 1
• Prior accounts provide a gradient-sink magnitude constraint and a regular growth curve, but leave the determinants of magnitude, its stopping point, and carrier identity unresolved.The paper frames these as questions about what sets the magnitude in practice, what stops its growth, and why one coordinate rather than another carries it.Source: §1 · p. 2

Massive activations and attention sinks

• Massive activations act as implicit attention biases, while attention sinks route disproportionate attention to an uninformative early token and may limit representational over-mixing.Prior work also uses sink tokens for streaming inference by retaining them in the KV cache.Source: §2 · p. 2
• Elhage et al. (2023) rule out LayerNorm and finite-precision arithmetic as causes of coordinate-aligned outliers, attributing their alignment instead to Adam’s per-dimension normalizers rather than to the architecture.Source: §2 · p. 2

The gradient-sink account

• A gradient-sink account links large residual magnitude to gradient attenuation, but its one-sided lower bound predicts neither a peak nor a subsequent decline.The account relates attenuation to the whole token’s rms rather than an individual coordinate.Source: §2 · p. 2

Emergence over training

• Gallego-Feliciano et al. (2025) fit a five-parameter law to activation-ratio trajectories, while leaving the mechanism behind its critical turnover point open.Gu et al. (2025) study when attention sinks appear; this paper instead follows the massive activation that carries the sink, including after its establishment.Source: §2 · p. 2

3 BACKGROUND

• The study combines densely checkpointed training runs with analyses of released Pythia checkpoints to examine massive-activation development and regulation.The background introduces the experiments and theory used to interpret them.Source: §3 · p. 3

3.1 ARCHITECTURE AND HIDDEN STATES

• In pre-norm residual blocks, large residual scale has little forward effect but changes backward gradients through the normalization Jacobian’s 1/rms factor.The setup indexes activations by token position and channel in the post-residual hidden state.Source: §3.1 · p. 3

3.2 THE SINK TOKEN

• The sink position is selected per model as the position carrying the largest activations, rather than assumed to be position zero.Under causal masking, the first token is visible to every later position, and attention mass can accumulate there when no position warrants it.Source: §3.2 · p. 3

3.3 THREE MAGNITUDES

• The analysis separates single-channel magnitude, token-level rms, and the max-to-median ratio, which can diverge as the median falls.The rms is an aggregate property of the token, whereas channel magnitude describes one coordinate.Source: §3.3 · p. 3

3.4 WHAT COUNTS AS A MASSIVE ACTIVATION

• Massive activations are identified by the largest-magnitude channel at the sink, while isolation measures the largest consecutive gap in the sorted channel magnitudes.Isolation near 1 indicates a populated continuum; a large gap indicates a small set standing apart from a collapsed bulk.Source: §3.4 · p. 4

3.5 GRADIENT QUANTITIES

• The sink-token gradient is attenuated in proportion to 1/rms(x), a prediction the paper tests with ablations under RMSNorm and LayerNorm.The LayerNorm relation holds up to centring.Source: §3.5 · p. 4
• Writer rows are output-projection rows that write a channel into the residual stream; their normalized loss-driven force G is compared with decay λ.A channel grows when G > λ and erodes otherwise.Source: §3.5 · p. 4
• AdamW preconditioning can amplify small persistent gradients, so raw maintaining forces below λ do not explain how large channels withstand decay.The paper therefore evaluates maintaining force after second-moment normalization.Source: §3.5 · p. 4

4.1 MASSIVE ACTIVATIONS EMERGE AND FOLLOW A GLOBAL LAW

• The dominant-layer max/median ratio follows a five-parameter law, while sizeable channels show a rise-and-decline pattern in raw magnitude.The ratio can keep rising as MA magnitude falls because the median decreases faster.Source: §4.1 · p. 5

4.2 WEIGHT DECAY CAUSES THE TURNOVER

• −11% with weight decay at constant learning rate versus −18% under a decaying schedule; without decay, global activation scale keeps rising.The MA’s peak location shifts approximately log-linearly with decay coefficient, and peak magnitude scales approximately as λ^-1/2.Source: §4.2 · p. 5, Figure 1 · p. 5

4.3 THE GRADIENT FLOOR IS ON THE TOKEN’S COLLECTIVE RMS

• Across channel ablations, ∥gx∥·rms(x) stays approximately constant, supporting gradient attenuation governed by the sink token’s collective rms.The product is 0.94 when the MA is ablated, within 0.89–0.95 for other large-channel ablations.Source: §4.3 · p. 5

4.4 APPARENT MA-SPECIFIC REGULATION RESOLVES TO MAGNITUDE

• Matched-rms perturbations produce similar pullback despite 11× different direction distortion, and the MA’s LayerNorm gain is not uniquely suppressed.The perturbations remove 73% and 71% of rms excess, respectively; a small random channel shows approximately 20% pullback.Source: §4.4 · pp. 5–6

4.5 THE MASSIVE ACTIVATION IS A STOCHASTIC ROLE, NOT A FIXED DIMENSION

• Sink-carrying coordinates vary across seeds but stabilize early within runs, consistent with stochastic symmetry-breaking followed by input-robust identity.Pythia-410m’s eventual top three reach ranks 1–3 by approximately 4k steps; the 51M model’s argmax locks by approximately 5k steps.Source: §4.5 · p. 6, Figure 3 · p. 7

4.6 SCALE AND TRAINING MATURITY SET WHETHER THE MA STANDS APART

• Magnitude profiles in the less-trained models remain continua, while fully trained Pythia models show a separated cohort; isolation grows even as separation settles earlier.The comparisons supporting magnitude-matched controls cover low-isolation training horizons, not the separated regime.Source: §4.6 · p. 7, Figure 4 · p. 7

4.7 BIRTH AND DEATH: COMPETITIVE CONSOLIDATION OF THE SINK

• From step 16k to 143k in Pythia-410m, the top three grow from 0.55 to 0.99 of sink-token squared norm as mid-rank channels erode; joint ablation reduces sink attention by 0.08.A single surviving-channel ablation reduces attention by only 0.004–0.015, supporting a redundant collective rather than conserved redistribution.Source: §4.7 · p. 8
• AdamW preconditioning lifts the large-channel cohort above the decay threshold, but overlapping winner and collapser forces leave survivor selection unresolved.Raw maintaining forces are approximately 10^-4, and tested mechanisms do not explain why some cohort members survive.Source: §4.7 · p. 8, Figure 5 · p. 8

5 DISCUSSION AND CONCLUSION

• Weight decay drives global activation-scale turnover, while the massive activation’s rise and peak are consistent with AdamW-preconditioned growth balancing decay.Sink-carrying channels vary across seeds but stabilize early; continued training erodes surrounding channels and concentrates the sink onto a small, redundant cohort. The decline reflects reduced activation per unit writer-weight norm, and the measured maintaining force does not distinguish eventual survivors.Source: §5 · p. 9
• Gradient attenuation follows the sink token’s collective RMS rather than any privileged coordinate, separating channel identity from aggregate magnitude.Magnitude-matched controls resolve apparent massive-activation-specific regulation into magnitude effects.Source: §5 · p. 9
• Because early channel winners are stochastic, suppressing individual channels may leave the role intact; aggregate-scale targeting is better motivated.The sink can reconstitute on different coordinates, as the seed experiments confirm.Source: §5 · p. 9
• The conclusions are bounded by experimental scale and training regime, and intra-cohort survivor selection remains unresolved.Identity and life-cycle analyses span 70M–6.9B-parameter Pythia models, while perturb-and-restore controls remain at approximately 51M; magnitude-matched comparisons require a populated tail.Source: §5 · p. 9

A THEORY: A BALANCE CONDITION, AND WHERE THE DECLINE LIVES

• The balance is calibrated by the λ-sweep because writer rows are only partially scale-invariant, unlike the units in prior AdamW analysis (Kosson et al. (2024)).That prior connection motivates expecting a balance between update-driven growth and decay, rather than transferring its result directly.Source: A THEORY: A BALANCE CONDITION, AND… · p. 12
• At the peak, measured preconditioned maintaining force balances decay (G = λ∥w*∥), and peak magnitude scales as λ^-1/2.The balance models m as proportional to writer-weight norm and uses the preconditioned update; the measured peak scaling fixes the effective drive exponent at q = −1, not an exponent for the loss landscape.Source: A THEORY: A BALANCE CONDITION, AND… · p. 12
• At λ = 0.1, writer-weight norm rises from 10.8 to 11.5 during activation decline while activation per norm falls from 4.2 to 3.2.Only at strong decay do the weights themselves contract; what causes the ratio to fall is not established.Source: A THEORY: A BALANCE CONDITION, AND… · p. 12
• The balance law covers rise and peak only; the decline is multi-channel competition, not a consequence of q = −1.The measured q = −1 comes from the massive activation’s peak scaling, not a universal cross-channel law.Source: A THEORY: A BALANCE CONDITION, AND… · p. 13
• Raw maintaining gradients are one to two orders below decay at every large channel, whereas AdamW’s second-moment normalization sustains the cohort.The regulation is visible under displacement or after preconditioning, not in raw gradient magnitude.Source: A THEORY: A BALANCE CONDITION, AND… · p. 13

• The five-parameter law uses amplitude A, decay rate κ, time scaling τ, time offset t0, and asymptotic baseline K.Source: B EMERGENCE-LAW FIT · p. 13
• The max/median ratio reflects changes in both the dominant channel’s magnitude and the median, rather than the dominant channel’s magnitude alone.Source: B EMERGENCE-LAW FIT · p. 13
• R2 = 0.999 over 40k steps in the 51M model; at layer 5, the ratio reaches approximately 400 while MA magnitude peaks near step 20k, then declines.The ratio rises monotonically because the median falls faster than the MA.Source: Figure 7 · p. 13, B EMERGENCE-LAW FIT · p. 13

• 98/98 channels at layer 2 and 382/398 at layer 5 rise and then decline in raw magnitude.Source: C GLOBAL CHANNEL-MAGNITUDE DYNAMICS · p. 13, Figure 8 · p. 14
• The rise-and-decline pattern is shared across sizeable channels, while the five-parameter ratio law also reflects the falling median.Source: C GLOBAL CHANNEL-MAGNITUDE DYNAMICS · pp. 13–14
• Peak location tends to increase with channel amplitude, with small channels peaking earlier and large channels later, but the per-channel scatter is wide.Channels peaking beyond the 40k-step window are absent, biasing the high-amplitude end downward; interpret this as a tendency, not a clean law.Source: Figure 8 · p. 14, C GLOBAL CHANNEL-MAGNITUDE DYNAMICS · p. 14

• The massive activation’s gain falls from approximately 1.0 early in training to 0.69 by the end while its magnitude rises and persists.Source: D LAYERNORM GAIN SUPPRESSION · p. 14
• The top eight channels have γ ≈0.65–0.84 against a bulk median of approximately 1.25; the MA at γ = 0.69 lies among equal-magnitude peers.At the layer where the MA is largest, gain decreases with magnitude across the whole channel population.Source: D LAYERNORM GAIN SUPPRESSION · p. 14

E.1 MODELS AND TRAINING

• Three decoder-only models (51M, 203M, and 380M parameters) were trained from scratch on Dolma, with LayerNorm primary and a matched RMSNorm variant.The runs use AdamW with λ = 0.1; released Pythia checkpoints are used for inference-only analyses, not training or fine-tuning.Source: §E.1 · pp. 14–15

E.2 PROBES

• Fixed validation inputs are probed across checkpoints for sink-token RMS, ranked channel magnitudes, argmax identity and layer, and leading-channel writer-row norms.Source: §E.2 · p. 15

E.3 PERTURBATION PROTOCOL

• Forks freeze learning rate and batch order, scale target-channel writer rows at every layer by α = 0, 1, or 2, and resume for 200–300 steps.Recovery is assessed against the control’s absolute trajectory as well as its gap, using fraction of gap closed and half-recovery time.Source: §E.3 · p. 15

Controlling for magnitude

• To separate channel identity from magnitude, the analyses fit trends below the massive activation, match perturbations by sink-token rms, and pool groups with at least 10 channels.Scoring leading channels out of sample avoids runner-up contamination; equal-rms perturbations differ in how much they rotate the token direction.Source: §E.4 · p. 16

Weight-decay test

• Near-peak continuations compare weight decay enabled versus disabled at constant learning rate, while a decay sweep measures peak timing across λ values.The sweep tests λ ∈ {0.05, 0.1, 0.2, 0.4}; peak locations are fit to tpeak ≈ a − b ln λ.Source: §E.4 · p. 16

Gradient floor test

• The gradient-floor test checks whether ∥gx∥·rms(x) stays constant across ablations; a near-one ratio supports attenuation governed by collective token scale.The comparison is between the observed gradient norm and the value predicted by the unablated control under the 1/rms law.Source: §E.4 · pp. 16–17

Apparent MA-specific regulation

• Floor and ceiling tests perturb the massive activation or a writer-row-norm-matched random channel, then measure recovery with learning rate frozen at its checkpoint value.Forks share batch order and run for 200–300 steps; direction-controlled kicks induce equal rms displacement but differ 11× in direction distortion.Source: §E.4 · p. 17

Scale and isolation profiles

• Scale profiles use max/median separation and the largest adjacent ratio in the top-15 magnitude profile as an isolation measure.The matched-magnitude control requires comparable reference channels, whose availability depends on the training regime.Source: §E.4 · pp. 16–17

Competitive consolidation

• Top-three winners rise then decline mildly, while mid-rank channels erode on staggered schedules and isolation builds continuously over training.The trajectory follows Pythia-410m from 16k to 143k steps.Source: Figure 10 · p. 18, §E.4 · p. 18
• Joint ablation of the top three sink channels produces an approximately 3× superadditive attention drop, while random-channel ablation has negligible effect.Individual surviving-channel ablations have small effects.Source: Figure 11 · p. 18, §E.4 · p. 18