Measuring Maximum Activations in Open Large Language Models

Luxuan Chen; Han Tian; Xinran Chen; Rui Kong; Fang Wang; Jiamin Chen; Yuchen Li; Jiashu Zhao; Shuaiqiang Wang; Haoyi Xiong; Linghe Kong; Dawei Yin

Source-linked AI summary

Measuring Maximum Activations in Open Large Language Models

Luxuan Chen, Han Tian, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Linghe Kong, Dawei Yin

arXiv:2605.15572v2May 15, 2026cs.CL

TL;DR

Activation-range evidence for post-LLaMA open LLMs is limited, despite its relevance to low-bit inference and scaling. The paper measures global and layerwise maxima across 27 checkpoints from eight families under a unified protocol, finding that peak magnitude varies by family, architecture, and training stage rather than parameter count alone. It concludes that maximum activation magnitude should be measured and reported before low-bit deployment.

Problem
Recent open LLMs vary across architectures and training stages, but their activation magnitudes have not been systematically mapped under a unified protocol despite their relevance to quantization and scaling.
Method
The paper measures global and layerwise maximum activations across 27 checkpoints from eight open families using a shared 5,000-sample corpus, family-specific tokenization, and identical hooks.
Results
Maximum activation magnitude is not predicted by parameter count alone, with cross-family, cross-generation, and cross-architecture comparisons breaking monotonic trends by orders of magnitude.
Takeaways & Limitations
Maximum activation magnitude is a family- and architecture-dependent model property, so deployment analysis should inspect residual-stream peaks and report activation statistics with open-weight releases.
Takeaways & Limitations
The paper does not claim a causal mechanism for observed differences and uses the absolute peak rather than a normalization-stack-specific re-analysis.

Abstract

from arXiv · show

The dynamic range of activations is a first-order constraint for low-bit quantization, activation scaling, and stable LLM inference. Prior work characterized outlier features and massive activations on pre-2024 LLaMA-style models, and the downstream activation-quantization stack inherits that picture without revisiting it for the post-LLaMA open-model boom. We ask the deployment-oriented question: how large can activations get in modern open LLMs, and how does this magnitude vary across families, generations, and training stages? Under a unified pipeline (5,000-sample multi-domain corpus, family-specific tokenization, identical hooks across embeddings, hidden states, attention, MLP/MoE, SwiGLU gates, and final norm), we measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants. We find that (i) global maxima span over nearly four orders of magnitude at comparable parameter counts, with Qwen3.5 and MoE checkpoints in the 10^2 to 10^3 range and Gemma3-27B-it reaching ~7 x 10^5; (ii) cross-family and cross-generation comparisons break simple monotonic scaling; and (iii) MoE checkpoints exhibit 14.0-23.4x lower peaks than matched-scale dense counterparts, while the residual stream carries the global maximum in 22/24 checkpoints. A lightweight INT-8 sanity check shows that measured maxima co-vary with low-bit reconstruction error via activation-scale selection. We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size - and should be measured and reported alongside any open-weight release before low-bit deployment. The code is publicly available at https://github.com/clx1415926/Max_act_llm.

1 Introduction

The paper revisits activation-range measurement for recent open LLMs, where parameter count alone may not predict peak magnitude. It proposes a unified survey spanning families, architectures, modalities, generations, and training stages.

Motivation: Activation maxima constrain the numerical range that inference systems, activation quantizers, and scaling rules must accommodate.Rare extreme values can dominate per-tensor scales and increase reconstruction error for ordinary activations.
Research gap: Recent open models differ from earlier LLaMA-style systems in normalization, gated MLPs, MoE routing, multimodal adaptation, intermediate training, and instruction tuning.
Approach: The study measures M = max |a| across 27 post-LLaMA checkpoints from eight families under a unified protocol.The analysis includes 24 main checkpoints plus three Qwen2.5-Instruct checkpoints.
Approach: The survey covers global and layerwise peaks across embeddings, hidden states, attention, MLP or MoE outputs, SwiGLU gates, and final normalization.
Contributions: The paper reports family-, architecture-, and training-stage comparisons with deployment implications, while explicitly making no causal-mechanism claim.

2 Measurement Protocol

The measurement protocol applies the same offline evaluation pipeline across model families while preserving family-specific tokenization. It records activation statistics from multiple components and tests peak stability through repeated subsampling.

Pipeline: Figure 1’s pipeline consists of Data Preparation, Activation Measurement, and Analysis using a shared offline evaluation protocol.The same text is tokenized per family, evaluated by forward inference, and converted into per-model statistics.
Corpus: 5,000 samples form a multi-domain corpus spanning mathematics or science, code, English web, knowledge text, Chinese, low-resource languages, and mixed web data.
Corpus: 93% of samples use 4096-token sequences, while 1%, 1%, 2%, and 3% use 256, 512, 1024, and 2048 tokens.The resulting corpus averages approximately 3899 tokens and about 19.5M tokens overall.
Controls: Family-specific tokenization keeps the underlying text semantically identical while reducing tokenizer-mismatch bias in activation statistics.
Model coverage: The 24-checkpoint main experiment spans eight families and includes dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.These variants enable comparisons of scale, family, architecture, modality adaptation, training progress, and supervised fine-tuning.
Activation measurement: PyTorch hooks collect embeddings, hidden states, attention outputs, MLP or MoE outputs, gate activations, and final normalization outputs during forward-only inference.
Peak stability: Repeated 1,000- and 2,000-sample subsamples reproduce the 5,000-sample peak order of magnitude, with maximum coefficients of variation of 10.1% and 8.2%.Each subsample size is repeated five times across four representative models.

3 From Binary Massive Activations to Continuous Peaks

The paper replaces a binary massive-activation diagnosis with the continuous global maximum M while retaining local sparsity as a descriptive bridge to prior work. Most checkpoints satisfy the prior criterion, but failures reveal distinct absolute-scale and local-density patterns.

Continuous peak metric: M = max |a| is computed across six hooked component classes and all layers, then used in bar charts and matched-pair ratios.
Binary diagnostic: The Sun criterion requires a same-token coordinate to exceed both an absolute-magnitude threshold and a local-ratio threshold in a hidden state.A checkpoint passes if any hidden layer contains at least one qualifying token-feature coordinate.
Existence: 20 of 24 main-analysis checkpoints pass the Sun criterion, showing that massive activations remain widespread in recent open LLMs.
Failure mechanisms: Qwen2.5-1.5B reaches 7,968 but fails the 1000× local-ratio threshold, indicating large yet relatively dense activations.
Failure mechanisms: Qwen3.5-0.8B, Qwen3.5-9B, and Qwen3.5-35B-A3B fail because their overall activation scale is systematically suppressed.All four failing checkpoints avoid Figure 2’s upper-right region, which requires both thresholds to be met.
Interpretation: The absolute peak remains the primary deployment metric because local ratio alone does not fully characterize activation-range risk.The binary criterion is retained mainly as a descriptive connection to prior work.

4 Where Maximum Activations Form

Maximum activations do not form at a universal depth or through a single trajectory: they either jump and plateau or accumulate gradually, while hidden states carry most global maxima.

4.1 Layerwise intensity distribution: Peak depth varies across architectures and even within families, so the full layerwise trajectory is more informative than peak-layer index alone.Figure 3 maps normalized depth against log-scale layerwise absolute peaks and marks each checkpoint’s peak depth bin.
4.2 Two layerwise patterns: Jump-and-plateau models rise sharply in early or middle layers and remain high across a long subsequent interval.Qwen2.5 and GPT-OSS exemplify this emergence pattern.
4.2 Two layerwise patterns: Gradual-accumulation models increase more smoothly with depth and often peak in later layers.Qwen3.5 and Gemma exemplify the gradual pattern.
4.2 Two layerwise patterns: The two trajectory patterns are associated with family and architecture rather than a monotonic function of parameter scale.The paper presents this as a qualitative distinction in depth-normalized trajectories.
4.2 Two layerwise patterns: The trajectory dichotomy is qualitative rather than a quantitative classifier.The analysis does not assign numerical class labels to the 24 trajectories.
4.3 Carrier components: 22/24 global maxima occur in layerwise hidden states, making the residual stream the dominant carrier of extreme values.GPT-OSS-20B peaks in an MLP output, while Qwen3.5-0.8B peaks at final LayerNorm output.

5 What Controls Peak Magnitude?

Peak magnitude often grows with size within a fixed family, but family and generation can override that trend, producing much larger cross-family differences.

5.1 Within-family scaling: Within-family global maxima usually increase with parameter count when model family and form are fixed.Gemma2 is a local exception: its 9B checkpoint peaks below its 2B checkpoint before the 27B checkpoint rises again.
5.2 Cross-family magnitude: Cross-family variation spans several orders of magnitude and exceeds typical within-family scaling variation.Qwen3.5 remains around hundreds to low thousands, whereas Gemma3-27B-it reaches 696,320.
5.2 Cross-family magnitude: 696,320 is the global maximum reported for Gemma3-27B-it.The value is shown on the log-scale cross-checkpoint comparison.
5.3 Non-monotonic generational evolution: Qwen maximum activations follow an inverted-V across generations, increasing from Qwen2.5 to Qwen3 before decreasing in Qwen3.5.The trend is reported across three matched size groups.
5.3 Non-monotonic generational evolution: Gemma maximum activations increase sharply from Gemma2 to Gemma3 across both matched size groups.The comparison is presented as a generational trend at similar model sizes.
5.4 Empirical rule of thumb: Family identity and generation are at least as important as parameter count for inference and quantization boundaries.Design and training differences can override within-family monotonicity.

6 Deployment Takeaways and Conclusion

The paper treats maximum activation magnitude as a deployment-relevant model property rather than a size-only statistic. It links the statistic to hidden-state carriers and low-bit reconstruction behavior.

Deployment takeaways: Maximum activation magnitude depends on family and architecture, with MoE routing, modality adaptation, instruction tuning, and training stage shifting peaks or their carriers.The conclusion frames these factors as model-property determinants rather than consequences of parameter count alone.
Deployment takeaways: Activation quantization and scaling policies should inspect hidden-state peaks because the residual stream dominates extreme-value locations.The residual stream carries the global maximum in 22/24 main checkpoints.
Deployment takeaways: Larger measured peaks can translate into lower reconstruction SQNR through activation-scale selection in the INT-8 probe.This motivates treating M as a practical model-card statistic.
Conclusion: Within-family scaling often increases M, but cross-family, cross-generation, and cross-architecture comparisons break monotonic trends by orders of magnitude.The conclusion defines M as max |a|.

A Supplementary Experiments Model Details

The supplementary comparison covers generational trends and checkpoint composition, with an important boundary on interpreting Gemma2 versus Gemma3.

Model details: The main analysis contains 24 checkpoints from eight families, and Gemma3 checkpoints are publicly released instruction-tuned models.Therefore, Gemma2/Gemma3 is a family-level public-checkpoint contrast, not a strict base-to-base ablation.
Generational comparison: Figure 7 compares maximum activation magnitudes across generations at similar model sizes for Qwen and Gemma.Qwen shows an increase followed by a decrease, while Gemma increases across both size groups.

C Architectural and Training Factors at Matched Scale

The paper compares maximum activations across architectural and training variations while holding model scale constant or approximately constant.

Controlled comparisons examine MoE versus dense, vision-language versus text-only, base versus instruction-tuned, and different Ling-mini training stages.

C.1 MoE vs. Dense Models

Matched-scale comparisons show substantial activation differences across MoE, vision-language, and instruction-tuned variants, with evidence strength varying by comparison.

MoE vs. Dense Models: 23.4× lower: Qwen3-30B-A3B peaks at 1,512 versus 35,328 for dense Qwen3-32B.Qwen3.5-35B-A3B is also 14.0× lower than dense Qwen3.5-27B, at 132 versus the dense counterpart's peak.
MoE vs. Dense Models: The MoE contrast is observational because it uses only n = 2 matched pairs and may include unobserved training-recipe differences.
Vision-language vs. Text-only models: 1.6× lower: Qwen2.5-VL-7B reaches 8,256 versus 13,248 for text-only Qwen2.5-7B.At 32B, Qwen2.5-VL reaches 22,144, 1.4× lower than the text-only counterpart at 30,848.
Base vs. Instruct: 1.4× reduction: Qwen2.5-32B-Instruct decreases from 30,848 to 22,144, while 1.5B remains unchanged at 7,968.The layerwise pattern indicates late-layer compression with stable middle-layer high-peak regions.

C.4 Training-stage evolution

Ling-mini checkpoints show increasing global maximum activations across training stages, while broader comparisons differ in magnitude and evidential strength.

C.4 Training-stage evolution: 1.34× increase: Ling-mini's global maximum rises monotonically from 7,648 at 5T tokens to 10,240 at 20T tokens.Intermediate values are 9,024 and 9,600 as training progresses.
C.4 Training-stage evolution: MoE, vision-language adaptation, SFT, and training stage are associated with measurable activation differences, but their magnitudes and evidence strengths vary.
C.4 Training-stage evolution: Maximum activation magnitude is the largest absolute activation observed across layers and key components under a unified evaluation corpus.The statistic is oriented toward dynamic range and deployment risk rather than testing for persistent abnormal feature dimensions.

D.2 Implications for quantization and deployment

The paper connects maximum activation magnitude to low-bit quantization through activation-scale selection and a lightweight INT-8 sanity check, while situating the measure alongside newer mitigation methods.

D.2 Implications for quantization and deployment: Higher peaks constrain activation-scale selection and can make per-tensor quantization more vulnerable to domination by extreme values.Quantization error also depends on the peak-layer distribution, effective signal magnitude, and clipping strategy.
D.2 Implications for quantization and deployment: The INT-8 sanity check evaluates eight representative models using 128 calibration and 256 evaluation samples at each model's peak hidden layer.It compares per-tensor symmetric max-abs scaling with 99.9% clipping.
D.2 Implications for quantization and deployment: The experiment is a deployment-relevance sanity check rather than a calibrated dose–response curve.The selected checkpoints span low-, medium-, and high-peak dense models plus MoE/dense contrasts.
D.2 Implications for quantization and deployment: Nearly four orders of magnitude: reported maximum activations vary across modern open releases, and mitigation costs remain monotone in the reported dynamic range.The measure therefore remains relevant as a per-checkpoint model-card entry even when quantization mitigations are applied.

D.3 Threats to validity and limitations

The study is observational and its conclusions are bounded by open-checkpoint coverage, extreme-statistic sensitivity, and an incomplete INT-8 evaluation. The authors identify broader data, context, training-stage, and quantization studies as future work.

The observational design identifies empirical relationships with family, architecture, and training stage but does not establish causal training mechanisms.
The study covers open LLMs only, so closed models and models with unreleased training recipes may exhibit different activation dynamics.
Maximum activations may change with longer contexts, rare inputs, or larger evaluation corpora, although repeated subsamples preserve representative peak orders of magnitude.
The evaluation uses English, Chinese, and code domains with sequences up to 4,096 tokens, leaving long-tail languages, reasoning chains, tool-use traces, and 32k–128k contexts untested.
The INT-8 sanity check covers 8 checkpoints, 1 layer each, and per-tensor recipes, so broader quantization comparisons remain necessary.
Public instruction checkpoints conflate supervised fine-tuning with downstream RLHF or DPO stages, preventing finer-grained training-stage decomposition.