Source-linked AI summary
The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
Hengjie Cao, Zhendong Huang, Mengyi Chen, Yifeng Yang, Fang Dong, Anrui Chen, Ruijun Huang, Xin Zhang, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Tun Lu, Fan Yang, Yixuan Chen, Li Shang
TL;DR
FP4 training remains fragile because activation outliers inflate blockwise quantization scales and compress long-tail signals. The paper identifies a coherent rank-one mean bias as a structural source of these outliers and proposes Averis, which separates mean and residual components before FP4 quantization. Across dense and MoE Qwen3 models, Averis narrows loss and downstream gaps with low overhead over vanilla NVFP4.
Problem
FP4 training is fragile because activation outliers inflate blockwise quantization scales and compress long-tail signals.
Method
Averis isolates the coherent mean component from activations and output gradients before FP4 quantization using mean reductions and elementwise subtractions.
Results
Across Qwen3-0.6B Dense and Qwen3-7B-A1.5B MoE, Averis reduces BF16 loss gaps to 1.19% and 0.81%, versus 2.05% and 1.10% for NVIDIA’s Hadamard-based method.
Takeaways & Limitations
Averis provides a practical, hardware-efficient path to stable FP4 LLM training and complements Hadamard transformation.
Takeaways & Limitations
Long-run experiments use FP4 simulation rather than direct end-to-end Blackwell execution, and broader validation remains future work.
Abstract
from arXiv · showhide
FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals. We identify a counterintuitive source of this failure: dominant activation outliers are not merely arbitrary sparse events, but are largely induced by a coherent rank-one mean bias, whose direction aligns with the leading anisotropic spectral component. This mean component strengthens during training, is amplified and reshaped by attention and FFN operators, and increasingly dominates top activation magnitudes. Crucially, this discovery reveals that a seemingly complex outlier-suppression problem admits a truly simple solution: isolate the coherent mean before quantization. We therefore propose Averis, a mean-residual splitting quantization method that separates the mean component using only reductions and elementwise subtractions before FP4 quantization. Across Qwen3 0.6B Dense trained on 100B tokens and Qwen3 7B A1.5B MoE trained on 50B tokens, Averis enables robust W4A4G4 FP4 training, reducing BF16 loss gaps to 1.19%/0.81% versus 2.05%/1.10% for NVIDIA's recently released Hadamard-based outlier-smoothing method, while limiting downstream gaps to 0.89/0.71 points. With only 2.20% end-to-end overhead over vanilla NVFP4, about 30% of NVIDIA's Hadamard-based design, Averis provides a hardware-efficient path to stable low-bit LLM training. Complementary to Hadamard, Averis further reduces the Qwen3-0.6B loss and downstream gaps to 0.94% and 0.73 points when combined. Code is available at: https://anonymous.4open.science/r/averis-504D.
1 Introduction
FP4 training is fragile because activation outliers distort blockwise quantization, and this work traces those outliers to a coherent rank-one mean bias. Averis isolates that mean before quantization, achieving more stable and efficient FP4 training than the compared Hadamard-based approach.
- FP4 training offers memory and compute savings but remains fragile because activation outliers inflate block scales and compress long-tail signals.These effects degrade forward representation fidelity and backward gradient propagation.
- A coherent rank-one mean bias induces dominant activation outliers, aligns with the leading spectral component, strengthens during training, and is amplified by attention and FFN modules.The mean component increasingly controls extreme activation magnitudes as training progresses.
- Approximately 95% median squared mean-share among top-0.1% activation entries in deep layers at late training indicates that extreme activations become mean-dominated.The reported shift is from residual-dominated early training to mean-dominated late training.
- Averis separates activations and output gradients into column-mean and residual components before FP4 quantization using mean reductions and elementwise subtractions.The split removes bias-induced dynamic range from the residual stream and reduces quantization error.
- 1.19% and 0.81% BF16 loss gaps for Qwen3-0.6B Dense and Qwen3-7B-A1.5B MoE compare with 2.05% and 1.10% for NVIDIA’s Hadamard-based method.Averis also limits downstream accuracy gaps to 0.89 and 0.71 points under NVFP4 forward evaluation.
- 2.20% end-to-end overhead over vanilla NVFP4 is about 30% of the Hadamard-based design, while combining Averis with Hadamard further reduces Qwen3-0.6B gaps.The combined method reaches 0.94% loss and 0.73-point downstream gaps.
2 Analysis
Mean bias is a coherent rank-one component aligned with the leading anisotropic direction, strengthens during training, and increasingly drives extreme activation magnitudes. These outliers distort blockwise quantization, while mean–residual separation exposes a simpler structure for analysis.
- Mean Bias Phenomenon: A feature-wise mean creates a shared token-representation offset, with mostly positive token-wise alignment along the mean direction.Non-mean directions show weaker and more mixed alignment.
- Mean Bias Phenomenon: The mean vector concentrates along the leading right singular direction because spectral anisotropy and token-wise coherence reinforce one another.Its cosine similarity with the leading direction approaches 0.99.
- Structural Emergence of Mean Bias: Mean-bias energy generally increases during training while remaining strongly aligned with the top singular direction across layers and stages.Attention and FFN modules amplify and reshape this coherent component within Transformer blocks.
- Structural Emergence of Mean Bias: At late training, FFN amplification dominates, whereas attention contributes more to directional reshaping of the mean component.At the early checkpoint, both attention and FFN increase the normalized mean-bias ratio, with FFN substantially redirecting the direction.
- Mean Bias as the Dominant Source of Activation Outliers: A nonzero column mean exponentially increases high-threshold exceedance probabilities relative to a zero-mean variance-matched residual baseline.After mean removal, residual fluctuations are substantially closer to a Gaussian marginal fit.
3 Method
Averis separates activation and output-gradient means from residuals before FP4 quantization, reducing bias-induced dynamic-range inflation through simple reductions and subtractions.
- Mean–residual splitting: Averis decomposes activations and output gradients into column-mean and residual components, then quantizes each component independently.The decomposition is applied in forward and backward GeMMs.
- Implementation: Averis requires only mean reductions and elementwise subtractions, keeping the method aligned with efficient accelerator execution.This design removes dominant bias-induced dynamic range from the residual stream.
- Forward pass: The forward pass reshapes activations into a matrix, computes the feature-wise mean, subtracts it to form residuals, and quantizes mean, residual, and weights separately.The mean matrix is broadcast across tokens without being materialized.
- Backward pass: Averis applies mean–residual splitting to output gradients in the backward pass before quantized gradient GeMMs.The supplied passages identify output gradients as the backward-pass input to this decomposition.
4 Experiments
Experiments evaluate Averis on dense and MoE Qwen3 models using W4A4G4 FP4 training, comparing loss, downstream performance, and runtime against BF16, vanilla NVFP4, and Hadamard smoothing.
- Training loss: Averis reduces BF16 loss gaps to 1.19% on Qwen3-0.6B and 0.81% on Qwen3-7B-A1.5B.The corresponding NVFP4-Hadamard gaps are 2.05% and 1.10%.
- Downstream performance: Averis narrows average downstream gaps to 0.89 points on Qwen3-0.6B and 0.71 points on Qwen3-7B-A1.5B under NVFP4 forward evaluation.Averis-Hadamard reaches a 0.73-point gap on Qwen3-0.6B.
- Complementarity: Combining Averis with Hadamard reduces the Qwen3-0.6B loss gap from 1.19% to 0.94% and achieves the best downstream average among FP4 variants.The two methods are described as orthogonal because Hadamard further smooths the residual signal.
- Runtime: Averis achieves 4.47× and 4.72× mean-latency speedups over tiled Hadamard on the two reported large activation shapes.The advantage increases with activation size.
- End-to-end overhead: Averis adds 2.01% end-to-end latency on Qwen3-0.6B and 2.20% on Qwen3-7B-A1.5B relative to vanilla NVFP4.These overheads are 29.6% and 28.9% of the corresponding tiled-Hadamard overheads.
5 Related Work
Prior low-bit quantization work addresses outlier dimensions through scale redistribution, compensation, absorption, or orthogonal transformations.
- Outlier mitigation: SmoothQuant redistributes activation and weight scales to reduce outlier concentration.
- Outlier mitigation: GPTQ, AWQ, QLoRA, and SVDQuant use strategies to absorb or compensate for large-magnitude components.
- Orthogonal transformations: QuaRot and HALO employ orthogonal rotations for low-bit quantization.
6 Conclusion
The conclusion identifies coherent mean bias as a structural source of FP4 activation outliers and presents Averis as a low-overhead mean–residual splitting approach. It also notes validation limits involving simulation, model breadth, and gradient-output bias.
- Averis isolates the coherent feature-wise mean before FP4 quantization to reduce bias-induced dynamic-range inflation.The method uses mean reductions and elementwise subtractions for activations and output gradients.
- 2.20% end-to-end overhead over vanilla NVFP4 is reported for Averis on Blackwell GPUs.
- The method is evaluated on Qwen3 dense and MoE models, with reported improvements in BF16 loss and downstream accuracy gaps.
- Long-run training experiments use FP4 simulation rather than direct end-to-end execution on Blackwell GPUs.
- Broader validation across model scales, data distributions, and architectures remains future work, and gradient-output mean bias receives no dedicated design.
A Additional Mean-Bias Visualizations Across Layers and Training Stages
Additional visualizations show that mean bias is visible early, persists across training, and occurs across shallow and deep layers. Its coherence becomes stronger at later checkpoints and is especially pronounced in deep late-stage activations.
- The leading spectral spike, one-sided token alignment, and mean–top-singular-vector alignment are visible in layer 0 at training step 10k.
- The same qualitative structure persists in layer 0 at training step 170k, with stronger mean-component coherence than at the early checkpoint.
- Across the examined layers and checkpoints, mean bias is presented as a robust structural feature rather than a single-layer artifact.
B Empirical Validation of the Diagonal Variance Approximation
Empirical variance measurements support the diagonal spectral approximation used in the tail model. The diagonal estimate closely tracks residual variance, while cross-mode covariance contributes little.
- The diagonal spectral estimate closely matches the empirical residual variance across columns.
- The cross-term contribution has median 0.006 and 95th percentile 0.036.
- These measurements support using the diagonal approximation in the main-text tail model.
C Residual Tail Contraction After Mean Removal
Mean removal visibly contracts the high-magnitude activation tail in representative shallow and deep late-stage examples. The residual distributions become more concentrated near zero and less heavy-tailed in the far-tail regime relevant to blockwise scales.
- Subtracting the feature-wise mean contracts high-magnitude tails in both shallow layer 0 and deep layer 27 at training step 170k.
- After mean removal, residual histograms are more concentrated near zero and exhibit less heavy-tailed behavior.
- The observed contraction provides empirical support for mean removal reducing the outlier population relevant to blockwise quantization scales.
D Output-Gradient Mean Centering
Output gradients show weaker mean-bias structure than activations, yet mean centering still consistently reduces NVFP4 quantization error. The effect is modest for gradients but contracts high-magnitude tails, especially in deep late-stage activations.
- Output gradients have weaker spectral dominance, less one-sided token alignment, and weaker alignment between the mean vector and top right singular vector than activations.
- Subtracting the feature-wise mean contracts high-magnitude tails in both shallow and deep layers, with the strongest effect in deep late-stage activations.
- 13.6% to 13.5%: mean centering slightly reduces relative NVFP4 quantization error for output-gradient matrices.The improvement is modest but directionally consistent with isolating the mean component.
- Mean centering still yields a slight NVFP4 quantization-error reduction despite the weaker mean-bias structure of output gradients.
E Proof of Theorem 1
The proof decomposes each sampled matrix entry into a column mean and centered residual, approximates the residual with a marginal Gaussian model, and derives mean-driven far-tail amplification. Under aligned leading-mode and variance assumptions, the amplification is attributed to a rank-one mean-bias component.
- Spectral decomposition: The proof begins from an SVD-based exact spectral expression for each column mean and identifies the leading-mode contribution.The matrix is analyzed through its singular-value decomposition and column-mean expansion.
- Mean–residual decomposition: Each row-sampled entry is split into a column mean and a centered row-coordinate residual by adding and subtracting the row-average contribution.The resulting fluctuation is explicitly centered.
- Residual model: The residual variance is approximated from diagonal mode-wise contributions while neglecting cross-mode covariance terms.The argument uses a marginal Gaussian approximation for a randomly sampled coordinate and does not require mutual independence of row residuals.
- Tail analysis: Under the marginal Gaussian model, the proof derives a two-sided tail formula and compares it with a variance-only zero-mean baseline.The far-tail steps establish one-sided dominance and then apply tail approximations to quantify amplification.
- Tail amplification: When |m_j|/τ_j ≫1, coherent mean shifts exponentially amplify high-threshold exceedances relative to same-variance zero-mean fluctuations.This conclusion follows from the far-tail exponent and averaging over the row index.
- Leading-mode attribution: Under aligned-spike dominance, the mean-driven amplification is controlled asymptotically by the rank-one contribution of the aligned leading singular mode.The proof identifies this rank-one mean-bias component as the source of the coherent shift.