Source-linked AI summary
Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs
Tanzila Rahman, Mehran Taghian Jazi, Yunke Peng, Zhuang Ma, Anandharaju Durai Raju, Yao Wang, Xing Huang, Hei Yi Mak, Shadan Golestan, Hoang Le, Yonghan Dong, Wei Guo, Yaoyuan Wang
TL;DR
Ultra-low-bit quantization promises more efficient MLLMs but suffers substantial FP4 degradation, especially from activation quantization and outliers. The paper systematically studies these effects and proposes RFQ, which reconstructs activation residuals through an auxiliary low-bit pathway. Across Wan2.2 and Qwen3-VL, RFQ recovers accuracy toward BF16 baselines while retaining ultra-low-precision computation.
Problem
Ultra-low-bit MLLM quantization faces substantial degradation because activation distributions are dynamic, heavy-tailed, and difficult to represent under extreme bit constraints.
Method
RFQ decomposes activations into a primary FP4 representation and an explicitly quantized residual reconstructed through a secondary low-bit pathway.
Results
RFQ consistently recovers MXFP4 and HiF4 performance toward BF16 baselines across Wan2.2 generation and Qwen3-VL reasoning experiments.
Takeaways & Limitations
Activation quantization is the dominant FP4 bottleneck, and residual-based activation reconstruction supports robust ultra-low-bit MLLM computation.
Abstract
from arXiv · showhide
Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models (MLLMs). Recent hardware support for low-precision formats, ranging from MXFP8 to ultra-low-bit formats such as MXFP4 and HiF4, has accelerated research into efficient MLLM training and deployment. In this work, we present a systematic study of these quantization schemes in representative MLLMs that span both video generation and reasoning tasks. Our analysis shows that MXFP8 achieves near-lossless performance, whereas aggressive 4-bit quantization leads to significant degradation. Through extensive ablations, we identify activation quantization as the primary source of this performance loss, contributing substantially more than weight quantization. Motivated by this observation, we propose Residual Fallback Quantization (RFQ), a lightweight activation reconstruction framework that supplements the primary ulta-low-bit activation representation with an auxiliary quantized residual pathway. By explicitly modeling and compensating for quantization errors, RFQ improves activation fidelity while preserving the efficiency advantages of ultra-low-bit computation. RFQ requires no architectural modifications and incurs negligible computational overhead. Extensive experiments on Wan2.2 and Qwen3-VL demonstrate that RFQ consistently recovers a substantial portion of the performance lost under the quantization of MXFP4 and HiF4, significantly narrowing the gap to BF16 baselines across both generation and 4 reasoning benchmarks. Our findings establish activation quantization as the dominant bottleneck in ultra-low-bit MLLMs and highlight residual-based activation reconstruction as an effective and practical strategy for robust 4-bit deployment.
1 Introduction
This section motivates ultra-low-bit MLLM quantization, diagnoses activation quantization and outliers as major FP4 failure sources, and introduces RFQ to recover accuracy while retaining FP4 efficiency.
- Motivation: Low-bit quantization reduces MLLM memory and computational costs by representing tensors with reduced-precision formats.The motivation spans weights, activations, gradients, and optimizer states.
- Motivation: FP4’s limited dynamic range and precision introduce numerical instability, degraded convergence, and accuracy loss.
- RFQ: RFQ re-quantizes activation residuals through a secondary low-bit pathway and accumulates their contributions during GEMM.It preserves FP4 computation for most operations while compensating for quantization error.
- Study Scope: The study systematically evaluates MXFP8, MXFP4, and HiF4 across Wan2.2 and Qwen3-VL generation and reasoning architectures.
- Findings: FP4 sensitivity is module-dependent, with visual components generally more susceptible to degradation than language components.Wan2.2 is most affected in WanDiT, while Qwen3-VL shows heightened vision-encoder sensitivity and non-trivial language-backbone degradation.
- Findings: Activation quantization is the primary driver of FP4 degradation, especially for heavy-tailed activation distributions and pronounced outliers.The resulting distortion amplifies downstream errors during training.
- Results: RFQ substantially recovers MXFP4 and HiF4 accuracy toward BF16 baseline performance while preserving ultra-low-precision efficiency benefits.
2 Related Work
Related work covers PTQ and QAT for language and multimodal models, while emphasizing unresolved challenges from dynamic activations, cross-modal statistics, and extreme bit constraints.
- PTQ and QAT: PTQ compresses pretrained models using calibration data without further parameter optimization, supporting deployment with low computational overhead.Representative methods include RTN, GPTQ, AWQ, and SmoothQuant.
- PTQ and QAT: QAT adapts models to low-precision noise during training, but ultra-low precision remains difficult because activation outliers amplify error propagation.
- MLLM Quantization: Multimodal architectures create quantization challenges because activation distributions differ substantially across modalities and components.Directly applying LLM quantization methods can therefore produce uneven degradation across modalities.
- MLLM Quantization: Existing multimodal methods address heterogeneous robustness using layer-wise, modality-aware, token-level, rounding, and reconstruction-based strategies.
- Open Challenge: Stable ultra-low-precision MLLM optimization remains open because activations evolve dynamically and cross-modal interactions intensify quantization limitations.The section motivates activation-level strategies that adapt beyond static calibration.
3 Preliminaries
The section introduces floating-point and uniform quantization, then explains block-based microscaling formats MXFP8, MXFP4, and HiF4 for ultra-low-bit representation.
- 3.1 Standard Floating-Point and Quantization Basics: Floating-point values use sign, exponent, and mantissa components to represent real-valued numbers.
- 3.1 Standard Floating-Point and Quantization Basics: Uniform quantization maps tensor values to discrete low-precision codes using a scaling factor and integer clipping bounds.
- 3.1 Standard Floating-Point and Quantization Basics: 4-bit quantization becomes less reliable because dynamic-range mismatches and activation outliers increase quantization error under fixed uniform scaling.
- 3.2 Microscaling (MX) Block Specifications: Microscaling partitions values into independent blocks, computes a shared exponent-based scale, and quantizes normalized elements using low-bit codes.The described block size is typically B = 32.
- 3.2 Microscaling (MX) Block Specifications: MXFP8 uses E4M3 for higher precision and E5M2 for wider dynamic range, while MXFP4 typically uses an E2M1 configuration with only 4 element-wise bits.
- 3.3 HiF4 (HiFloat4): HiF4 applies hierarchical scaling to 64-element blocks with a global scale and two tiers of binary micro-exponents for localized range refinement.Its hierarchy uses E1_8 and E1_16 corrections over progressively smaller subgroups.
- 3.3 HiF4 (HiFloat4): HiF4 expands intra-block dynamic range to 4.81 binades while retaining ultra-low precision storage, addressing persistent activation outliers in MLLMs.
4 Our Approach
This approach studies ultra-low-bit QAT in MLLMs, diagnoses activation quantization as the main degradation source, and introduces RFQ to reconstruct activation errors through a residual pathway. RFQ targets sensitive activation blocks while retaining low-bit computation and improves performance toward BF16 baselines.
- Mixed-Precision QAT: The study evaluates mixed-precision QAT for Wan2.2 and Qwen3-VL across MXFP8, W4A8, MXFP4, and HiF4 formats.The framework assigns numerical formats to components according to their quantization sensitivity.
- Mixed-Precision QAT: MXFP8 preserves performance close to BF16, while uniformly quantized MXFP4 and HiF4 produce substantially higher training loss.W4A8, with 4-bit weights and 8-bit activations, causes only marginal additional degradation.
- Cross-Modal Sensitivity: Vision components are more sensitive than language components to aggressive MXFP4 and HiF4 quantization in the QAT setting.The study reports this reverse trend relative to prior post-training quantization observations.
- Cross-Modal Sensitivity: Activation quantization dominates degradation because activation tensors have larger, more heterogeneous dynamic ranges and severe outliers, especially in vision layers.These outliers reduce effective precision for most values and cause distortion and underflow of small activations.
- Residual Fallback Quantization: RFQ preserves uniform low-bit arithmetic while re-quantizing activation residuals and accumulating their contributions, substantially recovering MXFP4 and HiF4 accuracy toward BF16.The fallback pathway is restricted to sensitive blocks, preserving FP4 computation for the majority of operations.
- Residual Fallback Quantization: RFQ decomposes each activation block into a primary FP4 representation and a conditional quantized residual that corrects errors in flagged outlier blocks.It computes residuals on the fly and accumulates the correction alongside the base GEMM, without threshold-based value selection.
5 Experimental Analysis
Experiments evaluate ultra-low-bit quantization and RFQ on Wan2.2 video generation and Qwen3-VL multimodal reasoning, using BF16 as the comparison baseline. RFQ improves downstream performance and training behavior under aggressive 4-bit quantization without increasing representation precision.
- 5.1 Experimental Setup: Experiments evaluate Wan2.2-5B and Qwen3-VL-30B under MXFP8, W4A8, MXFP4, and HiF4 quantization-aware fine-tuning.Embeddings and the final language modeling head remain in BF16, while other specified linear, MoE, and attention projection layers are quantized.
- 5.1 Experimental Setup: Wan2.2 video generation is evaluated with VBench, while Qwen3-VL reasoning is evaluated on RealWorldQA, MMStar, MMBench-EN, and SimpleVQA.The evaluations use the same inference protocol for BF16 and quantized models.
- 5.2 Evaluation Performance: RFQ raises Wan2.2 Dynamic Degree from 43.00 to 50.00 under MXFP4.Under HiF4, RFQ also improves Subject Consistency from 95.23 to 95.86 and Aesthetic Quality from 59.44 to 59.54.
- 5.2 Evaluation Performance: Under MXFP4, RFQ improves Qwen3-VL RealWorldQA from 70.98 to 72.16, MMStar from 69.67 to 70.73, and SimpleVQA from 15.16 to 16.44.Under HiF4, RealWorldQA improves from 72.42 to 72.81, exceeding the BF16 baseline.
- 5.2 Evaluation Performance: RFQ consistently improves training dynamics and downstream performance under aggressive FP4 quantization, narrowing the gap to BF16 without increasing representation precision.The reported training-loss comparison is relative to the BF16 baseline.
6 Conclusion
The conclusion finds that visual modules are generally more sensitive than language modules to aggressive quantization, while both contribute to degradation. It presents RFQ as residual-based compensation that recovers accuracy close to BF16 and identifies broader ultra-low-bit multimodal deployment as its supported direction.
- 6 Conclusion: Visual modules are generally more sensitive to aggressive quantization than language modules, although language quantization also causes non-negligible degradation.The conclusion therefore frames recovery as requiring attention to both components.
- 6 Conclusion: RFQ uses an auxiliary uniform quantization pathway to exploit residual quantization errors and compensate for information loss under ultra-low-bit quantization.The approach is described as simple and effective for multimodal LLM SFT.
- 6 Conclusion: RFQ consistently improves training dynamics and downstream performance, recovering accuracy close to the BF16 baseline.The conclusion positions residual-based compensation as promising for efficient deployment under tight memory and computation constraints.
- 6 Conclusion: Future work will explore 2-bit quantization and extend RFQ to broader multimodal architectures and tasks.