Source-linked AI summary
Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
Vage Egiazarian, Roberto L. Castro, Denis Kuznedelev, Andrei Panferov, Eldar Kurtic, Shubhra Pandit, Alexandre Marques, Mark Kurtz, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh
TL;DR
The paper asks whether MXFP4 and NVFP4 deliver practical benefits for LLM post-training quantization, despite limited rigorous evidence. It analyzes their format-specific errors and introduces MR-GPTQ with efficient GPU kernels; MR-GPTQ substantially improves MXFP4 and brings it near NVFP4 accuracy while enabling strong speedups.
Problem
The practical accuracy and performance benefits of MXFP4 and NVFP4 for LLM post-training quantization had not been rigorously established.
Method
MR-GPTQ adapts GPTQ to FP4 using block-wise Hadamard rotations and format-specific optimizations, supported by GPU kernels with negligible overhead.
Results
MR-GPTQ brings MXFP4 accuracy within 1–2% of NVFP4, while large models recover up to 98–99% of baseline FP16 accuracy.
Takeaways & Limitations
FP4 is not automatically better than INT4, but format-specialized quantization can improve its accuracy–performance trade-offs.
Takeaways & Limitations
The analysis assumes native weights and activations follow Laplace distributions and Hadamard-rotated tensors follow Normal distributions.
Abstract
from arXiv · showhide
The recent hardware-accelerated microscaling 4-bit floating-point formats such as MXFP4 and NVFP4, supported on NVIDIA and AMD GPUs, promise to revolutionize large language model (LLM) inference. Yet, their practical benefits remain unproven. We present the first comprehensive study of MXFP4 and NVFP4 for post-training quantization, revealing gaps between their promise and real-world performance. Our analysis shows that state-of-the-art methods struggle with FP4, due to two key issues: (1) NVFP4's small group size provably neutralizes traditional outlier mitigation techniques; (2) MXFP4's power-of-two scale quantization severely degrades accuracy due to high induced error. To bridge this gap, we introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the classic GPTQ quantization algorithm that tailors the quantization process to FP4's unique properties, by using block-wise Hadamard transforms and format-specific optimizations. We support our proposal with a set of high-performance GPU kernels that enable the MR-GPTQ format with negligible overhead, by rotation fusion into the weights, and fast online computation of the activations. This leads to speedups vs. FP16 of up to 3.6x layer-wise, and 2.2x end-to-end on NVIDIA B200, and of 6x layer-wise and 4x end-to-end on RTX5090. Our extensive empirical evaluation demonstrates that MR-GPTQ matches or outperforms state-of-the-art accuracy, significantly boosting MXFP4, to the point where it can near the accuracy that of NVFP4. We conclude that, while FP4 is not an automatic upgrade over INT4, format-specialized methods like MR-GPTQ can unlock a new frontier of accuracy-performance trade-offs.
1 INTRODUCTION
This paper studies whether hardware-oriented MXFP4 and NVFP4 formats deliver their promised accuracy and efficiency for LLM post-training quantization. It diagnoses format-specific weaknesses and introduces MR-GPTQ to improve FP4 accuracy while retaining practical GPU performance.
- The paper studies post-training quantization, which compresses existing pretrained models while aiming to preserve most of their accuracy.
- The work provides a first thorough study of NVFP4 and MXFP4 accuracy and performance limitations under state-of-the-art 4-bit weight-and-activation quantization.It examines how the formats interact with real parameter distributions and quantization algorithms.
- Rotations improve MXFP4 accuracy but hurt NVFP4 accuracy when combined with standard round-to-nearest quantization.The analysis considers both heavy-tailed Laplace-like distributions and Normal distributions arising after rotations.
- MR-GPTQ recovers MXFP4 accuracy to within 1–2% of NVFP4 while both formats recover up to 98–99% of FP16 accuracy for large models.The evaluation covers Llama-3 and Qwen-3 models across standard zero-shot tasks.
- The study finds that MXFP4 can cause major accuracy drops of approximately 10% relative, while existing methods do not consistently outperform RTN.GPTQ and MR-GPTQ provide consistently good recovery for NVFP4, whereas MR-GPTQ is particularly effective for MXFP4.
2 BACKGROUND ON MICROSCALING FLOATING-POINT FORMATS
MXFP4 and NVFP4 are hierarchical microscaling FP4 formats that quantize groups with shared compressed scales. They differ mainly in group size and scale representation, creating distinct accuracy and outlier-mitigation trade-offs.
- Microscaling formats represent groups of elements using a shared scale, with the scale itself stored in a compressed floating-point format.This hierarchical design supports efficient hardware implementation.
- MXFP4’s power-of-two scale quantization can introduce artifacts that significantly affect model accuracy.
- NVFP4 uses 16-element groups with E4M3 scales, while MXFP4 uses 32-element groups with E8M0 power-of-two scales.NVFP4 therefore preserves more precise scaling information, whereas MXFP4 simplifies hardware multiplication.
- Rotation-based methods target activation outliers, whose magnitudes can reach roughly 100× the average, but their effects differ across FP4 formats.
- Unlike prior work focused largely on INT quantization, rigorous evaluation of FP4 accuracy had remained absent despite industry claims of superiority.
3 A QUANTIZATION ERROR ANALYSIS OF NVFP4 AND MXFP4
The analysis compares quantization error in NVFP4 and MXFP4 under different distributions, group sizes, rotations, and metrics. It shows that dead-zone effects can dominate at large group sizes, while format-specific scale precision and outlier preservation determine how Hadamard transforms affect each format.
- Analytical MSE Bounds: The analysis uses per-element and top-element MSE, plus relative-error variants, because outlier error and higher-variance groups can influence quantized model accuracy.The top-element metric tracks outlier distortion, while relative metrics normalize errors for real weights and activations with non-unit variance.
- Asymptotic MSE Analysis: For any fixed quantization grid with a nonzero dead zone, per-element MSE approaches the full variance as group size grows for both Laplace and Normal models.As the block maximum diverges, normalized entries fall below the dead-zone threshold and are quantized to zero with high probability.
- Asymptotic MSE Analysis: Hadamard transforms are predicted to hurt native Laplace-distributed tensors at small group sizes but help transformed Normal tensors as group size increases.The analysis predicts a crossover because the Normal-model MSE eventually decreases faster than the Laplace-model MSE.
- Numerical Validation: For MXFP4, coarse E8M0 shared scales make top-value precision depend mainly on the FP4 element format, whereas NVFP4’s finer E4M3 scales preserve more precise outlier representation.Hadamard rotation spreads maximum-element error across the group, improving over pure E2M1 precision but remaining below pure E4M3 precision for heavy-tailed data.
- Discussion: With round-to-nearest quantization, NVFP4 has lower MSE and better outlier preservation than MXFP4, while MXFP4 has lower memory and computational costs and may benefit more from Hadamard normalization.These format-specific trade-offs motivate incorporating the analysis into GPTQ rather than treating FP4 formats as interchangeable.
4 MR-GPTQ: AN FP4-FOCUSED VARIANT OF THE GPTQ ALGORITHM
MR-GPTQ adapts GPTQ to FP4 formats through optimized grids, static activation reordering, and fused block-wise rotations, with QuTLASS kernels supporting efficient execution.
- Standard GPTQ: GPTQ minimizes output reconstruction error by iteratively quantizing weights and applying second-order updates to compensate for quantization errors.This reduces the computational burden of the original Optimal Brain Quantization framework by updating the inverse Hessian once per column.
- Adapting GPTQ to FP4 Formats: NVFP4 benefits from direct GPTQ on its standard grid, while rotated approaches trade higher local MSE for potentially unified treatment of NVFP4 and MXFP4.The analysis identifies distinct strategies because rotations improve some formats but can worsen NVFP4 under standard round-to-nearest quantization.
- 4.1 Adapting GPTQ to FP4 Formats: MR-GPTQ combines MSE-optimized grids, static activation reordering, and fused online Hadamard rotations for FP4 quantization.The method uses format-specific scale optimization and preserves the original microscaling group structure after reordering.
- Ingredient 2: Static Activation Reordering: Static activation reordering avoids the 10-20% end-to-end inference slowdown caused by dynamically reshuffling matrix columns at runtime.Columns are reordered after grid and scale computation, then restored while preserving the original microscaling structure.
- Kernel Support: QuTLASS provides fused online rotation and narrow-precision matrix-multiplication kernels for NVIDIA Blackwell GPUs.The library supports online Hadamard transforms, scale-factor rearrangement, and multiple matrix-multiplication backends.
5 EXPERIMENTAL RESULTS
Experiments compare FP4, INT4, and multiple quantization methods across simulated and real deployments, measuring both accuracy recovery and kernel performance.
- Experiments with Emulated Quantization: NVFP4 provides the best average accuracy, INT4 is second, and MXFP4 is third but benefits significantly from MR-GPTQ.NVFP4 and INT4 have similar average quality, while INT4 exhibits higher variance.
- Experiments with Emulated Quantization: Hadamard transforms improve INT4 and MXFP4 but hurt NVFP4 under round-to-nearest quantization, consistent with NVFP4’s smaller group size.GPTQ and SmoothQuant are consistently but moderately effective across all three formats.
- Real Quantization: Real-kernel accuracy tracks simulation within 0.2-0.3%, while Qwen3 NVFP4 models can exceed 99% average recovery.Standard GPTQ performs best on average for NVFP4, whereas MR-GPTQ is strongest among PTQ methods for MXFP4.
- 4. Kernel and Inference Performance: Up to ≈3.6× layer-wise speedup on B200 and ≈6× on RTX5090 demonstrates that QuTLASS approaches idealized FP4 efficiency in a single LLM layer.The measurements include Hadamards, quantization, and scale computation rather than only idealized matrix multiplication.
- Kernel and Inference Performance: Up to 2.2× end-to-end speedup over BF16 on B200 and nearly 4× on RTX5090 is achieved with MXFP4 in vLLM.The gains remain consistent across batch sizes for Llama-3.3-70B.
6 CONCLUSION
The paper studies practical accuracy and performance gaps in MXFP4 and NVFP4 and introduces MR-GPTQ with QuTLASS to address them.
- 6 CONCLUSION: The study identifies gaps between microscaling FP4 formats’ promise and their performance under state-of-the-art quantization methods.MR-GPTQ adapts GPTQ to these formats, while QuTLASS implements its micro-rotations with negligible overhead.
REPRODUCIBILITY STATEMENT
The paper provides code and pretrained MR-GPTQ models to support reproduction and reuse.
- Reproducibility Statement: Quantization and accuracy-evaluation code is available in the FP-Quant repository, with QuTLASS implementation access provided separately.MR-GPTQ quantized models are hosted in a dedicated Hugging Face collection.
A WEIGHT-ONLY QUANTIZATION RESULTS
Weight-only quantization shows similar performance for INT4 and NVFP4, while MXFP4 suffers a larger accuracy loss.
- MXFP4 suffers a larger accuracy drop than INT4 and NVFP4 under weight-only quantization.The average accuracy drop relative to the original model is approximately 2%.
B REAL QUANTIZATION RESULTS
The real-quantization evaluation covers multiple Llama and Qwen models, using QuTLASS-integrated vLLM scores and detailed recovery results.
- Real quantization is evaluated across four Llama-3 and three Qwen-3 model variants using QuTLASS vLLM integration.Qwen evaluations disable thinking mode because long reasoning chains reduced GSM8k and MMLU-CoT performance.
- The evaluation also reports results for the Qwen-3 8B, 14B, and 32B models.
- Per-model recovery results include real NVFP4 and MXFP4 alongside hypothetical NVINT4 and MXINT4 formats.
C SCALE QUANTIZATION ANALYSIS
Scale quantization introduces format-dependent error: MXFP4’s coarse E8M0 grid harms accuracy, while fitting the grid to data substantially improves MXFP4 performance toward NVFP4.
- E4M3 and E8M0 scales increase weight MSErel by 10% and 40% on average, respectively, whereas larger-mantissa FP8 and INT8 scales remain close to unquantized FP4.Activation results follow a similar pattern except for the heavy-tailed down_proj case.
- The hypothetical NVINT4 format performs close to NVFP4 without normalization, while Hadamard transforms improve NVINT4 but hurt NVFP4 under standard RTN.Micro-rotated NVINT4 is the most accurate analyzed format, whereas MXINT4 performs poorly.
- MR kernels reach about 15,000 tok/sec for large-batch inference with MR-MXFP4, delivering 2.2x the throughput of BF16 and 1.3x that of FP8.The kernels show little or no improvement over BF16 and FP8 for small batches.
- MXFP4’s E8M0 power-of-two scale grid is too coarse for observed weight and activation ranges, causing large scale-quantization error.E8M0 spans an extremely broad range, while actual model values occupy a narrower range.
- Fitting the MXFP exponent grid to tensor data reduces scale error and makes MXFP4† substantially better than vanilla MXFP4, approaching NVFP4 accuracy.MXFP4† uses 4.25 bits per parameter versus 4.5 for NVFP4.
I COMPARISON BETWEEN VARIANTS ON THE PLATINUM BENCHMARK
PlatinumBench comparisons show that MR-GPTQ component choices matter differently for NVFP4 and MXFP4, with NVFP4 achieving higher recovery and MXFP4 benefiting strongly from larger rotations.
- Had16 rotations with MSE scale optimization and ActOrder provide a statistically significant advantage for NVFP4 MR-GPTQ.Other NVFP4 variants appear within benchmark variance.
- NVFP4 reaches 96.6% top recovery versus 92.3% for MXFP4, a gap exceeding 4 points on average.
- MXFP4 shows a very large recovery gap between rotated and unrotated variants, with larger Hadamard rotations helping more than smaller ones.For NVFP4, matching rotation size to group size appears ideal.
- The study reports recovery averages and standard deviations across PlatinumBench, including per-task results and five-seed evaluations.
- Across five standard tasks, rotations slightly improve MXFP4 over identity but slightly degrade NVFP4 with RTN, while transform sizes perform similarly.