Source-linked AI summary
M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization
Weiming Hu, Zihan Zhang, Haoyan Zhang, Chen Zhang, Cong Guo, Yu Feng, Tianchi Hu, Guanglin Li, Guipeng Hu, Junsong Wang, Jingwen Leng
TL;DR
Low-bit MX formats improve efficiency but suffer accuracy loss from coarse shared power-of-two scaling, creating a need for bit-efficient accuracy recovery. M2XFP uses flexible metadata in an algorithm–hardware co-design with online encoding, reducing accuracy loss against MXFP4 and NVFP4 while improving accelerator performance and energy efficiency.
Problem
Existing low-bit MX formats can suffer substantial accuracy degradation because shared power-of-two scaling limits quantization precision, while metadata allocation remains underexplored.
Method
M2XFP allocates metadata asymmetrically between activations and weights, combines online encoding with lightweight hardware, and integrates the design into a systolic array.
Results
M2XFP reduces accuracy loss by 70.6% over MXFP4 and 37.3% over NVFP4, with up to 1.91× performance and 1.75× energy gains.
Takeaways & Limitations
Metadata-driven MX formats can improve low-bit quantization accuracy while retaining bit efficiency and practical accelerator support.
Takeaways & Limitations
For NVFP4, the added metadata raises effective bit-width from 4.5 to 5 bits, motivating further work on more bit-efficient encoding designs.
Abstract
from arXiv · showhide
Existing low-bit Microscaling (MX) formats, such as MXFP4, often suffer from substantial accuracy degradation due to the use of a shared scaling factor with the Power-of-Two format. In this work, we explore strategies that introduce minimal metadata to recover accuracy lost during quantization while maintaining high bit efficiency across a wide range of large language models. We propose a complete algorithm-hardware co-design based on flexible metadata, featuring an online quantization with simple encoding. To support the proposed method efficiently, we implement a lightweight hardware unit and integrate it into the accelerator. Evaluation results demonstrate that our method substantially narrows the accuracy gap, achieving on average a 70.63% reduction in accuracy loss compared to MXFP4 and a 37.30% reduction relative to the latest NVFP4 on LLM benchmarks. Furthermore, our design delivers up to 1.91$\times$ speedup and 1.75$\times$ energy savings over state-of-the-art accelerators. Our code is available at https://github.com/SJTU-ReArch-Group/M2XFP_ASPLOS26.
1 Introduction
LLM scaling has intensified memory and compute demands, making low-bit quantization important, but existing MX formats can lose substantial accuracy at 4-bit precision. M2XFP addresses this gap with metadata allocation and hardware co-design.
- LLM growth creates severe memory and compute demands, motivating low-bit quantization to reduce footprint, bandwidth, and energy while preserving quality.
- MX formats provide fine-grained quantization with shared scaling and hardware support, but accuracy degradation remains severe at 4-bit precision.
- Metadata remains underexplored as a flexible way to encode auxiliary precision or range information without changing the core data path.
- M2XFP systematically explores metadata allocation, treating metadata as extra mantissa or exponent bits to trade precision refinement against range extension.
- Element-level metadata suits dynamic activations, while subgroup-level metadata with scale search suits static weights, motivating a hybrid design.
- M2XFP uses 0.25 bits of metadata per element, reaches near-FP16 accuracy at effective 4.5-bit precision, and adds lightweight hardware support to systolic arrays.
2 Background
Quantization reduces representation cost by mapping values to low-precision grids, while MX extends group-wise quantization through shared power-of-two scales. This improves hardware efficiency but creates a central accuracy bottleneck when block maxima do not align with coarse E8M0 scaling.
- Quantization represents parameters with fewer bits to improve computational and memory efficiency.
- Group-wise quantization partitions weights into fixed-size blocks and quantizes each block independently with its own scaling factor.
- MX assigns k scalar elements a shared 8-bit E8M0 scale, restricting scaling to a hardware-friendly power-of-two representation.
- MX derives its shared scale from the block maximum and supports alternative exponent computation rules intended to reduce clipping or rounding effects.
- MX quantization reduces to shift-and-round operations, while dequantization can be integrated into GEMM for efficient accelerator execution.
- MX variants retain shared scaling, including hierarchical SMX exponents and NVFP scaling that trades greater precision for reduced exponent range.
- At 4-bit precision, a single shared power-of-two scale becomes an accuracy bottleneck because block maxima may fall between exponent bins, producing errors across the block.
3 Motivation
The analysis attributes low-bit MX accuracy loss primarily to coarse shared power-of-two scaling and finds limited benefit from further scaling-factor refinement. It motivates metadata as a flexible, underexplored alternative for addressing quantization error with lower overhead.
- Scaling Factor: Shared E8M0 scaling misaligns with block maxima, producing large rounding errors in low-bit MX quantization.FP16 scaling can map a group maximum tightly to the FP4 maximum, whereas power-of-two steps are coarse.
- Scaling Factor: Preserving each block maximum in FP16 drastically reduces MXFP4 perplexity, nearly matching FP4 and NVFP4.This experiment identifies mishandling of the block maximum as MX quantization’s primary weakness.
- Scaling Factor: Perplexity gains diminish beyond group size 32 as equivalent bit width increases, despite the additional scaling overhead.Reducing group size from 32 to 16 raises equivalent bit width through more scales per tensor, while accuracy improvements quickly plateau.
- Scaling Factor: Scaling-factor designs have largely converged on E8M0 and FP8, leaving limited room for breakthrough improvements along this dimension.The paper therefore examines metadata alongside scaling factors and data types as distinct design dimensions.
- Data Type: Custom data types offer representational flexibility but impose decoder, converter, and runtime overhead that is especially problematic for activation quantization.Supporting multiple formats can inflate area, latency, and energy, while dynamic tensors require low-latency processing.
- Metadata: Existing metadata methods are fragmented and often bit-inefficient, lacking principled allocation strategies for systemic block-maximum errors and activations.Examples include schemes focused on outliers and approaches such as MicroScopiQ with more than 40 bits of structural metadata per block.
4 M2XFP Analysis and Design
M2XFP explores metadata allocation across element and subgroup levels, under fixed and adaptive shared scales, to balance quantization accuracy, bit efficiency, and hardware feasibility. The resulting hybrid assigns subgroup-level refinement to weights and element-level refinement to activations.
- 4 M2XFP Analysis and Design: M2XFP systematically explores metadata allocation strategies and derives a hybrid design tailored to weights and activations.The analysis covers fixed and adaptive shared scales, accuracy–bit-efficiency trade-offs, and hardware-friendly quantization and encoding.
- 4.1 Framework for Design Space Exploration: Metadata is organized by precision-versus-range enhancement and element-versus-subgroup application within contiguous subgroups.A group of size k is divided into N contiguous subgroups, enabling localized metadata allocation.
- 4.1 Framework for Design Space Exploration: Elem-EM, Elem-EE, Sg-EM, and Sg-EE respectively add mantissa or exponent information at element or subgroup level.These four strategies restrict metadata to hardware-feasible mantissa or exponent augmentation.
- 4.2 Pareto-Optimal Analysis of Metadata Strategies: 4.5–4.75 EBW is the key fixed-scale region, where Elem-EM achieves the lowest MSE across all models.Top-1 and top-2 allocations are nearly identical, while Sg-EM is competitive only at EBW ≤4.375 and Sg-EE provides negligible or marginal gains.
- 4.2 Pareto-Optimal Analysis of Metadata Strategies: 4.6 bits meets the 1% accuracy-loss threshold for LLaMA2-7B, Falcon-7B, and Mistral-7B with Elem-EM, while Sg-EM requires ≥5.25 bits.Elem-EM requires 4.75 bits for LLaMA3-8B, whereas Sg-EE fails to meet the threshold entirely.
- 4.2 Pareto-Optimal Analysis of Metadata Strategies: Adaptive shared-scale optimization makes Sg-EM-2bit outperform Elem-EM in the 4.5–4.75 EBW region, motivating a hybrid strategy.The hybrid applies Sg-EM-2bit to weights and Elem-EM-top1 to activations; near-FP16 accuracy is achieved at an effective precision of 4.5 bits.
- 4.4 Quantization and Encoding Process: The encoding strategy addresses the risk that replacing a top-1 FP4 value with FP6 can change which element is maximal.This issue motivates an improved encoding strategy for preserving the intended top-1 selection.
5 Architecture
M2XFP extends a systolic-array accelerator with lightweight decode, processing, and quantization units that support metadata refinement while preserving efficient FP4 computation. Its streaming quantization engine enables deterministic online activation encoding without stalling the array.
- Architecture: M2XFP adds a Top-1 Decode Unit, augmented PE tile, and Quantization Engine to a conventional systolic-array pipeline.The units support subgroup metadata, extra mantissa processing, and online activation quantization.
- Data Layout: M2XFP stores packed 4-bit elements, an 8-bit shared scale, and 8-bit metadata in separate contiguous streams to preserve alignment and simplify access.The fixed-length organization supports parallel indexing of elements, scales, and metadata.
- Architecture: The decode unit maps FP4 values through a 16-entry lookup table and comparator tree, selecting one deterministic top-1 element per subgroup.Ties are resolved by choosing the lowest index, after which the selected index and metadata are forwarded to the PE array.
- Processing Element: The PE tile combines a baseline FP4 × FP4 MAC with auxiliary correction logic for extended mantissas and subgroup-scale refinement.The correction term is computed separately and accumulated with the baseline result, while subgroup multipliers 1.0, 1.25, 1.5, and 1.75 use shift-and-add operations.
- Accumulation: Final subgroup partial sums are dequantized into FP32, accumulated across subgroups, and combined with the shared scale using lightweight exponent alignment.For E8M0 scales, dequantization avoids full floating-point multiplication.
- Quantization Engine: The two-stage quantization engine computes group scales and candidates, then performs top-1 selection, bias-clamp encoding, and FP4 packing with 2-bit metadata.This deterministic, streaming-friendly pipeline supports real-time activation quantization without stalling the systolic array.
6 Evaluation
The evaluation compares M2XFP with hardware-supported quantization formats and accelerator baselines across language-model, reasoning, latency, energy, and format-extension experiments. M2XFP generally improves accuracy while retaining low hardware overhead and strong accelerator efficiency.
- Evaluation Setup: The study evaluates LLaMA, OPT, Mistral, Falcon, and DeepSeek-R1-Distill-Qwen models across language-model, commonsense, and reasoning benchmarks.The LLMs span 7B–70B parameters, while reasoning tests include AIME, MATH-500, GSM8K, GPQA-Diamond, and LiveCodeBench.
- LLM Accuracy: M2XFP reduces average accuracy loss on 7B/8B models to 1.58%, a 70.63% improvement over MXFP4 and 0.94% absolute gain over NVFP4 at 4.5 bits.NVFP4 remains better on some individual tasks, but M2XFP has higher average accuracy across benchmarks.
- Perplexity: M2XFP achieves the lowest Wikitext perplexity on all evaluated models except OPT-6.7B, where BlockDialect is better by 0.03.The comparison uses W4A4 quantization with group size 32 and an E8M0 shared scale.
- Reasoning Tasks: M2XFP reduces DeepSeek-R1-Distill-Qwen-1.5B’s average accuracy loss from MXFP4’s 12.12% to 4.59% and scales to 7B reasoning models.The results indicate improved robustness on advanced reasoning workloads.
- Hardware Cost: The decode units and quantization engine contribute only 0.26% area and 0.36% power overhead in the modeled 32 × 32 systolic array.Hardware estimates use Verilog synthesis with a TSMC 28 nm standard-cell library at 500 MHz and CACTI v7 for buffers.
- Performance and Energy: M2XFP delivers average 1.91× speedup and 1.75× energy reduction over MicroScopiQ, a state-of-the-art MX accelerator.The comparison uses identical systolic-array sizes, while some baselines require 8-bit tensors to match accuracy.
- NVFP4 Extension: Adding Elem-EM and Sg-EM metadata to NVFP4 lowers perplexity but increases its effective bit-width from 4.5 to 5 bits.The result motivates further exploration of more bit-efficient metadata encodings for NVFP4.
- Scaling Rules: M2XFP improves accuracy over MXFP4 across all five shared-scale computation rules, with ceil/RTNE achieving the lowest MXFP4 perplexity.RTN1 performs worse because it does not address block-maximum error and introduces additional nondeterminism.
7 Conclusion
M2XFP uses metadata-augmented microscaling to reduce 4-bit quantization accuracy loss while retaining efficient hardware execution. Its algorithm–hardware co-design combines bit-efficient metadata, dedicated encoding support, and systolic-array integration.
- Conclusion: M2XFP reduces accuracy loss by 70.6% over MXFP4 and 37.3% over NVFP4 while achieving up to 1.91× performance and 1.75× energy gains.These results support the practicality of metadata-driven MX formats for future LLM accelerators.