Source-linked AI summary
ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training
Janghwan Lee, Sihwa Lee, Jinseok Kim, Yongjik Kim, Jieun Lim, Jinwook Oh, Jungwook Choi
TL;DR
Large reasoning models face high inference costs and growing KV-cache footprints, while W4A4KV4 quantization can substantially degrade reasoning accuracy beyond what existing PTQ and QAT recover. ReQAT targets low-entropy token failures with trace-aligned training, selective entropy minimization, and quantization-friendly initialization. It matches or surpasses BF16 fine-tuning accuracy at the same training budget and reports up to 3.9× throughput speedup on production hardware.
Problem
Large reasoning models incur high serving costs from repeated weight loading, computation, and growing KV caches, while W4A4KV4 quantization can cause substantial accuracy degradation that existing PTQ and QAT do not close to BF16.
Source: §1 · pp. 1–2Method
Results
ReQAT matches or surpasses BF16 fine-tuning accuracy at the same training budget and achieves up to 3.9× throughput speedup on production hardware.
Source: §8 · p. 9Takeaways & Limitations
The reported results demonstrate that FP4 quantization can deliver high efficiency and strong reasoning performance together.
Source: §8 · p. 9TAQ depends on SFT supervision quality and may provide limited gains when reasoning traces are weak or noisy.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Large Reasoning Models (LRMs) achieve strong problem-solving through long chain-of-thought, but their deployment is constrained by the high cost of full-precision inference and growing KV cache footprints. Microscaled FP4 formats enable efficient FP4 deployment; however, fully quantizing weights, activations, and KV caches (W4A4KV4) causes severe reasoning degradation that existing PTQ and QAT fail to recover. We identify that FP4 failures concentrate on low-entropy tokens--precise symbolic commitments such as digits and operators--where quantization noise inflates sampling errors that cascade through reasoning traces. Based on this insight, we propose ReQAT, a reasoning-centric FP4 training framework with three components: (i) Trace-Aligned QAT (TAQ), which revisits identical reasoning traces to focus updates on critical low-entropy decisions; (ii) Selective Entropy Minimization (SEM), which reinforces confidence at low-entropy positions; and (iii) Q-FIT, a quantization-friendly initialization that jointly calibrates RoPE-consistent KV cache transformations to stabilize QAT. Under the same training budget, ReQAT not only recovers but surpasses BF16 fine-tuning accuracy, while delivering up to 3.9x throughput speedup on NVIDIA DGX Spark and 3.1x on B200.
1. Introduction
Large reasoning models face high serving costs from repeated weight loading, computation, and growing KV caches, while W4A4KV4 quantization can substantially degrade reasoning accuracy. ReQAT targets low-entropy token failures and reports accuracy gains over BF16 fine-tuning alongside faster serving.
1. Introduction
- W4A4KV4 quantization substantially degrades reasoning accuracy, and existing QAT and QAD improve over PTQ but remain below BF16 full fine-tuning.Quantized KV caches add layer-dependent distortions associated with channel-wise outliers and RoPE.Source: §1 · p. 2
- ReQAT targets low-entropy token failures with TAQ, which revisits identical traces, SEM, which reinforces confidence at those positions, and Q-FIT, which calibrates KV-cache transformations.The framework focuses on symbolic commitments such as digits and operators.Source: §1 · p. 2
- The evaluation covers multiple LRMs and AIME-120, MATH-500, and GSM8K benchmarks.Source: §1 · p. 2
- 65.94% AIME accuracy under NVFP4 W4A4KV4 exceeds the 56.83% BF16 baseline and 65.46% BF16 full FT; throughput speedups reach 3.1× on B200 and 3.9× on DGX Spark.The accuracy comparison is shown for R1-Qwen-14B; throughput is relative to BF16 baselines.Source: Figure 1 · p. 1, §1 · p. 2
2. Background
The background describes FP4 deployment settings and prior quantization approaches, then introduces token-level entropy as a way to characterize reasoning predictions.
2.1. Quantization for Efficient Inference
- MXFP4 and NVFP4 use E2M1 layouts with different scaling granularities; NVFP4 W4A4KV4 additionally quantizes the KV cache.The paper considers MXFP4 W4A16, MXFP4 W4A4, and NVFP4 W4A4KV4 inference regimes.Source: §2.1 · p. 2
- Prior work applies AWQ to W4A16 and QuaRot or FlatQuant to W4A4KV4, while reported reasoning-accuracy degradation motivates QAT.Source: §2.1 · p. 2
- KV-cache transformations use paired channel scaling before RoPE or channel shifting after RoPE to reduce quantization error while preserving attention computation; their inference parameters are fixed.The cited approaches are Lin et al. (2025) and Zhang et al. (2025b).Source: §2.1 · p. 2
2.2. Large Reasoning Models
- FP4 reasoning models have used PTQ, QAD, mixed precision, and QAT with simulated quantization noise, but prior work reports instability or limited effectiveness under W4A4KV4.The cited prior-work reports include Lv et al. (2026).Source: §2.2 · p. 3
- Token-level entropy is the Shannon entropy of the next-token distribution; low-entropy tokens typically represent confident predictions, while high-entropy tokens mark reasoning transitions.Entropy has been used to analyze the semantic structure of reasoning traces.Source: §2.2 · p. 3
3. Empirical Analysis
The analysis finds that low-entropy predictions—often symbolic tokens—are especially vulnerable to quantization-related perturbations. The experiments connect this sensitivity to increased probability of sampling alternatives even when the top prediction remains stable.
3. Empirical Analysis
- Approximately 20% of reasoning tokens have high entropy and mark transition points, while most tokens have low entropy.Source: §3 · p. 3
3.1. Semantic Grounding of Token Entropy
- Low-entropy tokens are dominated by digits and operators, while high-entropy tokens contain more discourse markers; FP4 flattens both types’ predictive distributions.The analysis groups over 1.5M generated tokens by average predictive entropy.Source: §3.1 · p. 3
3.2. Quantization Sensitivity via Token Entropy
- Routing low-entropy predictions to BF16 recovers a large fraction of quantization-lost accuracy, whereas routing high-entropy predictions yields relatively small improvement.The separate logit-noise experiment likewise finds a large AIME accuracy drop from perturbing low-entropy predictions and a much smaller effect from high-entropy perturbations.Source: §3.2 · p. 4
3.3. Low-Entropy Tokens Fail under FP4 Sampling
- Under FP4, low-entropy tokens retain nearly the same top-1 prediction but gain substantially more probability mass on non-top-1 alternatives, a sampling risk that prior PTQ and QAT methods may not fully address.Source: §3.3 · p. 4
4. Methods
ReQAT addresses FP4 reasoning errors concentrated at low-entropy predictions with three components: trace-aligned QAT, selective entropy minimization, and KV-cache calibration. Together, they form a pipeline from BF16 fine-tuning through Q-FIT initialization and QAT to an FP4 model.
4. Methods
- ReQAT combines trace-aligned QAT, selective entropy minimization, and Q-FIT to address quantization-induced errors in low-entropy predictions.The pipeline calibrates RoPE parameters, applies Q-FIT, then performs trace-aligned QAT with SEM loss on a TAQ subset.Source: §4 · pp. 4–5, Figure 3 · p. 5
4.1. TAQ: Trace-Aligned Quantization-Aware Training
- Reusing Stage-1 fine-tuning traces for Stage-2 QAT targets the same low-entropy token decisions with quantization-aware updates.A 70M-token TAQ subset can reach accuracy comparable to BF16 fine-tuning, with performance maintained at matched total fine-tuning budgets.Source: §4.1 · p. 5
4.2. SEM: Selective Entropy Minimization
- SEM selectively minimizes entropy at near-deterministic token positions, using soft weights to avoid over-penalizing tokens near the entropy threshold.The threshold τ is set to the 75th percentile of minibatch entropy values; the soft weighting is reported as more effective than a hard mask.Source: §4.2 · pp. 5–6
4.3. Q-FIT: Quantization-Friendly QAT Initialization
- Q-FIT jointly calibrates pre-RoPE channel-wise scaling and post-RoPE shifting before Stage-2 QAT to reduce KV-cache quantization error.It selects transformations by minimizing the distance between BF16 and KV4 attention outputs, adapting scaling or shifting to layer-specific key-cache characteristics.Source: §4.3 · p. 6
5. Experiments
Experiments compare ReQAT with quantization and fine-tuning baselines across FP4 settings, reasoning benchmarks, and two Blackwell platforms. They test its accuracy, throughput, and key design choices.
5.1. Experimental Settings
- Experiments cover MXFP4 W4A16, MXFP4 W4A4, and NVFP4 W4A4KV4, with R1-Qwen-14B and R1-Llama-8B evaluated on AIME-120, GSM8K, and MATH-500.TAQ uses a fixed 70M-token QAT stage, with total fine-tuning budgets matched across methods.Source: §5.1 · pp. 6–7
5.2. Experimental Results
- 65.94% AIME accuracy under NVFP4 W4A4KV4 exceeds both the BF16 baseline (56.83%) and BF16 FT (65.46%) at the same total training budget.ReQAT matches or exceeds the best BF16 FT accuracy across FP4 settings, and accuracy increases as TAQ, Q-FIT, and SEM are added.Source: §5.2 · p. 7, Table 1 · p. 7
- ReQAT achieves the highest AIME accuracy across evaluated settings, outperforming representative PTQ and QAT methods across W4A16 and W4A4KV4.PTQ methods recover some accuracy but remain below BF16 baseline or full fine-tuning; QAT and QAD gains over PTQ remain limited.Source: §5.2 · p. 7, Table 3 · p. 7
5.3. Throughput Evaluation
- NVFP4 reaches up to 3.93× throughput speedup on DGX Spark and 3.13× on B200 over BF16; ReQAT retains up to 3.90× and 3.05×, respectively.Reduced memory enables larger batches at longer outputs, while 4-bit GEMM improves compute efficiency; at 16K output tokens on B200, capacity limits speedup.Source: §5.3 · p. 8, Figure 6 · p. 8
5.4. Ablation Studies
- Two-stage QAT without trace alignment yields at most 1% accuracy gains, while trace-aligned two-stage QAT improves accuracy by about 5%.The comparison uses MXFP4 W4A16 on R1-Qwen-14B under different token budgets.Source: §5.4 · p. 8, Table 4 · p. 8
- Neither pre-RoPE scaling nor post-RoPE shifting alone gives reliable gains across budgets; the full Q-FIT design achieves the strongest accuracy.The ablation supports jointly calibrating key transformations rather than relying on one stabilization method.Source: §5.4 · p. 8, Table 5 · p. 8
- ReQAT maintains stronger accuracy than other FP4 methods on longer responses, achieving the best accuracy in the 24K–32K regime, and matches or slightly exceeds BF16 full fine-tuning on LiveCodeBench.The code-generation evaluation tests generalization from mathematical reasoning traces.Source: §5.4 · p. 8
6. Discussions
TAQ revisits aligned reasoning traces to focus QAT updates on low-entropy token predictions vulnerable to quantization. Its gains depend on the quality of the SFT supervision.
6. Discussions
- Aligned traces consistently increase the low-entropy tokens’ gradient contribution during QAT, while misaligned traces weaken this effect.Stage-2 QAT follows Stage-1 BF16 fine-tuning, which has already learned the overall reasoning structure.Source: §6 · p. 9, Figure 7 · p. 9
- TAQ’s main requirement is consistent supervision across stages, not unusually large or specialized reasoning datasets.The proposed mechanism is gradient reallocation toward quantization-sensitive low-entropy predictions.Source: §6 · pp. 8–9
- TAQ may provide limited gains when its SFT reasoning traces are weak or noisy, because it depends on SFT supervision quality.The authors suggest the mechanism may also apply to knowledge distillation, which they leave for future work.Source: §6 · p. 9
7. Related Works
Prior work has explored FP4 quantization and token entropy in reasoning models, but ReQAT distinguishes itself by targeting low-entropy predictions during QAT. The related work describes substantial reasoning degradation under W4A4KV4 settings and limited recovery from prior QAT approaches.
7. Related Works
- Prior deployments use PTQ or mixed-precision designs, while existing QAT approaches remain limited under W4A4KV4 settings.Reported reasoning degradation under aggressive FP4 quantization motivates further QAT work.Source: §7 · p. 9
- Earlier entropy-based studies use token entropy to analyze reasoning, modulate training, or control exploration, primarily in settings with limited reasoning capability.The cited work also examines quantization noise’s entropy-increasing effect in RL-based reasoning pipelines.Source: §7 · p. 9
- Unlike prior entropy-based studies, ReQAT identifies low-entropy predictions as a primary FP4 failure point and targets them during QAT.Source: §7 · p. 9
8. Conclusion
ReQAT targets low-entropy token failures for W4A4KV4 deployment and, under the same training budget, matches or surpasses BF16 fine-tuning accuracy. The accompanying appendix describes the scaling configurations of MXFP4 and NVFP4.
8. Conclusion
- Under the same training budget, ReQAT matches or surpasses BF16 fine-tuning accuracy and achieves up to 3.9× throughput speedup.The speedup is reported on production hardware.Source: §8 · p. 9
A. Microscaling Formats
- MXFP4 uses 32-element blocks with FP8 E8M0 scales and no explicit global scale; its larger blocks amortize scale overhead.It is commonly used in W4A16 and W4A4 deployments where activations may remain higher precision or the KV cache is not aggressively quantized.Source: A. Microscaling Formats · p. 15, Table 8 · p. 15
- NVFP4 uses 16-element blocks with FP8 E4M3 scales plus a global FP32 scale, trading higher metadata overhead for greater local adaptability.The global scale can stabilize overall magnitude calibration; NVFP4 can support W4A4KV4 settings.Source: A. Microscaling Formats · p. 15, Table 8 · p. 15
B.1. Token Entropy Analysis Settings (Fig. 2)
The appendix describes the entropy-analysis corpus and mixed-precision routing setup, then reports that perturbing low-entropy decisions has a substantially larger accuracy impact than perturbing high-entropy decisions. Its controlled-noise procedure uses a fixed perturbation budget to compare those token groups.
B.1. Token Entropy Analysis Settings (Fig. 2)
- More than 1.5M tokens from 90 AIME 2022–2024 prompts generated by R1-Qwen-14B were used to visualize frequently generated low- and high-entropy tokens.The visualization includes tokens appearing more than 100 times.Source: §B.1 · p. 15
- For BF16 and MXFP4 W4A16 R1-Qwen-14B models, entropy-aware decoding routes low-entropy tokens to full precision and high-entropy tokens to quantization, using τ = 0.6, approximately the top 25% highest-entropy tokens.Source: §B.1 · p. 15, Figure 8 · p. 16
- In one challenging example, BF16 produced the correct answer in 6 out of 8 trials, while the quantized model succeeded once.Across additional examples, letting the quantized model select only high-entropy tokens yielded higher accuracy than letting it select only low-entropy tokens.Source: §B.1 · p. 15, Figure 8 · p. 16
B.2. Controlled Logit-Noise Perturbation (Fig. 2(e))
- The controlled perturbation procedure selects the bottom or top q entropy quantiles, with q = 25% by default, to keep the fraction of perturbed decisions fixed.It injects multiplicative, scale-aware Gaussian noise into selected logits as a proxy for relative logit distortions from low-bit quantization.Source: §B.2 · p. 16
- Perturbing low-entropy decisions consistently causes a sharp drop in AIME-2025 accuracy across model sizes and architectures; high-entropy perturbations have substantially smaller effects and can be benign.Examples suggest low-entropy corruption can introduce symbolic errors that cascade into reasoning failure, while high-entropy changes may preserve the final answer.Source: §B.2 · p. 16
B.3. Qualitative Analysis of Entropy-Selective Logit Noise (Fig. 9)
The case study contrasts robust high-entropy connective tokens with fragile low-entropy symbolic decisions: perturbing the latter can derail reasoning and prevent a correct answer. High-entropy perturbations may change the trace’s wording while leaving its logical structure and answer intact.
B.3. Qualitative Analysis of Entropy-Selective Logit Noise (Fig. 9)
- Figure 9 contrasts low-entropy perturbations that cascade into incorrect answers with high-entropy perturbations that typically preserve the answer.Source: Figure 9 · p. 17
- High-entropy noise can alter wording and transitions while preserving correct arithmetic, symbolic relationships, geometric constraints, and the final answer.The examples include stylistic and lexical variation in connective phrases.Source: §B.3 · pp. 16–17
- Corrupting low-entropy tokens can introduce arithmetic or symbol-selection errors that make later reasoning incoherent.The trace may diverge into confusion, contradiction, or premature termination.Source: §B.3 · p. 17
- In all tested cases, the model does not recover or reach the correct answer after low-entropy anchors are corrupted.Source: §B.3 · p. 17
C.1. Multi-Epoch Training vs. Quantization
The section finds that longer fine-tuning or QAT alone does not close the quantization gap, while Q-FIT improves calibration and training behavior. It also describes the evaluation, training, and throughput settings used for the experiments.
C.1. Multi-Epoch Training vs. Quantization
- 15.42 points is the maximum gap widening reported at k = 4 after increasing fine-tuning epochs before NVFP4 W4A4KV4; extending QAT leaves a drop around 12–13 points.The results motivate emphasizing low-entropy anchors rather than relying on more iterations alone.Source: §C.1 · p. 17, Table 9 · p. 18
C.2. Detailed Algorithm and Design for Q-FIT
- Q-FIT consistently reduces post-RoPE calibration errors across layers and yields lower, more stable training loss than TAQ-only variants.Its layer-wise calibration and training-loss comparisons are shown in Figure 10.Source: §C.2 · p. 18, Figure 10 · p. 19
- For MXFP4, Q-FIT uses block-wise Hadamard transforms to spread activation outliers and achieves training loss nearly identical to leaving the MLP down-projection unquantized.The section says this rotation is not used for NVFP4, whose finer scaling factors mitigate outliers without it.Source: §C.2 · p. 19, Figure 11 · p. 19
D.1. Experimental Details
- The evaluation covers MXFP4 W4A16, MXFP4 W4A4, and NVFP4 W4A4KV4, with AIME-120 accuracy as the primary metric and MATH-500 and GSM8K as complementary benchmarks.It uses R1-Qwen-14B and transfer evaluation on R1-Llama-8B, averages results over eight seeds, and compares against PTQ baselines and FT+PTQ.Source: §D.1 · p. 20, Table 10 · p. 20
D.2. Training Details
- Training uses the Math subset of OpenThought-3; TAQ’s Stage-2 QAT budget is fixed at 70M tokens, and Q-FIT calibration uses 256 Wikitext-2 samples of length 512.For R1-Llama-8B, the stated Q-FIT calibration time is approximately 7 minutes on one H200 GPU.Source: §D.2 · p. 20, Table 11 · p. 21
D.3. Throughput Evaluation Details
- DGX Spark throughput tests use W4A4KV8 because TensorRT-LLM does not support its native NVFP4 KV-cache path, whereas B200 uses W4A4KV4.The benchmarks use the largest feasible batch size under a KV-cache memory-saturated regime.Source: §D.3 · pp. 20–21
D.4. Ablation Studies
The ablations compare SEM weighting and strength, additional TAQ epochs, decoding strategies, and Q-FIT against rotation alternatives. They report stronger benefits on reasoning-intensive tasks than on MMLU, where quantization degradation is smaller.
SEM: binary mask vs. soft weighting
- Soft weighting consistently yields higher AIME accuracy than a hard SEM binary mask, which can suppress flexibility for low-entropy tokens near the entropy boundary.Source: Table 12 · p. 21, §D.4 · p. 21
Multi-epoch QAT vs. SEM
- Even with up to 5 epochs (350M tokens), multi-epoch TAQ yields only marginal or degraded AIME performance, while ReQAT achieves higher accuracy with a single epoch.Source: §D.4 · p. 21, Table 13 · p. 22
ReQAT vs. non-reasoning benchmarks
- ReQAT and fine-tuning provide only marginal improvements over PTQ on MMLU, where PTQ causes substantially less accuracy degradation than on AIME-120.The reported pattern suggests ReQAT’s benefits are most pronounced on reasoning-intensive benchmarks while performance on MMLU remains comparable to PTQ and BF16.Source: §D.4 · p. 21, Table 14 · p. 22
Decoding strategy vs. Quantization
- Greedy decoding reduces PTQ’s accuracy drop relative to stochastic sampling but also lowers absolute accuracy, and decoding alone does not fully recover reasoning performance.Source: §D.4 · pp. 21–22, Table 15 · p. 22
Effect of SEM strength
- Across a broad range of SEM strengths, accuracy improves over the no-SEM baseline; λSEM = 0.1 gives the best average across GSM8K, MATH-500, and AIME-120.Smaller and larger values show only minor variation.Source: Table 16 · p. 22, §D.4 · p. 22
Q-FIT vs. rotation in NVFP4
- Under NVFP4 W4A4KV4, Q-FIT yields consistently higher accuracy than randomized Hadamard transformation.Source: Table 17 · p. 22, §D.4 · p. 22