INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

1 Introduction

• LLM computational and memory demands make quantization important for efficient deployment, while large, infrequent activation outliers challenge low-precision representations.The introduction notes the hardware industry's move toward low-precision FP formats to accommodate wide dynamic range.Source: §1 · pp. 1–2
• The paper identifies a gap in unified INT-versus-FP comparisons across granularities; prior studies often focus on one format or compare only coarse granularities.Block-wise quantization is described as a standard technique for mitigating outliers.Source: §1 · p. 2
• The study finds a granularity-dependent crossover: FP has an advantage at coarse granularity, while INT becomes competitive as block size shrinks, depending on bit width.The authors compare block-wise formats including MX blocks of 32 elements and NV blocks of 16.Source: §1 · p. 2
• The authors develop a theoretical and statistical QSNR framework to compare INT and FP performance trade-offs and clarify crossover points.Source: §1 · p. 2
• MXINT8 consistently outperforms MXFP8 in direct-cast inference and low-bit training, and symmetric clipping enables nearly lossless MXINT8 training.The clipping method resolves a gradient bias.Source: §1 · p. 2
• Fine-grained INT formats are reported to be more area- and energy-efficient than FP counterparts at matched throughput.Source: §1 · p. 2
• The authors argue that findings challenge the FP-centric hardware trajectory and support prioritizing fine-grained INT for a better accuracy-efficiency balance.Source: §1 · p. 2

2 Preliminaries

• Quantization maps a high-precision tensor to a lower-bit-width representation, and this section introduces INT and FP quantization, fine-grained block-wise schemes, and existing block formats.Source: §2 · p. 2

2.1 Low-Precision Integer Formats

• For b-bit integer quantization, values are scaled, rounded to the nearest integer, and clipped to the representable integer range.The cited table is identified as listing low-bit format ranges and scale factors.Source: §2.1 · p. 2, Table 1 · p. 3

2.2 Low-Precision Floating-Point Formats

• Floating-point representations use sign, exponent, and mantissa fields; ExMy denotes the numbers of exponent and mantissa bits, and quantization maps normalized values to a representable FP value.Source: §2.2 · p. 3

2.3 Quantization Granularity

• Quantization granularity specifies how scale factors apply: per-tensor, per-channel, or block-k, with block quantization identified as a key low-precision accuracy technique.Finer granularity usually improves accuracy but increases compute and memory overhead from additional scale factors.Source: §2.3 · p. 3

2.4 Block-Quantization Formats

• MX uses a shared UE8M0 scale for each 32-element block, while NVFP4 uses 16-element blocks, an E4M3 scale, and a second-level per-tensor scale.The paper introduces matching integer variants MXINT8, MXINT6, MXINT4, and NVINT4 for comparison with FP formats.Source: §2.4 · p. 3
• UE8M0 is an 8-bit unsigned floating-point format with eight exponent bits and zero mantissa bits.Source: §2.4 · p. 3

3 Quantization Recipe

• Figure 2 examines how clipping range affects final INT8 training loss for a 145M model trained on 20B tokens, with the scale factor held at BF16.The caption says this setup emphasizes the harm of asymmetric representation space during low-bit training.Source: Figure 2 · p. 4

3.1 Quantization Compute Flow

• Six quantization operations cover inputs, weights, gradients, and transposed tensors across the forward and backward passes.Block-wise quantization follows the GEMM reduction dimension, so paired operations use different axes.Source: §3.1 · p. 4

3.2 Quantization Operation

• The UE8M0 scale is rounded up rather than down because rounding down introduces extra clipping error.The scale is derived from the largest absolute value in the group sharing that scale.Source: §3.2 · pp. 4–5
• Symmetric integer ranges address the persistent negative gradient bias that asymmetric INT8 values can cause during training, especially with finer-grained quantization.The paper reports that BFloat16 arithmetic can still map a group minimum to −128 even when the scale is AbsMax(X)/127.Source: §3.2 · p. 5
• The section develops a theoretical framework for comparing low-bit INT and FP formats using QSNR analysis and theoretical format comparisons.The analysis is described as covering QSNR derivations and comparisons based on theoretical QSNR.Source: §3.2 · p. 5

4.1 Theoretical QSNR

• The analysis measures signal fidelity with QSNR in dB and assumes i.i.d. Gaussian block entries with blockwise AbsMax scaling.It models scale overhead with ρ; UE8M0 uses ρ ∈ [1, 2), while NV-format E4M3 scaling uses ρ = 1.Source: §4.1 · p. 5
• INT QSNR gains approximately 6.02 dB per added bit, incurs up to 6.02 dB loss from UE8M0 scale overhead, and decreases as crest factor κ grows.Smaller blocks usually reduce κ and improve QSNR.Source: §4.1 · pp. 5–6
• With ample dynamic range, FP QSNR is approximately 13.80 + 6.02M dB; larger κ increases the subnormal share and reduces QSNR.Finer-grained blocks reduce κ and the subnormal probability, improving QSNR.Source: §4.1 · p. 6

4.2 Theoretical Comparisons

• Table 2 summarizes crest-factor statistics by block size in boxplot form, reporting Q1 and Q3 quantiles.Q1 and Q3 correspond to the 25% and 75% quantiles, respectively.Source: Table 2 · p. 7
• MXINT8 exceeds MXFP8 QSNR when κ < 7.55, MXINT6 exceeds MXFP6 when κ < 1.96, MXINT4 exceeds MXFP4 when κ < 2.04, and NVINT4 exceeds NVFP4 when κ < 2.39.The comparison evaluates theoretical QSNR across crest factors; the figure marks the crossover points of INT and FP curves.Source: Figure 3 · p. 6, §4.2 · p. 7

5 FP v.s. INT

• The evaluation measures crest factors and QSNR across six intermediate tensors, then compares direct-cast inference and training under low-bit formats.The tensor analysis collects 10,752 tensors from 224 Llama-3.1-8B linear layers; direct-cast inference quantizes forward GEMMs.Source: §5 · p. 7, §5.1 · p. 7, §5.2 · p. 8

5.1 Tensor-wise Analysis

• For MX block size 32, Q3 is 2.96: below the MXINT8/MXFP8 crossover of 7.55 but above the MXINT6/MXFP6 and MXINT4/MXFP4 crossovers of 1.96 and 2.04.After rotation, Q3 falls to 2.39, preserving those format comparisons; NV-format Q3 is 2.39, equal to the NVINT4/NVFP4 crossover, then falls to 2.11 after rotation, favoring NVINT4.Source: §5.1 · pp. 7–8, Figure 4 · p. 8

5.2 Direct-Cast Inference

• With random Hadamard rotation, MXINT8 and NVINT4 outperform their FP counterparts on all 12 inference models, while MXINT4 loses all 12.Without rotation, only MXINT8 wins all 12; MXINT6, MXINT4, and NVINT4 perform worse than their FP counterparts.Source: §5.2 · p. 9

5.3 Training

• MXINT8 training curves nearly overlap BF16 and have approximately 0.001 lower loss than MXFP8; MXINT8 also achieves nearly BF16-level average accuracy across six tasks.Figure 5 compares BF16, MXFP8, and MXINT8 training on Llama-1B with 100B tokens.Source: §5.3 · pp. 9–10, Figure 5 · p. 9

• MXINT8 and NVINT4 reduce energy by 37% and 38%, respectively, versus MXFP8 and NVFP4; mixed MXINT8+NVINT4 reduces area by about 34% versus MXFP8+NVFP4.The mixed-format area reduction is attributed mainly to simpler circuit reuse in the INT pipeline.Source: §6 · p. 10

7 Conclusion

• Fine-grained MXINT8 consistently outperforms MXFP8 in both accuracy and hardware efficiency, whereas MXFP4 and NVFP4 generally hold the 4-bit accuracy advantage.NVINT4 can surpass NVFP4 when combined with random Hadamard rotation.Source: §7 · p. 10
• The paper calls for academia and industry to re-evaluate and prioritize fine-grained INT formats through algorithm-hardware co-design.Source: §7 · p. 10

A Related Work

• Related work largely focuses on single formats or lacks direct INT–FP comparisons; the paper also reviews hardware support for MXFP formats on NVIDIA Blackwell.The outline identifies sections on related work, QSNR theorem proofs, hardware-cost estimation, and supplementary model and ablation details.Source: A Related Work · p. 14, Outlines · p. 14

B.1 Common assumptions and notation

• The analysis treats each block vector as having i.i.d. Gaussian entries and denotes its block RMS by σ.Source: §B.1 · p. 14
• For MX, blockwise UE8M0 scaling is used, with s′ ≥ s to avoid upper clipping; BFloat16 or E4M3 scale factors use ρ = 1.Source: §B.1 · p. 14
• For FP with subnormals, Qref is Qmax, the largest finite normal magnitude; expectations are over data and quantization randomness unless stated otherwise.For E4M3, the passage gives Qmax = 448 as an example.Source: §B.1 · p. 14

B.2 Theorem 1 (INT quantization)

• The INT quantizer uses a symmetric uniform range [−Q, Q], with Q = 2^(b−1) − 1 and effective step Δ := s′.The stated examples are Q ∈ {127, 31, 7} for b ∈ {8, 6, 4}.Source: §B.2 · p. 15
• QSNR gains approximately 6.02 dB per added bit, but power-of-two scale overhead can cost up to 6.02 dB and crest-factor penalties typically rise with block size.The analysis uses Q ≈ 2^(b−1) and overhead ρ ∈ [1, 2).Source: §B.2 · p. 15
• With an E4M3 scale, ρ ≈ 1 and the block maximum maps with near-zero error, accounting for one near-error-free element per block.This scale-factor extension is described for NVINT4.Source: §B.2 · p. 15

B.3 Theorem 2 (FP quantization)

• In the ample dynamic-range regime, the simplified FP QSNR law is independent of block granularity and the distribution of X.For normal-region values, expected squared error is approximated as proportional to their signal energy, with coefficient α_M.Source: §B.3 · pp. 16–17
• With E4M3 scaling, the refined FP QSNR approximation excludes the block maximum’s energy, tightening the prediction when scale precision is sufficient.E4M3 scales closely track the ideal scale, so the block maximum maps with negligible error.Source: §B.3 · p. 17

C Hardware Cost Modeling

• The hardware model estimates combinational MAC, dequantizer, and FP32-accumulator costs, excluding quantizer, sequential, placement, routing, and interconnect costs.It models FP multipliers as typically more area- and energy-efficient than INT multipliers, while FP adders are more expensive than INT adders; the MAC uses lane multipliers and an adder tree.Source: C Hardware Cost Modeling · pp. 18–19

D.1 Used Models

• Inference evaluation prefers open-source base models over SFT versions when available; training evaluation uses Llama-3-style models with GQA and SwiGLU.Tables 8 and 9 list the inference model IDs and training architectures and hyperparameters.Source: §D.1 · p. 19, Table 8 · p. 20, Table 9 · p. 20

D.2 Necessity of Symmetric Integer Representation

• Asymmetric INT8 ranges consistently worsen training loss; with BFloat16 scales, block-32 quantization can be worse than block-256 quantization.The ablation reports 8-bit training loss for a 145M model trained on 20B tokens.Source: §D.2 · p. 20, Table 10 · p. 21
• Only block minima can map to −128 under asymmetric quantization; UE8M0 scales cause milder degeneration than BFloat16 because they are no smaller and fewer high-precision values map to Qmin.Smaller blocks create more opportunities for individual minima to map to Qmin.Source: §D.2 · p. 20
• BFloat16 scaling maps 16.82% of values to −128 despite a scale intended to map them to 127, motivating symmetric clipping for quantization correctness and stability.The analysis attributes the overflow to low-precision scaling; the effect is smaller with float16.Source: §D.2 · p. 21, Table 11 · p. 21

D.3 Detailed Results

• Tables 12 and 13 report average-token KL divergence, multiplied by 10^6, across low-bit formats in direct-cast inference for Qwen3 and Llama-3 models.The supplied passages identify the tables’ scope but do not include their cell values.Source: §D.3 · p. 21, Table 12 · p. 22, Table 13 · p. 22
• Tables 14 and 15 report WikiText2 perplexity across low-bit formats for Qwen3 and Llama-3 models, with results consistent with KL divergence in most cases.The supplied passages do not include the table cell values.Source: §D.3 · p. 21, Table 14 · p. 23, Table 15 · p. 23