PaperlayerView exact revision on arXiv

Source-linked AI summary

OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization

Neriah Ben David, Ori Meir, Or Ordentlich

arXiv:2610.08231v1cs.AIcs.ITmath.IT

TL;DR

NVFP4 offers compact storage and tensor-core acceleration, but preserving accuracy at FP4 precision requires careful quantization. OSFP4 jointly optimizes diagonal smoothing and block scales for the selected rounding procedure. It achieves the highest average accuracy among evaluated competitors while retaining approximately 94-97% of vendor NVFP4 prefill throughput on measured workloads.

  • Problem

    NVFP4’s absmax-to-6 scaling can zero entries 24 times smaller than a vector’s maximum, and alternative scale choices may reduce distortion.

    Source: §1 · p. 2

  • Method

    OSFP4 uses a multiplicatively dithered FP4 error approximation to jointly optimize diagonal smoothing and block scales for RTN or SIC, then selects E4M3 weight scales.

    Source: §1 · pp. 2–3

  • Results

    OSFP4 achieves the highest average accuracy among evaluated competitors while retaining approximately 94-97% of vendor NVFP4 prefill throughput on measured workloads.

    Source: §1 · p. 3

  • Takeaways & Limitations

    OSFP4’s reported results apply to NVFP4 quantization settings and measured prefill workloads.

    Source: §1 · p. 3

    The SIC error analysis assumes filtered quantization-noise energy is negligible relative to signal energy under a high-resolution approximation.

    Source: §2.2 · p. 6

Abstract

from arXiv · show

NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration. However, preserving accuracy using NVFP4 requires careful quantization. In this work we develop a novel quantization scheme called Optimized Smoothing and Scaling for NVFP4 (OSFP4). For each linear projection it uses a diagonal smoothing matrix whose entries are optimized to minimize the squared matrix-product quantization error under NVFP4, taking into account the rounding procedure that is used (either round-to-nearest, or GPTQ-style successive interference cancellation). This requires performing joint optimization on the smoothing entries as well as the block scales, which is facilitated by analyzing a multiplicative-dither FP4 quantizer instead of the fixed deterministic one. Experiments show that OSFP4 achieves the highest average accuracy among the evaluated competitors in the corresponding quantization settings, while retaining approximately 94-97\% of vendor NVFP4 prefill throughput on the measured workloads. Our code is available in https://github.com/neriahbd/OSFP4

1 INTRODUCTION

The introduction motivates OSFP4 by NVFP4’s accuracy challenge at FP4 precision and presents rounding-aware joint smoothing and block-scale optimization as its solution. It positions this approach against prior low-bit transformations, scale-selection methods, and quantization-error models.

1 INTRODUCTION

  • NVFP4’s absmax-to-6 scaling can zero entries 24 times smaller than a vector’s maximum, motivating optimization of block scales for unbalanced inputs.NVFP4 uses groups of 16 entries with E4M3 scales; the FP4 constellation’s largest value is 6.Source: §1 · p. 2
  • Unlike INT-oriented smoothing that balances infinity norms, FP4 smoothing must balance entries within 16-value weight and activation groups for E4M3 scale selection.The introduction notes that FP4’s limited dynamic range makes NVFP4 different from higher-range floating-point formats.Source: §1 · pp. 1–2
  • OSFP4 approximates matrix-product quantization error with a multiplicatively dithered FP4 quantizer, enabling joint smoothing and block-scale optimization tailored to RTN or SIC.It retains the optimized smoothing matrix, then selects weight scales over the E4M3 grid; activation scales use absmax-to-6.Source: §1 · pp. 2–3
  • OSFP4 achieves the highest average accuracy among evaluated competitors while retaining approximately 94-97% of vendor NVFP4 prefill throughput on measured workloads.The reported comparison covers the corresponding quantization settings.Source: §1 · p. 3

1.1 RELATED WORK

  • OSFP4 avoids rotations and jointly optimizes cheaper diagonal transformations with block scales; its W-SIC/X-RTN configuration has higher average W4A4 accuracy than MR-GPTQ under the common protocol.Prior work also explores randomized rotations and groupwise rotations with format-specific GPTQ optimization.Source: §1.1 · p. 3
  • OSFP4 combines diagonal smoothing with optimized NVFP4 block scales, while prior methods separately search scales, use activation moments, or model quantization error without this combined rounding-aware objective.Its analysis uses multiplicative randomization to retain finite-range effects and derives distinct objectives for RTN and SIC.Source: §1.1 · pp. 3–4

2 LOSS FUNCTIONS

The loss analysis uses a randomized FP4 quantizer to obtain a smooth error model, then derives matrix-product losses for RTN and SIC quantization. The resulting optimization separates across groups, making the smoothing and scale parameters independently optimizable per group.

2.1 SMOOTHED FP4 QUANTIZATION ERROR

  • The randomized FP4 normalized MSE reaches its minimum across the interval IFP4 = [7/4, 7].The minimum is approximately 0.010723, and the plotted curve shows a plateau over this interval.Source: §2.1 · p. 4

2.2 MATRIX-PRODUCT ERROR UNDER RTN AND SIC

  • The matrix-product error is approximated by the sum of weight-error and activation-error contributions, neglecting their second-order product.The approximation is used to construct the optimization loss for randomized quantization.Source: §2.2 · p. 5
  • The analysis derives distinct loss functions for weight and activation quantization under RTN and SIC, accounting for the rounding procedure used on weights.The losses are expressed as functions of the smoothing matrix and E4M3 scales.Source: §2.2 · pp. 5–7
  • The dependence of each loss on smoothing and scales decouples across groups, yielding independent optimization problems for each group.This separation applies to the diagonal smoothing entries and the corresponding weight and activation group scales.Source: §2.2 · p. 7

3 THE OSFP4 QUANTIZATION ALGORITHM

OSFP4 optimizes smoothing and scales for each linear projection using calibration activations, then refines weight scales for deterministic FP4 quantization. It supports weight-only and weight-plus-activation configurations with RTN or SIC weight rounding.

3 THE OSFP4 QUANTIZATION ALGORITHM

  • The calibration procedure retains activation second moments for weight-only settings, while activation-quantized settings use a random subset of calibration vectors.The subset limits the cost of optimizing the activation-related loss.Source: §3 · p. 7
  • OSFP4 optimizes smoothing and scales for each projection using calibration data, with separate optimization problems for each 16-entry group.The loss is selected for the chosen RTN/SIC and activation-quantization configuration.Source: §3 · p. 7
  • After joint optimization, OSFP4 fixes the optimized smoothing matrix and searches valid E4M3 weight scales for deterministic FP4 quantization.The search refines scales initially optimized using the randomized quantizer.Source: §3 · p. 8
  • For quantized activations, OSFP4 applies a global calibration-based scale and uses absmax-to-6 for each 16-entry group during inference.The global scale supports online quantization before the full activation tensor is computed.Source: §3 · p. 8

4 EXPERIMENTAL RESULTS

On Llama-3.1-8B-Instruct, OSFP4 achieved the highest average accuracy among evaluated methods in the reported groups, while retaining approximately 94–97% of vendor NVFP4 prefill throughput. Ablations show the lowest reported NVFP4 perplexity with optimized smoothing, scale selection, and SIC combined.

Experimental setup

  • The evaluation focuses on Llama-3.1-8B-Instruct, with locally prepared models sharing a checkpoint and calibration set and evaluated through vLLM on an RTX 5090.The authors provide a reproduction package with scripts, configurations, dependency versions, GPU specifications, measurements, and regeneration commands.Source: §4 · p. 8, REPRODUCIBILITY STATEMENT · p. 10

Accuracy under a common evaluation protocol

  • 77.03% for OSFP4 WSIC/X-RTN exceeds 76.34% for FP-Quant GPTQ and 76.18% for MR-GPTQ with absmax activation scaling.The weight-only configuration scores 78.42%, versus 78.00% for H-Scale and 79.22% for BF16; accuracy improvement is present both with and without activation quantization.Source: §4 · pp. 8–9, Table 1 · p. 9

Contribution of smoothing, scale selection, and SIC

  • 7.67 WikiText-2 perplexity for OSFP4 W-SIC is below 7.88 for absmax RTN and 7.77 for SIC with A = I and absmax scales.The full procedure combines optimized A, scales, and SIC; it attains the smallest projection-output MSE across layers and compared schemes.Source: Table 2 · p. 9, §4 · p. 9

Accuracy under a common evaluation protocol

  • Online activation-scale search is complementary to OSFP4, while OSFP4 W-SIC/X-RTN scores better than ScaleSweepMSE+GPTQ even with absmax activation scaling.Online search was evaluated mainly to enable a fair comparison with ScaleSweep.Source: §4 · p. 9

From predicted loss to deterministic quantization

  • For W-RTN/W4A16, randomized mean error agrees with predicted loss within 1%, and deterministic E4M3 scale search reduces mean error to approximately half the prediction.The deterministic search also improves over absmax weight scaling at the same optimized A.Source: §4 · p. 9, Figure 2 · p. 9

Inference efficiency

  • Approximately 94–97% of vendor NVFP4 prefill throughput is retained across measured workloads on an NVIDIA B200.The reported gap is primarily attributed to explicit diagonal smoothing before attention-output and MLP-down projections, where it cannot be absorbed into a preceding LayerNorm.Source: §4 · p. 10

A PROOF OF LEMMA 1

The proof represents smoothed relative error as an integral in logarithmic coordinates and uses a comparison quantizer to establish the stated lower bound and its equality range.

Integral representation

  • The smoothed relative error is expressed as an integral over a unit interval in logarithmic coordinates.The proof obtains the representation by substituting the definition and using a uniform random variable on [0, 1).Source: A PROOF OF LEMMA 1 · pp. 11–12

Proof of Lemma 1

  • The lemma proof uses an infinite-exponent model whose constellation contains the FP4 constellation to establish a lower bound on the error.It also invokes symmetry of the error under sign reversal.Source: A PROOF OF LEMMA 1 · p. 12
  • The lower bound is attained for every x in [7/4, 7], where the comparison quantizer matches FP4 across the relevant interval.The proof connects that interval match to equality in the preceding bound.Source: A PROOF OF LEMMA 1 · p. 12

B IMPLEMENTATION OF THE SCALE OPTIMIZATION

The implementation jointly optimizes smoothing and block scales using a smoothed loss, then separately selects deterministic E4M3 weight scales for deployment. The scale-selection procedure accounts for whether weights use RTN or SIC.

B IMPLEMENTATION OF THE SCALE OPTIMIZATION

  • Continuous optimization jointly fits diagonal smoothing and block scales using a smoothed loss, while deterministic E4M3 weight-scale selection is a separate step.For W4A4, the continuous objective also includes activation scales; weights, calibration activations, and second-moment statistics remain fixed during optimization.Source: B IMPLEMENTATION OF THE SCALE… · pp. 12–13
  • The trainable scale parameters are represented as log-scales and optimized together with Adam for 80 iterations, retaining the final iterate.The procedure seeks a low-loss solution of a nonconvex surrogate and does not impose monotonic descent.Source: B IMPLEMENTATION OF THE SCALE… · pp. 13–14
  • After optimization, scale normalization is applied once, smoothing entries are rounded for deployment, and E4M3 weight scales are selected using the rounded transformation.For W4A4, optimized activation scales guide smoothing but are discarded; deployment activation block scales are computed dynamically using absmax-to-6.Source: B IMPLEMENTATION OF THE SCALE… · p. 14

C DETAILED FINE-GRAINED E4M3 SCALE-SELECTION PROCEDURE

  • The final weight-scale search uses deterministic FP4 error and differs by rounding method: RTN treats block errors independently, while SIC accounts for feedback between coordinates.The SIC procedure chooses scales block by block using prior committed quantization decisions and approximates intra-block feedback for the current block.Source: C DETAILED FINE-GRAINED E4M3… · pp. 15–16

D NATIVE NVFP4 COMPARISON: EXPERIMENTAL SETUP

The native NVFP4 comparison evaluates locally prepared Llama-3.1-8B-Instruct models under a shared checkpoint and calibration protocol. It includes author-provided and locally implemented competitors, OSFP4 scale-selection ablations, and a separately prepared NVIDIA checkpoint as a reference.

D.2 METHODS AND IMPLEMENTATION PROVENANCE

  • The comparison separates W4A4 methods with absmax activation scaling, methods searching activation scales online, and weight-only W4A16 methods.Competitors include author-provided FP-Quant implementations and independently implemented methods; H-Scale W4A4 is the authors’ extension.Source: D NATIVE NVFP4 COMPARISON:… · p. 17, §D.2 · pp. 17–18

D.1 MODEL AND CALIBRATION

  • All locally prepared models use the same BF16 Llama-3.1-8B-Instruct checkpoint and 1024 FineWeb-Edu sequences, with all seven decoder-layer linear projections quantized and no fine-tuning or distillation.NVFP4 uses E2M1 values, E4M3 scales per 16 input channels, and an FP32 tensor scale; W4A16 retains BF16 activations.Source: §D.1 · p. 17

D.2 METHODS AND IMPLEMENTATION PROVENANCE

  • Activation-ScaleSweep ablations reuse OSFP4 W4A4 checkpoints and change only the online activation block-scale rule, isolating that choice.A separately released NVIDIA checkpoint is included as a reference because its calibration and preparation are outside the controlled procedure.Source: §D.2 · p. 18

E PROJECTION-LEVEL DISTORTION AND DETERMINISTIC SCALE SELECTION

Projection-level experiments compare predicted randomized loss with measured errors for four RTN/SIC and W4A16/W4A4 configurations. They report close agreement between randomized pseudo-MSE and predicted loss, while deterministic OSFP4 reaches approximately half the predicted loss; the appendix also includes additional model comparisons.

E PROJECTION-LEVEL DISTORTION AND DETERMINISTIC SCALE SELECTION

  • The study measures local projection error on BF16-model inputs for three Llama-3.1-8B-Instruct projections, using 32 fresh dithers to estimate randomized pseudo-MSE.Optimization and measurement use the same empirical activation population, so this is a calibration diagnostic rather than a held-out generalization test.Source: E PROJECTION-LEVEL DISTORTION AND… · pp. 18–19
  • Deterministic W4A16 curves compare OSFP4 with absmax-to-6 weight scales at fixed A; W4A4 curves compare activation-scale rules with OSFP4 weights and transformations fixed.The W4A4 comparison uses ScaleSweepMSE versus absmax-to-6 activation block scales.Source: E PROJECTION-LEVEL DISTORTION AND… · p. 19, Figure 4 · p. 19, Figure 5 · p. 20
  • Deterministic OSFP4 has 0.48–0.53 ratios to predicted loss, while mean randomized pseudo-MSE has 0.96–1.01 ratios across configurations and projections.The W4A4 result uses ScaleSweepMSE activation scales; Figure 4 shows the full W-RTN/W4A16 comparison.Source: E PROJECTION-LEVEL DISTORTION AND… · p. 19, Table 4 · p. 21

F.1 LLAMA-3-8B PERPLEXITY

  • The appendix includes a WikiText-2 perplexity comparison for Llama-3-8B and an evaluation comparison for Qwen3-8B, with quantized configurations using NVFP4.Source: Table 5 · p. 22, Table 6 · p. 22

F.3 QWEN3-30B-A3B-INSTRUCT ACCURACY

For Qwen3-30B-A3B-Instruct, the reported evaluation is limited to W4A16 because route-aware activation-scale serving for OSFP4 W4A4 was not implemented. The appendix also specifies a separate multi-task evaluation protocol for Gemma-4-31B-IT.

F.3 QWEN3-30B-A3B-INSTRUCT ACCURACY

  • Qwen3-30B-A3B-Instruct reports only W4A16 because the route-aware serving path needed for OSFP4 W4A4 was not implemented.W4A4 would require expert-specific activation global scales and a vLLM plugin to select the appropriate scale after routing.Source: §F.3 · p. 23

F.4 GEMMA-4-31B-IT ACCURACY

  • Gemma-4-31B-IT evaluation uses zero-shot text-only prompts and averages scores over three seeds for GPQA-D, MATH-500, and IFEval, and eight for AIME 2025.The evaluation uses the Gemma chat template with thinking disabled; all table entries, including BF16, are local measurements.Source: §F.4 · p. 23, Table 8 · p. 23

G INFERENCE EFFICIENCY

The section compares OSFP4 with vendor NVFP4 RTN, unit smoothing, and BF16 throughput on Llama-3.1-8B-Instruct. OSFP4 closely matches unit smoothing across workloads; its gap to the vendor path primarily reflects runtime smoothing implementation.

G INFERENCE EFFICIENCY

  • Three timing repetitions per method and workload each consist of 30 inference runs, with smoothing applied at runtime only on o_proj and down_proj.The workloads use fixed synthetic batches of random-token prompts.Source: G INFERENCE EFFICIENCY · p. 24
  • OSFP4 closely matches the unit-smoothing control across the workload sweep, while its gap to vendor NVFP4 primarily reflects runtime smoothing implementation.End-to-end throughput also includes attention with a BF16 KV cache, activation quantization, output writes, and scheduling overhead.Source: G INFERENCE EFFICIENCY · p. 24
  • Table 9 compares OSFP4, vendor NVFP4 RTN, unit smoothing, and BF16 throughput on Llama-3.1-8B-Instruct using an NVIDIA B200.Prefill measures time to first token; decode timing subtracts the duration for one output token from that for 256 output tokens.Source: Table 9 · p. 24, G INFERENCE EFFICIENCY · p. 24

H COMPONENT ABLATION

The component ablation evaluates NVFP4 configurations under W4A16 and W4A4, reporting projection errors and WikiText-2 perplexity against precision-matched FP8 RTN references. The supplied passages describe the comparisons but do not provide table cell results.

H COMPONENT ABLATION

  • The ablation evaluates configurations calibrated on 1,024 FineWeb sequences of 2,048 tokens, with perplexity and projection-output errors measured on WikiText-2.It covers both W4A16 and W4A4 quantization.Source: H COMPONENT ABLATION · p. 25
  • For W4A16, the comparisons include AbsMax, unweighted-MSE-optimal E4M3 scales, W-RTN, and W-SIC with either identity smoothing and AbsMax scales or the proposed method.The configurations distinguish smoothing and scale optimization from the baselines.Source: H COMPONENT ABLATION · p. 25
  • For W4A4, the comparisons include AbsMax, W-RTN, and W-SIC with either identity smoothing and standard AbsMax scales or the proposed method.The identity-smoothing baseline uses standard AbsMax scaling for both weights and activations.Source: H COMPONENT ABLATION · p. 25

F ˆY FP8 RTN

I DETAILED LOSS DERIVATIONS AND SIC PROCEDURE

The derivation formulates randomized NVFP4 quantization losses over diagonal smoothing and E4M3 block scales, with separate forms for RTN and SIC and optional activation quantization. Partitioning the matrices into 16-wide blocks makes the optimization separable across blocks.

I DETAILED LOSS DERIVATIONS AND SIC PROCEDURE

  • The appendix distinguishes the E4M3 scale constellation from the FP4 constellation and defines Cscale as the positive E4M3 points.It also neglects a third error term described as much smaller than the first two under NVFP4 quantization.Source: I DETAILED LOSS DERIVATIONS AND SIC… · p. 26
  • The derivation uses the activation covariance from k calibration activations, with kXX⊤ defining the covariance-related quantity used in the formulation.The text treats X as a matrix of calibration activations when optimizing the diagonal scaling matrix.Source: I DETAILED LOSS DERIVATIONS AND SIC… · p. 26
  • For SIC, the coupling introduced by the activation covariance makes optimal FP4 quantization computationally difficult, so the procedure uses a SIC/GPTQ approach.Its loss derivation assumes filtered quantization noise has negligible energy beside the signal energy.Source: I DETAILED LOSS DERIVATIONS AND SIC… · p. 28
  • The randomized NVFP4 loss functions are expressed in terms of the smoothing matrix and E4M3 scales for weight-only and weight-and-activation quantization under RTN or SIC.The section assumes a multiplicatively dithered quantizer with zero-mean, unit-variance random noise independent of other quantities.Source: I DETAILED LOSS DERIVATIONS AND SIC… · pp. 27, 29
  • The loss dependencies on smoothing and E4M3 scales decouple into L independent optimization problems, one for each 16-entry block.Weights are partitioned into L matrices of shape m × 16; the smoothing matrix is block diagonal with 16 × 16 blocks.Source: I DETAILED LOSS DERIVATIONS AND SIC… · pp. 26, 29

Related papers

Loading 2610.08231v1…