KBBQ: A Predictive Noise Law and the Limits of Spectrum Flattening in FP4 Quantization

Lexington Whalen; Yuki Ito; Ryo Sakamoto

Source-linked AI summary

KBBQ: A Predictive Noise Law and the Limits of Spectrum Flattening in FP4 Quantization

Lexington Whalen, Yuki Ito, Ryo Sakamoto

arXiv:2609.08135v1Sep 8, 2026cs.LGcs.AI

TL;DR

Four-bit block-scaled quantization motivates a theory that explains why transforms behave differently for integer and floating-point formats. The paper derives a variance-profile noise law and a participation-factor ceiling, then introduces KBBQ to interpolate toward that ceiling; at W4A4, it outperforms prior state of the art across four base models and two FP4 formats without additional deployment-time computation.

Problem
Block-scaled four-bit floating-point quantization can substantially degrade quality, while existing transforms have unpredictable effects across models and formats.
Method
The paper derives a variance-profile noise functional, reduces the floating-point case to the participation factor κ, bounds it by κ*, and introduces KBBQ as a parameterized transform family.
Results
At W4A4, KBBQ outperforms the prior state of the art across four base models and two FP4 formats without additional deployment-time computation.
Takeaways & Limitations
The analysis provides a format-specific account of transform behavior and a theoretical ceiling for floating-point quantization improvements.
Takeaways & Limitations
The ceiling and its idealized maximizer rely on assumptions about rounding errors and access to population second moments, which deployed quantizers may violate.

Abstract

from arXiv · show

We develop a second-order theory of quantization noise in matrix multiplication in which the quantization format is characterized by the variance it assigns to each element. The constant variance profile of integer quantization recovers existing integer-noise theory, while the multiplicative profile of floating-point rounding reduces the data dependence to a scalar, the participation factor $κ$, yielding a closed-form signal-to-noise-ratio law. The resulting functional also admits a closed-form upper bound $κ^{*}$ that no function-preserving linear transform can exceed and that is attained by a recent state-of-the-art method. Building on this analysis, we introduce KBBQ (\textbf{K}appa-\textbf{B}raked \textbf{B}lockwise \textbf{Q}uantization), which parameterizes the extent to which a transform approaches this ceiling. At W4A4, across four base models and two FP4 formats, KBBQ outperforms the prior state of the art without additional deployment-time computation.

1. Introduction

The paper develops a format-agnostic noise theory that distinguishes integer and floating-point quantization through their variance profiles, then uses the participation factor κ to analyze transform limits and design KBBQ. At W4A4, KBBQ is evaluated across four base models and two FP4 formats against prior methods without additional deployment-time computation.

Motivation: Four-bit block-scaled floating-point quantization is challenging because each block shares a scale set by its largest magnitude and has only a one-bit mantissa.Quality degrades substantially when both weights and activations use four-bit precision unless tensors are made easier to round.
Theory: The floating-point law reduces data dependence to the participation factor κ, the ratio of aligned signal energy to accumulated elementwise-product energy.κ measures constructive alignment across dot-product coordinates against independently accumulating noise.
Transform limits: A closed-form ceiling κ* bounds what any function-preserving transform can achieve, and it is computable per layer from second-order statistics.The numerator of κ is invariant under invertible function-preserving transforms.
KBBQ: KBBQ parameterizes movement from the untransformed baseline toward the idealized transform attaining κ*, while preserving the kernel, storage layout, and inference arithmetic.Its best operating point is reported as interior rather than at the ceiling.
Theory: The proposed noise functional characterizes quantized matrix multiplication through the elementwise variance profile assigned by the format.Constant variance recovers integer-noise theory, while multiplicative variance yields the floating-point case.
Evaluation: At W4A4, KBBQ outperforms the prior state of the art across four base models and two FP4 formats without additional deployment-time computation.The paper also predicts and verifies its closed-form SNR laws across six model families in two formats.

2. A Noise Law for Floating-Point Quantization

The paper derives a format-agnostic noise functional whose variance profile recovers integer and floating-point theories. For floating-point quantization, the resulting SNR depends on the participation factor κ and admits format-specific predictions and transform bounds.

General noise functional: A general noise functional characterizes quantized dot-product noise through the elementwise variance profile of the format.It is derived under unbiased, uncorrelated errors with variance functions for weights and activations.
Variance profiles: Constant variance recovers integer quantization theory, whereas multiplicative variance yields the floating-point law.The two profiles differ because integer noise depends on grid amplitude while floating-point noise scales locally with operand magnitude.
Floating-point law: The floating-point law reduces data dependence to the participation factor κ, the ratio of aligned signal energy to incoherent product energy.κ measures constructive alignment across dot-product coordinates against independently accumulating noise.
Transform bound: The closed-form ceiling κ* bounds improvement from any function-preserving transform, while transform behavior depends on whether amplitude or alignment governs the format.The theory predicts that amplitude-only changes cannot improve floating-point quantization, whereas alignment-only changes cannot improve integer quantization.
Empirical validation: Measured SNR follows the predicted κ relationship across six model families and two formats, with format-dependent intercepts.Per-tensor E4M3 has a median intercept of 28.42 dB, while the measured cross-format spacing is 13.24 dB over 3968 records.

3. Applications of the Noise Law

The applications analysis explains why rotations help integer quantization but can be ineffective for floating-point formats. Because trained floating-point networks already exceed the Haar attractor, generic rotations provide little benefit.

Integer formats: Integer rotations reduce noise by spreading outliers, shrinking the shared grid step set by the largest coordinate.The signal and total energy remain unchanged, while amplitude flattening lowers the step for every element sharing the scale.
Floating-point formats: Floating-point rotations affect SNR only through κ because the multiplicative noise law does not depend on the maximum amplitude.Function-preserving transforms are therefore limited by the gap κ*−κ.
Haar attractor: Across four benchmarked models, median native κ lies 0.012–0.071 dex above the Haar attractor.These models were not trained with a quantization-aware objective, yet their native participation factors already exceed the generic random-rotation value.
Empirical consequence: A dense data-free Haar rotation does not exceed the identity method on any model–format pair.This outcome is consistent with the networks already lying above the random-rotation attractor.

4. KBBQ: A Braked Optimum

KBBQ replaces fixed spectrum flattening with a tunable brake that interpolates between no flattening and the κ* construction. Although the idealized objective favors full flattening, experiments show that an interior operating point performs best without deployment-time cost.

Construction: KBBQ introduces a free flattening exponent that interpolates between the untouched spectrum and complete square-root spectrum inversion.The parameter λ contracts the paired log spectrum from no flattening at λ = 0 to full flattening at λ = 1.
Idealized objective: Under assumptions A1–A3, κ(λ)/κ* rises monotonically from 0.73 at λ = 0 to 1.00 at λ = 1, making full flattening optimal for the idealized objective.The bound depends on assumptions that may not hold for deployed quantizers.
Operating point: Accuracy peaks before λ = 1 on both swept models, while λ = 0 is the weakest setting.This indicates a gap between the idealized assumptions and deployed-quantizer behavior.
Construction: At λ = 1, KBBQ reproduces the full-flattening construction attaining κ*, while lowering λ partially flattens the paired spectrum.At λ = 0, the transform retains data-dependent basis and root factors but applies no flattening.
Deployment: KBBQ matches the reference method’s transform shape, storage, kernels, and inference arithmetic, so λ adds no deployment-time cost.Transform blocks match quantization groups of 16 for NVFP4 and 32 for MXFP4.

5. Experiments

The experiments evaluate KBBQ at W4A4 across four base models and two FP4 formats, including native NVFP4 execution and emulated MXFP4 quantization. KBBQ improves over identity on every model–format pair and usually surpasses RTN-WUSH.

Setting: KBBQ is evaluated at W4A4 on four base models under NVFP4 and MXFP4, with native NVFP4 and emulated MXFP4 evaluation.The models are Llama-3.2-3B-Base, Llama-3-8B-Base, Qwen3-4B-Base, and Qwen3-8B-Base.
Main results: RTN-KBBQ improves over identity on all eight model–format pairs, with gains from 0.5 points to 7.1 points.The smallest gain is on Llama-3-8B-Base with NVFP4, and the largest is on Qwen3-8B-Base with MXFP4.
Main results: RTN-KBBQ outperforms RTN-WUSH on seven of eight model–format pairs.For Qwen3-8B-Base under NVFP4, it achieves 71.54 accuracy and 96.4% recovery versus 70.73 and 95.3%.

6. Robustness and Ablations

Ablations show that the best brake is interior rather than fully flattened, while performance remains robust across calibration corpora and calibration budgets. These results support a geometry-dependent operating point rather than sensitivity to calibration quantity or domain.

Brake ablation: The best five-task average is 66.44 at λ = 0.75, exceeding unbraked λ = 1 by 1.93 points.The sweep ranges from λ = 0 with no flattening to λ = 1 corresponding to WUSH.
Brake ablation: κ(λ)/κ* increases monotonically from 0.73 to 1.00 as λ rises, while accuracy peaks earlier and declines.Every tested interior setting exceeds λ = 1, including averages of 65.22, 65.61, and 66.44 at λ = 0.25, 0.50, and 0.75.
Calibration corpus: Changing the calibration corpus moves likelihood benchmarks by at most 1.8 points, while KBBQ’s GSM8K margin over WUSH remains 1.8 to 5.8 points.The margin persists across the default, GSM8K, and random corpora.
Calibration budget: Across a 256-fold calibration-budget range, KBBQ’s five-task averages vary within 0.3 points among the four budget settings.KBBQ retains its advantage over WUSH at every budget, including a 6.2-point GSM8K margin with 256 tokens.

7. Conclusion

The paper unifies integer and floating-point quantization-noise analysis through the elementwise variance profile, identifying κ as the floating-point data-dependent quantity and κ* as a per-layer ceiling. KBBQ makes progress toward that ceiling tunable rather than assuming the idealized optimum is operationally best.

Conclusion: The elementwise variance profile is the single format-dependent object governing expected quantized-dot-product noise.Constant profiles recover integer theory, while multiplicative profiles reduce data dependence to the participation factor κ.
Conclusion: The analysis yields a closed-form per-layer ceiling κ*, but the construction attaining it optimizes an objective that deployed quantizers may not actually face.KBBQ instead parameterizes the distance traveled toward this ceiling.

B. Results on Other Models

Table 6 extends the evaluation from Section 5 to the two 8B base models.

Table 6 repeats the Section 5 evaluation for Llama-3-8B-Base and Qwen3-8B-Base.
The repeated evaluation covers both 8B models named in the section.
The section reports evaluation results rather than introducing a new experimental protocol.

C. Full Proofs for Section 2

The proof analyzes quantization noise under three assumptions, including conditional independence of weight- and activation-side errors. The activation-side error may depend on the activation draw subject to these assumptions.

The derivation uses three standard assumptions on quantization errors, denoted A1–A3.
Weight-side error depends on weights and quantizer randomization, whereas activation-side error depends on the activation draw and activation quantizer randomization.
Weight- and activation-side errors are conditionally independent given the operands.
A2 asserts uncorrelated errors, while factoring expectations across operands requires Eq. 16.
The activation-side error may depend on x in any way consistent with A1–A3.

C.1. The Noise Functional

The noise functional decomposes quantization error into first-, second-, and higher-order contributions, then reduces floating-point data dependence to the participation factor κ. Its range and format dependence support a closed-form FP4 comparison.

The quantized dot-product error follows by expanding each quantized operand into its true value plus quantization error.
Each correction term is tied to specific assumptions, so failures of the noise law can be traced to A1–A3.
When A1 fails under deterministic round-to-nearest on structured operands, shared error signs create a coherent-bias channel through R1.
Under the multiplicative profile, the surviving second-order terms factor into a data-dependent participation factor κ, while the diagonal fourth-order correction is approximated away.
κ lies in [0, D], approaching 0 under cancellation and equaling D only when products have equal RMS and pairwise correlation one.
Floating-point rounding has a multiplicative variance profile, whereas integer quantization retains dependence on the scale-group amplitude.
A mantissa-bit difference of p2 − p1 predicts an intercept gap of approximately 6.02(p2 − p1) dB under a shared operand distribution.

C.4. Recovery of the Integer Theory (CAT)

The integer-theory recovery connects per-tensor concentration and alignment to SQNR, while the matched-ensemble analysis identifies the participation-factor ceiling and its attaining transform class.

Recovery of the integer theory: The integer quantization assumption block is recovered by uniform cell errors with a constant variance profile.
Recovery of the integer theory: Per-tensor SQNR scales with bit-dependent grid resolution and concentration, while composed weight and activation channels combine reciprocally in SNR.
Alignment and κ: For a row, κ and alignment A normalize the same signal by different energies, becoming proportional under the equal-diagonal condition.
Alignment and κ: Under the matched ensemble, the relation between κ and alignment holds without the equal-diagonal condition after averaging numerator and denominator separately.
The ceiling and its attainer: The ceiling κ* depends only on the spectrum of Σ, is rotation-invariant, and equals D times the ensemble alignment of the layer.
The ceiling and its attainer: The matched-ensemble optimum completely flattens the paired spectrum, while reparameterized-coordinate orthogonal transforms preserve their objective.
The ceiling and its attainer: TWUSH satisfies the equal-diagonal condition and therefore belongs to the attainer class of the theorem.

C.6. Transforms: Invariance, Blindness, and the Ceiling

Function-preserving transforms leave the dot product and the participation-factor numerator unchanged, while changing the denominator determines how closely the layer approaches a closed-form ceiling. The ceiling is attained by scaled orthogonal transforms with equal transformed covariance diagonals, but blockwise deployment targets a lower per-block ceiling and actual per-cell measurements may exceed the ensemble bound.

Invariance: Invertible preprocessing preserves each dot product pointwise, so the participation-factor numerator is invariant under the transform.The transformed operands satisfy ŵ=T^-⊤w and x̂=Tx, giving ŵ^⊤x̂=w^⊤x for every realization.
Blindness: Diagonal scaling is invisible to floating-point participation and SNR, but it changes integer quantizer steps through the operand maxima.The products remain pointwise unchanged under paired diagonal rescaling, whereas integer per-tensor amax values generally move.
The ceiling: The participation-factor ceiling bounds the ensemble aggregate for every invertible transform and is derived from second-order input statistics.The proof uses a Rayleigh-quotient reduction and bounds the aggregate quantity, not necessarily the κ measured on an individual trained cell.
Measurement caveat: A per-cell κ measured from actual trained weights may exceed the ensemble ceiling, because the bound applies to the ensemble aggregate.This distinction matters when interpreting empirical measurements against κ*.
The ceiling: The ceiling is attained by transforms T=diag(s)R where R is orthogonal and the transformed covariance has equal diagonal entries.A suitable orthogonal transform exists, and randomized Hadamard transforms attain the equal-diagonal condition in expectation.
Blockwise scope: For blockwise transforms, the relevant ceiling is computed independently within each quantization-matched block rather than at the full layer dimension.The blockwise theorem substitutes block size d and covariance block Σ_b; the layer-level ceiling applies only to dense transforms.