PaperlayerView exact revision on arXiv

Source-linked AI summary

WUSH: Near-Optimal Adaptive Transforms for LLM Quantization

Jiale Chen, Vage Egiazarian, Roberto L. Castro, Torsten Hoefler, Dan Alistarh

arXiv:2512.00956v3cs.LGcs.CL

TL;DR

Extreme outliers degrade low-bit LLM quantization, while fixed Hadamard transforms leave their optimality unclear. WUSH derives adaptive blockwise transforms and improves accuracy over prior transform-based baselines.

  • Problem

    Extreme weight and activation outliers degrade low-bit resolution, while Hadamard-based transforms are fixed and data-agnostic, leaving their quantization optimality unclear.

    Source: §1 · p. 1

  • Method

    WUSH derives closed-form, data-aware non-orthogonal blockwise transforms for joint weight-activation quantization, combining a Hadamard backbone with second-order statistics.

    Source: §1 · p. 1, §7 · p. 9

  • Results

    WUSH significantly improves end-to-end accuracy over prior transform-based baselines and delivers up to 5.8× per-layer speedups over BF16 with FP4 matrix multiplication.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The results clarify Hadamard rotations’ empirical success and indicate that principled data-aware block transforms can yield significant improvements, including making MXFP competitive with NVFP.

    Source: §7 · p. 9

    The analysis assumes block-diagonal transforms aligned with quantization groups and stochastic, unbiased quantization.

    Source: §3 · p. 2, §4.1 · p. 4

Abstract

from arXiv · show

Quantizing LLM weights and activations is a standard approach for efficient deployment, but a few extreme outliers can stretch the dynamic range and amplify low-bit quantization errors. Prior transform-based mitigations (e.g., Hadamard rotations) are fixed and data-agnostic, and their optimality for quantization has remained unclear. We derive closed-form optimal linear blockwise transforms for joint weight-activation quantization under standard RTN AbsMax-scaled block quantizers, covering both integer and floating-point formats. The resulting construction, WUSH, combines a Hadamard backbone with a data-dependent second-moment component to form a non-orthogonal transform that is provably near-optimal for FP and INT quantizers under mild assumptions while admitting an efficient fused GPU implementation. Empirically, WUSH improves W4A4 accuracy over the strongest Hadamard-based baselines (e.g., on Llama-3.1-8B-Instruct in MXFP4, it gains +2.8 average points with RTN and +0.7 with GPTQ) while delivering up to 5.8$\times$ per-layer throughput over BF16 via FP4 MatMul. Source code is available at https://github.com/IST-DASLab/WUSH.

1. Introduction

WUSH replaces fixed, data-agnostic rotations with a closed-form, adaptive blockwise transform for joint weight-activation quantization. It is theoretically optimal for floating-point block quantizers and asymptotically optimal for integer block quantizers, with reported accuracy and throughput gains.

Introduction

  • Outliers degrade low-bit resolution by expanding dynamic range, while widely used Hadamard rotations spread variance but do not adapt to weight or activation statistics.Blockwise Hadamard transforms have performed strongly in practice; Egiazarian et al. (2026) reported the best empirical quantization performance among a large set of existing transforms.Source: §1 · p. 1
  • +2.8 average points with MXFP4 RTN and +0.7 points with MXFP4 GPTQ on W4A4 Llama-3.1-8B-Instruct over the Hadamard-based baseline.The paper also reports that WUSH substantially narrows the accuracy gap between NVFP and MXFP formats and provides consistent benefits with GPTQ.Source: §1 · p. 1
  • WUSH derives closed-form optimal linear blockwise transforms for joint weight-activation quantization; its generally non-orthogonal, adaptive construction is optimal for FP and asymptotically optimal for INT block quantizers.Unlike calibration-data methods such as SpinQuant (Liu et al., 2025) and FlatQuant (Sun et al., 2025), WUSH is a closed-form calibration-driven transform.Source: §1 · p. 1
  • Up to 5.8× per-layer speedup over BF16 is achieved with FP4 matrix multiplication, while WUSH’s fused GPU kernel matches optimized blockwise Hadamard-kernel throughput.The kernel supports a distinct data-aware transform per block.Source: §1 · p. 1

2. Related Work

Prior work mitigates quantization outliers through explicit outlier storage or transforms, but these approaches can require irregular formats, costly learning, or rely on Hadamard transforms alone. WUSH instead derives a closed-form, data-aware optimal blockwise transform for FP and INT block quantizers.

Post-training quantization (PTQ) and outliers

  • RTN, OBQ (Frantar & Alistarh, 2022), and GPTQ are sensitive to heavy-tailed LLM outliers that can set quantization scales and increase error.A small number of extreme weight or activation values can determine the scale.Source: §2 · p. 2

Non-uniform bitwidths and explicit outlier storage

  • LLM.int8(), SpQR, and QUIK (Ashkboos et al., 2024a) store outliers at higher precision, while HPTQ (Chen et al., 2026) Huffman-compresses them, but formats are irregular and need specialized kernels.These methods can achieve high accuracy at low effective bitwidth.Source: §2 · p. 2

Transform-based outlier mitigation

Blockwise transforms and emerging FP formats

  • For emerging MXFP (MX Alliance, 2023) and NVFP, blockwise Hadamard transforms can outperform full-layer transforms, but Hadamard alone has limited gains linked to weight and activation distributions.WUSH derives a closed-form, data-aware optimal blockwise transform and analyzes its interaction with FP and INT block quantizers.Source: §2 · p. 2

3. Methodology

WUSH chooses blockwise weight and activation transforms to reduce quantized linear-layer output loss, deriving closed-form transforms that are optimal for FP and asymptotically optimal for INT quantization. The method integrates second-order statistics with a Hadamard component and is designed to fit RTN quantization and GPTQ workflows with low overhead.

Problem setup

  • The objective is to choose weight and activation transforms that minimize L2 output loss after quantizing both transformed matrices.For a linear layer with weights W and calibration activations X, the quantized output is q(T_W W)^⊤q(T_X X).Source: §3 · p. 2

Block-independent constraint

  • Transforms are constrained to block-diagonal form aligned with RTN AbsMax quantization groups of size d, a power-of-two factor of d_in.The constraint limits dynamic activation-side transformation overhead; the text attributes this practice to Egiazarian et al. (2026).Source: §3 · p. 2

Optimal block-diagonal transforms

  • WUSH derives closed-form optimal transforms for each block, provably optimal for FP and asymptotically optimal for INT quantization.Its construction uses a Hadamard matrix as the only data-agnostic ingredient and jointly balances weight and activation scales.Source: §3 · pp. 2–3

GPTQ integration

  • WUSH can use calibration-activation second moments or the GPTQ Hessian, but its coupling to GPTQ weight updates requires interleaving transform construction and error propagation.GPTQ applies standard intra-block updates to transformed weights and usual blockwise updates across blocks; calibration layers are processed sequentially with activations propagated through each quantized layer.Source: §3 · pp. 3–4

Complexity and inference costs

  • WUSH adds negligible offline overhead when combined with GPTQ and negligible inference runtime overhead when the activation transform is fused with quantization.Weight transforms are absorbed into pre-quantized weights, while the online activation transform is fused with activation quantization.Source: §3 · p. 4

4. Theoretical Derivation

The derivation reduces joint weight–activation quantization to a one-sided transform problem and identifies a Hadamard-based, data-dependent transform as optimal for FP and near-optimal for INT under stated distributional conditions.

4.1. Problem Setup and General Proof Approach

  • Assuming stochastic unbiased quantization, the analysis treats weight and activation columns as i.i.d. samples and reduces the two-sided loss to a one-sided transform optimization.A first-order approximation splits the loss into two non-negative terms, which can be optimized individually.Source: §4.1 · pp. 4–5
  • For FP, the optimal transform uses U′ = H, S′ = S^1/2, and R = I; the same configuration is near-optimal for INT, with tighter guarantees for zero-mean Gaussian or Laplacian data.For INT, the theorem gives a d factor for general distributions and an o(1) factor for the stated Gaussian or Laplacian distributions.Source: §4.1 · p. 5

4.2.1. QUANTIZATION ERROR MODELING

  • For FP types, quantization error is modeled as proportional to each value’s absolute magnitude, with i.i.d. zero-mean noise.Source: §4.2.1 · p. 6

4.2.2. LOSS MINIMIZATION

  • For the FP loss, the lower bound is attained with U′ = H, S′ = S^1/2, and R = I.Source: §4.2.2 · p. 6

4.2.3. DISCUSSION

  • Under the FP error model, orthogonal transforms do not reduce quantization error, while WUSH can reduce it by at most d-fold in the extreme-outlier scenario.Both U′ = H and S′ = S^1/2 are essential; choosing either U′ = I or S′ = I yields the same suboptimal trace term as an orthogonal transform.Source: §4.2.3 · p. 6

4.3.1. QUANTIZATION ERROR MODELING

  • For INT types, quantization error is modeled as proportional to the maximum absolute value within each quantization group, with i.i.d. zero-mean noise.Source: §4.3.1 · p. 6

4.3.2. LOSS MINIMIZATION

  • The INT construction’s quantization loss is near-optimal within a factor of d for general distributions and within o(1) for tail-bounded data.The tail-bounded condition includes zero-mean multivariate Gaussian or Laplacian distributions.Source: §4.3.2 · pp. 6–7

4.3.3. DISCUSSION

  • For INT, orthogonal transforms are suboptimal, and Hadamard can have error bounds up to d times those of the optimal transform in the extreme-outlier scenario.Source: §4.3.3 · p. 7

5. GPU Kernel Support

WUSH requires an online, per-block transform, complicating reuse and fusion compared with data-independent Hadamard rotations. Its fused GPU kernel mitigates the added transform cost, closely matching the optimized Hadamard baseline while retaining substantial speedups over BF16.

GPU Kernel Support

  • Unlike data-independent Hadamard rotations, WUSH assigns a distinct transform to each block, complicating reuse and fusion in existing low-level libraries.The online transform is required during inference; its additional cost is almost entirely mitigated by the kernel design and L1 and L2 cache reuse.Source: §5 · pp. 7–8

Fused WUSH + Quant kernel implementation

  • WUSH matrices are stored offline as (G,G,C), and the memory-bound small-group transform is computed as C independent dense transforms, loading one (G,G) matrix per thread block.A CUTLASS GEMM template maps activation M and transform G to GEMM M and N, fixes K’=G, and handles C through a per-block offset.Source: §5 · p. 7

Performance

  • 5.8× over BF16 is the peak speedup for WUSH + Quant + FP4-MatMul, which averages only 1.3% lower throughput than H + Quant + FP4-MatMul.Some configurations achieve identical performance; caching largely mitigates the cost of loading per-block transform matrices.Source: §5 · pp. 7–8, Figure 2 · p. 7

6. Experiments

Experiments evaluate WUSH’s layerwise quantization loss and end-to-end accuracy across model families, formats, and quantization procedures. WUSH reduces layerwise losses and improves benchmark accuracy, with the clearest reported gains on MXFP4.

6. Experiments

  • Experiments compare WUSH with prior methods and ablate transform components and RTN versus GPTQ across Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, and Qwen3-8B/14B/32B.Accuracy is evaluated with Platinum Benchmarks and the LM Evaluation Harness.Source: §6 · p. 8

6.1. Superior Layerwise Quantization Loss

  • WUSH yields the smallest layerwise RTN loss for MXFP4 and INT4, while WUSH and WUS are almost equal for NVFP4.The reported layerwise comparison uses Qwen3-8B’s 18th transformer block; results were consistent across blocks and datasets. NVFP4 errors tend to be lower due to its smaller group size.Source: Table 1 · p. 8, §6.1 · p. 8
  • The layerwise results support the stochastic unbiased quantization surrogate, while format-specific effects explain why transform components help or hurt differently across MXFP4, INT4, and NVFP4.MXFP4’s hybrid behavior can benefit from Hadamard mixing; WUS can amplify coordinates and increase INT4 AbsMax scales, while WUSH overcomes NVFP4’s Hadamard-only harm from top-element preservation.Source: §6.1 · p. 8

6.2. End-to-End LLM Accuracy Benchmarks

  • On Qwen3-8B Platinum Benchmarks, WUSH improves MXFP4 average accuracy by up to +1.3 (RTN) and +0.7 (GPTQ) points; NVFP4 gains are smaller and often within run-to-run variability.The comparison covers both NVFP4 and MXFP4 transforms.Source: Figure 3 · p. 9, §6.2 · p. 9
  • For Qwen3-14B, WUSH reduces the NVFP4-MXFP4 average gap to 0.36 points, with above 98% recovery.The paper also reports reduced KL divergence and stability across calibration datasets.Source: §6.2 · p. 9

7. Conclusion

WUSH derives closed-form linear transforms for FP and INT block quantization, combining a fixed Hadamard backbone with a second-order-statistics component. The resulting non-orthogonal blockwise transform is provably near-optimal and supports efficient GPU kernels.

7. Conclusion

  • WUSH provides closed-form linear transforms for weight-activation quantization with both FP and INT block quantizers.The work studies LLM quantization in the presence of heavy-tailed outliers.Source: §7 · p. 9
  • Combining a fixed Hadamard backbone with a second-order-statistics component yields a non-orthogonal blockwise transform that is provably near-optimal and GPU-kernel-friendly.Source: §7 · p. 9
  • The results clarify Hadamard rotations’ empirical success and indicate that principled data-aware block transforms can make MXFP competitive with NVFP.Source: §7 · p. 9

A. Detailed Theoretical Results · A.1. Second Moment Decompositions · A.2. Multivariate vs. Univariate Probabilistic Modeling

The theoretical results establish invariance of the second-moment decomposition choices and motivate multivariate, spectrum-aware transforms for group quantization. They distinguish changing coordinate marginals from reducing the underlying anisotropy that drives outlier-dominated scaling.

A.1. Second Moment Decompositions

  • The matrix products W′(i)V(i) in Eq. (7) are invariant to decomposition choices up to sign and permutation changes.The stated invariance concerns any full-rank W′(i) satisfying Eq. (5).Source: §A.1 · p. 13
  • Factors cW and cX with the same respective Gram matrices as W and X induce orthogonal QW and QX, with cW = WQW and cX = XQX.This establishes the orthogonal relationship used to compare valid decompositions.Source: §A.1 · p. 13

A.2. Multivariate vs. Univariate Probabilistic Modeling

  • Prior transform methods emphasize univariate Gaussianization, whereas this analysis models each quantization group as a joint random vector and uses its second-order spectrum.The spectrum comprises eigenvalues or singular values and supports transforms that depend on that structure.Source: §A.2 · p. 13
  • Hadamard rotation can make coordinate marginals nearly identical and their average more Gaussian-like, but this does not establish that joint-space outliers have been removed.The figure contrasts the original anisotropic joint density with the rotated coordinate marginals.Source: §A.2 · p. 13
  • Orthogonal transforms preserve global energy and covariance eigenvalues, so points extreme along principal directions remain extreme in the joint space.Such transforms rotate eigenvectors rather than changing the covariance spectrum.Source: §A.2 · p. 13
  • Because group quantization error depends on joint mass, dominant directions, and interactions with groupwise scaling, marginal statistics alone cannot certify it.The multivariate formulation targets the spectrum to reduce effective anisotropy responsible for outlier-dominated scaling, rather than merely redistributing it across coordinates.Source: §A.2 · p. 13

A.3. Equalization Effect

WUSH equalizes per-channel RMS across weight and activation blocks by balancing their scales, producing flatter profiles than the illustrated alternatives. The section explains this effect through the transformed second moments and contrasts it with rotations that preserve multivariate outliers.

A.3. Equalization Effect

  • Orthogonal rotation can make coordinate marginals nearly identical but preserves the joint spectrum, leaving extreme multivariate points extreme.The figure motivates transforms that act on joint geometry rather than relying on marginal Gaussianization.Source: Figure 4 · p. 14
  • WUSH maps each blockwise second moment to HSH^T, whose diagonal is constant, equalizing per-channel RMS within weight and activation blocks.The transformed activation second moment has a constant diagonal, and the analogous property holds for the transformed weight second moment.Source: §A.3 · p. 13, Figure 5 · p. 14
  • Unlike WUSH, orthogonal rotations cannot transfer scale between weights and activations, while SmoothQuant transfers scale through diagonal rescaling and can unevenly amplify channels.Hadamard equalizes within-block RMS more effectively than random rotation; WUS lacks the Hadamard factor and leaves per-channel RMS unequal.Source: Figure 5 · p. 14, §A.3 · p. 15
  • WUSH balances weight and activation scales and produces the flattest RMS profiles among the illustrated transforms.Figure 5 shows absolute-value slices and per-input-channel RMS profiles after identity, random rotation, Hadamard, WUS, WUSH, and SmoothQuant transforms.Source: Figure 5 · p. 14, §A.3 · p. 15

A.4. Equation Derivations

The derivations express a second-order quantity as US^2U^T and identify a Hadamard-based choice attaining equality. They also use tail-distribution inequalities to tighten a bound and equalize coordinate-wise quantities.

A.4. Equation Derivations

  • W′ᵀX′X′ᵀW′ simplifies to US^2Uᵀ using the stated equations.The derivation writes the product as USVᵀVSUᵀ before simplifying.Source: §A.4 · p. 15
  • The equality can be attained by choosing U′ = H and S′ = S.Source: §A.4 · p. 17
  • For zero-mean Gaussian or Laplacian variables, the stated maximum inequalities hold without requiring independence.The passage identifies these as Lemma A.2's conditions.Source: §A.4 · p. 19
  • For tail-bounded zero-mean multivariate Gaussian or Laplacian Dy, Lemma A.2 is used to further tighten an upper bound.Source: §A.4 · p. 19
  • With U′ = H, the derivation reports a coordinate-wise quantity equalized across all k.The supplied passage does not expose the full expression for that quantity.Source: §A.4 · p. 19

A.5. Quantization Error Modeling Justifications … B. Further Experimental Results

The appendix justifies integer and floating-point quantization-error models, then develops survival-function-based maximum bounds for Gaussian and Laplacian distributions.

A.5.1. INTEGER (INT) TYPES

  • INT quantization error is modeled as pseudo-noise with i.i.d. samples, using an AbsMax scale for a symmetric b-bit integer quantizer.For a vector α, the scale is s = 2^(1-b) ||α||∞.Source: §A.5.1 · p. 19

A.5.2. FLOATING-POINT (FP) TYPES

  • The smoothed FP format SEaMb approximates 1 + x by 2x for 2^-b_m in [0, 1].The approximation is visualized in Figure 6.Source: §A.5.2 · p. 20, Figure 6 · p. 20
  • Under the FP casting-error model, the expected quantization error is zero.The model treats the error components for values and scales using random variables ξ and ξ̃.Source: §A.5.2 · p. 21

A.6. Maximum Inequalities for Tail-Bounded Distributions

  • Lemma A.3 expresses expectations through the survival function, assuming the boundary term vanishes at infinity.The lemma relates an integral involving a function of a nonnegative random variable to an integral weighted by its survival function.Source: §A.6 · p. 21

A.6.1. GAUSSIAN DISTRIBUTION

  • A sub-Gaussian variance proxy is at least the variance, and for a Gaussian distribution the optimal proxy equals the variance.The Gaussian maximum-bound derivation applies to zero-mean variables without requiring their independence.Source: §A.6.1 · p. 22
  • The Gaussian section derives a maximum bound for zero-mean Gaussian variables without requiring independence.The derivation uses a common variance σ².Source: §A.6.1 · p. 22

A.6.2. LAPLACIAN DISTRIBUTION

  • For zero-mean Laplacian variables with potentially different scale parameters, the maximum-bound derivation does not require independence.The section specifies scale parameters b1, …, bd.Source: §A.6.2 · p. 23

B.1. Additional Accuracy Results · B.2. Computational and Memory Costs

The appendix reports additional W4A4 and benchmark evaluations across Llama and Qwen models, alongside descriptions of WUSH preprocessing cost and activation-transform storage overhead.

B.1. Additional Accuracy Results

  • Evaluations span Llama 3 and Qwen 3; Qwen thinking mode is disabled, quantization is emulated in BF16/FP16, and results aggregate multiple random seeds.Source: §B.1 · p. 24
  • Table 3 covers W4A4 LM Evaluation Harness accuracy for Llama models using RTN, GPTQ, and identity, Hadamard, or WUSH transforms; MMLU-CoT is reported in the MMLU column.Source: Table 3 · p. 24
  • Figure 7 compares transforms for Qwen3-14B with NVFP4 and MXFP4 on Platinum Benchmarks, showing task accuracies and average scores with standard deviations.Source: Figure 7 · p. 25
  • Figure 8 compares transforms for Qwen3-32B with NVFP4 and MXFP4 on Platinum Benchmarks, showing task accuracies and average scores with standard deviations.Source: Figure 8 · p. 25
  • Figure 9 compares transforms for Llama-3.2-3B-Instruct with NVFP4 and MXFP4 on Platinum Benchmarks, showing task accuracies and average scores with standard deviations.Source: Figure 9 · p. 26
  • Figure 10 compares transforms for Llama-3.1-8B-Instruct with NVFP4 and MXFP4 on Platinum Benchmarks, showing task accuracies and average scores with standard deviations.Source: Figure 10 · p. 26

B.2. Computational and Memory Costs

  • Table 6 reports WUSH with RTN offline preprocessing cost, including calibration and transform construction; offloading and recomputing activations can reduce GPU memory use while increasing runtime.Source: §B.2 · p. 27, Table 6 · p. 27
  • Table 7 reports the additional whole-checkpoint storage overhead incurred by storing activation-side WUSH transforms.Source: §B.2 · p. 27, Table 7 · p. 27
Loading 2512.00956v3…