PaperlayerView exact revision on arXiv

Source-linked AI summary

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

Mahsa Salmani, Sayeh Sharify, Hesham Mostafa

arXiv:2610.13767v1cs.LGcs.AI

TL;DR

Low-bit inference benefits from quantizing both weights and activations, but activation outliers make calibration-free activation quantization challenging. DUALQUANT derives dual scales by minimizing weight-reconstruction MSE, using its column scales to rescale activations without calibration data. Across Llama and Qwen evaluations it matches or surpasses SmoothQuant across most formats and models, and its packed W4A4 path improves memory use and throughput over BF16.

  • Problem

    Activation outliers make low-bit activation quantization challenging, and calibration-based methods require representative data, limiting calibration-free joint W+A quantization across modern MX and NV formats.

    Source: §1 · p. 1

  • Method

    DUALQUANT minimizes weight-reconstruction MSE with dual row and column scales, using weight-derived column scales as activation rescalings and a BCD solver with closed-form updates.

    Source: §1 · p. 2

  • Results

    DUALQUANT matches or surpasses SmoothQuant across most models and formats and outperforms calibration-free QuaRot and SINQ+, while packed Llama-3.1-8B W4A4 reduces memory 2.51–2.61× and improves speed 1.39–2.46× over BF16.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Weight-derived scales provide effective activation rescaling without calibration data across the evaluated Llama and Qwen models and supported formats.

    Source: §1 · p. 2, §6 · p. 9

    The convergence result assumes that no row is mapped entirely to zero by the quantizer and that the quantized matrix stabilizes at a finite iteration.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requires calibration data. In this work, we introduce DUALQUANT, a dual-scale, calibration-free post-training quantization framework that directly minimizes the weight-reconstruction Mean Squared Error (MSE) and, as an immediate by-product, yields per-channel activation rescalings prior to activation quantization. Our key observation is that the scaling factors obtained by minimizing weight-reconstruction error provide effective activation rescalings, enabling competitive activation quantization without calibration data. We formulate DUALQUANT as a non-differentiable weight-reconstruction problem and develop an efficient solver based on variable splitting and block coordinate descent, where each subproblem admits a closed-form update. DUALQUANT provides a unified algorithm supporting a wide range of formats, including the full MX family and NVFP4. To our knowledge, DUALQUANT is the first dual-scaling method to formally support MX and NV block formats. Across Llama and Qwen models evaluated on WikiText-2, C4, and zero-shot downstream tasks, DUALQUANT matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats, and substantially outperforms calibration-free baselines, with large gains for MX and NV block formats. We validate these gains with a real packed W4A4 inference path for Llama-3.1-8B on an RTX 4090. The resulting implementation closely matches the accuracy of fake quantization while delivering up to 2.61x memory reduction and up to 2.46x throughput speedup over BF16.

1 INTRODUCTION

DUALQUANT addresses the lack of competitive calibration-free joint weight-and-activation quantization across modern MX and NV formats. It derives activation rescalings from weight-reconstruction scales and reports competitive results against calibration-based and calibration-free methods.

1 INTRODUCTION

  • Activation outliers make low-bit activation quantization difficult, while calibration-based methods require representative data that can impose distribution sensitivity and data-access overhead.The introduction identifies privacy-sensitive and on-device settings as cases where that overhead can be prohibitive.Source: §1 · p. 1
  • DUALQUANT jointly quantizes weights and activations without calibration data, using per-row and per-column scales obtained by minimizing weight-reconstruction MSE.The weight-derived column scales also serve as activation rescalings, playing a role similar to SmoothQuant’s calibrated scales.Source: §1 · pp. 1–2
  • Its BCD solver uses closed-form per-coordinate updates and format-aware scale projection, and supports INT4, MXINT4, MXFP4, and NVFP4.The method requires no inner loop, Hessian, or calibration data.Source: §1 · p. 2
  • DUALQUANT matches or surpasses SmoothQuant across most models and formats, outperforms QuaRot and SINQ+ especially on MX and NV, and its packed Llama-3.1-8B path reduces memory 2.51–2.61× and improves speed 1.39–2.46× over BF16.The packed W4A4 implementation matches fake-quantization accuracy within 0.01 perplexity on WikiText-2.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior PTQ methods address weight and activation quantization in different ways, but the cited approaches leave a gap in calibration-free joint W+A quantization. DUALQUANT targets weight-reconstruction MSE and uses its column scales to rescale activations.

2.1 POST-TRAINING WEIGHT QUANTIZATION

  • Weight-only PTQ methods include second-order correction, salient-channel scaling, reconstruction objectives, adaptive LASSO, and SINQ’s dual scaling, but leave activations in BF16.SINQ minimizes matrix imbalance as a proxy for weight quantization error.Source: §2.1 · p. 2

2.2 ACTIVATION QUANTIZATION

  • Activation outliers can dominate the quantization grid, and existing approaches include mixed precision, calibrated rescaling, dynamic quantization, learned transformations, and rotations.Most rotation-based approaches depend on calibration data.Source: §2.2 · p. 2
  • DUALQUANT’s weight-reconstruction MSE optimization yields column scales that also rescale activations, enabling calibration-free W+A quantization that matches or surpasses SmoothQuant across most formats and models.The paper reports that direct MSE targeting lowers reconstruction error versus proxy objectives such as SINQ and that this translates to better downstream performance.Source: §2.2 · p. 3, Figure 1 · p. 3

3 QUANTIZATION PRELIMINARIES

The section defines quantization against a format-specific representable grid and describes symmetric, asymmetric, and block-wise variants. It specifies the scale constraints used by the MX and NV formats considered.

3 QUANTIZATION PRELIMINARIES

  • Quantization maps tensor values to a finite format-specific grid by scaling, projecting to the nearest representable value, and rescaling.Integer formats use round-and-clip, while floating-point formats use nearest-level projection on a non-uniform exponent-mantissa grid.Source: §3 · p. 3
  • Symmetric quantization uses zero-point z = 0 and a scale based on the maximum magnitude, whereas asymmetric quantization uses nonzero z to span the tensor’s range.The symmetric scale is Δ = max(|W|)/qmax.Source: §3 · p. 3
  • Block-wise quantization assigns independent scales to groups of contiguous row elements; MX formats use 32-element groups with E8M0 scales, while NVFP4 uses 16-element groups with FP8 UE4M3 scales.MX scales are restricted to powers of two.Source: §3 · p. 3

4 METHODOLOGY

DUALQUANT replaces activation-statistics-based calibration with weight-derived column rescaling while optimizing row and column scales for weight-reconstruction MSE. Variable splitting and block coordinate descent provide closed-form scale updates, with inference applying the column rescaling before activation quantization.

4.1 PROBLEM DESCRIPTION

  • The joint layer-output error contains activation- and weight-quantization terms, but minimizing them independently need not minimize the joint error, and direct optimization requires calibration data on X.SmoothQuant uses calibration-derived per-channel rescaling; DUALQUANT instead derives the rescaling from static weights.Source: §4.1 · pp. 3–4

4.2 DUAL-SCALE PARAMETERIZATION AND THE ACTIVATION FACTORIZATION

  • DUALQUANT adds a column scale to standard row scaling, distributing quantization burden across both dimensions and reducing peak element-wise reconstruction error.The quantized weight is reconstructed using inverse row and column scales around a format-specific quantizer.Source: §4.2 · p. 4
  • The column scales both reduce weight quantization error and rescale activations before quantization, using weight information rather than activation statistics.With quantized weights fixed, applying the scales measurably reduces activation quantization error; their correlation with activation statistics is heterogeneous and projection-dependent.Source: §4.2 · p. 4

4.3 MSE OPTIMIZATION VIA BLOCK COORDINATE DESCENT

  • Variable splitting separates scale roles for grid assignment and reconstruction, enabling BCD to alternate outer-scale updates and re-quantization with closed-form scalar subproblems.The updates are projected onto format-admissible scale sets after each update.Source: §4.3 · p. 5
  • Under the non-degenerate quantization assumption, if the quantized matrix stabilizes at a finite iteration, every limit point is coordinate-wise stationary.The assumption excludes iterations in which an entire weight row quantizes to zero; empirically, the relative change nears zero within about 15 iterations and MSE decreases monotonically to a plateau.Source: §4.3 · p. 5, Figure 2 · p. 6

4.4 INFERENCE-TIME PROCEDURE AND OVERHEAD

  • At inference, the column scale rescales activations before format-native row-wise quantization, while the row scale is absorbed into weight dequantization and adds no runtime cost.The per-channel multiply is applied inside the activation-quantization kernel without a separate kernel launch.Source: §4.4 · p. 6

5 EXPERIMENTS

Across Llama and Qwen evaluations, DUALQUANT generally matches or surpasses calibration-based SmoothQuant on joint 4-bit weight-and-activation quantization, while avoiding calibration data. Packed W4A4 deployment and scale ablations further test its accuracy, efficiency, and design choices.

5 EXPERIMENTS

  • Across the evaluated models and formats, the setup covers Qwen and Llama models, four 4-bit formats, perplexity datasets, and five zero-shot tasks.Calibration-dependent baselines use 256 WikiText-2 training sequences; evaluation sets are disjoint, and packed inference efficiency is measured separately from fake quantization.Source: §5 · p. 6

5.1 JOINT WEIGHT-AND-ACTIVATION QUANTIZATION

  • DUALQUANT matches or surpasses SmoothQuant on Llama perplexity across most formats and models, with consistent trends on Qwen and especially large advantages over calibration-free baselines on MX and NV formats.On NVFP4, SINQ+ and QuaRot diverge; the authors attribute their failures to format-unaware scales and incompatible activation rotations, respectively.Source: Table 1 · p. 7, §5.1 · p. 7
  • On Llama-3.1-8B zero-shot tasks, DUALQUANT achieves the best calibration-free accuracy across nearly all settings and beats every baseline on all five NVFP4 tasks.On INT4, it leads calibration-free methods on MMLU, HellaSwag, and ARC-Easy; Qwen3-14B results show similar trends.Source: Table 2 · p. 8, §5.1 · p. 8

5.2 RUNTIME AND MEMORY ANALYSIS

  • For packed Llama-3.1-8B W4A4 on an RTX 4090, speedup reaches 2.46× and memory use falls by 2.51–2.61× versus BF16.The packed path records WikiText-2 perplexity of 6.74 versus 6.73 for fake quantization; activation quantization is the only per-step quantization overhead.Source: §5.2 · pp. 8–9, Table 3 · p. 9

5.3 ABLATION STUDIES

  • DUALQUANT’s weight-derived β aligns near-perfectly with SmoothQuant’s scale at λ=0, while correlation decreases toward the activation-dominated λ=1.0 setting.The authors caution that correlation is descriptive and does not imply equivalent quantization behavior.Source: Table 5 · p. 9, §5.3 · p. 9
  • Combining row and column scales gives the best ablation results, with roughly additive gains because the scales address complementary error sources.Column scaling yields consistently larger gains than row scaling alone; row scaling helps primarily on NVFP4, whose FP8 scale grid is finer than MXFP4’s E8M0 grid.Source: §5.3 · p. 9, Table 4 · p. 9
  • The experiments use 15 BCD iterations, with initialization chosen to reduce the iterations needed despite convergence from any starting point.The initialization uses format-native row absmax scales and per-column L1-norm scales.Source: §5.3 · p. 9

6 CONCLUSION

DUALQUANT jointly quantizes weights and activations without calibration data, using weight-reconstruction MSE and BCD with closed-form updates across INT4, MX, and NVFP4 formats.

6 CONCLUSION

  • DUALQUANT matches or surpasses SmoothQuant and outperforms calibration-free baselines on Llama and Qwen, while its packed W4A4 path reduces memory use by up to 2.61× and raises throughput by up to 2.46× over BF16.The packed path is validated on an RTX 4090 for Llama-3.1-8B.Source: §6 · p. 9

A.1 VARIABLE SPLITTING

  • Variable splitting gives each scale separate inner and outer copies for grid assignment and reconstruction, then enforces consistency between them.This separates the conflicting roles that make joint scale optimization non-differentiable and non-convex.Source: §A.1 · p. 13

A.2 BLOCK COORDINATE DESCENT

  • BCD alternates optimization of variable blocks while holding the others fixed; here, the outer row and column scales have independent scalar least-squares updates.Each update has a closed-form solution, making BCD an efficient solver for the dual-scale problem.Source: §A.2 · p. 13

B CLOSED-FORM OPTIMAL SOLUTIONS FOR SCALES

  • With the quantized matrix and other scale fixed, each outer row or column scale is updated by solving its own least-squares problem and projecting to the nearest representable value.The row and column updates are derived separately as closed-form solutions.Source: B CLOSED-FORM OPTIMAL SOLUTIONS FOR… · p. 14
  • The relative change in Q approaches zero by about iteration 16, while normalized MSE decreases monotonically and plateaus within about 10 iterations.The reported down_proj projection has the highest residual MSE among the compared projections.Source: Figure 3 · p. 15, B CLOSED-FORM OPTIMAL SOLUTIONS FOR… · p. 15

C PROOF OF THEOREM 1.

The convergence proof establishes monotone objective descent and coordinate-wise stationarity of limit points after the quantized matrix stabilizes. The appendix also explains why format-aware reconstruction optimization differs from SINQ’s matrix-imbalance approach.

C PROOF OF THEOREM 1.

  • After Q stabilizes, the proof establishes monotone objective descent and coordinate-wise stationarity of every limit point, assuming positive scales and unique block minimizers.The proof uses continuity, unique per-block minimizers, and bounded iterates in the post-stabilization regime.Source: C PROOF OF THEOREM 1 · p. 16

D.1 BACKGROUND

  • SINQ minimizes matrix imbalance by alternating row and column normalization, and its original formats store per-block scales in unconstrained BF16.The imbalance ratio compares the largest and smallest per-axis standard deviations; a value of 1 denotes a perfectly balanced matrix.Source: §D.1 · p. 17

D.2 EXTENDING SINQ TO W+A AND MX/NV FORMATS

  • The MX/NV extension projects SINQ’s format-unaware scales after optimization, degrading MX results and causing divergence on NVFP4; the authors call these extensions best-effort baselines.The extensions are the paper authors’ own and were not validated by the original authors.Source: §D.2 · p. 17

D.3 WHY SINKHORN ITERATION DIVERGES UNDER FORMAT CONSTRAINTS

  • Under discrete scale constraints, projected Sinkhorn iterates can oscillate between nearby representable values instead of converging; DUALQUANT derives updates within the format constraint and preserves monotone descent.The paper reports stabilization within about 12–16 iterations for the evaluated MXINT4 model-layer settings.Source: C PROOF OF THEOREM 1 · p. 16, §D.3 · p. 17

D.4 COMPARISON OF OBJECTIVES

  • Matrix imbalance does not measure post-quantization reconstruction error, whereas DUALQUANT’s MSE objective includes the format-specific quantizer and adapts scale assignment to non-uniform grids.The authors connect reconstruction MSE more directly than imbalance proxies to downstream perplexity and accuracy.Source: §D.4 · p. 18

E ISOLATED ACTIVATION-SIDE EFFECT OF β

With quantized weights held fixed, applying β improves activation SQNR across the tested projections. The appendix also describes the calibration and quantization procedures used for the compared baselines.

E ISOLATED ACTIVATION-SIDE EFFECT OF β

  • Applying β raises activation SQNR by +0.33 dB on average across projections and +0.53 dB at block output, with gains across all seven projection types.The largest gains are +0.81 dB for k_proj and +0.65 dB for v_proj; quantized weights were held fixed in the comparison.Source: E ISOLATED ACTIVATION-SIDE EFFECT OF β · p. 18, Table 6 · p. 18

F ADDITIONAL DETAILS ON BASELINES

  • The baselines include calibration-based SmoothQuant and AWQ+, calibration-free RTN and QuaRot, and SINQ+ extended by the authors to W+A and MX/NV formats.The calibration-based methods use 256 sequences from WikiText-2 training, disjoint from evaluation splits.Source: F ADDITIONAL DETAILS ON BASELINES · p. 19

G ADDITIONAL W+A QUANTIZATION RESULTS

  • Supplementary results report 4-bit W+A quantization on Qwen and 8-bit W+A quantization across both model families, supporting generalization across architectures, formats, and bit-widths.The reported scope complements the main experiments.Source: G ADDITIONAL W+A QUANTIZATION RESULTS · p. 19

G.1 4-BIT W+A QUANTIZATION

Across Qwen 4-bit evaluations, DUALQUANT is a strong calibration-free method, often matching or surpassing SmoothQuant on MX and NV formats and performing well on downstream tasks. At 8 bits, it remains near-lossless on MX formats.

G.1 4-BIT W+A QUANTIZATION

  • On Qwen 4-bit WikiText-2 and C4 evaluations, DUALQUANT often matches or surpasses SmoothQuant on MX and NV, while outperforming calibration-free baselines in most settings.On NVFP4 WikiText-2, it scores 6.53 vs. 6.70 for Qwen2.5-7B and 23.77 vs. 24.16 for Qwen3-0.6B; SINQ+ diverges on all three evaluated models.Source: §G.1 · pp. 19–20, Table 7 · p. 20
  • On Qwen3-14B downstream tasks, DUALQUANT beats both calibration-free baselines on every MXFP4 and MXINT4 task and stays close to SmoothQuant, surpassing it on WinoGrande.On MXFP4 WinoGrande, accuracy is 72.06 vs. 69.14; on MXINT4 it is 70.64 vs. 66.30.Source: §G.1 · pp. 20–21, Table 8 · p. 21

G.2 8-BIT W+A QUANTIZATION

  • At 8 bits, DUALQUANT is effectively lossless and matches or surpasses SmoothQuant on MXINT8 and MXFP8 for every model except MXFP8 Qwen3-14B, where it trails by 0.01.For MXINT8 Qwen2.5-7B, WikiText-2 perplexity is 6.15 vs. 6.15 for BF16.Source: §G.2 · p. 21

H WEIGHT-ONLY QUANTIZATION RESULTS

In weight-only evaluation, DUALQUANT achieves the best calibration-free results on nearly all format–model combinations and compares closely with calibration-based methods. Its advantage over SINQ is especially pronounced for MX and NV formats.

H WEIGHT-ONLY QUANTIZATION RESULTS

  • Table 10 evaluates weight-only quantization with activations kept in BF16 to test whether dual scaling also yields competitive weight reconstruction.DUALQUANT applies β-rescaling without calibration data.Source: H WEIGHT-ONLY QUANTIZATION RESULTS · p. 21
  • DUALQUANT achieves the best calibration-free results on nearly all format–model combinations, with Qwen3-14B exceptions on INT4 and NVFP4.It closely matches calibration-based AWQ on INT4 and MX formats and remains within a small margin of GPTQ without calibration data or Hessian computation.Source: H WEIGHT-ONLY QUANTIZATION RESULTS · p. 22
  • On Qwen3-0.6B, SINQ reaches 112.00 on MXINT4 and 94.84 on MXFP4, where DUALQUANT substantially outperforms it.The passage identifies these as severe degradations from SINQ's format-unaware scales.Source: H WEIGHT-ONLY QUANTIZATION RESULTS · p. 22
  • DUALQUANT outperforms the original SINQ on nearly all weight-only format–model combinations, including INT4 on three of four models.SINQ’s output scales are projected to MX and NV formats after its algorithm completes; the reported gap widens on those block formats.Source: H WEIGHT-ONLY QUANTIZATION RESULTS · p. 22

I REAL PACKED W4A4 DEPLOYMENT

The packed W4A4 path for Llama-3.1-8B closely reproduces fake-quantization accuracy. Its INT4 deployment keeps quantized weights packed during inference and reduces the reported weight footprint.

I REAL PACKED W4A4 DEPLOYMENT

  • The implementation targets INT4 with group size G=64 and packs weights offline for all 224 linear layers into the Nunchaku/SVDQuant packed W4A4 backend.A CUDA kernel folds β into activation quantization, and an allocation-free GEMM entry point enables CUDA Graph capture; embedding, lm_head, and normalization remain BF16.Source: I REAL PACKED W4A4 DEPLOYMENT · p. 22
  • Measurements use one RTX 4090, an 82-token prefill prompt without KV cache, and 100 generated decode tokens.WikiText-2 perplexity uses sequence length 8192; latency and memory measurements are from warm runs stable within 0.2%.Source: I REAL PACKED W4A4 DEPLOYMENT · p. 23
  • 6.74 WikiText-2 perplexity for packed W4A4 compares with 6.73 for fake quantization, a difference of 0.01.The BF16 baseline measured with the same harness has perplexity 5.61.Source: I REAL PACKED W4A4 DEPLOYMENT · p. 23
  • 8.82 ms versus 17.33 ms for quantized linear-layer execution during batch-1 prefill; at batch 16, the times are 32.98 ms versus 121.49 ms.Activation quantization, including β pre-scaling, costs 0.70–3.09 ms and accounts for 4.3–6.1% of total W4A4 GPU time.Source: I REAL PACKED W4A4 DEPLOYMENT · p. 23, Table 11 · p. 24
  • 5.92 GB total weight footprint for DUALQUANT INT4 compares with 16.20 GB for BF16 on Llama-3.1-8B.The 224 linear layers use 3.49 GB plus 0.22 GB for group scales; BF16 embedding and lm_head use 2.10 GB, while activation and KV-cache components remain BF16.Source: I REAL PACKED W4A4 DEPLOYMENT · pp. 23–24, Table 12 · p. 24

J LICENSES FOR EXISTING ASSETS

The work uses publicly available models, datasets, and software libraries under their respective licenses, solely for academic research purposes.

J LICENSES FOR EXISTING ASSETS

Related papers

Loading 2610.13767v1…