PaperlayerView exact revision on arXiv

Source-linked AI summary

SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs

Wenhua Cheng, Weiwei Zhang, Heng Guo, Haihao Shen, Zaner Ma

arXiv:2512.04746v2cs.CLcs.AI

TL;DR

LLM scaling creates memory, bandwidth, and latency barriers, while extremely low-bit quantization remains difficult to recover accurately. SignRoundV2 combines gradient- and distortion-guided mixed precision with lightweight tuning stabilization; it reports competitive weight-only accuracy against high-cost alternatives at lower cost and strong weight-activation performance.

  • Problem

    Extremely low-bit quantization can severely degrade LLM performance, while specialized training or distillation for very low-bit models is less suited to fast, scalable, hardware-efficient PTQ.

    Source: §1 · pp. 1–2

  • Method

    SignRoundV2 uses gradient information and quantization-induced deviations for layer-wise bit allocation, alongside loss filtering and pre-tuning scale search.

    Source: §1 · pp. 2–3

  • Results

    SignRoundV2 achieves competitive accuracy to recent high-cost QAT and vector-quantization methods in weight-only quantization at substantially lower quantization cost, while maintaining strong weight-activation performance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    SignRoundV2 maintains high-fidelity performance at 4–5 bits and yields robust results at 2 bits under aggressive compression.

    Source: §5 · p. 9

    Uniform 2-bit settings still show a noticeable accuracy gap from full-precision baselines; bit-widths also remain static during tuning.

    Source: §6 · p. 10

Abstract

from arXiv · show

Extremely low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2 bits and even at 4 bits (e.g., MXFP4). We present SignRoundV2, a post-training quantization framework designed to maintain high performance even under aggressive compression. SignRoundV2 introduces (1) a simple yet efficient adaptive mixed-precision strategy that leverages gradient information and quantization-induced reconstruction errors to guide layer-wise bit allocation, and (2) a set of lightweight stabilization techniques, including loss filtering and a pre-tuning scale search, to improve tuning effectiveness in extremely low-bit regimes. Our approach takes a significant step toward closing the performance gap between quantized and full-precision models. Experimental results across diverse LLMs demonstrate that SignRoundV2 achieves near-lossless performance in mixed MXFP settings, narrowing the gap to $\sim$1\% at an average of 4.5 bits, while substantially improving accuracy in challenging 2-bit weight-only quantization. The source code is available at \url{https://github.com/intel/auto-round}.

1 Introduction

LLM scaling raises memory, bandwidth, and latency barriers, while extremely low-bit quantization remains difficult to recover accurately. SignRoundV2 combines adaptive mixed precision with lightweight stabilization and reports competitive accuracy against high-cost alternatives.

  • LLM scaling increases memory consumption, bandwidth pressure, and inference latency, constraining deployment in resource-limited settings.Source: §1 · p. 1
  • Specialized training or distillation can support roughly 1.58-bit representations, but these approaches are less suited to fast, scalable, hardware-efficient post-training quantization.Source: §1 · p. 2
  • Existing layer-sensitivity heuristics can correlate imperfectly, model-wise distortion calibration can be slow, and learning-based allocation adds compute overhead.Source: §1 · p. 2
  • Its adaptive mixed-precision strategy combines gradient information and quantization-induced deviations to guide layer-wise bit allocation via dynamic programming.Source: §1 · p. 2
  • Loss filtering and a lightweight pre-tuning scale search improve stability and accuracy in extremely low-bit regimes.Source: §1 · pp. 2–3
  • SignRoundV2 achieves competitive accuracy to recent high-cost QAT and vector-quantization methods in weight-only quantization, at substantially lower quantization cost, while maintaining strong weight-activation performance.Source: §1 · p. 3

2 Related Work

Related work contrasts the cost and accuracy trade-offs of QAT, PTQ, and adaptive mixed-precision methods, alongside approaches to optimized rounding.

  • QAT optimizes task loss and may update non-quantizer parameters, but can be sensitive to fine-tuning data, require careful tuning, and incur higher cost and data demands.Source: §2 · p. 3
  • PTQ avoids task-loss optimization and additional parameter fine-tuning, offering a resource-efficient pipeline, but weight-only methods still lose accuracy at extremely low bit-widths such as 2 bits.Source: §2 · p. 3
  • Hessian-based and reinforcement-learning assignment can be prohibitively expensive at LLM scale; AMQ remains relatively slow, while SFMP and MicroMix may require specialized hardware or kernels.Source: §2 · p. 3
  • AdaRound [55] formulates rounding as per-weight binary optimization with a Taylor approximation of task-loss perturbations; SignRound [10] jointly optimizes rounding and clipping with sign-based gradient descent.Source: §2 · pp. 3–4

3 Methodology

SignRoundV2’s methodology uses DeltaLoss to measure quantization sensitivity, allocates layer-wise bit-widths under a target budget, and stabilizes quantization-parameter tuning. It also specifies the calibration and optimization settings used for these procedures.

3.1 DeltaLoss Sensitivity Metric

  • DeltaLoss sensitivity varies across data types, and down_proj layers show consistently higher sensitivity, making manual bit assignment and block-wise fallback potentially suboptimal.The metric combines quantization deviations with task-aware gradients so layers with both large perturbations and large gradients receive higher precision under a fixed budget.Source: §3.1 · pp. 4–5, Figure 2 · p. 4

3.2 Layer-wise Bit Allocation

  • Layer-wise bit allocation assigns one allowed bit-width to each layer while minimizing DeltaLoss under a target average-bit budget, solvable by dynamic programming or integer linear programming.The formulation assumes identical weight and activation bit-widths, though it can be extended to different bit-widths or data types across layers.Source: §3.2 · pp. 5–6

3.3 Tuning Quantization Parameters

  • A pre-tuning scale search initializes quantization scales from candidate values informed by channel-wise average activations, then refines the selected scale with a learnable parameter.For symmetric quantization, candidates are searched around 2^(b−1); for W2A16, t=0.9 and δ=0.01, with α constrained to [0, 2.0].Source: §3.3 · p. 6

3.4 Tuning Stabilization via Loss Filtering

  • Tuning stability is improved by excluding the top 0.1% of batch losses when computing the block-output mean squared error.The effective optimization loss averages the remaining losses after removing the largest values.Source: §3.4 · p. 6

3.5 Hyperparameters

  • DeltaLoss uses 16 calibration samples of sequence length 256; tuning uses 200 steps per transformer block, batch size 8, and sequence length 2048.The lower-cost recipe reduces tuning calibration samples from 512 to 128; Ours* instead uses 500 steps, learning rate 2.0/steps, and 512 samples, with AMP throughout.Source: §3.5 · pp. 6–7

4 Experiments

SignRoundV2 performs strongly across weight-only, adaptive mixed-bit, and MXFP quantization settings, with reported accuracy gains at very low bit-widths and near-full-precision performance at higher average MXFP bit-widths. The experiments also evaluate stabilization techniques and quantization costs.

4.1 Experimental Setup

  • Evaluations cover LLaMA and Qwen across low-bit and mixed-precision settings using zero-shot standard benchmarks.The study uses the Pile for calibration, symmetric quantization, and NVIDIA A100 (80GB) GPUs.Source: §4.1 · p. 7

4.2 Comparison of Weight-Only Quantization Methods

  • In uniform W2A16 settings, SignRoundV2 consistently outperforms established PTQ baselines and is competitive with higher-complexity methods on large-scale models.At slightly higher average bit-widths, such as 2.5 bits, its adaptive layer-wise strategy preserves model quality.Source: §4.2 · pp. 7–8

4.3 Comparison of Adaptive Bit-Width Quantization

  • On Llama-3.1-8B at 2.5 bits, SignRoundV2 improves recovery rate over AMQ by 12.24 percentage points.SignRoundV2 is competitive with SFMP while using per-tensor granularity that avoids SFMP’s per-group kernel overhead.Source: §4.3 · p. 8, Table 2 · p. 8

4.5 Ablation Studies

  • Loss filtering boosts average accuracy by 0.82% on Qwen3, while combining it with proper initialization yields a 3.41% absolute improvement on Llama3.1.Both strategies enhance performance under W2A16G64.Source: §4.5 · pp. 8–9, Table 4 · p. 9

4.4 Performance under MXFP Quantization

  • At average bit-widths of 4.5 and 5 bits, SignRoundV2 approaches full-precision performance on most evaluated models under MXFP quantization.The study reports performance gains in the 4-bit regime, surpassing RTN and outperforming SignRoundV1 in most scenarios.Source: §4.4 · p. 8, Table 3 · p. 9

4.6 Quantization Cost

  • For Llama2-70B, SignRoundV1 takes 2.2 hours, SignRoundV2 2.5 hours, and the enhanced version (Ours*) 6 hours.QAT-based and vector-quantization methods typically require tens to hundreds of GPU hours; Table 5 compares runtime cost.Source: Table 5 · p. 9, §4.6 · p. 9
  • DeltaLoss’s additional VRAM and runtime overhead for weight-only quantization, such as W2A16, remains comparable to MXFP4.Table 6 reports adaptive bit-width assignment costs excluding quantization overhead.Source: Table 6 · p. 9, §4.6 · p. 9

5 Conclusion

SignRoundV2 is a post-training quantization framework for extremely low-bit LLM deployment. The conclusion reports high-fidelity performance at 4–5 bits and robust results at 2 bits.

5 Conclusion

  • SignRoundV2 combines a gradient-informed sensitivity metric with lightweight pre-tuning to support stable quantization under aggressive compression.The framework is designed for extremely low-bit LLM deployment.Source: §5 · p. 9
  • SignRoundV2 maintains high-fidelity performance at 4–5 bits and yields robust results even at 2 bits.Source: §5 · p. 9
  • PTQ can significantly reduce LLM computational and memory footprints for large-scale inference.Source: §5 · p. 10

6 Limitations

SignRoundV2 has limitations in uniform 2-bit settings, bit-allocation optimization, and compatibility with inference-only frameworks.

6 Limitations

  • Uniform 2-bit SignRoundV2 still has a noticeable accuracy gap versus full-precision baselines.The passage says extremely low-bit PTQ remains challenging without mixed precision or additional training.Source: §6 · p. 10
  • Bit-width configuration is fixed before block-wise tuning and remains static during optimization.Jointly updating allocation and rounding parameters may improve accuracy, but would increase search complexity.Source: §6 · p. 10
  • DeltaLoss requires gradient computation, limiting its immediate applicability in inference-only frameworks such as ONNX Runtime.Source: §6 · p. 10

Ethics Statement

The research uses existing open-source models and public datasets under their respective licenses and usage terms, and reports no significant new ethical risks.

Ethics Statement

  • The research uses open-source models and publicly available datasets under their respective licenses and usage terms.Source: Ethics Statement · p. 10
  • The method requires minimal tuning of pre-existing models and is not tied to specific high-risk applications.Source: Ethics Statement · p. 10
  • The authors report no significant new ethical risks and acknowledge the resource creators and maintainers with proper citations.Source: Ethics Statement · p. 10

A Comparison with Naive Mixed-bit Methods

SignRoundV2's DeltaLoss strategy without tuning outperforms head- and tail-layer allocation heuristics across the reported precision budgets and models.

A Comparison with Naive Mixed-bit Methods

B Comparison with Prior Mixed-bit Methods

The appendix provides task-level accuracy tables for mixed-bit weight-only quantization and mixed MXFP4/6 quantization across multiple models.

B Comparison with Prior Mixed-bit Methods

  • The mixed-bit weight-only comparison reports six-task accuracies and average accuracy for Llama-3.1-8B, Qwen3-8B, Qwen3-14B, and Qwen3-32B.Source: B Comparison with Prior Mixed-bit… · p. 16
  • The Qwen3-8B mixed-bit table identifies its gray rows as uniform 3-bit and 4-bit SignRoundV2 baselines.Source: Table 10 · p. 16

C.1 Mixed MXFP4/6 Results

  • The mixed MXFP4/6 subsection lists task-wise accuracies across 10 tasks for each model in Tables 13–17.Source: §C.1 · p. 16

C.2 Mixed MXFP4/8 Results

SignRoundV2 generally improves over RTN in mixed MXFP4/8 quantization and achieves the best recovery in most higher-bit settings. Under NVFP4, it leads on four of five models and remains competitive on Qwen3-32B.

C.2 Mixed MXFP4/8 Results

  • Across 10 tasks, SignRoundV2 generally improves over RTN and is competitive with or better than SRV1 at 4 bits; at 4.5–6 bits, it achieves the best recovery in most scenarios.Table 18 reports average accuracies and recovery rates; Avg. Bits is average per-weight bit-width, and DL means untuned DeltaLoss.Source: §C.2 · p. 19, Table 18 · p. 19
  • Table 19 provides task-level accuracies for Llama-3.1-8B-Instruct with mixed MXFP4/8 precision.It complements the aggregate results with detailed accuracies across 10 tasks.Source: Table 19 · p. 19, §C.2 · p. 19

E INT2/4 Mixed-bit Results

This appendix gives task-wise results for INT2/4 mixed-bit weight-only quantization across uniform and mixed-precision settings. It covers five tasks and provides separate detailed tables for Llama3 and Llama2 models.

E INT2/4 Mixed-bit Results

  • The INT2/4 mixed-bit experiments report accuracies on ARC-Challenge, ARC-Easy, HellaSwag, PIQA, and WinoGrande across model scales and uniform and mixed-precision settings.These task-wise results complement averaged comparisons in Table 1 and provide a finer-grained view of method behavior under extreme low-bit quantization.Source: E INT2/4 Mixed-bit Results · p. 19
  • Table 25 reports detailed accuracies for Llama3 models under INT2/4 mixed precision.The table covers five tasks and reports average bit-width.Source: Table 25 · p. 22
  • Table 26 reports detailed accuracies for Llama2 models under INT2/4 mixed precision.The table covers five tasks and reports average bit-width.Source: Table 26 · p. 23
Loading 2512.04746v2…