Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs

1 Introduction

• LLM deployment on constrained devices is difficult because models require substantial memory and storage and impose computational demands that hinder real-time use.The introduction frames quantization as important for more efficient deployment.Source: §1 · p. 1
• QAT can improve accuracy over PTQ but adds training complexity, time, and hyperparameter tuning, while applying it to LLMs is resource-intensive.PTQ avoids simulated training or fine-tuning but is susceptible to significant accuracy drops.Source: §1 · p. 1
• Weight-only quantization is presented as practical for LLMs because activation quantization is challenging and token generation often faces memory-bandwidth limits.It reduces memory-footprint and bandwidth demands by quantizing weights while retaining activations in floating-point precision.Source: §1 · p. 1
• RTN rounds each weight independently and misses weight and activation relationships, while adaptive rounding may be inaccurate when higher-order loss terms matter.The cited concern is that rounding can substantially change weights, making a second-order-only approximation inadequate.Source: §1 · pp. 1–2
• SignRound combines QAT and PTQ strengths by using SignSGD to tune weight rounding and clipping without additional inference overhead.SignSGD is selected for reaching an optimal rounding solution in limited steps with minimal hyperparameter tuning.Source: §1 · p. 2
• SignRound improves performance over recent methods across 2- to 4-bit configurations and achieves nearly lossless 4-bit results in most scenarios.The authors also report strong generalization across models and further gains from constrained model-specific hyperparameter tuning.Source: §1 · p. 2

Quantization Aware Training; Post-training Quantization (PTQ)

• QAT methods enable fine-tuning and often improve accuracy over PTQ, whereas PTQ simplifies quantization by avoiding additional training and has low resource requirements.The passages also describe QAT as resource-intensive for LLMs and PTQ as vulnerable to accuracy drops.Source: §2 · p. 2

Large Language Models Quantization

• LLM quantization methods include mixed precision, additive or low-rank techniques, channel clustering, QAT, and equivalent transformations.LRQ learns fewer parameters while allowing individual weight scaling to improve generalization capability.Source: §2 · p. 2

Weight Only Quantization

• Weight-only methods include GPTQ, AWQ, TEQ, OmniQuant, and HQQ, while some alternatives use extra inference overhead to improve accuracy.This work focuses on methods without inference overhead; the figure depicts SignRound tuning rounding and clipping through block-wise output reconstruction.Source: §2 · pp. 2–3, Figure 1 · p. 3
• SignRound’s illustration contrasts RTN’s direct rounding with signed-gradient tuning of rounding and weight clipping through block-wise output reconstruction.The caption describes lightweight forward and backward steps and identifies quantization and dequantization as standard operations.Source: Figure 1 · p. 3

Rounding Methods

• Adaptive Rounding uses a second-order Taylor approximation, while FlexRound adds element-wise division and Oscillation-free discusses possible weight oscillation from learnable parameters.The passage also describes AQuant as making the rounding border depend on activation values.Source: §2 · p. 3

Signed Gradient Descent

• Sign-based gradient methods are used in particular settings, with cited work linking their advantages to Hessian structure and reporting variants with faster convergence or optimal asymptotic rate under stated conditions.The cited stochastic sign-descent result assumes bounded variance.Source: §2 · p. 3

3 Methodology

• Algorithm 1 takes calibration data, learning rate, step count, model, block module, and batch size, and outputs the best V, α, and β.The pseudocode updates the best setting when the loss is lower.Source: §3 · p. 4
• Trainable V adjusts rounding values beyond independent RTN rounding, which loses the ability to model correlations among weights or activations.The method builds on prior rounding research by introducing V for this adjustment.Source: §3 · pp. 3–4
• Trainable α and β fine-tune weight clipping, while block-wise output reconstruction trains these parameters.The optimization uses each block's input X; the Frobenius norm is used in the reconstruction expression.Source: §3 · p. 4
• SignSGD updates parameters using gradient signs, suited to bounded rounding and clipping ranges and requiring less memory and computation than Adam.The rounding range is [-0.5, 0.5], while the clipping range is [0, 1].Source: §3 · p. 4
• The default setup uses 200 steps, a 5e-3 learning rate, and linear weight decay, with the same optimizer hyperparameters across experiments unless stated otherwise.The stated maximum adjustment per parameter is 200 × 0.005/2 = 0.5.Source: §3 · p. 4

4.2 Comparing With Recent Methods

• SignRound achieved scores of 30/32 against GPTQ, 27/32 against AWQ, 15/16 against HQQ, and 29/32 against OmniQuant across tested models and quantization settings.The comparison covers LLaMA-V1, LLaMA-V2, and Mistral-7B at W4G-1, W4G128, W3G128, and W2G128.Source: Table 1 · p. 5, §4.2 · p. 7

4.1 Experimental Settings

• The evaluation uses 11 zero-shot tasks as its primary accuracy metric, with perplexity on three language datasets treated as supplementary because outliers can inflate it.Models included LLaMA-V1, LLaMA-V2, and Mistral, spanning 7B to 70B parameters.Source: §4.1 · pp. 5–6

4.2 Comparing With Recent Methods

• At W2G128, SignRound yielded absolute average accuracy improvements ranging from 6.91% to 33.22%; performance could be enhanced by tuning hyperparameters from a selection of eight choices, denoted as ours*.Source: §4.2 · p. 7

4.3 Comparing with Rounding Methods

• SignRound achieved better results in 200 steps than other rounding methods required in 5000 steps, with comparison limited to models of 13B parameters or smaller.The other rounding-method implementations were notably slow, especially under their official 5000-step settings.Source: §4.3 · p. 7

4.4 Ablation Studies

• SignSGD outperformed Adam on average accuracy; rounding and clip tuning each benefited over RTN, although clip tuning sometimes lowered W2G128 accuracy, and default hyperparameters were balanced.The ablations examine average accuracy across 11 tasks; hyperparameter sensitivity tests cover calibration, batch size, tuning steps, and learning rate.Source: §4.4 · pp. 7–8

4.5 Generalization to Other Models

• With 1000 tuning steps and model-wise hyperparameters, int4 models across Gemma, Phi, Mistral, Mixtral, and Llama3 stayed within 1% of FP16 or BF16 accuracy.The generalization evaluation used an updated lm-eval-harness version and real quantized models, which may cause minor discrepancies from other benchmark data.Source: §4.5 · p. 8

• SignRound outperforms other quantization methods across various models and weight bit widths in the majority of tested scenarios.Source: §5 · p. 8
• SignRound tunes weight rounding and clipping with signed gradient descent in 200 steps, quantizing LLaMA-V2-70B in approximately 2.5 hours.Source: §5 · p. 8
• SignRound shows promising generation capabilities in recent models, with performance further enhanced by model-specific hyperparameter tuning.Source: §5 · p. 8

• A noticeable accuracy gap from the original model remains at ultra-low bit widths, particularly with 2-bit quantization.Source: §6 · p. 9
• The authors leave exploration of non-uniform and mixed-precision quantization for future work as potential ways to address the gap.Source: §6 · p. 9

• The study uses open-source models and publicly available datasets, with minimal fine-tuning on original models, and reports no potential ethical implications from the method's technical details.Source: §7 · p. 9

• Disabling low_gpu_mem_usage speeds SignRound tuning but raises memory use; LLaMA-2-70B still runs on the 80GB A100.Source: A Quantization Cost · p. 12
• The authors report their methods as faster than other methods, while noting that HQQ is exceptionally fast.Table 9 also compares FlexRound, Adaptive Round, and SignRound costs.Source: A Quantization Cost · p. 12
• The quantization-cost comparison measures methods on a single NVIDIA A100 GPU with 80GB of memory.Table 8 reports costs at W4G-1 for LLaMA-V2; OmniQuant 70B uses 128 calibration samples and the other methods use 512.Source: Table 8 · p. 12, A Quantization Cost · p. 12

B View of distribution of tuned parameters

• The distributions cover V in Eq. 3 and α and β in Eq. 4 for Mistral-7B-v0.1 and Llama-2-7B at W4G-1.Figure 2 reports these parameter distributions by layer.Source: B View of distribution of tuned… · p. 12
• Most layers have flat distributions, while a few layers at the beginning and end are exceptions.The reported pattern applies to the distributions of V, α, and β.Source: B View of distribution of tuned… · p. 12
• Table 9 reports quantization times for rounding methods at W4G-1 with 200 steps for LLaMA V2 models and Mistral-7B.The passage gives the table’s scope, not comparative time values.Source: Table 9 · p. 12

C More results

• SignRound shows a clear accuracy advantage, particularly in ultra-low-bit quantization, with state-of-the-art performance versus several popular weight-quantization methods.The accuracy evaluations cover 11 zero-shot tasks for LLaMA and Mistral at W4G-1, W4G128, W3G128, and W2G128.Source: C More results · p. 12, Table 10 · p. 13, Table 11 · p. 14
• Figure 2 depicts V, α, and β distributions for Mistral-7B-v0.1 and Llama-2-7B at W4G-1, with colors indexing layers from shallow blue to deeper red.The caption describes the layer-color encoding but does not state distribution outcomes.Source: Figure 2 · p. 17
• SignRound outperformed all other methods in 83 of 124 PPL scenarios, while quantization algorithms including SignRound showed sensitivity across models and tasks.The PPL evaluation covers Wikitext2, PTB, C4, and Wikitext tasks for LLaMA and Mistral.Source: C More results · p. 12, Table 14 · p. 18
• Table 15 reports detailed accuracy results across 11 tasks, described in Section 4.1, for LLMs evaluated at W4G128 using 1,000 steps.Source: Table 15 · p. 19