PaperlayerView exact revision on arXiv

Source-linked AI summary

G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

Ruikang Liu, Haoli Bai, Yuxuan Sun, Qian Zhang, Wenzheng Cai, Yanqi Hao, Feiyu Wang, Weidong Zhong, Zhuang Wang, Tong Yang, Xiangsheng Zhou

arXiv:2609.31009v1cs.CLcs.AI

TL;DR

GPTQ-based PTQ methods lack either global supervision or current first-order and Hessian guidance as quantization proceeds. G2PTQ combines refreshed first- and second-order information under block-wise supervision; experiments report better full-precision alignment than state-of-the-art baselines.

  • Problem

    Existing methods either use local layer-wise objectives or global objectives with fixed Hessians and no first-order gradients.

    Source: §1 · pp. 1–2

  • Method

    G2PTQ uses block-wise supervision and refreshes first- and second-order estimates before quantizing each Transformer block.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Across various model families and bit-widths, G2PTQ achieves better alignment with full-precision models than state-of-the-art baselines.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The framework provides globally informed guidance for quantization weight updates without fine-tuning.

    Source: §1 · p. 2, §6 · p. 10

    FOEM’s Taylor gradient approximation is exact only for layer-wise MSE and introduces error for block-wise MSE or KL divergence.

    Source: §3 · p. 4

Abstract

from arXiv · show

Post-training quantization (PTQ) is a practical approach to reducing the memory and computational footprint of large language models (LLMs) without retraining. GPTQ-based methods have become the de facto standard, yet they suffer from two complementary limitations. Methods with local, layer-wise objectives lack global supervision; while methods with global objectives fix their Hessian estimates at the start and ignore first-order gradients, so their guidance grows stale as quantization proceeds. This paper presents G$^2$PTQ, a unified PTQ framework with Generalized Gradient Compensation that integrates both first- and second-order information under a globally supervised, block-wise optimization objective. By refreshing gradient and Hessian estimates before quantizing each Transformer block, G$^2$PTQ avoids the staleness of prior global methods. Furthermore, to stabilize the exact first-order compensation, we introduce a trust-region scaling mechanism that dynamically bounds the gradient step to prevent exploding weight updates. Finally, we derive efficient implementations for block-wise Hessian approximation and exact gradient compensation. Experimental results on various model families and bit-widths demonstrate that G$^2$PTQ enables better alignment with the full-precision model, outperforming state-of-the-art baselines. Code is available at: https://github.com/G2PTQ/G2PTQ.

1 INTRODUCTION

Existing GPTQ-based methods separate global objectives from first-order information, while fixed Hessians and approximate gradients limit their guidance. G2PTQ combines refreshed first- and second-order information under block-wise supervision, with experiments showing closer alignment to full-precision models.

1 INTRODUCTION

  • Global-objective methods keep Hessians fixed and omit first-order gradients, while first-order methods remain limited to local objectives.GPTQ-style local MSE objectives may yield suboptimal global performance; Taylor-approximated gradients can introduce error under expressive objectives.Source: §1 · pp. 1–2
  • G2PTQ combines first- and second-order quantization-error compensation under a globally supervised, block-wise objective.It uses block-wise MSE for intermediate blocks and KL divergence for the final block.Source: §1 · p. 2
  • G2PTQ refreshes gradient and Hessian estimates before each Transformer block and uses trust-region scaling to bound gradient compensation.It also derives efficient implementations for block-wise Hessian approximation and exact gradient compensation.Source: §1 · p. 2
  • Quantizing an 8B model takes 2.24 hours and 12.54 GB of memory on a single accelerator.These are reported as an example of the efficient implementation.Source: §1 · p. 2
  • Across diverse model families, G2PTQ achieves lower KL divergence than state-of-the-art baselines and improves downstream task accuracy.The reported alignment is with full-precision models.Source: §1 · p. 2

2 RELATED WORK

Related work builds from second-order quantization-error compensation toward both more efficient GPTQ variants and objectives with broader supervision. These approaches differ in how they compensate errors, model dependencies, and approximate global curvature.

2 RELATED WORK

  • PTQ is presented as simpler to deploy than QAT, and GPTQ is a widely adopted representative PTQ method.The broader related-work discussion is deferred to Appendix A.Source: §2 · p. 2

Optimal Brain Quantization

  • OBQ established second-order compensation by adjusting remaining weights after quantizing one weight; GPTQ improved its efficiency, QuIP derived LDLQ, and FOEM added a first-order term.OBQ and GPTQ form the foundation; QuIP and FOEM extend the line.Source: §2 · p. 2

From Local to Global Optimization Objectives

  • Global-objective methods replace local layer-wise MSE with end-to-end NLL or KL objectives, with OAC, GuidedQuant, and YAQA using different Hessian approximations.OAC assumes row-wise independence, GuidedQuant groups channels, and YAQA uses a Kronecker-product approximation.Source: §2 · pp. 2–3
  • GPTAQ adds cumulative quantization error, while BOA models attention-aware inter-layer dependencies to reduce attention-output distortion.GPTAQ compensates for errors in previously quantized linear layers; BOA derives attention-aware Hessian matrices.Source: §2 · pp. 2–3

3 PRELIMINARIES

The preliminaries describe GPTQ’s second-order weight compensation, FOEM’s first-order extension, and global-objective Hessian methods. Their complementary limitations motivate combining first- and second-order information under block-wise supervision.

GPTQ: Second-order Weight Compensation

  • GPTQ quantizes a weight and updates remaining weights in its row to compensate for the induced error using second-order information.It shares a Hessian across rows, applies Cholesky-based updates, and defers updates to remaining columns in batches.Source: §3 · p. 3

FOEM: First-order Gradient Compensation

  • FOEM restores the first-order term omitted by GPTQ’s convergence assumption, but approximates gradients with a Taylor expansion.When the gradient is zero, FOEM reduces to GPTQ.Source: §3 · pp. 3–4
  • FOEM’s Taylor gradient approximation is exact for layer-wise MSE but introduces error for block-wise MSE or KL divergence, confining it to local objectives.The passage identifies this as the limitation of FOEM’s approximation under more expressive objectives.Source: §3 · p. 4

GuidedQuant: Global-objective Hessian

  • GuidedQuant uses Fisher information to approximate a global NLL or KL Hessian and shares averaged Hessians across groups of output channels.It partitions channels into groups to reduce memory overhead.Source: §3 · p. 4
  • GuidedQuant fixes its Hessian before quantization and omits the first-order gradient term, leaving its estimates increasingly stale as weights change.FOEM has first-order information but a local objective, whereas GuidedQuant has a more expressive objective but stale Hessians and no first-order term.Source: §3 · p. 4

4.1 GENERALIZED GRADIENT COMPENSATION

  • G2PTQ addresses these complementary gaps by computing first- and second-order information under block-wise supervision.The section formulates the objective as matching quantized and full-precision Transformer-block outputs.Source: §3 · p. 4

4 METHOD

G2PTQ quantizes Transformer blocks by combining first- and second-order information under objectives that compare quantized and full-precision block outputs. It refreshes gradient and Hessian estimates block by block and uses trust-region scaling to bound exact gradient compensation.

4.1 GENERALIZED GRADIENT COMPENSATION

  • G2PTQ minimizes block-wise output discrepancy using block-wise MSE for intermediate blocks and KL divergence for the final block.These objectives provide stronger supervision than prior layer-wise MSE objectives.Source: §4.1 · p. 4
  • Fresh gradient and Hessian estimates are computed before each Transformer block is quantized, keeping compensation guidance aligned with the partially quantized model.Each estimate requires a backward pass through that block alone, rather than a full-model backward pass.Source: §4.1 · pp. 4–5
  • Block-wise MSE and KL Hessians are approximated with outer products of gradient vectors; separate or grouped Hessians reduce storage and computation.The method partitions output channels into groups and reuses one Hessian per group.Source: §4.1 · pp. 4–5
  • Trust-region scaling dynamically restricts the exact gradient step to a predefined loss-change budget, addressing oversized updates that can violate the Taylor approximation’s local validity.The exact gradient is computed by backpropagation rather than approximated with a first-order Taylor expansion.Source: §4.1 · p. 5

4.2 AN EFFICIENT IMPLEMENTATION

  • G2PTQ quantizes weight columns sequentially, using a Cholesky reformulation and row-wise trust-region scaling in the weight update.The algorithm groups columns into blocks and updates the remaining columns in a batched manner after each block.Source: §4.2 · pp. 6–7
  • A recursive computation of F(t) avoids repeated Hessian inversion and brings weight-update complexity in line with vanilla GPTQ.The implementation also uses lazy-batch updates, deferring updates to remaining columns in blocks to improve memory-bandwidth utilization.Source: §4.2 · p. 7

5 EXPERIMENTS

Experiments evaluate G2PTQ against representative quantization baselines across diverse model families, bit-widths, and weight-only and weight-activation settings. The reported results show gains across these settings, alongside ablations and calibration-resource measurements.

5.1 EXPERIMENTAL SETTINGS

  • Evaluation covers 13 dense and 2 MoE models ranging from 0.6B to 125B, using perplexity, KL divergence, and accuracy on seven commonsense QA benchmarks.Baselines include RTN, GPTQ, GuidedQuant, and GPTAQ; the comparison uses a uniform scalar quantizer and established rotation, clipping, and activation-reordering techniques.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • Across weight-only settings, G2PTQ variants achieve the best reported performance across evaluated metrics; under W2A16, G2PTQ* halves KL divergence versus GPTAQ, from 1.83 to 0.90, and raises QA accuracy by 6.45%.At higher bit-widths, G2PTQ further narrows the performance gap with full-precision models.Source: Table 1 · p. 8, §5.2 · p. 8
  • On LLaMA3-70B under 4-bit weight-activation quantization, G2PTQ improves QA accuracy over GPTAQ by 4.56%.The authors attribute the result to compensating for accumulated activation quantization errors through weight updates.Source: Table 2 · p. 8, §5.2 · p. 8
  • On the 125B Qwen3.8-Flash-Next model under 4-bit quantization, G2PTQ incurs only a 0.33% decrease in QA accuracy versus the full-precision model.The comparison covers two MoE models and reports that G2PTQ further improves accuracy over GPTQ.Source: Table 3 · p. 8, §5.2 · p. 9

5.3 DISCUSSIONS

  • On Qwen3-0.6B with 3-bit quantization, block-wise alignment raises average QA accuracy from 33.09% to 37.40% over layer-wise GPTQ; refresh plus exact gradient compensation adds another 1.75%.The ablation reports that omitting trust-region scaling causes the exact gradient step to diverge and downstream accuracy to collapse.Source: Table 4 · p. 9, §5.3 · p. 9
  • Calibration uses more memory and computation than GPTQ because G2PTQ performs backward passes, but it avoids end-to-end backpropagation and remains practical for deployment.Quantizing an 8B model takes 2.24 hours and 12.54 GB of memory on a single accelerator.Source: §5.3 · p. 9

6 CONCLUSIONS

G2PTQ combines refreshed first- and second-order information with block-wise supervision to guide quantization without fine-tuning. The appendices derive Hessian approximations and discuss when first-order gradient approximations are reliable.

6 CONCLUSIONS

  • G2PTQ refreshes first- and second-order information under block-wise supervision, and experiments across model families and bit-widths show better alignment with full-precision models than state-of-the-art baselines.The framework also derives efficient implementations for block-wise Hessian approximation and exact gradient compensation.Source: §6 · p. 10

A RELATED WORK

  • The related-work discussion covers remedies for outlier channels and massive activations, including channel smoothing, pivotal-token preservation, and architectural changes.It distinguishes fixed-channel activation outliers from large activations on pivotal tokens.Source: A RELATED WORK · p. 14

B.1 HESSIAN APPROXIMATION FOR KL DIVERGENCE AND MSE LOSS

  • For KL divergence, the Hessian approximation uses the Fisher information matrix; for block-wise MSE, it uses expected gradient outer products with Gaussian-noise targets.The KL derivation assumes quantized weights are close to full-precision weights; the MSE derivation uses noise ϵ ∼ N(0, dI).Source: §B.1 · pp. 15–16
  • A single forward and backward pass can approximate per-token gradient squares for Hessian computation when future-token gradient contributions are negligible under sparse causal attention.The derivation assumes squared gradients from future tokens with respect to earlier-token activations are approximately zero.Source: §B.1 · p. 16

B.2 DISCUSSIONS ON GRADIENT APPROXIMATION

  • The FOEM first-order Taylor gradient approximation is exact for layer-wise MSE but may incur error with other loss types.Exactness requires a strictly quadratic weight loss; global NLL, KL, and block-wise MSE objectives are non-quadratic, and the convergence assumption can fail for task-specific NLL under calibration-domain gaps.Source: §B.2 · pp. 17–18

B.3 ANALYSIS OF THE EXPLODING FIRST-ORDER WEIGHT UPDATE

Exact first-order compensation can produce unstable updates because its magnitude depends strongly on the relative loss scales. The paper bounds that contribution with trust-region scaling and derives batched update formulas.

B.3 ANALYSIS OF THE EXPLODING FIRST-ORDER WEIGHT UPDATE

  • The squared norm of exact first-order updates scales by a^2b^-4 when the two losses are scaled by a and b, making update magnitude unbounded and especially sensitive to small b.This scale sensitivity can destabilize optimization across architectures and quantization configurations.Source: §B.3 · p. 18

B.4 DERIVATION OF THE TRUST-REGION SCALING FACTOR

  • Trust-region scaling constrains the exact gradient step with β ∈ [0, 1], bounding its loss contribution relative to the GPTQ loss change.The selected scaling factor is the largest feasible β; when c ≤ 2α · |∆LGPTQ|, β = 1 requires no scaling.Source: §B.4 · pp. 18–19

B.5 EFFICIENT WEIGHT UPDATE WITH GRADIENT COMPENSATION

  • The weight-update derivation applies a recursive relation across a column block to obtain batched updates for the remaining columns.It also expresses the complete lazy-batch update formula through the block’s initial state.Source: §B.5 · pp. 20–21

C IMPLEMENTATION DETAILS

The implementation refreshes block-level information, tunes trust-region thresholds, and uses fused kernels and MoE-specific optimizations. Evaluation covers diverse model architectures and uses KL divergence as a degradation metric.

C.2 TRUST-REGION THRESHOLD AUTOTUNING

  • The trust-region threshold α is grid-searched per block over [0, 3] in 0.1 steps using 64 calibration samples, with early stopping when loss increases.The block-wise quantization loss is used to choose the threshold.Source: §C.2 · p. 21

C.3 QUANTIZATION EFFICIENCY OPTIMIZATION

  • Kernel fusion reduces redundant memory traffic, while MoE quantization groups small experts, distributes work across devices, and offloads calibration hidden states to conserve GPU memory.The fused Triton kernels handle column-wise quantization and intra-block weight and F-matrix updates.Source: §C.3 · p. 21

C.1 PSEUDOCODE

  • For each Transformer block, the pseudocode computes Hessian and gradient information, selects MSE objectives for intermediate blocks and KL/NLL objectives for the final block, then applies trust-region scaling before quantization.The procedure refreshes these estimates before quantizing each block.Source: §C.3 · p. 22

D.1 MODEL COVERAGE

  • Evaluation spans dense, hybrid-attention, and MoE models from 0.6B to 125B parameters, including architectures with gated residuals and n-gram embeddings.The coverage includes 15 models across several model families.Source: §D.1 · p. 22

D.2 DISCUSSION OF THE KL DIVERGENCE METRIC

  • KL divergence directly measures distributional shift from the full-precision model and is presented as a more dependable degradation proxy than potentially noisy task-specific metrics.The passage reports that prior studies find a strong correlation between KL divergence and downstream task accuracy.Source: §D.2 · p. 23

D.3 BENCHMARKS

The benchmark section specifies downstream evaluations, model sampling, and quantization settings. It also identifies the scope of calibration-set-size and output-channel-group ablations.

D.3 BENCHMARKS

  • Evaluation reports PPL and KL divergence on WikiText2, UltraChat, and NuminaMath1.5, plus accuracy on seven commonsense QA benchmarks.Qwen3.8-Flash-Next is also evaluated on four long-context reasoning benchmarks.Source: §D.3 · p. 23
  • Figure 1 compares calibration-set size for G2PTQ and GPTQ in 3-bit weight quantization on Qwen3-0.6B.The caption places G2PTQ on the left and GPTQ on the right.Source: Figure 1 · p. 24
  • Table 7 reports average accuracy metrics and relative calibration cost for 3-bit Qwen3-0.6B quantization across output-channel-group counts.The group count g controls the comparison.Source: Table 7 · p. 24

D.4 QUANTIZATION SETTINGS AND BASELINES

  • Experiments cover 2-, 3-, and 4-bit weight-only quantization and 4-bit weight-activation quantization, using 1,024 dense-model calibration sequences and 8,192 for MoE models.The stated baselines include RTN, GPTQ, GuidedQuant, and GPTAQ.Source: §D.4 · p. 24

E.1 MORE ABLATIONS

The ablations show that exact gradients matter under block-wise objectives, while calibration size and Hessian grouping involve accuracy-related trade-offs.

E.1 MORE ABLATIONS

  • G2PTQ generally requires more calibration samples than GPTQ to attain its best accuracy, despite competitive results with a small set.The authors assume additional samples can reduce variance in estimated first- and second-order information.Source: §E.1 · p. 24
  • 2.09% lower QA accuracy results when exact gradient compensation is replaced by a Taylor approximation under 3-bit quantization on Qwen3-0.6B.The approximation also degrades distribution alignment; the comparison supports exact gradients for block-wise objectives.Source: Table 8 · p. 25, §E.1 · p. 25
  • Increasing output-channel groups beyond g = 1 improves quantization accuracy through finer Hessian estimates, at the cost of memory and computation.The experiments use g = 4, following GuidedQuant (Kim et al., 2025).Source: §E.1 · p. 25

E.2 MORE DISCUSSIONS

The discussions report resource costs and threshold-transfer results, find Hessian-based clipping benefits vary by model family, and summarize strong results across weight-only and weight-activation settings.

E.2 MORE DISCUSSIONS

  • For the 125B-parameter Qwen3.8-Flash-Next, G2PTQ* uses 86.72 GB per device and 20.30 hours, versus GPTQ's 19.48 GB and 7.98 hours.Quantization uses 8,192 calibration samples distributed across eight accelerators.Source: §E.2 · p. 25
  • 1.20–1.33× speedup follows from transferring the threshold tuned on 3-bit Qwen3-0.6B to other settings; optimal thresholds differ by less than 0.4 on average across layers.The transfer covers 2- and 4-bit quantization and Qwen3-0.6B-Base without re-tuning, with comparable performance.Source: Figure 2 · p. 26, Table 10 · p. 26, §E.2 · p. 26
  • On Qwen3-4B, Hessian-based H-Scale clipping raises average QA accuracy by 2.12%, but it offers no tangible benefit and marginally degrades performance on Qwen3.5 models.The reported contrast suggests clipping effectiveness depends on model family; the paper retains weight MSE-based clipping and leaves optimal method selection for future work.Source: Table 11 · p. 26, §E.2 · p. 27

E.3 COMPLETE RESULTS ON WEIGHT-ONLY QUANTIZATION

  • G2PTQ variants achieve the lowest average KL divergence in 38 of 39 weight-only settings and the highest QA accuracy on 12 of 13 models under 2-bit quantization.The evaluation covers 13 dense models from 0.6B to 70B parameters across 2-, 3-, and 4-bit settings.Source: §E.3 · p. 27

E.4 COMPLETE RESULTS ON WEIGHT-ACTIVATION QUANTIZATION

  • G2PTQ variants achieve the highest QA accuracy on all seven evaluated models and the lowest KL divergence on six under 4-bit weight-activation quantization.The models span three families.Source: §E.4 · p. 27

E.5 COMPLETE RESULTS ON MOE QUANTIZATION

On two large MoE models, G2PTQ variants outperform GPTQ in distributional alignment and downstream accuracy; 4-bit G2PTQ* also preserves reasoning benchmark accuracy closely.

E.5 COMPLETE RESULTS ON MOE QUANTIZATION

  • G2PTQ variants achieve better distributional alignment and downstream task accuracy than GPTQ on Qwen3-30B-A3B and Qwen3.8-Flash-Next.The evaluated MoE models range from 30B to 125B parameters.Source: §E.5 · p. 27, Table 16 · p. 31, Table 17 · p. 32
  • The evaluation reports weight-only and weight-activation quantization results on Qwen3-30B-A3B, alongside 4-bit weight-only quantization results on Qwen3.8-Flash-Next, as specified in the cited table captions.Source: Table 16 · p. 31, Table 17 · p. 32
  • 0.19% decrease in average accuracy is reported for G2PTQ* under 4-bit quantization on Qwen3.8-Flash-Next reasoning benchmarks.Mean accuracy and variance are reported across three random seeds.Source: §E.5 · p. 27, Table 18 · p. 32

Related papers

Loading 2609.31009v1…