PaperlayerView exact revision on arXiv

Source-linked AI summary

KronQ: LLM Quantization via Kronecker-Factored Hessian

Donghyun Lee, Yuhang Li, Ruokai Yin, Priyadarshini Panda

arXiv:2607.07964v2cs.LG

TL;DR

Existing PTQ methods largely use input activation statistics while overlooking output-side gradient information. KronQ incorporates gradient covariance through bidirectional incoherence processing and Hessian-trace-based mixed-precision allocation, achieving 7.93 perplexity at 2-bit weight-only quantization on LLaMA-3-70B where GPTQ and GPTAQ fail.

  • Problem

    Existing second-order PTQ methods largely use input activation covariance while overlooking output-side gradient covariance, treating output directions as equally important.

  • Method

    KronQ incorporates gradient covariance through bidirectional incoherence processing and joint Hessian-trace sensitivity for inter-layer mixed-precision allocation.

  • Results

    7.93 perplexity at W2 on LLaMA-3-70B, while GPTQ and GPTAQ diverge or fail to produce valid quantizations; KronQ shows consistent gains across LLaMA-2 and LLaMA-3 models.

  • Takeaways & Limitations

    KronQ’s gradient-aware quantization is especially beneficial at 2-bit precision, where activation-covariance-only methods degrade severely.

  • Takeaways & Limitations

    KronQ adds 8–11 seconds per layer and temporarily stores the full gradient covariance matrix, creating latency and memory overhead during preprocessing.

Abstract

from arXiv · show

Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Most existing second-order PTQ methods, including GPTQ, construct quantization objectives from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), \KronQ{} achieves 7.93 perplexity.

1 Introduction

KronQ addresses the deployment challenges of increasingly large LLMs by incorporating output-side gradient covariance into post-training quantization. Under the K-FAC approximation, it combines bidirectional incoherence processing with joint-Hessian-trace sensitivity for mixed-precision allocation.

  • Motivation: Hundreds-of-billions parameter counts make LLM memory and computation exceed commodity and edge hardware capabilities.Quantization is motivated as a way to reduce these deployment requirements.
  • Problem: Existing compensation-based methods rely solely on input activation covariance HX, omitting output-side Hessian information.K-FAC factorizes the full weight Hessian as H ≈ HX ⊗ HG, where HG = E[gg⊤].
  • Method: KronQ incorporates gradient covariance HG into the quantization objective under the K-FAC approximation.This accounts for output-side Hessian information that existing compensation-based methods largely overlook.
  • Kronecker-Factored Quantization Error: KronQ extends the layer-wise objective with HG and introduces bidirectional incoherence processing while preserving base-quantizer efficiency.HG cancels in the quantization update, so the method retains the efficiency of the base quantizer.
  • Inter-Layer Mixed Precision via Joint Hessian Traces: KronQ derives a sublayer sensitivity metric from tr(HG) · tr(HX) for principled inter-layer mixed-precision allocation.The metric differentiates sublayers that share identical input statistics.

2 Related Work

Prior work in post-training quantization spans Hessian-guided compensation, output-aware Fisher statistics, rotation-based incoherence, and mixed-precision allocation. These methods extend input-only objectives through saliency, Kronecker-factored statistics, orthogonal transforms, and layer-ranking or search strategies.

  • Compensation-Based Post-Training Quantization: GPTQ scales OBS to LLMs with column-wise updates using input activation covariance H_X as a proxy Hessian.This established the de facto standard for one-shot LLM quantization.
  • Compensation-Based Post-Training Quantization: GuidedQuant reweights H_X with per-output-channel end-loss saliency, while YAQA estimates KL-divergence Fisher information through two Kronecker-factored sketches.YAQA’s sketches differ in token handling: Sketch A power-iterates under token independence, whereas Sketch B is an unbiased single-pass sketch retaining cross-token structure.
  • Rotation-Based Post-Training Quantization: QuaRot and SpinQuant use orthogonal transforms on weights and activations to suppress outliers for low-bit weight-and-activation quantization.QuIP provides the theoretical foundation by linking incoherent weights and Hessians to lower quantization error; QuIP# adds randomized Hadamard transforms and lattice-based vector quantization.
  • Mixed-Precision Quantization: Mixed-precision methods allocate bit-widths within layers using weight saliency or activation norms, or across sublayers using Hessian traces, evolutionary search, or multiple-choice knapsack formulations.The cited intra-layer methods are SliM-LLM and CMPQ; inter-layer methods include HAWQ-V2, AMQ, Q-Palette, and HIGGS.

3 Preliminary

This section formulates layer-wise PTQ through the input activation covariance and describes GPTQ’s greedy error compensation. It then identifies the remaining limitation: GPTAQ corrects input drift, while existing incoherence processing still omits output-side gradient statistics.

  • Layer-wise PTQ formulation: The standard PTQ objective minimizes squared layer-output reconstruction error using H_X = XX^⊤ as a proxy for the layer-wise Hessian.X contains calibration inputs, and W and its quantized counterpart define the layer transformation.
  • Layer-wise PTQ formulation: GPTQ greedily quantizes columns and uses an OBS update to distribute each column’s quantization error optimally across remaining unquantized weights.The compensation uses entries of the inverse Hessian associated with the quantized column and later columns.
  • Limitations of existing objectives: GPTAQ corrects input drift through asymmetric calibration, but inherits H_X and therefore leaves output-side gradient statistics unaccounted for.Layer-wise errors accumulate because the objective is evaluated on activations already degraded by prior-layer quantization.
  • Incoherence processing: Incoherence processing spreads weight magnitudes across dimensions by applying random orthogonal rotations, yet existing methods use only input-side Hessian information through H_X.QuIP and QuIP# provide Kronecker-structured and randomized Hadamard implementations, respectively, with the latter reducing inference overhead to Θ(d_in log d_in + d_out log d_out).

4 Method

KronQ uses a Kronecker-factored Hessian that jointly incorporates activation and gradient covariances into quantization. It exploits this structure through bidirectional incoherence processing, efficient weight compensation, and a gradient-aware mixed-precision sensitivity score.

  • Kronecker-factored objective: KronQ factorizes the Hessian into activation and gradient covariance factors under the K-FAC independence assumption, recovering output-side statistics omitted by activation-only objectives.For a linear layer, the per-sample weight gradient factorizes as gx⊤, while the joint expectation is otherwise intractable.
  • Weight compensation: Under column-wise OBS updates, HG cancels algebraically from weight compensation, allowing KronQ to retain gradient-aware preprocessing with a one-sided solver.The resulting compensation applies after quantizing column p, and KronQ inherits GPTAQ’s computational efficiency.
  • Mixed-precision allocation: KronQ ranks sublayers with the sensitivity score tr(HG) · tr(HX), assigning higher bit-widths to layers with larger scores.The score is computed after BiIP and extends activation-only sensitivity to both Kronecker factors.
  • Mixed-precision allocation: The gradient-aware score distinguishes Q, K, and V projections sharing the same input, producing substantially different rankings from tr(HX) alone.Activation-only metrics assign identical sensitivity to these projections, whereas their downstream gradients differ under KronQ.

5 Experiments

Experiments across LLaMA models and multiple quantization regimes show that KRONQ generally improves perplexity and accuracy over GPTQ, GPTAQ, and other baselines. The evaluations also validate its mixed-precision sensitivity score, efficiency benefits, and component contributions while identifying calibration overhead.

  • Evaluation Setup: KRONQ is evaluated on LLaMA-2-7B/13B/70B and LLaMA-3-8B/70B against GPTQ and GPTAQ under weight-only, group, and weight-and-activation regimes.Activation quantization uses the QuaRot framework.
  • Weight-only Quantization: KRONQ achieves the lowest perplexity across nearly all per-channel settings, with its largest gains at W2 and W3 where activation-covariance-only methods degrade severely.Its zero-shot accuracy is highest or comparable across benchmarks and bit-widths.
  • Group Quantization: 7.60 PPL is achieved by KRONQ on LLaMA-2-7B under W2 group quantization, versus 274.0 PPL for GPTQ.The comparison uses 128-column blocks.
  • Weight-and-Activation Quantization: 9.46 PPL is achieved by KRONQ on LLaMA-2-7B under W2A4 quantization, reducing perplexity from 36.74.KRONQ consistently outperforms both baselines across the reported LLaMA-2-7B/13B and LLaMA-3-8B settings.
  • Mixed-precision Allocation: The joint sensitivity score s = tr(HG) · tr(HX) produces better mixed-precision allocations than the activation-only score tr(HX) on LLaMA-2-7B.The evaluation incrementally upgrades the most sensitive sublayers from W2 to W3 across all layers.
  • Inference Efficiency: 3.5–3.9× and 4.0–7.5× reductions in peak inference VRAM are measured at W4 and W2, respectively, while W4 decoding is 1.25–2.51× faster than bf16.A 70B model requiring 138–141 GB in bf16 fits on a single A100 at W4 using 35–39 GB.
  • Ablation and Overhead: 8.19/6.99/11.92 PPL rises to 10.18/7.95/14.52 when GPTAQ is replaced with GPTQ on LLaMA-2-7B/13B and LLaMA-3-8B at W2.Additional ablations find diagonal rescaling necessary alongside Hadamard rotations, while preprocessing adds +8–11s per layer over GPTAQ.

6 Conclusion … A.2 Proof of Theorem 1 (Rotation Invariance)

KRONQ incorporates gradient covariance into bidirectional incoherence processing and mixed-precision allocation while retaining GPTAQ’s computational efficiency. Its proofs establish gradient-covariance cancellation in weight compensation and rotation invariance of the full quantization objective.

  • 6 Conclusion: KRONQ incorporates gradient covariance HG into bidirectional incoherence processing and inter-layer mixed-precision allocation.The framework extends beyond activation-covariance-only quantization methods.
  • 6 Conclusion: Experiments on LLaMA-2 and LLaMA-3 from 7B to 70B show consistent gains across W2/W3/W4, with the largest improvements at 2-bit.
  • 6 Conclusion: HG cancels algebraically in the quantization updates, allowing KRONQ to inherit GPTAQ’s computational efficiency.
  • A.1 Proof of Proposition 1 (KRONQ Weight Compensation): Under the K-FAC approximation H = H_X ⊗ H_G, the OBS derivation produces KRONQ’s weight-compensation update from column quantization errors.The proof starts from the general OBS optimal weight update and specializes it using the Kronecker-factored Hessian.
  • A.1 Proof of Proposition 1 (KRONQ Weight Compensation): The full objective adds a linear drift term from ∆X, yielding a constrained minimizer that reduces to the OBS update when ∆X = 0.
  • A Proofs: HG cancels from every term in the constrained minimizer, making it coincide with the unweighted GPTAQ objective and yielding the correction matrix P and Equation (10).
  • A.2 Proof of Theorem 1 (Rotation Invariance): Under the transformation W′ = UWV⊤ with orthogonal U and V, the transformed weights, covariances, and activation drift preserve the KRONQ objective terms.The proof verifies invariance term by term using U⊤U = I, V⊤V = I, and the cyclic property of the trace.
  • A.2 Proof of Theorem 1 (Rotation Invariance): Both terms are individually invariant, so the full KRONQ objective (9) is invariant under transformation (12).

A.3 Proof of Proposition 2 … C.2 Comparison on Mistral-7B

The paper proves Proposition 2 for KRONQ’s one-sided solver, specifies the BiIP preprocessing and quantization loop, and evaluates weight-only quantization on reasoning benchmarks and Mistral-7B. The proof links stochastic-rounding loss to diagonal gradient-Hessian terms, while experiments verify the approximation and assess generalization beyond LLaMA.

  • A.3 Proof of Proposition 2: Under stochastic rounding, row-wise residuals are conditionally independent and mean-zero, enabling an exact transformed loss decomposition.The decomposition uses the transformed quantities eH_X and eH_G and per-step rounding residuals.
  • A.3 Proof of Proposition 2: The expected loss depends on the diagonal of eH_G, because its off-diagonal terms do not contribute.The proof states that row residual independence removes the off-diagonal contribution from the expected loss.
  • A.3 Proof of Proposition 2: RHT-based row mixing concentrates row energies and ranges, and the optimal rescaling satisfies s_j ∝ (h_j/a_j)^1/4.This rescaling is identified as the SG of Equation (11), up to a global scale that leaves the product invariant.
  • A.3 Proof of Proposition 2: Numerical verification on Llama-2-7B layer-0 matrices finds Equation (24) accurate to machine precision and E[L_q] within 0.6% of Equation (25).The exponent sweep is minimized at α = 1/4, while nearest-rounding diagonal predictions remain within 2.3% across RHT sign draws.
  • B KRONQ Algorithm: BiIP samples independent input- and output-side random Hadamard rotations, rescales both dimensions, transforms H_X and H_G, and releases H_G after preprocessing.The procedure constructs online activation statistics, applies U and V, and returns the transformed weights and rescaling factors.
  • B KRONQ Algorithm: The KRONQ quantization loop damps H_X, computes its inverse Cholesky factor, applies the GPTAQ correction, and performs blockwise column quantization with error feedback.After quantization, the algorithm reverses the incoherence rotations and rescaling before returning Q.
  • C.1 Zero-shot Accuracy on Weight-only Quantization: The zero-shot evaluation covers PiQA, ArcE, ArcC, HS, WG, BoolQ, and OBQA for WxA16 weight-only quantization at W4, W3, and W2.NaN denotes diverged quantization in this evaluation.
  • C.2 Comparison on Mistral-7B: On Mistral-7B, Table 11 extends the weight-only WikiText-2 perplexity evaluation beyond the LLaMA family, testing KRONQ’s generalization.The reported metric is WikiText-2 perplexity ↓ under WxA16 quantization.

C.3 Generalization to Newer Models and Harder Benchmarks · C.4 Comparison of KRONQ and YAQA Hessians · C.5 Comparison with Previous Works on Group Quantization

KRONQ generalizes across newer model families and harder benchmarks, while its Hessian estimator shows model- and bit-width-dependent tradeoffs against YAQA. Group-quantization comparisons extend the evaluation to prior methods under WikiText-2 perplexity.

  • C.3 Generalization to Newer Models and Harder Benchmarks: KRONQ wins all eight W4/W2 WikiText-2 perplexity settings across Gemma-3-12B, DeepSeek-R1-Distill-Llama-8B, and Phi-4-mini-instruct.The margin over GPTQ and GPTAQ widens sharply at W2.
  • C.3 Generalization to Newer Models and Harder Benchmarks: Table 13 evaluates KRONQ beyond perplexity on four harder benchmarks using W4A16 accuracy.MMLU, GPQA-Diamond, and AIME-2024 use zero-shot lm-evaluation-harness evaluation, while LiveCodeBench uses its official runner with vLLM.
  • C.3 Generalization to Newer Models and Harder Benchmarks: The evaluation uses per-channel asymmetric weight-only W4 quantization calibrated on 128 WikiText-2 sequences without fine-tuning.Identical settings are used across methods.
  • C.4 Comparison of KRONQ and YAQA Hessians: Under a fixed KRONQ+GPTQ pipeline, Hessian-estimator swaps show that neither KRONQ nor YAQA dominates.Drift correction is disabled so external Hessians can be compared consistently.
  • C.4 Comparison of KRONQ and YAQA Hessians: YAQA’s Sketch-B Hessians are evaluated either recomputed on the same WikiText-2 tokens or used as released from RedPajama.The comparison separates Hessian estimation from the rounding algorithm by swapping only the Hessian pair.
  • C.4 Comparison of KRONQ and YAQA Hessians: Sketch-B is better on L2-7B, whereas the K-FAC estimate is better on L3.1-8B, with the largest gaps at W2.Differences are small at W3 and W4.
  • C.5 Comparison with Previous Works on Group Quantization: Table 15 compares WikiText-2 perplexity for weight-only group quantization with group size g=128.The comparison includes OmniQuant and AWQ, whose results are taken from their papers.
  • C.5 Comparison with Previous Works on Group Quantization: The group-quantization comparison reports WxA16 WikiText-2 perplexity against prior methods OmniQuant and AWQ.These results are presented in Table 15.

C.6 Additional Bit-width Results on Group Quantization … C.10 Comparison with Inter-Layer Mixed-Precision Quantization

Across group, weight-and-activation, and mixed-precision settings, KRONQ maintains strong performance and improves allocation tradeoffs. Matched comparisons further show that its allocation methodology lowers perplexity against competing inter-layer approaches under fixed quantizers.

  • C.6 Additional Bit-width Results on Group Quantization: KRONQ maintains its lead across W2/W3/W4 group-quantization results, with especially large W2 margins where per-channel methods struggle.The evaluation uses group quantization with g=128.
  • C.7 Comparison with Previous Works on Weight-and-activation Quantization: Table 17 compares WikiText-2 perplexity for weight-and-activation quantization, with all methods using the QuaRot rotation framework for activation quantization.QuaRot and SpinQuant results are taken from their respective papers.
  • C.8 3 and 4-bit Results on Weight-and-activation Quantization: KRONQ achieves the lowest perplexity across all W3A4 and W4A4 settings, with gains increasing at lower bit-widths.These results extend the weight-and-activation evaluation to 3- and 4-bit configurations.
  • C.9 Mixed Precision Quantization on LLaMA: On LLaMA-3-8B and LLaMA-2-13B, KRONQ’s joint score yields strictly better PPL–bits tradeoffs than the activation-only score.The comparison extends the mixed-precision analysis and includes SliM-LLM and CMPQ.
  • C.10 Comparison with Inter-Layer Mixed-Precision Quantization: Inter-layer allocation comparisons use the same 128 WikiText-2 sequences of length 2048 for every method, isolating allocation methodology from calibration-data access.The matched data-aware setting compares AMQ, Q-Palette, and HIGGS with KRONQ.
  • C.10 Comparison with Inter-Layer Mixed-Precision Quantization: At a matched 3.1-bit budget with GPTQ fixed on LLaMA-2-7B, KRONQ’s allocation beats AMQ’s data-aware NSGA-II search.The comparison uses context length 2048 and varies only allocation.
  • C.10 Comparison with Inter-Layer Mixed-Precision Quantization: With Q-Palette’s quantizer fixed on LLaMA-3.1-8B, KRONQ’s allocation achieves lower perplexity at every budget.This comparison uses context length 8192 and Q-Palette’s data-aware actual-loss allocation.
  • C.10 Comparison with Inter-Layer Mixed-Precision Quantization: The quantizer-isolation comparison contrasts KRONQ’s scalar quantizer with HIGGS’s vector quantizer on LLaMA-2-7B.This comparison uses context length 4096.

D Why GPTQ and GPTAQ Fail on LLaMA-3-70B · E Per-Sublayer Calibration Memory

KronQ attributes GPTQ and GPTAQ’s failure on LLaMA-3-70B to extreme input-column outliers that cause degenerate column-wise OBS updates. Its per-sublayer calibration memory accounting includes block slices, triangular factors, and temporary storage of the gradient covariance during BiIP.

  • D Why GPTQ and GPTAQ Fail on LLaMA-3-70B: Figure 6 evaluates original weights, column-side incoherence using HX only, and bidirectional incoherence using HX + HG.The figure reports column norms in the top histograms and row norms in the right histograms, with CVin and CVout denoting their coefficients of variation.
  • D Why GPTQ and GPTAQ Fail on LLaMA-3-70B: CVin measures variation in column norms, whereas CVout measures variation in row norms.These metrics are used to characterize weight-magnitude distributions under the three configurations shown in Figure 6.
  • D Why GPTQ and GPTAQ Fail on LLaMA-3-70B: LLaMA-3-70B exhibits extreme column-wise weight outliers concentrated on specific input dimensions, with CVin = 9.21.These outliers are identified as unique to the LLaMA-3/3.1-70B family in the cited discussion.
  • D Why GPTQ and GPTAQ Fail on LLaMA-3-70B: Orders-of-magnitude expansion of the quantization range causes GPTQ and GPTAQ’s column-wise OBS updates to produce degenerate solutions.The passage contrasts this behavior with LLaMA-2-70B, but its comparison is truncated in the supplied text.
  • E Per-Sublayer Calibration Memory: Table 23 reports per-sublayer calibration memory for LLaMA-2-7B, LLaMA-3-8B, and LLaMA-2-13B using block slices of size B = 128.The accounting follows the convention of Li et al. (2025), Table 9.
  • E Per-Sublayer Calibration Memory: The Cholesky factor L is stored as a lower-triangular matrix, while the correction matrix P is stored as an upper-triangular matrix.These storage formats are included in the reported calibration-memory accounting.
  • E Per-Sublayer Calibration Memory: During BiIP, KronQ additionally stores HG as a full dout × dout matrix.The passage states that HG is released afterward, after which the remaining memory footprint is identical to G.

F Limitations

KronQ adds offline computation and peak memory overhead because it estimates and stores the full gradient covariance before quantization, although the precomputed covariance adds no computation during quantization itself.

  • Computational and memory overhead: KronQ requires a calibration-set backward pass to estimate the gradient covariance HG, adding offline computation compared with activation-only GPTQ and GPTAQ.The extra computation occurs before quantization.
  • Computational and memory overhead: The full dout × dout gradient covariance HG increases peak memory during bidirectional incoherence preprocessing.HG is stored after precomputation.
  • Computational and memory overhead: After HG is precomputed and stored, it adds no computation during quantization because it cancels algebraically in the column-wise updates.This cancellation is stated for the quantization stage itself.
Loading 2607.07964v2…