A Unified Hessian-Guided Framework for Post-Training Quantization

1 INTRODUCTION

• Quantization reduces memory and computational costs, while the central challenge is preserving full-precision predictions.Source: §1 · p. 1
• Local reconstruction measures layer-output error, but its magnitude is not necessarily a good proxy for changes in the model output.Source: §1 · p. 1
• Output Fisher-weighted error correlates more strongly with downstream KL divergence than conventional local proxies, and its Kronecker factorization retains this correlation.Fisher-weighted weight error alone fails to capture activation-quantization effects.Source: Figure 1 · p. 2, §1 · p. 2
• The framework introduces an Output Kronecker-factorized approximation to the layer-output Fisher as a quadratic proxy for KL divergence that accounts for weight and activation quantization.Source: §1 · p. 2
• HZ-PTQ jointly optimizes transformation design, weight-range selection, adaptive rounding, and activation clipping under a common downstream-sensitivity objective.Its objective accounts for downstream effects of both weight and activation quantization.Source: §1 · p. 2
• The paper reports improvements over existing PTQ baselines across multiple model families and quantization granularities.Source: §1 · p. 2

2 BACKGROUND

• For a linear layer, activations X and weights W produce output Z = XW.X has s tokens and i input features; W has i input features and o output features.Source: §2 · p. 2
• The grid step trades clipping against rounding resolution: a larger step reduces clipping but coarsens the grid, while a smaller step improves resolution but may clip large values.Source: §2 · p. 2
• Activation quantization uses an optimized shared clipping factor to set input-dependent quantization steps, with symmetric or asymmetric clipping bounds.Source: §2 · pp. 2–3
• Adaptive rounding coordinates weight-rounding decisions so later choices can compensate for errors introduced earlier, unlike independent round-to-nearest.Source: §2 · p. 3
• An invertible transformation preserves the full-precision layer output but generally changes the quantized output; reducing a weight group's range can enable a finer grid.Source: §2 · p. 3

3.1 JOINT OBJECTIVE

• PTQ is framed as minimizing downstream prediction divergence, approximated by a quadratic loss proxy that weights layer-output perturbations by output sensitivity.For KL divergence, the Hessian corresponds to the Fisher Information Matrix.Source: §3.1 · pp. 3–4

3.2 MODELING THE HESSIAN MATRICES

• The Output Kronecker Fisher factorization approximates otherwise intractable output curvature and exactly reproduces sampled gradient outer products when the output gradient is rank 1.The approximation separates sequence and output-feature curvature.Source: §3.2 · p. 4

3.3 HZ-PTQ: OPTIMIZATION PROCEDURE

• HZ-PTQ optimizes transformations first, then uses the resulting coordinates and a shared output-curvature proxy to determine the remaining quantization parameters.Its stages share the same layer-output curvature proxy.Source: §3.3 · pp. 4–5

3.4 OPTIMAL TRANSFORMATIONS

• The transformation is optimized through a two-step procedure: choose a positive-definite component to reduce transformed gradient norms, then apply a Hadamard rotation to reduce quantization ranges.A block-wise approximation reduces memory and computation cost.Source: §3.4 · p. 5

3.5 WEIGHT SCALES AND ROUNDING

• Weight scales and rounding are optimized iteratively by quantization tile, using input- and output-side curvature to select scales and adjust rounded weights.The procedure sets scales in anti-diagonal order; each tile scale is selected by a grid search over candidates.Source: §3.5 · p. 6

3.6 ACTIVATION CLIPPING.

• Activation clipping uses the Hessian proxy to account for downstream sensitivity, selecting the clipping parameter on a small calibration set from a small candidate set.The method uses only diagonal sequence curvature to reduce memory requirements, reporting little error from this approximation.Source: §3.6 · p. 6

4.1 CHANNEL-WISE W4A4KV4 QUANTIZATION

• Across W4A4KV4 channel-wise quantization, HZ-PTQ achieves the lowest perplexity across evaluated architectures while matching or improving reasoning accuracy.It has lower perplexity than FlatQuant on all four models and consistently outperforms CAT.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 BLOCK-WISE QUANTIZATION

• Across W4-g32 and W4A4-g32 block-wise settings, HZ-PTQ achieves the lowest perplexity and highest accuracy across evaluated architectures.The reported gains persist despite reduced optimization headroom and remain stable across model families.Source: §4.2 · p. 7, Table 3 · p. 8

4.3 ABLATION STUDIES

• Removing any HZ-PTQ component degrades W4A4KV4 performance, with transformations contributing most, followed by activation clipping and Hessian modeling.Replacing the output Kronecker approximation with a GPTQ-style Hessian estimate also degrades performance.Source: §4.3 · p. 8, Table 4 · p. 8
• The Hessian-weighted range objective matches or outperforms the best manually selected Lp objective without hyperparameter tuning.The best Lp varies substantially with quantization setting and transformation choice.Source: §4.3 · p. 8, Figure 2 · p. 9
• The output Kronecker Hessian proxy performs on par with or better than the YAQA-style estimate and outperforms gradient-free GPTQ-style Hessian sketches.This comparison separates the curvature estimate from the rounding algorithm.Source: §4.3 · pp. 8–9
• For a fixed Hessian estimate, HZ-PTQ’s Hessian-aware rounding and range setting improve performance over conventional GPTQ/LDLQ and YAQA optimization.The passage notes that the output Kronecker and YAQA-style Hessians are equivalent for GPTQ rounding because only the first Kronecker component determines weight corrections.Source: §4.3 · p. 9

5 DISCUSSION AND CONCLUSION

• Across diverse architectures and quantization granularities, jointly optimizing PTQ components consistently improved performance over existing state-of-the-art methods.The framework derives the quantization decisions from a common approximation of downstream prediction loss.Source: §5 · p. 9
• The downstream objective is approximated using a second-order expansion, a rank-one Kronecker Fisher model, error decomposition, and rounding-noise and covariance assumptions.The method therefore does not directly optimize the ideal downstream objective.Source: §5 · p. 9
• Estimating output gradients adds calibration time and memory relative to gradient-free reconstruction methods.The discussion points to richer curvature models and adaptive accuracy–calibration-cost trade-offs as future work.Source: §5 · p. 10

A.1 DERIVATION OF THE TRANSFORMATION OBJECTIVE

• The derivation starts from a second-order KL approximation with the Output Kronecker factorization, using calibration inputs and sampled labels to define the output-gradient quantities.It separates activation and weight errors to make the optimal-transform derivation tractable.Source: §A.1 · p. 13
• Rounding errors are modeled as conditionally independent and uniform within their quantization bins, with negligible clipping and zero conditional means.The output perturbation drops the product of activation and weight errors as third- and fourth-order terms, while conditional independence removes the mixed quadratic term.Source: §A.1 · p. 13
• The activation contribution uses the chain-rule gradient GX = GZ W^T and an asymmetric per-token grid whose range is each token's maximum minus minimum activation.The derivation retains matching scalar error indices when taking the conditional expectation.Source: §A.1 · p. 13
• The weight contribution uses GW = X^T GZ and the configured symmetric weight-grid step to define each column's grid span.Separating range and gradient terms assumes their covariance is approximately zero; combining the activation and weight contributions with Eq. 8 yields Eq. 11.Source: §A.1 · pp. 13–14

A.2 GEOMETRIC-MEAN SOLUTION FOR GRADIENT BALANCING

• Writing P = MM^T makes the gradient-norm objective depend on M only through a positive-definite P, with every such P represented by its unique positive-definite square root.The derivation assumes the gradient second-moment matrices A and B are positive definite.Source: §A.2 · p. 14
• The resulting trace minimization has a unique minimizer expressed using the Pusz–Woronowicz geometric mean.The geometric-mean notation is identified as such in the derivation.Source: §A.2 · p. 14
• Across invertible transformations, the minimizers retain a free rotation after the positive-definite component is fixed.That rotation is left for the subsequent range-reduction step.Source: §A.2 · p. 14

A.3 RANGE CLIPPING FOR WEIGHTS DERIVATION

• The chain rule transforms existing gradient estimates to obtain Fisher quantities in transformed coordinates without an additional backward pass.The relation G_WT = T^T G_W is used in Eq. 13 and in the weight contribution to Eq. 11.Source: §A.3 · p. 14
• The Table 2 experimental setup uses transform blocks of size 128 for CAT and the proposed method.The configuration details are reported in Table 6.Source: Table 6 · p. 15
• The weight-error proxy treats the input-side curvature factor as constant and assumes input- and output-side curvature correlations contribute negligibly.Substitution of the P and Q definitions followed by row vectorization gives the stated expression.Source: §A.3 · p. 15

• The evaluation spans Llama v3 8B, Llama v3.2 1B instruct, Ministral 8B instruct, and Qwen v3 8B.Source: §B.1 · p. 15
• Transformation baselines include Quarot, FlatQuant, CAT, and WUSH.Source: §B.1 · p. 15
• Rounding baselines include GPTQ/LDLQ and YAQA Sketch B.Source: §B.1 · p. 15

• Block-wise experiments omit activation clipping because its impact is limited at the small block size.Source: §B.2 · p. 15
• Scale and clipping searches use sampled candidates, while rounding processes 32 output channels at a time and batches scale searches across 32 blocks.The text says batching reduces runtime without significantly affecting quantization performance.Source: §B.2 · p. 15
• Table 7 specifies the settings used for Table 3, with transformations referring to a single block of size 32.Source: Table 7 · p. 16
• Figure 3 compares activation-clipping objectives on Llama 3.2 1B Instruct under 4-bit per-token quantization, with and without Hadamard transformations.Source: Figure 3 · p. 16

• The proposed Hessian proxy performs strongly against manual clipping-percentage selection and clipping with simpler Hessian proxies.Source: C ADDITIONAL RESULTS · p. 16
• Using the same offline-estimated average curvature for all tokens may be suboptimal compared with estimating curvature separately for each token.Source: C ADDITIONAL RESULTS · p. 16
• HZ-PTQ outperforms the baselines without weight-rounding adjustment, highlighting the effectiveness of its transformations and activation clipping.Source: Table 8 · p. 16, C ADDITIONAL RESULTS · p. 16