PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Preserving Adaptive Rounding

Albert Tseng, Zhaofeng Sun, Christopher De Sa

arXiv:2505.22988v3cs.LGcs.AI

TL;DR

Quantization seeks to preserve a model’s output distribution, but common layerwise objectives ignore how rounding affects future layers. YAQA directly targets end-to-end error with adaptive rounding and theoretically motivated Hessian sketches, reducing KL by approximately 30% versus GPTQ/LDLQ and outperforming QAT while adding no inference overhead.

  • Problem

    Existing quantization methods often optimize immediate layerwise activation error as a proxy for end-to-end output-distribution error, despite ignoring future-layer effects.

  • Method

    YAQA is an adaptive rounding algorithm that generalizes LDLQ to optimize full-model error using theoretically motivated Kronecker-factored Hessian approximations and sketches.

  • Results

    ≈30% lower KL than GPTQ/LDLQ is reported for YAQA, with lower KL than QAT and state-of-the-art PTQ downstream performance.

  • Takeaways & Limitations

    YAQA provides end-to-end quantization error bounds and improves PTQ quality without adding inference overhead.

  • Takeaways & Limitations

    Several prior full-model approaches assume models are trained to convergence, an assumption stated to be almost never true for LLMs.

Abstract

from arXiv · show

The goal of quantization is to produce a compressed model whose output distribution is as close to the original model's as possible. To do this tractably, most quantization algorithms minimize the immediate activation error of each layer as a proxy for the end-to-end error. However, this ignores the effect of future layers, making it a poor proxy. In this work, we introduce Yet Another Quantization Algorithm (YAQA), an adaptive rounding algorithm that directly considers the error at the network's output. YAQA introduces a series of theoretical results that culminate in the first end-to-end error bounds for quantization algorithms. First, we characterize the convergence time of adaptive rounding algorithms via the structure of their Hessian approximations. We then show that the end-to-end error can be bounded by the approximation's cosine similarity to the true Hessian. This admits a natural Kronecker-factored approximation with corresponding near-optimal Hessian sketches. YAQA is provably better than GPTQ/LDLQ and empirically reduces the error by $\approx 30\%$ over these methods. YAQA even achieves a lower error than quantization aware training. This translates to state of the art performance on downstream tasks, all while adding no inference overhead.

1. Introduction

YAQA targets end-to-end output-distribution error rather than immediate layerwise error, combining adaptive rounding with theoretically justified Hessian approximations. It provides end-to-end error bounds and reports lower KL than existing rounding methods and QAT, with state-of-the-art PTQ downstream results.

  • Quantization seeks to preserve the original model’s outputs, but directly solving the discrete objective is intractable, motivating approximate rounding or QAT methods.
  • Existing GPTQ-style rounding uses immediate layerwise activation error as a proxy, but this ignores quantization effects on future layers.
  • ≈30% lower KL than LDLQ is reported for YAQA, which also achieves lower KL than QAT and state-of-the-art PTQ downstream results.
  • YAQA uses a Kronecker-factored Hessian approximation and sketches motivated by convergence analysis and cosine similarity to the true Hessian.
  • YAQA generalizes local adaptive rounding to optimize full-model KL and is provably better than GPTQ and LDLQ.
  • The method characterizes structural requirements for tractable adaptive rounding and proves the first theoretical bounds on end-to-end quantization error.

2. Background

The background contrasts PTQ and QAT, then explains how local layerwise Hessian approximations support efficient rounding but can miss full-model effects. Prior alternatives add information through empirical-Fisher or block-structured sketches, though their assumptions and structures lack consistent theoretical justification.

  • PTQ compresses trained models at relatively low cost, whereas QAT can produce better models but requires significantly more compute.
  • GPTQ and LDLQ use H1, the immediate activation-error Hessian, with iterative column rounding and linear feedback from its Cholesky decomposition.
  • H1 is cheap to compute from forward passes, but it ignores future-layer effects and therefore does not always reduce KL.
  • OBS, AdaRound, SqueezeLLM, and GuidedQuant incorporate empirical-Fisher information, but assume converged training and use block-diagonal approximations for tractability.
  • Prior structured approximations generally lack error bounds and use largely ad-hoc Hessian sketches, while YAQA bounds KL through cosine similarity to the true Hessian.
  • Kronecker-factored Hessian sketches have been widely studied in optimization, often using Fisher-information constructions.

3. Yet Another Quantization Algorithm

YAQA combines full-model adaptive rounding with structured Hessian sketches to minimize end-to-end KL efficiently. Its theory links tractable convergence and quantization error to Hessian structure, while experiments evaluate scalable sketches and their similarity to the true Hessian.

  • 3. Yet Another Quantization Algorithm: YAQA rounds each layer to minimize full-model KL using a principled adaptive algorithm and tractable near-optimal Hessian sketches.It can use any quantizer, so the quantizer determines inference efficiency rather than YAQA.
  • 3.1.1. WHEN IS ADAPTIVE ROUNDING FAST?: The structural nilpotence degree bounds adaptive-rounding convergence, with Equation 3 converging in at most snd(L) steps.For a general dense lower-triangular matrix, snd(L)=mn, motivating structured approximations with low SND.
  • 3.1.1. WHEN IS ADAPTIVE ROUNDING FAST?: Kronecker factorization ẼH=H_O⊗H_I yields snd(L)≤m+n−1 and fast multiplication, enabling symmetric input- and output-side feedback.The decomposition gives snd(L)=snd(L_O)+snd(L_I), while YAQA runs in m+n iterative steps with parallelizable matrix multiplications.
  • 3.1.2. END-TO-END BOUNDS & HESSIAN DESIDERATA: YAQA’s true second-order error is bounded using the proxy error and cosine similarity between H_O⊗H_I and the true Hessian H.Theorem 3.4 further favors factors with high directional similarity, low incoherence, and approximately low rank; when H_O is low rank, YAQA has a lower error bound than LDLQ.
  • 3.2. Scalable Near-Optimal Kronecker Factored Hessian Sketches: Power iteration constructs Kronecker factors by maximizing cosine similarity to H, while empirical spectra show real-world H_O matrices from Sketches A and B are approximately low rank.The objective also seeks low incoherence and rank, and Figure 1 reports similar spectra for the two sketches.
  • 3.2. Scalable Near-Optimal Kronecker Factored Hessian Sketches: Sequence dependence forces Hessian estimation over entire sequences, increasing variance; Sketch A trades bias for lower variance, whereas Sketch B uses one power-iteration round and outperforms A with sufficient data.Both sketches have higher cosine similarity to H than the LDLQ Hessian baseline, with A sometimes preferable in data-limited settings.

4. Experiments

YAQA is evaluated against local, global, gradient-based, and QAT baselines across multiple quantizers, models, and finetuning settings. It consistently lowers KL divergence while maintaining strong downstream performance and state-of-the-art results.

  • Experimental Setup: The experiments evaluate KL divergence alongside perplexity and zeroshot accuracy because KL measures the full output distribution rather than only the ground-truth token.The main setup focuses on weight-only post-training quantization without additional finetuning, while also testing recovery finetuning.
  • Second Order Quantization Algorithms: YAQA consistently outperforms LDLQ and GuidedQuant across the reported metrics, with YAQA-B outperforming YAQA-A.The experiments compare local and global second-order information during adaptive rounding.
  • Second Order Quantization Algorithms: 30% lower KL than LDLQ is reported for YAQA with the QTIP quantizer and no finetuning, while outperforming LDLQ and GuidedQuant.Table 1 uses incoherence processing and the QTIP quantizer; the comparison covers local and global adaptive rounding baselines.
  • First Order Quantization Methods: YAQA remains effective with standard INT4 quantization, demonstrating that the rounding algorithm is quantizer agnostic.The INT4 evaluation uses Llama 3.1 8B Instruct without finetuning.
  • First Order Quantization Methods: YAQA achieves lower KL divergence than the Gemma 3 QAT model despite QAT using 100× more data, while preserving a smaller downstream-performance gap.The comparison indicates that QAT can produce a model farther from the original output distribution even when downstream results are strong.
  • Finetuning: Recovery finetuning narrows the LDLQ–YAQA gap, but YAQA still reduces KL by ≈10-20% and maintains a large downstream-task advantage.With recovery finetuning, YAQA sets a new state of the art across the evaluated PTQ methods and quantizers.

5. Conclusion

YAQA generalizes local layerwise quantization to minimize end-to-end error, supported by theoretical bounds and Hessian constructions. It outperforms established methods empirically and without inference overhead.

  • 5. Conclusion: YAQA generalizes state-of-the-art local layerwise quantization methods to minimize end-to-end error.
  • 5. Conclusion: YAQA provides the first end-to-end error bounds for quantization algorithms.These results motivate a natural Kronecker-factored Hessian structure and corresponding near-optimal Hessian sketches.
  • 5. Conclusion: YAQA empirically reduces KL by ≈30% versus GPTQ/LDLQ and achieves lower KL than QAT.The method is also reported to be provably better than GPTQ/LDLQ.
  • 5. Conclusion: YAQA sets a new empirical state-of-the-art among post-training quantization methods on downstream tasks without adding inference overhead.

A.1. Experimental Setup and Implementation Details

The experiments use specified datasets, sequence lengths, matrix constructions, and chat-template evaluation procedures for Hessian, KL, perplexity, and zero-shot measurements.

  • A.1. Experimental Setup and Implementation Details: Hessians are collected using the RedPajama v1 dataset.
  • A.1. Experimental Setup and Implementation Details: KL divergence and perplexity use the OPTQ Wikitext2 and C4 dataset splits with sequence length 8192.
  • A.1. Experimental Setup and Implementation Details: The implementation uses Hadamard matrices from Neil Sloane’s website, and zero-shot evaluations apply the chat template.

A.2. Hessian Sketch Memory and Compute Requirements

YAQA’s Hessian sketches require more storage than layerwise activation methods and depend on numerically stable FP32 Hessian computation with diagonal regularization.

  • A.2. Hessian Sketch Memory and Compute Requirements: YAQA requires O(n^2) memory for H_I and O(m^2) memory for H_O, storing only their lower-triangular parts.The triangular storage is sufficient because both Hessians are symmetric.
  • A.2. Hessian Sketch Memory and Compute Requirements: YAQA requires roughly twice the Hessian storage of LDLQ and other layerwise activation methods because those methods store only H_I.
  • A.2. Hessian Sketch Memory and Compute Requirements: FP32 Hessian computation with diagonal regularization ≈10^-4 tr(H)/n maintained numerical stability and positive-definiteness, whereas TF32 computation did not.

A.3. GuidedQuant as LDLQ

GuidedQuant’s grouped LDLQ corresponds to a block-diagonal Hessian approximation whose sequential depth is determined by its largest block.

  • A.3. GuidedQuant as LDLQ: GuidedQuant partitions the output channels of W* into g groups that share an input Hessian for LDLQ.
  • A.3. GuidedQuant as LDLQ: The grouped construction is a block-diagonal Hessian approximation with g ≤ m unique n × n blocks.
  • A.3. GuidedQuant as LDLQ: Because the LDL factor is block lower unit triangular, the approximation’s SND equals the largest SND of any block, namely n.This matches an optimal implementation in which LDLQ runs in parallel across groups.

A.4. KL Divergence vs. Perplexity

KL divergence compares two full output distributions, whereas perplexity measures probability assigned to a ground-truth target within one distribution. Consequently, similar perplexities do not imply similar models.

  • KL divergence measures differences between two distributions over their full support, while perplexity measures target probability within a single distribution.
  • Llama 1 7B and Llama 2 7B have Wikitext 2 perplexities of 5.68 and 5.47 despite being pretrained separately from scratch.
  • Their KL divergence is 0.197, much higher than the perplexity difference would suggest.

A.5. Incoherence Processing with the Random Hadamard Transform

Incoherence processing applies a Random Hadamard Transform to weights and Hessians to support the paper’s incoherence framework. The definition covers both Hessian eigenspaces and individual weight entries.

  • A Hessian is µ-incoherent when its eigenvectors have bounded coordinate magnitudes, while a weight matrix is µ-incoherent when its entries are bounded similarly.
  • Incoherence processing with the RHT applies a Random Hadamard Transformation to both the weight matrix W and Hessian H.
  • The RHT transforms x as x ← H^log2 n Sx using a random sign vector, with n required to be a power of 2 for the standard construction.
  • For non-power-of-two dimensions, the implementation uses a fixed small Hadamard matrix as the base instead of H1.

A.6. YAQA Rounding Algorithm

YAQA rounds weight blocks using output- and input-side Hessian factors, iterating until the quantized weights stabilize. Its accompanying results characterize convergence and Hessian-approximation effects, while low-data sketches can retain strong performance.

  • YAQA can also be implemented iteratively, and its blockwise implementation traverses target blocks before quantizing and placing them into the output matrix.
  • YAQA initializes a quantized weight matrix, computes block LDL decompositions of positive-definite input- and output-side Hessians, and repeatedly updates until convergence.
  • The fixed-point update quantizes W + L_OΔW + ΔWL_I, where ΔW is the difference between the original and current quantized weights.
  • The convergence lemma states that Equation 3 converges in at most snd(L) steps, while the dependency-graph proof identifies this quantity with the graph depth.
  • The Kronecker-product result gives snd(L1 ⊗ L2) = snd(L1) + snd(L2), supporting convergence analysis for separable input- and output-side factors.
  • Theoretical bounds analyze blockwise nearest or stochastic rounding under positive-definite Hessian factors and incoherence assumptions, with error expressed through trace terms and Hessian approximation similarity.
  • The implementation collects input- and output-side Hessian information from activations and gradients during custom forward and backward passes.
  • Using as few as 2K sequences for Sketch B yields state-of-the-art KL divergence and downstream performance, despite 64K sequences being used for the main results.
Loading 2505.22988v3…