PaperlayerView exact revision on arXiv

Source-linked AI summary

Align, Then Correct: Training-Free Two-Stage Low-Rank Compensation for Extremely Quantized Large Language Models

Seobin Song, Geonho Lee, Janghwan Lee, Jungwook Choi

arXiv:2610.08164v1cs.LGcs.CL

TL;DR

At 2 bits, weight quantization still incurs severe accuracy degradation, while existing training-free compensators use symmetric calibration and omit the remaining first-order signal. This paper introduces a two-stage closed-form compensation framework; on held-out C4, it closes 51% of the FP16 perplexity gap on Qwen3-8B and 84% on Qwen3-4B, versus 31% and 63% for the strongest baseline.

  • Problem

    At 2 bits, post-training quantization still incurs severe accuracy degradation, while existing training-free compensators share symmetric calibration and omit the remaining first-order signal.

    Source: §1 · pp. 1–2, Remaining First-Order Signal after… · p. 4

  • Method

    A two-stage closed-form framework first uses Fisher-weighted asymmetric alignment, then applies a rank-constrained natural-gradient step on the compensated model.

    Source: §1 · pp. 1–2

  • Results

    On held-out C4, it closes 51% of the FP16 perplexity gap on Qwen3-8B and 84% on Qwen3-4B, versus 31% and 63% for the strongest baseline, with gains across seven-task accuracy, bit-widths, and quantizers.

    Source: §1 · p. 2, Main Results · p. 7

  • Takeaways & Limitations

    The results show substantial held-out C4 recovery alongside consistent gains across evaluated tasks, bit-widths, and quantizers.

    Source: §6 · p. 8

    The framework relies on layer-wise K-FAC curvature approximations and a fixed quantized backbone, while Stage 2 adds per-block forward and backward construction cost.

    Source: §6 · p. 8

Abstract

from arXiv · show

Low-rank quantization error compensation (LQEC) recovers the accuracy lost under aggressive weight quantization by attaching a closed-form rank-$r$ adapter beside each frozen quantized weight, without any training. We show that existing compensators are limited by two shared simplifications. They calibrate symmetrically, evaluating the full-precision and compensated weights on the same activation, which yields a compensation target that is inherently high-rank -- so a fixed rank budget captures only a small fraction of it. And they minimize only the second-order term of the loss, although the compensated model is not stationary: a first-order descent direction larger than the applied compensation itself remains in every layer, and no reconstruction objective can absorb it. We propose a two-stage closed-form framework that removes both simplifications. Stage 1 aligns each layer's output with the full-precision model under a Fisher-weighted asymmetric objective, concentrating the rank budget on a rank-compressible target. Stage 2 re-measures statistics on the compensated model and applies a rank-constrained natural-gradient step that absorbs the remaining first-order signal. Every adapter is the result of a single truncated SVD; backward passes serve only to collect statistics. At 2 bits under QuIP#, our method reduces WikiText-2 perplexity from 12.43 to 10.26 on Qwen3-8B and from 21.11 to 13.22 on Qwen3-4B. On the held-out C4 corpus, it recovers 51% and 84% of the gap to FP16, versus 31% and 63% for the strongest baseline, with consistent gains in the seven-task zero-shot average, at higher bit-widths, and under a distinct quantizer.

1 Introduction

The paper proposes a training-free, closed-form two-stage low-rank compensation method: first align quantized-layer outputs to the full-precision model, then recover the remaining first-order loss signal. At 2 bits under QuIP#, it improves perplexity and held-out C4 gap recovery over the strongest baseline.

Background

  • LQEC attaches closed-form low-rank adapters beside frozen quantized weights, offering training-free compensation with small memory overhead and compatibility with any quantizer.Adapters are computed from a small calibration set in minutes on a single GPU, while training-based alternatives require renewed training for each bit-width, rank, and model.Source: §1 · pp. 1–2

Motivation and limitations

  • Existing training-free compensators use symmetric calibration and only the quadratic loss term, yielding rank-dispersed targets and leaving a dominant first-order descent signal unaddressed.A rank-64 adapter captures only a small fraction of the symmetric target, while the remaining first-order direction exceeds the applied compensation in every layer and by more than an order of magnitude in most layers.Source: §1 · p. 1

Stage 1

  • Stage 1 uses a Fisher-weighted asymmetric objective to match full-precision outputs on quantized activations, aligning the rank budget with loss sensitivity.It refreshes input statistics while solving blocks front to back; the paper identifies it as the first LQEC objective to use the full output Fisher and distinguish the two activations.Source: §1 · pp. 1–2

Stage 2

  • Stage 2 remeasures statistics on the compensated model and applies a rank-constrained natural-gradient step to recover the remaining first-order term.Including the existing adapter in the target yields a rank-r solution by construction; the paper describes this as the first training-free compensation step solved against the compensated model itself.Source: §1 · p. 2

Results

  • At 2 bits under QuIP#, perplexity falls from 12.43 to 10.26 on Qwen3-8B and from 21.11 to 13.22 on Qwen3-4B; C4 gap recovery is 51% and 84% versus 31% and 63%.The gains also carry over to seven-task zero-shot commonsense results, higher bit-widths, and a distinct quantizer; the contribution summary reports the largest gain from Stage 2 using 8K calibration tokens.Source: §1 · p. 2

2 Background

The background frames quantization compensation as minimizing curvature-weighted loss and describes low-rank adapters as a closed-form way to recover accuracy. Existing LQEC methods share symmetric calibration and a quadratic-only objective, while differing in their approximations to curvature and activation statistics.

Second-Order Analysis of Quantization Error

  • Quantization-induced loss is analyzed with a second-order Taylor expansion, but existing methods assume the pretrained model has converged and retain only the quadratic term.Because the weight-space Hessian is too large for LLM-scale computation, the quadratic term is expressed in layer-output space and uses a calibration-averaged empirical Fisher.Source: Second-Order Analysis of Quantization… · p. 2
  • The curvature approximation proceeds from the Hessian to the Fisher, empirical Fisher, and K-FAC, which approximates the empirical weight Fisher as Hx ⊗ Hz.Hz is the empirical Fisher defined from output gradients, and Hx is the input second moment.Source: Second-Order Analysis of Quantization… · p. 2

Low-Rank Quantization Error Compensation

  • Existing LQEC methods attach a rank-r adapter beside each frozen quantized weight but share symmetric calibration and minimize only the quadratic loss term despite quantization moving the model from convergence.Symmetric calibration evaluates the full-precision and compensated weights on a shared input, leaving the discrepancy between full-precision and quantized-model activations unaddressed.Source: Low-Rank Quantization Error Compensation · pp. 2–3
  • At rank 64, the asymmetric objective captures substantially more energy than the symmetric objective, making it more compatible with low-rank compensation.This comparison uses mlp.down_proj in layer 0 of 2-bit Qwen3-8B, calibrated on 16,384 WikiText-2 tokens.Source: Low-Rank Quantization Error Compensation · p. 3, Figure 1 · p. 3
  • LQEC methods differ in their Fisher and input-statistic approximations: LoftQ uses identity factors, several methods whiten the input axis, LQ-LoRA uses a diagonal Fisher approximation, and no existing method uses full Hz.LQER, QERA, EoRA, and CLoQ use the full activation second moment; ProjQ also jointly re-optimizes the quantized weight and adapter.Source: Low-Rank Quantization Error Compensation · p. 3

3 Motivations

Under aggressive 2-bit quantization, symmetric calibration leaves a high-rank compensation target, while second-order compensation leaves a substantial first-order descent signal. These limitations motivate asymmetric alignment and an explicit first-order correction.

3 Motivations

  • In the 2-bit regime, the section examines two simplifications—symmetric calibration and omitting the first-order loss term—and finds neither justified.Symmetric calibration creates an inherently high-rank target, while after strong compensation the model remains non-stationary and has a task-loss descent direction.Source: §3 · p. 3

High-Rank Targets under Symmetric Calibration

  • At rank 64, the asymmetric target concentrates more energy in leading directions than the symmetric target, making it more compatible with low-rank compensation.The asymmetric target retains both full-precision and quantized activations; the benefit depends on combining it with the full output Fisher.Source: High-Rank Targets under Symmetric… · p. 4

Remaining First-Order Signal after Compensation

4 Methodology

The method uses two closed-form stages with frozen quantized weights: asymmetric Fisher-weighted output alignment, followed by a rank-constrained natural-gradient correction. Each stage uses one truncated SVD per layer, with no iterative optimization or training.

Stage 1: Fisher-Weighted Output Alignment

  • Stage 1 aligns each layer’s output with the full-precision model under an asymmetric objective, directing the rank budget toward a compact compensation target.The quantized weights remain frozen; only rank-r adapters are added.Source: §4 · p. 4

Stage 2: Low-Rank Natural-Gradient Correction

Fisher-weighted asymmetric objective.

  • The Fisher-weighted objective ranks output-error directions by their contribution to loss and reduces to a reduced-rank regression solved by one truncated SVD.The target accounts for the adapter acting on quantized inputs, and the output Fisher is estimated once on the original quantized model for reuse throughout Stage 1.Source: Fisher-weighted asymmetric objective · p. 4

Shrinking the closed-form solution.

  • Stage 1 shrinks its closed-form adapter by a fraction η because finite-sample singular values can overestimate their population counterparts.Held-out calibration confirms that re-evaluated singular values are consistently lower, with the gap narrowing as the calibration set grows.Source: Shrinking the closed-form solution · p. 4

Sequential block-wise calibration.

  • Stage 1 processes decoder blocks front to back, refreshing input moments before each block so downstream adapters use updated activations and retain the asymmetric cross moment.All linear layers within a block share the same measurements.Source: Sequential block-wise calibration · p. 4, Figure 3 · p. 5

First-order objective.

  • Because the Stage-1 model is not stationary, the Stage-2 loss approximation retains a first-order gradient term alongside the curvature-weighted quadratic term.The gradient is measured at the compensated effective weight; reconstruction objectives omit this first-order signal.Source: First-order objective · p. 5

Closed-form solution.

  • The Stage-2 rank-r adapter is the curvature-weighted best approximation to the Stage-1 adapter moved by a natural-gradient step; truncating them together preserves rank r.Truncating only the step and adding it to the original adapter could produce rank 2r.Source: Closed-form solution · pp. 5–6

Trust-ratio step scaling.

  • The trust ratio scales the natural-gradient step relative to the Stage-1 adapter and clips at 1, the exact minimizer of the quadratic model.A sweep found a broad stable optimum at τ = 0.2–0.3; τ = 0 reproduces Stage-1 perplexity, while τ = 1 leaves the regime where the local quadratic model is accurate. Stage 2 freshly collects gradients and input and output statistics at every block.Source: Trust-ratio step scaling · p. 6, Block-wise re-measurement · p. 6

5 Experiments

The two-stage method improves quantized-model performance across bit-widths, corpora, and quantizers, with especially large gains at 2 bits. Ablations show that its aligned Stage 1 and first-order Stage 2 correction work best together.

Main Results

  • At 3 and 4 bits, Qwen3-8B reaches lower WikiText-2 and held-out C4 perplexity than FP16, with CSQA within 0.3 of FP16.WikiText-2 perplexity is 8.54 and 8.22 versus FP16 9.73; C4 is 14.56 and 14.15 versus 15.24; CSQA is 69.11 and 69.41 versus 69.38. Under 3-bit groupwise RTN on LLaMA-3.1-8B, CSQA rises from 56.26 to 62.50, the highest among methods.Source: Table 2 · p. 6, Table 3 · p. 6, Main Results · p. 7

Models and tasks.

  • Evaluation covers Qwen3-8B, Qwen3-4B, LLaMA-3.1-8B, and LLaMA-3.2-1B, using perplexity on WikiText-2 and C4 and mean zero-shot accuracy across seven common-sense tasks.Perplexity is reported at sequence length 2048; the CSQA average is unweighted.Source: Models and tasks · p. 6

Quantization.

  • QuIP# experiments use 2-, 3-, and 4-bit quantization with no layerwise finetuning; the study also tests 3-bit groupwise RTN with group size 128.For the RTN setting, baselines and Stage 1 use 262K WikiText-2 calibration tokens, while Stage 2 uses 8K; η = 0.3 and τ = 0.1 apply only to RTN.Source: Quantization · p. 6

Baselines.

  • All methods use rank-64 adapters per linear layer and WikiText-2 training-split calibration; baselines include EoRA, QERA-Approx, ProjQ’s low-rank compensation stage, and Fisher-weighted LQ-LoRA.Stage 1 uses 262K calibration tokens and Stage 2 uses 8K for both Qwen3 models; all methods share one quantizer.Source: Baselines · pp. 6–7, Our method · p. 7

Main Results

  • At 2 bits, WikiText-2 perplexity falls from 12.43 to 10.26 on Qwen3-8B and 21.11 to 13.22 on Qwen3-4B; held-out C4 gap recovery reaches 51% and 84%, versus 31% and 63%.CSQA rises from 62.81 to 64.51 on Qwen3-8B and from 57.47 to 60.05 on Qwen3-4B; a gap to the FP16 reference remains for Qwen3-4B on C4 and CSQA.Source: Main Results · p. 7
  • At 2 bits, EoRA, QERA, and ProjQ are within 0.07 perplexity of one another, while LQ-LoRA reaches 11.62 and the proposed method reaches 10.26.The paper attributes the remaining improvement to asymmetric calibration and including the first-order term; the correction is in the adapter, so the procedure applies unchanged across quantizers.Source: Main Results · p. 7
  • Stage 1’s asymmetric target alone does not improve perplexity; the full output Fisher lowers it to 10.26, and Stage 1 followed by Stage 2 outperforms either stage alone or reversed order.The reported perplexities are 12.50 for the asymmetric target without output Fisher versus 12.34 for the symmetric objective; Stage 1 alone reaches 12.48, Stage 2 alone 12.81, and reversed order 12.55.Source: Main Results · p. 7, Table 4 · p. 7, Table 5 · p. 7
  • The procedure uses closed-form adapters without iterative optimization, and Stage 2 collects statistics from only 8K calibration tokens—1/32 of Stage 1’s 262K.The experiments ran on a single A100 GPU with 40 GB memory.Source: Main Results · p. 7

6 Conclusion · A Notation and Preliminaries

The paper concludes that training-free low-rank compensation works by aligning layer outputs with a Fisher-weighted asymmetric objective, then applying a rank-constrained natural-gradient step. At 2 bits, it recovers 51–84% of the held-out C4 perplexity gap to FP16 on Qwen3 models, versus at most 63% for the strongest baseline.

6 Conclusion

  • The training-free framework uses Fisher-weighted asymmetric alignment to form rank-compressible targets, then a rank-constrained natural-gradient step to recover remaining first-order loss signals.Each compensation is closed-form via truncated SVD.Source: §6 · pp. 7–8
  • At 2 bits, the method recovers 51–84% of the held-out C4 perplexity gap to FP16 on Qwen3 models, versus at most 63% for the strongest baseline.The conclusion also reports consistent gains across bit-widths and quantizers.Source: §6 · p. 8
  • The framework uses layer-wise K-FAC curvature approximations with a fixed quantized backbone, and Stage 2’s per-block forward and backward passes add construction cost.The conclusion identifies joint optimization of the quantizer and compensator as a natural next step.Source: §6 · p. 8

A Notation and Preliminaries

  • The notation defines the current effective weight as the quantized weight plus the low-rank adapter factors, with factor matrices sized by the rank budget.The symbols are W_eff, W_hat, A, and B; A and B have shapes m × r and n × r.Source: A Notation and Preliminaries · p. 10
  • The notation distinguishes full-precision and low-precision inputs, full-precision outputs, outputs before and after the current adapter, and their output error and perturbation.These quantities are represented by x_t, x_hat_t, z_t, z_hat_0,t, z_hat_t, e_t, and delta z_t.Source: A Notation and Preliminaries · p. 10
  • Stage 1 distinguishes the symmetric target, formed from the weight difference on quantized inputs, from the asymmetric target, formed using full-precision and quantized model outputs.The targets are denoted T_sym and T_asym, respectively.Source: A Notation and Preliminaries · p. 10
  • Table 6 states that hatted inputs come from the quantized model in Stage 1 and the compensated model in Stage 2; subscript 0 marks the pre-adapter state, and tilde marks a cross-model statistic.These conventions clarify how the paper’s notation changes across stages.Source: Table 6 · p. 10

B Proof of Theorem 1

The proof reduces the damped objective to a rank-r Frobenius approximation in whitened coordinates, then obtains the closed-form solution by truncated SVD and transforms it back.

B Proof of Theorem 1

  • The proof expands the token-level output perturbation and quadratic objective, with the two scalar cross terms equal because they are transposes.Source: B Proof of Theorem 1 · p. 10
  • Damping makes the covariance matrices positive definite, enabling invertible symmetric square roots and a completed-square reduction to Frobenius approximation in whitened coordinates.Source: B Proof of Theorem 1 · pp. 10–11
  • Because whitening preserves rank, the feasible matrices remain rank at most r; the truncated SVD solution is then transformed back to yield the closed form before shrinkage.Source: B Proof of Theorem 1 · p. 11

C Proof of Theorem 2

The proof derives a tractable K-FAC quadratic model for Stage 2 and shows that its rank-constrained solution comes from whitening the target, truncating its SVD, and transforming back. It also explains why the rank constraint must apply to the stored adapter sum.

C Proof of Theorem 2

  • A second-order output-space Taylor model uses an empirical-Fisher outer product, while K-FAC factorizes the expected Kronecker term to avoid storing full weight-space curvature.K-FAC replaces the expected Kronecker product with the product of its expectations.Source: C Proof of Theorem 2 · p. 11
  • Completing the square cancels terms linear in the Stage-2 factors; whitening shows the target incorporates the unconstrained K-FAC natural-gradient step added to the Stage-1 adapter.The resulting whitened target is the Stage-1 adapter plus that natural-gradient step.Source: C Proof of Theorem 2 · p. 12
  • Because invertible whitening factors preserve rank, Eckart–Young yields a rank-r truncated SVD of the whitened target residual, whose retained factors are transformed back into Stage-2 adapter factors.The retained singular values are split symmetrically when forming the factors.Source: C Proof of Theorem 2 · p. 12
  • Constraining the stored adapter sum is essential: a rank-r increment may yield a sum of rank 2r, while a full-rank increment may leave the sum rank r.Projecting the sum after optimizing the increment can lose the stated-objective minimizer and provides no descent guarantee.Source: C Proof of Theorem 2 · p. 12

D Damping and Constants · E Algorithm · F Diagnostics for the Two Stages

The appendix specifies damping, hyperparameters, and the two-stage sweep, then reports diagnostics motivating Stage-1 shrinkage and Stage-2 step-size scaling. It distinguishes the damping rules used in the two stages and identifies the scope of the calibration settings.

D Damping and Constants

  • Stage 1 damps the input and output second-moment factors before eigendecomposition, using εi = 3×10−2 and εo = 10−2.The ridge is proportional to each factor’s mean diagonal; neither the gradient nor the cross moment is damped.Source: D Damping and Constants · pp. 12–13
  • Stage 2 clamps negative second-moment eigenvalues to zero, then adds εi = 10−2 and εo = 10−2 times the largest eigenvalue before constructing H±1/2.Damping is applied to the input and output factors after eigendecomposition.Source: D Damping and Constants · p. 13
  • All compensated linear layers use rank r = 64, and Stage 2 uses 16 × 512 calibration tokens in every setting.For QuIP# experiments, η = 0.1 across models and bit-widths, while τ varies by model and bit-width; the 3-bit RTN LLaMA-3.1-8B experiment uses η = 0.3 and τ = 0.1.Source: D Damping and Constants · p. 13, Table 7 · p. 13

E Algorithm

  • Algorithm 1 performs a Stage-1 sweep solving the Fisher-weighted asymmetric objective, followed by a Stage-2 sweep applying rank-constrained natural-gradient corrections to the same target layers.The sweeps operate on target linear layers in each block.Source: E Algorithm · p. 13

F Diagnostics for the Two Stages

  • Stage-1 singular-value stability motivates shrinkage η, while the Stage-2 relative-step-size sweep motivates trust-ratio scaling and clipping it at one.These are the two diagnostics reported in the appendix.Source: F Diagnostics for the Two Stages · p. 13

Stage-1 Shrinkage Diagnostic

On Qwen3-4B quantized to 2 bits with QuIP#, held-out singular values show finite-sample inflation that lessens with more calibration data. The shrinkage sweep supports using η = 0.1 in the full two-stage pipeline to leave room for Stage 2.

Stage-1 Shrinkage Diagnostic

  • Only 0.68% of top-64 held-out-to-in-sample singular-value ratios exceed one; quadrupling calibration data raises their mean from 0.6403 to 0.8517.Every sampled linear layer moves toward one, and η = 0.1 lies below these ratios, suggesting it also controls the magnitude of cascaded Stage-1 updates.Source: Figure 4 · p. 14, Stage-1 Shrinkage Diagnostic · p. 14
  • Moderate Stage-1 shrinkage consistently improves held-out C4 perplexity over no compensation, while η = 0.5 is unstable; η = 0.2 gives the best Stage-1-only perplexity.The full method uses η = 0.1 because shrinkage is selected for the two-stage pipeline, where a more conservative Stage-1 adapter leaves greater scope for Stage 2’s natural-gradient correction.Source: Table 8 · p. 14, Stage-1 Shrinkage Diagnostic · p. 14

Stage-2 Relative Step Size … H Computation Count

Stage 2 performs best across a broad range of relative step sizes, while adapter construction adds offline time and reusable-cache storage. The computation is dominated by Stage 1’s paired model forwards, even though Stage 2 performs backward passes at every block.

Stage-2 Relative Step Size

  • Perplexity is 13.22, 13.28, and 13.24 at τ = 0.2, 0.25, and 0.3, but rises to 83.38 at τ = 1.At τ = 0, perplexity is 19.8200, matching Stage 1 at 19.8194; the sweep supports scaling the step relative to the current adapter and clipping the full-step coefficient at one.Source: Stage-2 Relative Step Size · p. 15

G Adapter Construction Cost

  • At inference, every method considered stores one rank-64 adapter per linear layer, so this method’s additional cost is confined to offline adapter construction.The reported timings use one run on a single NVIDIA A100 GPU with 40 GB of memory.Source: G Adapter Construction Cost · p. 15

Construction Time

  • For Qwen3-8B at 2 bits, construction takes approximately 53 minutes with a precomputed Ĥz0 cache, or 1 hour 3 minutes including its approximately 10-minute first-run build.The cache is reusable while the model, calibration data, and calibration size remain unchanged; baseline methods take 16–37 minutes. Table 10 reports Qwen3-8B construction time, with Stage 1 assuming a precomputed cache.Source: Construction Time · p. 15, Table 10 · p. 15, Table 9 · p. 15

Storage and Memory

H Computation Count

  • Stage 1 uses 2Nb + 1 forward passes and one backward pass; Stage 2 uses Nb forward and Nb backward passes.Stage 2 jointly collects g, Ĥx, and Ĥz for all Nm target linear layers at each block-level backward pass; Nm = 7 for the Qwen3 models.Source: H Computation Count · p. 15, Table 16 · p. 15
  • Stage 1’s paired full-precision and compensated-model forwards dominate total computation because its passes use 32 times more tokens than Stage 2’s.For Qwen3-4B, forward-token volumes are 19,136,512 versus 294,912, while backward volumes are similar at 262,144 versus 294,912; reusing a compatible Ĥz0 cache omits Stage 1’s initial backward pass.Source: H Computation Count · p. 15

I Full Per-Model and Per-Task Results

The section presents full per-model and per-task result breakdowns, including individual zero-shot tasks and, for Qwen3, WikiText-2 and held-out C4 perplexity. The supplied passages describe result coverage and table scope but do not provide cell values.

I Full Per-Model and Per-Task Results

  • The breakdown covers all three bit-widths for Qwen3-8B and the 2-bit setting for Qwen3-4B and LLaMA-3.2-1B, with seven zero-shot tasks used for CSQA Avg.For Qwen3, it also reports WikiText-2 and held-out C4 perplexity.Source: I Full Per-Model and Per-Task Results · p. 16
  • At 2 bits under QuIP#, the Qwen3-4B table reports WT2 and C4 perplexity, seven zero-shot accuracies, and their unweighted mean as CSQA Avg.The caption says best results are shown in bold, but the supplied passage does not include the table’s cell values.Source: Table 14 · p. 17
  • For LLaMA-3.2-1B at 2 bits under QuIP#, the table reports seven zero-shot tasks and unweighted CSQA Avg; Stage 1 and Stage 2 use distinct calibration settings.Stage 1 uses 128 × 2048 WikiText-2 calibration tokens with η = 0.1, and Stage 2 uses 16 × 512 tokens with τ = 0.2; FP16 is a reference.Source: Table 15 · p. 17

Related papers

Loading 2610.08164v1…