PaperlayerView exact revision on arXiv

Source-linked AI summary

SchurQuant: Groupwise Discrete Optimization for Layer-Wise LLM Quantization

Gunjun Lee, Sehwan Son, Younjoo Lee, Byungjun Kim, Jung Ho Ahn

arXiv:2608.15567v1cs.LG

TL;DR

Low-bit weight-only PTQ can sharply degrade LLM accuracy, while existing backpropagation-free optimizers do not fully account for continuous suffix corrections or adaptive quantization grids. SCHURQUANT addresses this with Schur-conditioned group optimization and teacher-informed reconstruction, achieving the highest mean zero-shot accuracy among evaluated backpropagation-free PTQ baselines across eight Llama and Qwen models.

  • Problem

    Weight-only PTQ supports memory-constrained LLM deployment, but conventional methods can collapse in perplexity and downstream accuracy at 2–3 bits.

  • Method

    SCHUROPT eliminates the optimal continuous suffix response to form a Schur-reduced group objective, then refits quantization parameters and codes; SCHURQUANT adds teacher-informed reconstruction components.

  • Results

    Across eight Llama and Qwen models, SCHURQUANT achieves the highest mean zero-shot accuracy among evaluated backpropagation-free PTQ baselines, reaching 50.83%, 65.11%, and 67.54% at 2–4 bits.

  • Takeaways & Limitations

    Layer-wise reconstruction is a loose surrogate for final language-model loss, so tighter reconstruction does not consistently improve end-model metrics at higher precision.

  • Takeaways & Limitations

    SCHURQUANT has higher offline quantization cost than existing PTQ methods, limiting scalability to larger models.

Abstract

from arXiv · show

Weight-only post-training quantization (PTQ) enables the deployment of large language models under tight memory budgets, but accuracy often collapses at 2-3 bits. Existing backpropagation-free PTQ optimizers have two limitations: group decisions ignore the correction that the remaining continuous suffix can absorb, and discrete refinements typically keep the affine quantization grid fixed. We introduce SCHUROPT, which analytically eliminates the suffix's optimal continuous response, yielding an exact groupwise quadratic with Schur-complement curvature. It then alternates closed-form row-wise scale/zero-point refitting with coordinate descent over integer codes. With the GPTQ objective fixed, SCHUROPT improves mean zero-shot accuracy on 2-bit Qwen3-4B by 11.88 percentage points (pp). At higher precision, however, tighter reconstruction does not consistently improve end-model metrics. SCHURQUANT therefore combines SCHUROPT with quantized-prefix teacher reconstruction, reference-weight regularization, residual-add targets, and teacher-decision token weighting. Across eight Llama and Qwen models, SCHURQUANT achieves the highest mean zero-shot accuracy among the evaluated backpropagation free PTQ baselines, outperforming the strongest baseline by 9.65 pp at 2 bits.

1 INTRODUCTION

Weight-only PTQ addresses LLM memory constraints but often fails at 2–3 bits, motivating SCHUROPT’s Schur-conditioned group optimization and SCHURQUANT’s improved reconstruction surrogate. With GPTQ’s objective fixed, SCHUROPT substantially improves 2-bit Qwen3-4B results, while SCHURQUANT leads evaluated backpropagation-free baselines across eight models.

  • Motivation: Weight-only PTQ reduces pretrained weights to low precision using limited calibration data, but conventional methods sharply degrade perplexity and downstream accuracy at 2–3 bits.Deployment constraints are increasingly driven by memory capacity and bandwidth, especially on edge devices.
  • SCHUROPT: SCHUROPT conditions each group decision on a fixed quantized prefix and optimally responding continuous suffix, replacing raw curvature Gcc with Schur-complement curvature S.Analytically eliminating the suffix yields the exact conditional quadratic and linear term for the current group.
  • SCHUROPT: SCHUROPT jointly refits row-wise scales and zero-points in closed form while refining integer codes through row-parallel coordinate descent.The method partitions the input dimension into chunks corresponding to one quantization group and keeps the suffix continuous during current-chunk optimization.
  • Results: 11.88 pp: SCHUROPT improves mean zero-shot accuracy on 2-bit Qwen3-4B while reducing WikiText-2 perplexity from 258.96 to 81.95 under GPTQ’s fixed objective.At 3 and 4 bits, perplexity worsens and mean zero-shot accuracy changes only marginally, exposing a mismatch between layer reconstruction and final language-model loss.
  • SCHURQUANT: SCHURQUANT augments SCHUROPT with reference-weight regularization, residual-add reconstruction, and teacher-decision token weighting without changing the optimizer.These additions modify the surrogate’s sufficient statistics to better address the gap between layer-wise reconstruction and end-model performance.
  • Results: Across eight Llama and Qwen models at 2–4 bits, SCHURQUANT achieves the highest mean zero-shot accuracy among the evaluated backpropagation-free PTQ baselines, with its largest advantage at 2 bits.The experiments separate optimizer effects from objective effects.

2 BACKGROUND AND MOTIVATION

This section defines grouped affine weight quantization and motivates SCHURQUANT from GPTQ’s nearest-level discrete decisions, limited operation-local scope, and the opportunity to account exactly for continuous suffix corrections. It situates the approach among coordinate-descent, quantization-aware, calibration-optimized, and representation-changing PTQ methods.

  • Quantization setup: Grouped affine quantization maps weights to integer codes and assigns each row–group its own quantizer, with group size g = 128 used throughout.Unsigned and symmetric code ranges differ, while FP16 metadata yields approximately b + 16/g effective bits.
  • Limitations of existing PTQ: GPTQ uses inverse-Hessian corrections after nearest-level rounding, so curvature changes subsequent weight corrections rather than the discrete decision itself.Its sequential, operation-local procedure does not account for activation error from already quantized prefixes.
  • Discrete optimization: Existing discrete PTQ methods use coordinate optimization, but the cited prior updates do not first analytically eliminate the adjustable suffix before applying parameter and code updates.QuantEase proves objective non-increase for cyclic coordinate descent, while CDQuant contrasts greedy coordinate descent with GPTQ’s predetermined pass and extends it to group quantization.
  • Schur-complement motivation: The Schur complement gives the exact local curvature after the continuous suffix makes its optimal correction, subtracting the curvature that the suffix can absorb.For quantization, the current group is the discrete block and the continuous suffix is the adjustable block.
  • Related quantization approaches: QAT is more expensive because it requires repeated forward and backward passes, while calibration-optimized and representation-changing PTQ methods modify objectives, transformations, rotations, or codebooks instead.Examples include EfficientQAT, OmniQuant, AWQ, rotation-based methods, and vector-quantized methods.

3 SCHUROPT AND SCHURQUANT

SCHUROPT optimizes each quantized group after analytically accounting for the continuous suffix response, then refits affine parameters and integer codes using reduced curvature. SCHURQUANT augments this optimizer with reference-weight anchoring, residual-add targets, and teacher-decision token weighting while leaving the optimizer unchanged.

  • SCHUROPT: SCHUROPT eliminates the suffix’s optimal continuous response, so group selection minimizes residual loss after the best suffix correction rather than isolated group error.The resulting Schur-complement curvature captures the suffix-absorbable component, while the linear term accounts for prefix error and suffix correction.
  • SCHUROPT: SCHUROPT uses reduced curvature to fit each row’s scale and zero-point and to choose integer codes, including cross-column coupling within the group.This differs from independent rounding and from GPTQ’s scalar-curvature nearest-level decision.
  • SCHURQUANT: SCHURQUANT adds a reference-weight anchor, residual-add targets, and teacher-decision token weighting to reduce overfitting of the layer-wise surrogate without changing SCHUROPT.These choices modify the sufficient statistics or objective terms, while the optimizer remains unchanged.
  • SCHUROPT: The algorithm alternates exact row-wise affine refitting with coordinate descent over integer codes, followed by a final refit; each block update is non-increasing.With finite code space, the procedure converges to a coordinate-wise fixed point and requires no explicit suffix error-propagation step.

4 EXPERIMENTS

Across eight Llama and Qwen models, SCHURQUANT delivers its largest gains at 2 bits, where it leads both perplexity and mean zero-shot accuracy, while improvements narrow at higher precision. Ablations attribute the strongest low-bit gains to Schur conditioning, grid refitting, and the full teacher-reconstruction configuration.

  • Main results: SCHURQUANT achieves the lowest PPL on all eight models at 2 bits, preventing collapses such as Llama3.2-3B’s 10549.22 to 29.39.At 3 bits, it is best on seven of eight models; Qwen3-8B instead favors GPTQ, 10.92 versus 11.16.
  • Main results: 50.83% mean zero-shot accuracy at 2 bits makes SCHURQUANT best on all eight models, exceeding QEP by 9.65 pp and GPTQ by 11.58 pp.At 3 bits, it averages 65.11%, 2.16 pp above QEP, and wins on seven of eight models.
  • Main results: 67.54% mean accuracy at higher precision leaves SCHURQUANT only 0.68 points ahead of the strongest baselines, reflecting diminishing discrete-assignment differences on finer grids.The section reports mean zero-shot accuracy across PIQA, ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, and BoolQ.
  • Ablations: With the grid fixed, Schur conditioning reduces controlled loss from 1.645% to 1.165% and PPL from 2344.82 to 324.81; with refitting, loss falls from 0.872% to 0.550%.The refitted loss reduction is 36.9% relative, showing the gain is not attributable to coordinate descent or grid refitting alone.
  • Ablations: 11.88 pp is SCHUROPT’s 2-bit accuracy gain under the fixed GPTQ objective, while the full configuration gives the best 2-bit mean and best 2/3-bit PPL.SCHUROPT changes PPL from 258.96 to 81.95 and accuracy from 36.77% to 48.65%; refinement perplexity improves rapidly and saturates around R = 4, but R = 16 is used elsewhere for stability.

5 CONCLUSION

SCHUROPT analytically removes the optimal continuous suffix response, optimizes groups under a Schur-reduced quadratic, and refits row-wise scales and zero-points in closed form. On 2-bit Qwen3-4B with the GPTQ objective fixed, replacing GPTQ’s optimizer substantially lowers perplexity and improves mean accuracy.

  • SCHUROPT: SCHUROPT eliminates the optimal continuous suffix response and optimizes each group using the resulting Schur-reduced quadratic.The method also includes a closed-form row-wise scale/zero-point refit.
  • SCHUROPT: 11.88 pp mean accuracy improvement results on 2-bit Qwen3-4B when GPTQ’s optimizer is replaced while keeping the GPTQ objective fixed.Perplexity decreases from 258.96 to 81.95 under the same comparison.
  • SCHURQUANT: SCHURQUANT pairs SCHUROPT with quantized-prefix teacher reconstruction and reference-weight regularization.The supplied passage introduces this combination but does not provide further outcome details here.

6 LIMITATIONS

SCHURQUANT has higher offline quantization cost than existing PTQ methods, limiting scalability to larger models; layer-wise surrogate optimization also does not always improve end-task performance, and evaluation covers only asymmetric quantization.

  • Computational cost: SCHURQUANT incurs higher offline quantization cost than existing PTQ methods because it repeatedly calibrates and performs discrete refinement, limiting scalability to larger models.The paper identifies repeated calibration and discrete refinement as the sources of the added cost.
  • Objective-to-task transfer: Optimizing the layer-wise surrogate does not always translate into improved end-task performance.The limitation concerns the relationship between surrogate-objective optimization and downstream task results.
  • Evaluation scope: The evaluation is limited to asymmetric quantization, leaving broader quantization settings for future work.The paper explicitly identifies broader quantization settings as outside the current evaluation scope.

A CALIBRATION

Calibration quantization to model outputs weights errors by the activation distribution rather than treating all weight directions equally. Token weighting and residual-add targets modify the empirical curvature and cross-Gram statistics used by the quantizer.

  • Calibration objective: Quantization targets preservation of model outputs, measuring weight error through the calibration inputs’ total squared output error.The weight error is defined as E = W_ref − W, with calibration inputs collected in X = [x_1, ..., x_N].
  • Activation-weighted curvature: The Gram matrix XX^⊤ weights errors by activation strength, whereas data-free round-to-nearest implicitly replaces it with the identity.Strongly activated directions therefore receive greater importance than equally large errors in nearly inactive directions.
  • Activation-weighted curvature: Calibration estimates missing activation curvature from a small sample, while standard calibration assigns equal explicit mass to token columns.Nonnegative token weights ω_i generalize this derivation by changing the empirical activation distribution seen by the quantizer.
  • Weighted calibration: Teacher-decision weighting modifies weighted curvature, while the residual-add target independently modifies the weighted cross-Gram YΩX^⊤.These changes preserve the quadratic form while altering the sufficient statistics used for calibration.
  • Weighted calibration: The unweighted special cases recover the Hessians used by GPTQ and QEP.The cited baselines are GPTQ and QEP (Arai & Ichikawa, 2025).

B DETAILED DERIVATIONS · B.1 PROOF OF PROPOSITION 1

The proof isolates terms involving the fixed prefix, derives the suffix’s optimal continuous response, and completes the square. Combining the resulting terms yields the curvature S and linear coefficient T in Eq. 12, proving Proposition 1.

  • B.1 PROOF OF PROPOSITION 1: Terms containing only the fixed prefix WF are constant in the quadratic expansion.
  • B.1 PROOF OF PROPOSITION 1: Cross terms involving WF and Wc contribute to the effective linear coefficient Ceff.
  • B.1 PROOF OF PROPOSITION 1: The suffix linear coefficient is adjusted as r = Cr −WF GF r.
  • B.1 PROOF OF PROPOSITION 1: Collecting all terms that depend on (Wc, Wr) produces the joint expression up to an additive constant.
  • B.1 PROOF OF PROPOSITION 1: Holding Wc fixed and differentiating with respect to Wr identifies the stationary condition for the suffix.
  • B.1 PROOF OF PROPOSITION 1: The stationary condition gives the optimal continuous suffix response, and suffix-dependent terms can be completed to a square.
  • B.1 PROOF OF PROPOSITION 1: The first completed-square term vanishes at Eq. 26, while the remaining terms yield curvature S and linear coefficient T of Eq. 12.

B.2 PROOF OF PROPOSITION 2

For fixed zero-point o, the proof substitutes q_i = a_i u_i(o) into Eq. 15 and optimizes the resulting objective over the scale variables. It then establishes global exactness for fixed codes by projecting the unconstrained optimum and enumerating all finite b-bit zero-points.

  • Fixed zero-point: For fixed zero-point o, substituting q_i = a_i u_i(o) into Eq. 15 reduces the optimization to a scale-dependent substituted objective.This substitution is the proof’s starting step.
  • Scale optimization: When A_i(o) > 0, the unconstrained optimum is d_i(o)/A_i(o), projected onto a_i ≥ ϵ_a as specified by Eq. 16.The projection enforces the scale lower bound.
  • Global exactness: Enumerating the finite set of b-bit zero-points and selecting the smallest substituted objective is globally exact for fixed codes.This concludes the proof for fixed codes.

C QUANTIZATION RUNTIME · D QUANTIZATION SETUP

SCHURQUANT incurs a substantial offline quantization-time cost because it performs repeated refinement sweeps and full-model calibration before each Transformer layer. Experiments use a standardized GPTQ-style calibration setup with group size 128, row-wise asymmetric quantization, 16 refinement steps, and float16 computation.

  • C QUANTIZATION RUNTIME: R = 16: SCHURQUANT performs 16 sweeps and one current full-model calibration pass before each Transformer layer, in addition to the clean-cache pass.This makes the method an offline accuracy–quantization-time trade-off rather than an inference-speed result.
  • C QUANTIZATION RUNTIME: 45.8×/31.8×: At 2 bits, SCHURQUANT takes this long as GPTQ on Qwen3-4B/Llama2-7B.The corresponding 2-bit ratios versus QEP are 4.6×/5.8×.
  • C QUANTIZATION RUNTIME: 37.0/51.1× and 26.5/33.0×: The 3/4-bit SCHURQUANT-to-GPTQ ratios on Qwen3-4B/Llama2-7B are respectively these values.Table 6 measures end-to-end wall-clock time, including statistics collection and discrete optimization.
  • C QUANTIZATION RUNTIME: Statistics, optimizer time, and peak memory are separately instrumented, while dense evaluator checkpoints do not provide packed-size evidence.These measurements accompany the end-to-end runtime comparison.
  • D QUANTIZATION SETUP: g = 128: Unless stated otherwise, all weight-only experiments use group size 128, a row-wise asymmetric quantizer, R = 16 refinement steps, and float16 computation.These are the default quantization settings.
  • D QUANTIZATION SETUP: 128 random windows of length 2,048: Calibration samples are drawn from long documents in the C4 training split following GPTQ’s standard calibration protocol.The final SCHURQUANT configuration builds top-mismatch weights from all 128 calibration windows before quantizing each Transformer layer.
Loading 2608.15567v1…