PaperlayerView exact revision on arXiv

Source-linked AI summary

SandwichQuant: Which Parameters Matter Before and After Quantization?

Peng Xia, Junbiao Pang

arXiv:2608.24173v1cs.CV

TL;DR

Quantization correction lacks clear evidence about which parameter subspaces matter most under extreme low-bit settings. SandwichQuant studies these subspaces and adapts normalization-affine parameters before and after PTQ; across multiple LLM families and regimes, it consistently improves perplexity and mean downstream accuracy without adding an inference-time operator.

  • Problem

    Quantization correction lacks clear evidence about which parameter subspaces provide effective correction under extreme low-bit deployment settings.

  • Method

    The paper decomposes trainable parameters into backbone weights, normalization-affine parameters, and quantizer parameters, comparing these subspaces under matched budgets and a common target graph.

  • Results

    Across three LLM families and weight-only or joint weight–activation–KV regimes, SandwichQuant consistently improves perplexity and mean downstream accuracy without adding an inference-time operator.

  • Takeaways & Limitations

    Normalization-affine parameters are unusually high-leverage response controls for correcting structured quantization distortion within the studied settings.

  • Takeaways & Limitations

    The evidence is limited to dense 7B–8B LLMs and one deterministic seed, while severe clipping or rounding remains irreversible.

Abstract

from arXiv · show

Quantization correction methods usually optimize weights, quantization parameters, or reconstruction objectives, while the underlying parameter subspaces responsible for effective correction remain unclear. In this work, we study quantization correction from a parameter subspace perspective and reveal that correction capability is highly non-uniform across parameter groups. By decomposing trainable parameters into backbone weights, normalization-affine parameters, and quantization parameters, we show that the low-dimensional normalization-affine subspace provides a highly efficient correction direction under matched budgets. Based on this finding, we propose SandwichQuant, a two-stage normalization-affine correction framework that performs adaptation before and after quantization. The pre-stage improves quantization robustness, while the post-stage compensates residual errors after the quantized graph is fixed. Extensive experiments on vision models and large language models demonstrate consistent improvements under various low-bit quantization settings, validating the effectiveness of subspace-aligned correction.

1 Introduction

Quantization correction is studied as a parameter-subspace problem, revealing that normalization-affine parameters offer high-leverage correction before and after PTQ. SandwichQuant combines both stages and consistently improves low-bit LLM performance without adding an inference-time module.

  • Motivation: PTQ dominates deployment because it calibrates pretrained models with limited data, yet extreme low-bit precision can sharply degrade accuracy.QAT can provide strong accuracy but becomes increasingly expensive at LLM scale.
  • Motivation: Updating normalization-layer affine parameters can recover substantial functional error after quantization and alter the distribution presented to calibration before PTQ.This motivates examining whether affine adaptation should occur before PTQ, after PTQ, or on both sides.
  • Parameter-subspace analysis: The study decomposes trainable parameters into backbone weights W, normalization-affine parameters Φ, and quantizer parameters Ω under matched budgets and a common target graph.Response analysis, controlled interventions, and task-aware error geometry assess correction leverage across these subspaces.
  • SandwichQuant: SandwichQuant learns affine parameters before PTQ, reruns PTQ from scratch, then learns independent affine parameters on the fixed low-bit graph.The pre-stage performs distribution preconditioning, while the post-stage corrects residual response errors without adding an auxiliary inference branch or deployment module.
  • Parameter-subspace analysis: Normalization-affine parameters provide unusually effective correction directions despite being orders of magnitude fewer than backbone weights.Each affine parameter applies a channel-wise transformation across tokens or spatial locations, influencing downstream response trajectories.
  • Results: Across three LLM families, multiple PTQ backends, and weight-only or joint low-bit regimes, the complete pipeline consistently lowers perplexity and improves mean zero-shot accuracy.The framework addresses correctable response mismatch while distinguishing it from irreversible low-bit information loss.

2 Related Work

Related work spans PTQ correction, normalization adaptation, QAT, robustness, and low-dimensional parameter updates. SandwichQuant differs by measuring subspace leverage under the actual structured and anisotropic quantization graph.

  • Quantization correction: Classical and recent PTQ methods improve fixed models through equalization, reconstruction, sensitivity modeling, error propagation analysis, or auxiliary correction paths.Other approaches reshape quantization difficulty through scaling, weight protection, rotations, or invertible transforms.
  • Normalization adaptation: Normalization-based methods update affine parameters or perform global affine calibration to realign quantized and floating-point activations.Related work also links normalization to smoother optimization, reduced sharpness, expressive restricted training, and test-time adaptation.
  • QAT and low-dimensional subspaces: QAT, perturbation-robust optimization, normalization-restricted SAM, intrinsic-dimension studies, and parameter-efficient adaptation all motivate low-dimensional or restricted updates.These lines of work concern fake quantization, sharpness, conditioning, or useful small update subspaces.
  • Empirical positioning: Figure 1 indicates that the same affine coordinates are useful before PTQ reconstruction and after a converged QAT checkpoint.The post-QAT result diagnoses useful residual directions under a fixed protocol rather than greater asymptotic capacity than full-space training.
  • Distinction: The study differs by measuring subspace leverage under structured, anisotropic quantization noise produced by rounding, clipping, and activation quantization.This setting is distinct from generic isotropic parameter perturbations.

3 Method

The method studies correction through parameter subspaces and applies normalization-affine adaptation before and after PTQ. Matched-budget and intervention analyses show that this structured, low-dimensional subspace provides efficient, propagation-aware correction.

  • Parameter-subspace formulation: The trainable state is decomposed into backbone weights W, normalization-affine parameters Φ, and quantizer variables Ω.Controlled subspace comparisons restrict updates to one parameter group under the same quantized initialization and optimization budget.
  • Subspace comparison: Under matched budgets, Φ achieves the strongest overall recovery and slightly outperforms gradient-selected weight controls on Qwen3-8B W2A4KV4.The comparison uses identical scalar budgets, calibration data, optimization steps, and training objectives.
  • Target-aligned affine stages: SANDWICHQUANT learns Φpre before PTQ, rebuilds the backend from the restored dense checkpoint, then learns independent Φpost on the fixed quantized graph.The pre-stage changes responses presented to the backend, whereas the post-stage corrects residual response on one realized quantized solution.
  • Functional error geometry: Post-PTQ affine correction reduces task-relevant degradation more consistently than isolated local operator error, supporting propagation-aware rather than uniform layerwise reconstruction.The structured-source interventions are non-additive, and task-aware recovery can exceed recovery of an isolated local operator.
  • Target-aligned affine stages: The stages optimize task loss plus teacher distillation on the same target graph used for evaluation, with all non-affine tensors and graph topology fixed during each affine update.Changing the target graph changes both the residual and the correction directions, so affine checkpoints are backend-specific.
  • Implementation and scope: The framework adds no inference operator or deployment module, but requires two offline affine optimizations and a second PTQ pass.Affine states remain specific to the backend, bit-width, and calibration setting.
  • Parameter-subspace formulation: Normalization-affine parameters provide broadcast channel-wise response control across tokens or spatial locations despite their small parameter count.For BN, Φ contains channel-wise γ and β; for RMSNorm, it contains γ only.
  • Functional error geometry: Affine adaptation can repair structured response mismatch but cannot restore information destroyed by severe clipping or rounding.The recoverable component depends on alignment between quantization error and the affine subspace’s functional directions.

4 Experiments

Experiments evaluate affine correction across QAT checkpoints, vision tasks, and low-bit LLM quantization. Results support complementary pre- and post-quantization adaptation, with the complete pipeline strongest in demanding settings.

  • Cross-Domain and Checkpoint Controls: Affine-only post-QAT adaptation improves several QAT checkpoints at W4A4 and W2A4, but the W2A4 OOQ-Freeze exception shows it is not universally beneficial.This supports structured correction rather than unconditional post-hoc improvement.
  • Cross-Domain and Checkpoint Controls: At W4A4, affine-only correction restores MobileNetV2 from 0.33% to 66.11% top-1 and U-Net from 3.27% to 67.21% mIoU.At W3A4, affine preconditioning combined with QDrop is strongest, indicating greater complementarity under more destructive quantization.
  • LLM Quantization: Post-PTQ correction is the more stable single stage, while the full composition is strongest in three of four settings and gains most under W2A4KV4.The complete composition is therefore used as the headline SANDWICHQUANT configuration.
  • LLM Quantization: The complete two-sided pipeline improves every tested backend and model family, with the largest gains for fragile RTN and measurable gains after GPTAQ/ResComp.These results indicate correction of residual response geometry rather than simple replacement of the base quantizer.
  • LLM Quantization: Joint activation and KV-cache quantization produces especially large perplexity and Avg. gains on Llama3-8B and Qwen3-8B.The reported gains are attributed to complementary preconditioning and frozen-graph correction rather than a longer post-tuning run alone.

5 Discussion and Limitations

The discussion frames affine correction as effective for reachable response distortion but bounded by irreversible low-bit information loss. Its evaluation and deployment scope also remains limited.

  • Evidence and Interpretation: Normalization-affine coordinates can reduce task-aware damage even when an isolated local discrepancy changes little or increases.Figures 2 and 3 do not establish exact projection or uniform local reconstruction.
  • Scope and Cost: The pre-stage changes the model presented to PTQ, whereas the post-stage adapts to the realized frozen graph; their composition is most useful under aggressive joint quantization.The method adds no inference operator but requires two affine optimizations, a second offline PTQ pass, and a trainable target graph during calibration.
  • Scope and Cost: Evaluation is limited to dense 7B–8B LLMs and one deterministic seed, leaving larger and MoE models, deployment latency, and multi-seed confidence intervals for future work.Affine states are also backend-, bit-width-, and calibration-specific.

6 Conclusion

The paper identifies normalization-affine coordinates as high-leverage controls and uses them before and after PTQ. Across tested models and quantization regimes, this improves metrics without adding an inference-time operator, while remaining bounded by irreversible information loss.

  • Conclusion: The paper formulates low-bit quantization correction as a parameter-subspace problem and identifies normalization-affine coordinates as unusually high-leverage response controls.It adapts this native affine subspace independently before and after PTQ.
  • Conclusion: Across three LLM families and weight-only and joint weight–activation–KV regimes, the two-stage construction consistently improves perplexity and mean downstream accuracy without adding an inference-time operator.The conclusion reports the result at the paper level rather than for a single benchmark.
  • Conclusion: The method corrects structured, reachable response distortion but not information irreversibly lost to clipping or rounding.This defines the principal boundary of the proposed correction framework.

Large Language Model (LLM) Usage

The authors used LLMs for language polishing and limited manuscript-production support, while retaining responsibility for scientific decisions, experiments, verification, and conclusions.

  • LLM Usage: LLMs assisted with language polishing and limited manuscript-production support.The statement assigns scientific responsibility to the authors.
  • LLM Usage: The authors performed or approved all scientific decisions, experiments, result verification, and conclusions.They state that they take full responsibility for the manuscript.

A Reproducibility and Complete Experimental Protocol

The protocol separates disposable PTQ probing, pre-PTQ affine adaptation, PTQ rebuilding, and post-PTQ correction, with matched evaluation and controlled parameter branches. Experiments use fixed calibration and reproducibility controls across models, backends, and subspaces.

  • Protocol: The two-stage procedure transfers only Φpre to the dense checkpoint, discards the first run’s weights and optimizer state, reruns PTQ, then optimizes fresh Φpost.Both affine stages operate on normalization parameters while the completed quantized graph is used for post-correction.
  • Optimization: Each affine stage uses 200 AdamW steps with batch size 1, sequence length 256, learning rate 5×10−4, zero weight decay, and gradient clipping at 1.0.The objective combines equal-weight cross entropy and full-vocabulary teacher distillation at temperature 1.
  • Evaluation: Evaluation covers WikiText-2 and C4 perplexity plus six lm-evalharness zero-shot tasks with documented splits, block lengths, and metric aggregation.Avg. is the unweighted mean across PIQA, ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, and BoolQ.
  • Cost: Table 7 decomposes construction cost into two PTQ executions and two short affine stages, excluding identical downstream evaluation.The two affine stages update only native normalization parameters.
  • Reproducibility: Baseline and paired affine variants share dense checkpoints, tokenizers, dataset snapshots, calibration indices, preprocessing, and metric implementations within each model–regime pair.Run manifests record code revisions and generated artifacts for unmarked results.
  • Controls: The order ablation reuses the same checkpoint, backend, calibration tensor, and seed while matching compute per stage rather than across rows.It compares PTQ, post-only, pre-only, and full two-stage construction.

B Supplementary Mechanism Derivations and Limits

The derivations characterize affine correction as a weighted projection onto locally expressible response directions. They also identify limits: residual error outside that subspace, especially information lost through clipping or rounding, remains unrecoverable.

  • Exact BN Scale–Shift Cancellation: Under fixed BN statistics, the derivation treats pre-BN perturbations as structured scale–shift effects plus a residual term.The assumption fixes BN statistics µc and σc while analyzing local perturbations.
  • Exact BN Scale–Shift Cancellation: The exact cancellation identity is specific to BN with fixed statistics; RMSNorm provides multiplicative response control but no bias term for general additive offsets.Thus the normalization type changes which error components are directly expressible.
  • Weighted Projection: For the local problem, the minimum-norm affine update is obtained from the weighted pseudoinverse expression in Eq. (16).The expression maps the error through the affine Jacobian and Hz weighting.
  • Weighted Projection: The decomposition e∥ = JΦ∆Φ∗ and e⊥ = ez − e∥ separates locally expressible error from the residual orthogonal in the Hz geometry.This is a first-order characterization rather than an exact description of finite-step nonlinear optimization.
  • Limits: Affine adaptation can repair structured response mismatch but cannot generally reconstruct information removed by severe clipping or rounding.Correction depends on alignment between the available subspace and the recoverable quantization-error component.

C.1 SANDWICHQUANT after QAT and Alternating SandwichQuant-QAT

The supplementary variants test affine correction after saturated QAT and separate quantization-grid adaptation from response adaptation. These controls retain frozen backbone weights and distinguish the LLM formulation from full-parameter QAT.

  • After QAT: Post-QAT adaptation fixes backbone and quantization parameters while updating only normalization-affine parameters of a saturated QAT checkpoint.This tests whether the checkpoint retains an under-exploited affine adaptation direction.
  • Alternating SandwichQuant-QAT: SandwichQuant-QAT alternates group-wise quantization-scale updates with RMSNorm-scale updates while keeping the complementary branch fixed.The alternating variant decouples quantization-grid adaptation from response adaptation.
  • Alternating SandwichQuant-QAT: QScale-QAT adapts only Ω, RMSNorm affine adaptation adapts only Φ, and SandwichQuant-QAT alternates between both subspaces.The formulation explicitly does not perform full-parameter QAT.
  • Vision Controls: CNN experiments use MQBench with per-output-channel symmetric weight quantization and per-tensor asymmetric activation quantization.ImageNet and Cityscapes quantize the full network, while the CIFAR-100 diagnostic keeps the first and last layers at 8 bits.
  • Vision Controls: Some vision controls add a channel-scale regularizer, whereas headline LLM experiments set its coefficient to zero.The regularizer discourages a few channels from dominating activation ranges.
  • After QAT: CIFAR-100 post-QAT adaptation freezes convolutional, linear, and quantizer parameters and updates only BatchNorm scales and biases.The setup uses a fixed model-selection subset and a shared CE+KD optimization objective.
  • Scope: Vision results are cross-architecture controls, while the headline LLM formulation uses the complete Φpre → PTQ → Φpost pipeline.Vision rows identify whether intervention is post-PTQ or pre-reconstruction.

D.1 Additional Controls and Practical Boundaries

Additional controls show that normalization-affine structure, backend alignment, and tuning-data coverage materially shape correction effectiveness. The experiments also distinguish complete-pipeline results from post-only components and test a learned-rotation backend.

  • Matched-Size Controls: On ImageNet, 34.2K affine coordinates reach 66.11% W4A4 accuracy versus 18.35% for the strongest equal-size weight control.On Qwen3-8B, Φ slightly outperforms gradient-selected weights under an identical 299,008-parameter budget.
  • Matched-Size Controls: Low dimensionality alone does not explain the effect; the normalization-affine structure matters under matched parameter budgets.The comparison rules out sparsity or coordinate count as the sole explanation.
  • Alignment and Distillation: Removing KD reduces W4A4 accuracy from 66.11% to 61.50%, while transfer to a mismatched fake-quantized graph yields 48.72%.The reported 17.39-point gap indicates backend-specific compensation rather than generic channel rescaling.
  • Practical Boundaries: SANDWICHQUANT maintains optimizer states for 34.2K parameters and requires 1.00× normalized time versus 1.83× for full tuning.Parameter efficiency does not imply data efficiency in the current ImageNet implementation.
  • Practical Boundaries: Using 5K or 10K ImageNet images fails to recover the collapsed model, whereas full-data tuning reaches 66.11%.The present implementation is therefore not calibration-only or few-shot.
  • Practical Boundaries: The adaptation is backend-specific, can conflict with some QAT solutions, depends on sufficient tuning-data coverage, and cannot recover information destroyed by severe clipping and rounding.These are stated scope boundaries of the overall correction approach.
  • Backend Extension: SpinQuant experiments replace QuaRot with model-specific learned rotations while retaining the same two-stage state transitions.The rotation parameters remain fixed during both affine stages.
  • Component Results: Tables 11 and 12 isolate Φpost on frozen W3A16 and W2A4KV4 graphs, and are reported as component results rather than the full method.The complete pipeline remains the headline formulation.
Loading 2608.24173v1…