PaperlayerView exact revision on arXiv

Source-linked AI summary

A Target-Centric Survey of Quantization-Aware Training

Jiamin Song, Mengjie Zhao, Zijing Wang, Yongkang Liu, Qian Li, Shi Feng, Feiliang Ren, Daling Wang, Hinrich Schütze

arXiv:2608.29667v1cs.LG

TL;DR

Large models impose substantial memory and computational demands, while low-bit quantization can introduce accuracy loss from limited representational range and structured outliers. This paper surveys QAT from a target-centric perspective, combining theoretical foundations, target-specific methods, and evaluation practices. It concludes that QAT assessment must account for target-dependent errors, reasoning behavior, and deployment fidelity rather than relying only on nominal compression or perplexity.

  • Problem

    Large-scale models require resources beyond many edge devices and consumer GPUs, while directly quantizing pretrained models can cause performance degradation from low-bit representational limits and activation outliers.

  • Method

    The paper establishes QAT foundations and organizes methods by quantization target, synthesizing cross-target differences, strategy transferability, evaluation paradigms, and reporting practices.

  • Results

    The survey reports that INT8 QAT is largely mature, but perplexity and nominal compression do not reliably capture reasoning quality or real deployment acceleration.

  • Takeaways & Limitations

    QAT evaluation should include target-specific error diagnostics, reasoning-aware measures, and measured latency or throughput alongside task accuracy or perplexity.

  • Takeaways & Limitations

    Gradient QAT remains an optimization-stability problem requiring range control, direction preservation, and structure-aware selectivity.

Abstract

from arXiv · show

The rapid development of LLMs incurs prohibitive memory footprints and intensive computational demands. Quantization-Aware Training (QAT) techniques have emerged as a promising solution to address these challenges by explicitly simulating quantization effects during model training, yielding low-bit models that achieve accuracy comparable to their full-precision counterparts. In this work, we provide a target-centric survey of QAT, aimed at clarifying both its theoretical foundations and its evolving implementation landscape. We systematically review existing QAT methods through a target-centric taxonomy and synthesize cross-target differences in error characteristics, numerical formats, and strategy transferability. We further summarize QAT evaluation paradigms and discuss challenges in optimization and deployment, outlining potential directions for future research.

1 Introduction

Large foundation models create memory and computation bottlenecks, motivating quantization and QAT. This survey addresses those challenges through a target-centric review of QAT foundations, methods, and evaluation.

  • Large foundation models can exceed the memory and computational capacity of edge devices and consumer-grade GPUs.
  • Quantization reduces model size by lowering parameter bit-width, while PTQ can degrade accuracy because low-bit formats poorly represent high-dynamic-range outliers.
  • QAT simulates quantization during training with fake-quantization modules to optimize models under low-precision inference constraints.
  • The survey organizes QAT methods by quantization target rather than treating QAT only within broader quantization or architecture taxonomies.
  • The paper establishes QAT theory, surveys target-specific methods, synthesizes cross-target differences, and presents evaluation protocols and a reporting checklist.

2 Preliminary Knowledge

QAT represents low-precision effects through quantization operators while training remains centered on floating-point computation and parameters. Its optimization relies on surrogate gradients because rounding is non-differentiable, with granularity and clipping shaping the quantizer.

  • Quantization maps real-valued tensors to low-precision discrete values, trading memory and computation cost against representational capacity.
  • Uniform affine quantization uses a step size, zero-point, and integer range to map a real-valued tensor to an integer tensor.
  • Fake quantization dequantizes quantized values for floating-point computation, producing a proxy tensor and quantization error relative to the original tensor.
  • Quantization granularity determines the scope over which scale and zero-point parameters are shared, such as per-tensor or per-channel quantization.
  • QAT inserts fake-quantized weights and activations into the forward pass while retaining full-precision master weights during training.
  • STE treats rounding as an identity operation in backpropagation because direct differentiation would yield vanishing or ill-defined gradients.
  • STE is a coarse approximation, motivating improved surrogate gradients, learned quantizer parameters, optimization stabilization, and differentiable relaxations.

3 Taxonomy

The survey organizes QAT methods by quantization target, emphasizing that target-specific numerical behavior, reuse, lifetime, and deployment constraints shape method design. It covers weights, activations, joint weight–activation quantization, and other targets while comparing transferable and target-specific strategies.

  • Taxonomy: The target-centric taxonomy groups QAT methods by quantization target rather than primarily by architecture or bit-width.It categorizes multi-target methods under the targets where their key ideas, algorithmic designs, and ablations are centered.
  • Model Weights: Weight QAT spans binary, ternary, ultra-low-bit, selective, mixed-precision, adapter-integrated, and pre-training-integrated approaches.At very low bit-widths, threshold-adjacent weights can repeatedly flip between quantization levels, causing oscillation and instability.
  • Activations: Activation QAT must address input-dependent distributions, outliers, non-stationary statistics, and mismatch between smooth training surrogates and discrete inference.Adaptive clipping thresholds, step sizes, offsets, and asymmetric ranges are used to track shifting activation distributions.
  • Weight + Activation: Joint weight–activation QAT is a co-design problem because quantized weights reshape activations, clipping affects updates, and deployment imposes operator- and accumulator-aware constraints.Integer-only Transformer methods therefore redesign operations such as Softmax, GELU, and LayerNorm, rather than quantizing individual tensors alone.
  • KV Cache: Promising directions include distribution-aware, token-wise, vector-quantized, and retrieval-oriented approaches for compressing KV states.These approaches respond to variability from KV, RoPE, outliers, token importance, and attention relevance.
  • Cross-Target Analytical Synthesis: QAT strategies transfer best when they address target-agnostic numerical issues, whereas persistence, temporal reuse, optimization semantics, and graph execution require target-specific mechanisms.Fine-grained scaling, adaptive range control, and mixed-precision allocation generalize relatively well; KV-specific, gradient-direction, and integer nonlinear redesign strategies do not transfer directly.

4 Evaluation

The survey evaluates QAT through performance, structural diagnostics, and deployment evidence, emphasizing that outcomes depend on quantization settings, targets, architectures, and backends. Its findings show mature INT8 QAT, practical but architecture-sensitive W4 QAT, severe sub-3-bit degradation, and a gap between compression, accuracy, and acceleration.

  • Evaluation Framework: QAT evaluation should combine task performance, structural diagnostics, and deployment evidence rather than relying on accuracy or perplexity alone.The proposed dimensions expose activation outliers, gradient mismatch, overflow, reasoning degradation, and latency regressions.
  • Empirical Findings: INT8 QAT is largely mature, but fake-quantized training graphs still require validation against real integer execution.Unified INT8 training reports only a 0.26 Top-1 drop on ResNet-50, while I-BERT achieves essentially lossless W8A8 RoBERTa inference.
  • Empirical Findings: W4 QAT is practical, yet failure modes vary across architectures and targets.Residual CNNs can match or exceed full precision, whereas MobileNetV2, ViTs, and joint LLM weight-activation or KV-cache quantization show distinct sensitivities.
  • Empirical Findings: -7.80 Top-1 on DeiT-S at W2A2 illustrates the sharp degradation of sub-3-bit QAT across model families.EfficientQAT W2g64 reports -4.72 zero-shot ability and +1.39 perplexity on LLaMA-2-7B, alongside optimization and error-accumulation failure modes.
  • Empirical Findings: Perplexity alone misses long-horizon reasoning degradation, while compression alone does not guarantee acceleration.Realized speedups depend on kernels, compiler support, and integer-only execution; I-ViT achieves a 3.72–4.11ˆ inference speedup over FP32 baselines.
  • Minimal Reporting Checklist: The checklist requires full configuration reporting, matched baselines across at least two bit-widths, structural diagnostics, efficiency measurements, and reasoning-intensive LLM benchmarks.It is intended to improve reproducibility while distinguishing where quantization works, why it fails, and whether compression is deployable.

5 Challenges and Future Directions

The survey identifies optimization instability, activation outliers, simulation-to-deployment gaps, training overhead, homogenization, and format-target mismatch as persistent QAT challenges. It proposes hardware-aware, target-aware, and more efficient approaches as future directions.

  • Optimization Instability: STE-based surrogate gradients can bias optimization, while threshold crossings and coarse bins cause instability or trap weights in ultra-low-bit training.Better alignment between backward gradients and the discrete objective remains open.
  • Activation Outliers: Input-dependent activation outliers skew quantization ranges and accumulate across layers, especially under joint WA-QAT in Transformers and LLMs.Knowledge distillation and rotation-based transformations address the issue but add significant overhead.
  • Simulation-to-Deployment Gap: FP32 fake quantization omits accumulator overflow, rounding modes, and operator fusion, creating a simulation-to-deployment gap.Hardware-aware constraints during QAT are proposed to improve execution on integer-only accelerators.
  • High Training Overhead: QAT training can be prohibitive for billion-parameter models because it retains full-precision master weights, optimizer states, and often distillation teachers.This creates tension between compression objectives and the cost of producing compressed models.
  • Homogenization: Uniform quantizers, fixed bitwidths, and layer-agnostic choices can suppress fine-grained features and downstream adaptability through model homogenization.The effect intensifies with increasing model scale.
  • Format–Target Co-Design: Emerging FP8, FP4, and microscaling formats motivate jointly optimizing numerical format, scaling, and rounding for each quantization target.The design must account for distinct precision and dynamic-range requirements across W/A/KV/G.

6 Conclusion

The survey presents a target-centric synthesis of QAT methods and evaluation practices. It connects target-specific properties with error propagation, numerical formats, strategy transferability, and persistent optimization and deployment challenges.

  • Conclusion: The survey organizes QAT methods by weights, activations, KV cache, and gradients, linking target-specific properties to error propagation, numerical formats, and strategy transferability.It also reviews evaluation practices and proposes a minimal reporting checklist.
  • Conclusion: Persistent challenges include optimization instability, activation outliers, simulation-to-deployment gaps, training overhead, and model homogenization.The survey uses these challenges to outline future research directions.

Limitations

The survey primarily covers QAT methods, evaluations, challenges, and future directions, while deployment-oriented systems and hardware-specific optimization may be underrepresented. Its quantitative evidence reflects general trends rather than strict head-to-head comparisons, and emerging KV-cache QAT literature remains sparse.

  • Scope: Deployment-oriented systems and hardware-specific optimization techniques may be underrepresented because the survey focuses primarily on QAT methods and evaluations.The authors frame this as a scope boundary rather than a claim of comprehensive deployment coverage.
  • Evidence Comparability: Quantitative evidence should be interpreted as general trends rather than strict head-to-head comparisons because existing studies differ in setup.New methods, benchmarks, and deployment backends may also not be fully reflected.
  • Coverage Boundary: Direct QAT literature remains sparse for emerging targets such as KV-cache quantization, reflecting the current maturity of these areas.The survey covers representative works, but this literature is still developing.

A Survey Methodology

The survey defines its scope through a systematic search and inclusion criteria, then reviews quantization fundamentals and contrasts symmetric with asymmetric schemes. Its methodological foundation emphasizes how quantization ranges and formats are selected for different tensor distributions.

  • Search scope and coverage: The search covers QAT, low-bit training, fake quantization, integer-only training, LLM QAT, KV-cache quantization, and gradient quantization through citation tracing.Coverage spans early low-bit QAT work through May 2026, with emphasis on studies published since 2020.
  • Inclusion and exclusion criteria: The taxonomy includes methods that simulate quantization during training, optimize quantization parameters, or construct low-bit training graphs, while excluding pure PTQ and communication-only compression.Multi-target methods are categorized by their central technical design, primary ablations, and main evaluation target.
  • Uniform quantization: Uniform affine quantization maps a real interval to an integer interval using a step size and zero-point derived from the selected clipping range.The clipping range may come from calibration statistics, local reconstruction-error optimization, or joint learning.
  • Quantization formats: Symmetric quantization fixes the zero-point to 0 and uses a range centered on zero, whereas asymmetric quantization accommodates shifted or nonnegative distributions.Symmetric quantization is commonly used for weights; asymmetric quantization is often useful for activations.

B.2 Gradient Derivation for Learnable Step Size

The section explains how learnable quantization parameters receive surrogate gradients and surveys strategies that improve QAT optimization beyond vanilla STE. These methods address gradient mismatch, parameter adaptation, threshold oscillation, and hard-rounding non-differentiability.

  • Learnable step-size gradient: LSQ treats the quantization step size as learnable and uses an STE-based piecewise derivative to update the quantization grid under task loss.Gradient scaling keeps step-size updates comparable in magnitude to weight updates.
  • Learnable step-size gradient: The surrogate gradient balances clipping-induced distortion against rounding-induced error by adapting the quantization step size.LSQ additionally scales gradients to stabilize step-size optimization relative to weight updates.
  • Surrogate-gradient strategies: Clipped or masked STE aligns backward propagation with forward saturation but can create dead zones when weights or activations remain saturated.This refinement is widely used in practical QAT pipelines.
  • Surrogate-gradient strategies: Learnable scales, offsets, and clipping thresholds adapt the quantization grid to tensor distributions, including asymmetric activations.PACT learns activation thresholds, LSQ learns step sizes, LSQ+ learns offsets, and TQT optimizes thresholds.
  • Surrogate-gradient strategies: Oscillation-aware methods reduce repeated quantized-value flips near thresholds, while generalized activations and soft quantization modify forward behavior or rounding to improve optimization.These approaches target unstable low-bit regimes and replace or complement vanilla STE.
  • Stabilization strategies: Knowledge distillation and regularization complement surrogate-gradient changes by smoothing supervision or constraining representations against rounding noise.Distillation is especially useful when aggressive quantization causes large prediction shifts.

D.1 Benchmarks and Metrics

QAT evaluation combines task performance with structural and deployment evidence, but protocol maturity varies across model families. CNN and BERT-style encoder benchmarks are relatively standardized, whereas newer model families require more diagnostics.

  • Protocol maturity: Evaluation protocol maturity is relatively high for CNN and BERT-style encoder quantization but lower for ViT, diffusion, and LLM QAT.Less standardized settings require diagnostics for activation outliers, attention maps, nonlinear operators, KV cache, or denoising timesteps.
  • Benchmark coverage: QAT benchmarks span ImageNet, GLUE, SQuAD, WikiText-2, MMLU, GSM8K, HumanEval, and FID across model families.The listed benchmarks cover vision, language understanding, language modeling, reasoning, code generation, and generative quality.
  • Comparability: Coverage rates are descriptive rather than leaderboard comparisons because studies differ in backbones, budgets, benchmarks, kernels, accelerators, deployment assumptions, and reporting protocols.The tables therefore support cautious interpretation of cross-paper coverage.

D.2 Extended Structural Evaluation

Extended evaluation traces quantization error from tensor distortion through model behavior, training stability, reasoning, and hardware feasibility. It emphasizes target-specific diagnostics and separates actual storage and deployment evidence from nominal bit-width or isolated accuracy.

  • Structural diagnostics: Structural evaluation identifies where quantization error enters and how it propagates through the model.The survey treats structural diagnostics as complementary to task-level performance.
  • Tensor-level distortion: Normalized MSE, cosine similarity, clipping ratio, saturation ratio, and layer-wise profiles measure tensor distortion and localize error in sensitive modules.Per-layer curves are preferred to a single aggregate number because error is often concentrated in a few modules.
  • Functional propagation: Transformer evaluation should pair tensor distortion with block-output drift, hidden-state similarity, logit KL divergence, and perplexity or cross-entropy changes.These metrics connect local perturbations to functional degradation after attention, residual, normalization, and nonlinear operations.
  • Training stability: Training stability diagnostics include gradient cosine similarity, sign agreement, gradient-norm drift, and repeated quantized-value flips, especially below 3 bits.These measures target instability caused by surrogate gradients and threshold oscillation.
  • LLM reasoning: LLM reasoning should be evaluated separately from perplexity using fixed prompting and decoding settings, with answer accuracy, invalid-output rate, and generation length reported.Recommended tasks include mathematical reasoning, multi-step knowledge reasoning, and code generation.
  • Target-specific LLM evaluation: Comparing W-only, W+A, W+A+KV, and KV-only settings isolates whether reasoning loss is associated with weights, activations, KV-cache accumulation, or their interaction.This comparison is particularly important for long-context generation.
  • Deployment evidence: Compression reporting should include scales, zero-points, preserved full-precision tensors, adapters, and KV-cache storage rather than nominal weight bit-width alone.Training feasibility, model compression, and realized inference speed should also be reported separately.

W W3, W4 Group LLM

The survey organizes QAT across model targets, numerical formats, and evaluation settings, showing that low-bit accuracy and failure patterns vary substantially across vision, language, and generative models.

  • Taxonomy and formats: The survey distinguishes INT8/INT4-based QAT from FP8/FP4-based quantization-aware and low-precision training approaches.This comparison is presented as part of the survey’s taxonomy and cross-target analysis of numerical formats and adaptation strategies.
  • Vision models: 71.23 ImageNet Top-1 accuracy is reported for OCTAV MobileNet-V2 at W4A4, while DoReFa-Net AlexNet reaches 46.10 at W1A2G6.The CNN results also include ResNet-18, ResNet-50, and MobileNetV2 configurations with varying accuracy drops.
  • Cross-target evaluation: QAT results span CNNs, vision transformers, diffusion models, BERT-family models, and LLMs, with each target evaluated using task-specific metrics.The referenced tables report ImageNet accuracy, FID, MNLI and SQuAD scores, perplexity, and zero-shot accuracy.
  • BERT-family models: 83.3 MNLI-m accuracy and 87.4 SQuAD F1 are reported for TernaryBERT at W2E2A8, compared with 84.2 and 88.3 for BinaryBERT at W1E1A8.The table also reports INT8 I-BERT and dynamic-bit OCTAV results across BERT and RoBERTa backbones.
  • LLM results: 69.88 average zero-shot accuracy is reported for LR-QAT on LLaMA-2-7B at W4 g128, versus 60.14 for EfficientQAT at W2g64.The LLM table pairs these scores with WikiText-2 perplexity values and reports results for LLaMA-1 and LLaMA-2 models.
Loading 2608.29667v1…