PaperlayerView exact revision on arXiv

Source-linked AI summary

QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization

Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, Fengwei Yu

arXiv:2203.05740v2cs.CVcs.AI

TL;DR

Extremely low-bit PTQ remains challenging despite its lower computational cost than QAT, especially when activation quantization is ignored during reconstruction. The paper introduces QDROP, which randomly drops activation quantization while tuning weights to pursue flatness on calibration and test data. Across vision and language tasks, QDROP improves low-bit PTQ, reaches 2-bit post-training quantization, and reports a new state of the art.

  • Problem

    Existing PTQ methods remain inaccurate for extremely low-bit activation quantization, while available calibration data are limited.

  • Method

    QDROP randomly drops activation quantization during PTQ reconstruction, using a theoretical flatness framework to guide weight tuning.

  • Results

    QDROP improves accuracy across evaluated ImageNet models and tasks, with 2-bit post-training quantization becoming possible for the first time.

  • Takeaways & Limitations

    QDROP is a simple plug-and-play module that consistently boosts existing PTQ methods across CNNs, Transformers, and multiple vision and language tasks.

  • Takeaways & Limitations

    Reported comparisons include setting-dependent first and last layers, and some NLP gains are limited or affected by training-data representativeness.

Abstract

from arXiv · show

Recently, post-training quantization (PTQ) has driven much attention to produce efficient neural networks without long-time retraining. Despite its low cost, current PTQ works tend to fail under the extremely low-bit setting. In this study, we pioneeringly confirm that properly incorporating activation quantization into the PTQ reconstruction benefits the final accuracy. To deeply understand the inherent reason, a theoretical framework is established, indicating that the flatness of the optimized low-bit model on calibration and test data is crucial. Based on the conclusion, a simple yet effective approach dubbed as QDROP is proposed, which randomly drops the quantization of activations during PTQ. Extensive experiments on various tasks including computer vision (image classification, object detection) and natural language processing (text classification and question answering) prove its superiority. With QDROP, the limit of PTQ is pushed to the 2-bit activation for the first time and the accuracy boost can be up to 51.49%. Without bells and whistles, QDROP establishes a new state of the art for PTQ. Our code is available at https://github.com/wimh966/QDrop and has been integrated into MQBench (https://github.com/ModelTC/MQBench)

1 INTRODUCTION

PTQ reduces computation relative to QAT but remains inaccurate at extremely low activation bit-widths. QDROP incorporates activation quantization into reconstruction and uses flatness analysis to improve PTQ across vision and language tasks.

  • Motivation: PTQ requires much less computation than QAT because it avoids end-to-end training.QAT uses the whole dataset and GPU effort, whereas PTQ does not require end-to-end training.
  • Motivation: Extremely low-bit activation quantization remains inaccurate despite model-space reconstruction methods that succeed at 4-bit quantization and sometimes 2-bit weight quantization.Existing theoretical analyses model weight quantization as perturbation while ignoring activation quantization.
  • Approach: QDROP randomly drops activation quantization during PTQ reconstruction to pursue flatter optimized models on calibration and test data.The paper reports that activation quantization benefits calibration-data flatness, while partial dropping contributes to test-data flatness.
  • Approach: The paper establishes a theoretical framework linking the flatness of optimized low-bit models on calibration and test data to final accuracy.The framework motivates incorporating activation quantization and dropping it partially during reconstruction.
  • Scope and contribution: QDROP is presented as a plug-and-play method for CNNs and Transformers across image classification, object detection, text classification, and question answering.The authors report a new state of the art for PTQ and make 2-bit post-training quantization possible for the first time.

2 PRELIMINARIES

PTQ replaces continuous values with fixed-point representations, but minimizing quantization error in parameter space does not necessarily minimize task loss. With only a tiny calibration subset, reconstruction methods therefore model quantization effects to tune weights more directly.

  • Quantization basics: Uniform quantization maps continuous values to fixed-point integers using a rounding-to-nearest operator and a step size.The activation quantizer is written as ˆx = ⌊x/s⌉·s.
  • Quantization basics: Rounding-to-nearest minimizes mean squared error, but parameter-space minimization does not equal minimization of the final task loss.The final objective is expressed as Ex∼Dt[L(ˆw, x)].
  • Calibration constraint: PTQ reconstruction is difficult because only a tiny calibration subset Dc containing 1k images is available.The limited subset makes direct minimization of the final loss objective difficult.
  • Prior reconstruction: Prior methods learn rounding schemes for weights and use Taylor expansion to analyze quantization-induced loss changes and weight interactions.These methods reconstruct each block or layer output by finetuning only the weights.
  • Gap: Existing reconstruction methods ignore activation quantization during output reconstruction, producing the same optimized weights across activation bit-widths.The paper argues that activation and weight quantization noise should be considered coherently.

3 METHODOLOGY

QDROP studies activation quantization during PTQ reconstruction, showing that partial involvement can improve extremely low-bit accuracy by producing flatter optimized models. It randomly drops activation quantization during reconstruction and demonstrates gains across vision and language tasks.

  • 3.1 EMPIRICAL OBSERVATIONS: Activation quantization during weight tuning substantially improves extremely low-bit PTQ, whereas separate optimization of weights and activations can fail.In W2A2 experiments, Case 1 barely converges while Case 2 achieves good accuracy.
  • 3.1 EMPIRICAL OBSERVATIONS: Partial block-wise activation quantization outperforms quantizing all activations during reconstruction.Case 3 leaves activations in the current tuning block unquantized and performs better than Case 2.
  • 3.2 HOW DOES ACTIVATION QUANTIZATION AFFECT WEIGHT TUNING: The theoretical framework models activation quantization jointly with weight perturbation and links final accuracy to loss-landscape flatness on calibration and test data.Case 2 and Case 3 suffer less loss degradation than Case 1 under similar perturbation magnitudes on calibration data, while calibration–test flatness can mismatch.
  • 3.3 QDROP: QDROP randomly enables or disables activation quantization on each forward pass to cover more flatness directions.It is element-wise, whereas Case 3 drops quantization in a block-wise manner; QDROP produces a smoother test-data loss surface than Case 3.
  • 4. EXPERIMENTS: QDROP improves results beyond ImageNet, including up to 6.5 mAP on MS COCO and superiority over No Drop on tested GLUE and SQuAD tasks.For calibration studies, dropping some quantization remains better across smaller-data settings and can be comparable with No Drop using half the original calibration data.

5 CONCLUSION

The paper introduces QDROP for post-training quantization and targets accurate low-bit models using a tiny calibration set. It achieves nearly lossless 4-bit quantization and significantly improves 2-bit results.

  • QDROP is introduced as a post-training quantization mechanism designed to achieve good test accuracy with a tiny calibration set.It optimizes toward a flat minimum.
  • QDROP comprehensively demonstrates effectiveness across a large variety of tasks.
  • QDROP achieves a nearly lossless 4-bit quantized network and significantly improves 2-bit quantization results.

A NOISE FORM CHOICE

The paper compares additive and multiplicative noise formulations for activation quantization. It adopts multiplicative noise because its range is independent of activation range and quantizer step size.

  • The traditional additive activation noise e = ˆa − a is represented as c · s, coupling its range to activation range and step size.This coupling complicates unified analysis across the network.
  • Parameter perturbations should account for parameter magnitude because large perturbations can switch signs and significantly change loss.
  • The paper uses multiplicative activation noise ˆa = (1 + u) · a to eliminate influence from activation range.In the quantization formulation, u is derived from the rounding error and quantized integer value.
  • The range of multiplicative noise u depends on rounding error and bit-width, not activation range or step size s.The paper therefore argues that this form has physical meaning in the quantization setting and benefits analysis.

B PROOF OF LEMMA 1 AND THEOREM 1

The proof establishes that activation quantization noise can be transformed into an equivalent weight perturbation in fully connected and convolutional networks. Taylor expansion then connects the resulting losses under the two perturbation views.

  • The analysis separately considers fully connected and convolutional networks, with additional illustration focused on convolutional layers.
  • Fully Connected Networks: For fully connected layers, activation noise u can be viewed as a weight perturbation v with a proper construction.
  • Fully Connected Networks: For every layer, optimizing loss with quantized activations can be approximated by optimizing loss with multiplicatively perturbed quantized weights and unquantized activations.
  • Convolutional Networks: For convolutional layers, the proof introduces G1 with activation noise and G2 with weight noise, then compares their losses and first-order derivatives.
  • Convolutional Networks: The proof assumes identical activations at u = 0 and v = 0 for the two networks before establishing their derivative relationship.
  • Proof: Taylor expansions around u = 0 and v = 0 transform the activation-noise loss into a weight-perturbation loss approximation.
  • Proof: Applying expectation over calibration samples completes the equivalence used to prove Lemma 1 and derive Theorem 1.

C.1 SUPPLEMENTARY EXPERIMENTS OF SEC. 3.1

The experiments compare PTQ and QAT settings for weight and activation quantization on ResNet-18 W2A2, revealing that broader weight tuning can substantially improve accuracy even under a restricted optimization space.

  • Quantization settings: QAT uses whole-ImageNet, end-to-end training for weight quantization, whereas PTQ uses 1024 samples and block-by-block training.Both methods retain the same rounding-up-or-down parameter optimization space.
  • Quantization settings: QAT activation quantization uses whole-ImageNet, LSQ, and five epochs of end-to-end training.
  • Findings: A large accuracy boost results from using the whole dataset for weight tuning, despite keeping the weight optimization space very restricted.
  • Findings: The activation quantization step size may be less important than weight tuning for accurate PTQ.

C.2 SUPPLEMENTARY EXPERIMENTS OF SEC. 3.3

Experiments on ResNet-18 W2A2 show that calibration accuracy can mask overfitting, while partial activation quantization improves test-data flatness; Hessian measurements support QDROP's flatter solution.

  • Overfitting phenomenon: Both Case 2 and Case 3 perform well on calibration data, but Case 2 performs worse than Case 3 on test data.This is identified as clear evidence that Case 2 suffers more severe overfitting.
  • Theoretical interpretation: Case 2 fully introduces activation quantization noise according to calibration data, so its resulting flatness fits calibration data but does not generalize well on test data.
  • Overfitting phenomenon: Case 3 drops part of the activation quantization noise and improves the possibility of flatness on test data.The two QDROP accuracies also support this phenomenon with more diverse directions of flatness.
  • Theoretical interpretation: Case 1 has low accuracy on both calibration and test data because it excludes activation quantization during weight tuning.It therefore does not account for the flatness term even on calibration data.
  • Hessian information: QDROP has the smallest top-1 and top-5 Hessian eigenvalues and Hessian trace among the compared cases.These Hessian measurements match the theoretical framework and loss-landscape observations.

C.3 SUPPLEMENTARY EXPERIMENTS OF SEC. 4

A comparison with AdaQuant attributes accuracy differences to activation-quantization placement and pretrained-model accuracy, while noting that AdaQuant's placement is impractical for deployment.

  • Activation quantization placement: AdaQuant inserts activation quantization nodes differently, using two quantizers for the same input, whereas QDROP follows the alternative placement shown in Figure 7.
  • Activation quantization placement: The alternative node placement brings approximately 0.4% gains on ResNet-18 and -50 W4A4 and improves more at ultra-low bit widths.
  • Deployment consideration: Using two different quantizers for the same input makes AdaQuant's placement incompatible with the Requantize procedure in real deployment.
  • Pretrained-model settings: AdaQuant's pretrained models have 71.97% and 77.2% FP32 accuracy, compared with 71.06% and 77.0% for the authors' models.
  • Pretrained-model settings: Replacing the authors' settings with AdaQuant's two pretrained models and node-placement settings reaches 71.07% and 76.67%.

C.4 FLATNESS AND POST-TRAINING QUANTIZATION

The paper connects activation quantization and weight flatness in PTQ, relating smoother loss surfaces and distinct full-precision models to quantization performance and robustness.

  • Flatness and quantization: The paper connects flatter quantized weights with activation quantization from a PTQ perspective, while noting that their interaction has been little explored.
  • Smoother full-precision models: SWA20 applies stochastic weight averaging to ResNet-18 through 20 epochs of fine-tuning on the whole ImageNet.
  • Smoother full-precision models: SWA20 surpasses the naive full-precision model by a large margin under lower-bit quantization, where quantization noise is larger.Its generalization improvement alone does not fully represent the quantization enhancement induced by SWA.
  • Related robustness perspective: Prior work uses random bit training to improve DNN robustness, while this paper relates bit width to perturbation.
  • Post-training quantization: PTQ requires much less computation than QAT because it does not require end-to-end training, but lower-bit quantization remains difficult.
  • Flatness and quantization: Flatness has been studied in relation to generalization, adversarial training, robustness under perturbation, and distribution shift.

E IMPLEMENTATION DETAILS

Experiments use block-wise reconstruction with jointly tuned quantization parameters, calibrated on sampled data across vision and language tasks. Baselines and task-specific settings largely follow BRECQ for comparison.

  • The implementation considers three activation-introduction cases, whose outcome differences are reported as negligible, and parameterizes activation step sizes in each case.
  • ImageNet experiments calibrate with 1024 training examples and use BRECQ-aligned optimization settings, including 20,000 reconstruction iterations.
  • QDROP jointly learns weight and activation parameters while dropping 50% of activation quantization during reconstruction.
  • MS COCO experiments calibrate with 256 training samples and reconstruct the backbone block-wise and neck layer-wise, leaving the detection head unquantized.
  • BERT experiments sample 1024 training examples, use sequence lengths 128 for GLUE and 384 with doc stride 128 for SQuAD1.1, and quantize BERT components except embedding weights beyond 8-bit.
  • Baselines include AdaQuant, BRECQ, and Adaround, with settings such as per-channel quantization aligned where possible for fair comparison.
Loading 2203.05740v2…