PaperlayerView exact revision on arXiv

Source-linked AI summary

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, Shi Gu

arXiv:2102.05426v2cs.LGcs.CV

TL;DR

PTQ seeks accurate neural-network quantization without end-to-end retraining, but low precision can cause severe degradation. BRECQ analyzes second-order error and reconstructs blocks, achieving INT2 quantization and 4-bit performance comparable with QAT.

  • Problem

    PTQ avoids full retraining but suffers severe accuracy degradation at low precision, limiting quantization without extensive data and computation.

  • Method

    BRECQ reconstructs neural-network blocks using second-order error analysis and incorporates mixed precision through inter-layer and intra-layer sensitivity measures.

  • Results

    BRECQ is the first PTQ method reported to quantize weights to INT2 without significant accuracy loss and reaches performance on par with QAT at 4-bit precision.

  • Takeaways & Limitations

    Block reconstruction balances cross-layer dependency and generalization error, while mixed precision supports hardware-constrained quantized models.

  • Takeaways & Limitations

    The theoretical analysis does not determine the optimal reconstruction granularity, and the block-wise choice is supported by experiments.

Abstract

from arXiv · show

We study the challenging task of neural network quantization without end-to-end retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantized models than Quantization-Aware Training (QAT). In this work, we propose a novel PTQ framework, dubbed BRECQ, which pushes the limits of bitwidth in PTQ down to INT2 for the first time. BRECQ leverages the basic building blocks in neural networks and reconstructs them one-by-one. In a comprehensive theoretical study of the second-order error, we show that BRECQ achieves a good balance between cross-layer dependency and generalization error. To further employ the power of quantization, the mixed precision technique is incorporated in our framework by approximating the inter-layer and intra-layer sensitivity. Extensive experiments on various handcrafted and searched neural architectures are conducted for both image classification and object detection tasks. And for the first time we prove that, without bells and whistles, PTQ can attain 4-bit ResNet and MobileNetV2 comparable with QAT and enjoy 240 times faster production of quantized models. Codes are available at https://github.com/yhhhli/BRECQ.

1 INTRODUCTION

Deep learning models create substantial computation and memory demands, motivating compression methods such as quantization. BRECQ addresses low-precision PTQ by analyzing second-order error and reconstructing network blocks.

  • Deep learning's computation cost and memory footprint motivate quantization and pruning of pretrained networks.
  • End-to-end quantization pipelines require full training data and substantial computation, while PTQ uses limited calibration data without retraining.
  • 39% top-1 accuracy is achieved by DFQ for 4-bit ResNet-18, compared with 69.7% at 8-bit without accuracy loss.
  • Second-order analysis motivates reconstructing layer outputs, but prior methods could not quantize weights to INT2 because cross-layer Hessian dependencies matter.
  • BRECQ adopts block reconstruction as an intermediate tradeoff between cross-layer dependency and generalization error.
  • BRECQ combines Fisher-based importance measures and mixed-precision search, and reports INT2 quantization without significant accuracy loss across varied tasks and models.

2 PRELIMINARIES

The preliminaries frame quantization as weight perturbation and explain how second-order loss analysis motivates feature-map reconstruction. PTQ operates with limited calibration data, making practical approximations necessary.

  • A feedforward network represents convolutional or fully connected layer inputs and outputs as x and z, with weights W and flattened weights w.
  • ReLU is the activation function used in the paper, and bias terms can be merged into the activation for analysis.
  • Uniform symmetric quantization maps floating-point values to evenly spaced, symmetrically distributed fixed-point grids controlled by step size and bit-width.
  • Minimizing parameter-space quantization error does not necessarily optimize task performance, motivating learned or task-aware step-size and loss objectives.
  • PTQ starts from full-precision weights and uses only a small training-data subset for calibration, without end-to-end finetuning.
  • Weight quantization is treated as perturbation, and Taylor expansion approximates the resulting loss degradation through gradient and Hessian terms.
  • Layer-independence and constant diagonal pre-activation curvature are assumptions used to simplify the Hessian structure.
  • Second-order analysis transforms the objective into feature-map change, enabling layer-by-layer reconstruction with few calibration images.

3 PROPOSED METHOD

BRECQ transforms second-order quantization-error analysis into output reconstruction and selects block-wise reconstruction to balance cross-layer dependency against generalization. It extends this framework with Fisher-based sensitivity measures and hardware-constrained mixed-precision search.

  • 3.1 CROSS-LAYER DEPENDENCY: The Gauss–Newton formulation transforms large-scale second-order error into changes in network outputs, avoiding explicit Jacobian storage during optimization.The transformation applies to weight quantization and, by the stated remark, activation quantization.
  • 3.1 CROSS-LAYER DEPENDENCY: BRECQ evaluates net-wise, stage-wise, block-wise, and layer-wise reconstruction granularities corresponding to essential CNN components.These choices differ in which Hessian dependencies they retain during reconstruction.
  • 3.2 BLOCK RECONSTRUCTION: Block-wise reconstruction retains intra-block dependency while ignoring inter-block dependency, producing less generalization error than full network-output reconstruction.The block objective reconstructs intermediate outputs using a block-diagonal Hessian approximation.
  • 3.2 BLOCK RECONSTRUCTION: Block-wise optimization outperforms the other tested granularities because the main off-diagonal Hessian loss is concentrated within blocks while inter-block loss is small.The authors state that the theoretical analysis does not determine the optimal granularity; the block choice is supported by experiments and has no hyperparameters in their setup.
  • 3.2 BLOCK RECONSTRUCTION: BRECQ reconstructs each block by collecting calibration inputs and full-precision outputs, updating weight rounding, and optionally updating activation step sizes.Algorithm 1 repeats this process across all blocks and optimization iterations.
  • 3.4 MIXED PRECISION: The method uses diagonal Fisher Information to measure pre-activation importance and combines diagonal and off-diagonal sensitivity for mixed-precision allocation.A genetic algorithm searches 2-, 4-, and 8-bit assignments under hardware performance constraints, while 2-bit permutations capture block-level sensitivity.

4 EXPERIMENTS

Experiments evaluate BRECQ across classification, detection, mixed precision, and QAT comparisons. BRECQ supports low-bit quantization while retaining strong accuracy and reducing training requirements.

  • Ablation study: Block-wise reconstruction outperforms net-wise, stage-wise, and layer-wise alternatives in 2-bit ImageNet experiments.The result suggests net-wise and stage-wise generalization errors outweigh their off-diagonal-loss benefits; differences are less significant for ResNet-18.
  • ImageNet: Across ResNet, MobileNetV2, RegNet, and MNasNet, BRECQ consistently achieves the lowest accuracy degradation for 2-bit weight quantization.The first and last layers remain 8-bit, while activations are initially kept at full precision.
  • ImageNet: 4-bit activation quantization can substantially affect RegNet and MobileNet, but BRECQ still outperforms other state-of-the-art methods.The fully quantized experiments use 4-bit activations alongside low-bit weights.
  • Comparison with quantization-aware training: BRECQ achieves comparable accuracy to QAT models and surpasses them on 4-bit MobileNetV2 while using less than one training GPU hour.Compared with HAQ, BRECQ has comparable accuracy and avoids end-to-end searches for each hardware performance threshold.
  • MS COCO: On MS COCO detection, 4-bit weight quantization nearly preserves performance, including only a 0.21% mAP decrease for 4-bit ResNet-18 Faster R-CNN.With 4-bit ResNet-50 RetinaNet, BRECQ outperforms ZeroQ by 3% mAP, and 2-bit weights still achieve near-original mAP.
  • Mixed precision: Mixed precision consistently outperforms unified precision and offers hardware-adaptable bit configurations.At the same latency as a 2-bit model, mixed precision can increase accuracy by up to 10%; unified precision provides only two fixed models.

5 RELATED WORKS

Quantization research contrasts QAT and PTQ, while second-order analysis uses Hessian-based curvature information to study perturbation sensitivity and optimization.

  • Quantization: QAT and PTQ are the two main categories of neural-network quantization.QAT typically uses gradient approximations such as the Straight-Through Estimator, whereas PTQ avoids retraining.
  • Quantization: QAT methods can achieve promising results but usually require more than 100 GPU hours, motivating PTQ.PTQ is important when retraining data and computation are limited.
  • Quantization: Most models can be safely quantized to 8-bit without retraining, but parameter-space methods generally perform poorly at 4-bit quantization.Prior layer-wise 8-bit PTQ can even operate without data, while lower-bit settings remain challenging.
  • Second-order Analysis and Optimization: Hessian matrices provide curvature information for pruning, quantization sensitivity analysis, and second-order optimization.HAWQ, for example, uses the largest Hessian eigenvalue to determine sensitivity.

6 CONCLUSION

BRECQ is a second-order-analysis PTQ framework that reconstructs quantization at block granularity. It targets very low-bit quantization, supports mixed precision, and reports performance comparable to QAT at 4-bit.

  • Conclusion: BRECQ uses block-granularity reconstruction to balance cross-layer dependency and first-order approximation.The framework is designed to address especially difficult 2-bit weight quantization.
  • Conclusion: BRECQ supports mixed precision and can reduce the search cost.The conclusion presents mixed precision as part of the framework's practical scope.
  • Conclusion: BRECQ reaches the highest reported PTQ performance and is the first to match QAT using 4-bit quantization.These are the paper's stated overall conclusions.

A.1 PROOF OF THEOREM 3.1

The proof represents second-order weight perturbation error as a quadratic form and relates it to network outputs through Jacobian-based first-order approximation.

  • A.1 PROOF OF THEOREM 3.1: The proof represents the second-order perturbation term Δθ^T H(θ) Δθ as a quadratic form.The weight vector has shape θ ∈ R^d, and the network output vector has shape z^(n) ∈ R^m.
  • A.1 PROOF OF THEOREM 3.1: The derivation uses the cross-entropy loss when connecting the quadratic-form expression to the theorem's equations.The passage identifies L as the cross-entropy loss.
  • A.1 PROOF OF THEOREM 3.1: Substitution carries the preceding second-order expression into the next equation of the proof.This is an algebraic transition within the derivation.
  • A.1 PROOF OF THEOREM 3.1: The Jacobian is defined as the derivative of x with respect to y and is used in the output-change approximation.The proof applies first-order Taylor expansion to approximate the change in network output.

B.1 EFFECT OF THE FIRST AND THE LAST LAYER

The first and last layers affect accuracy, model size, and latency differently across bitwidths and architectures. These observations motivate more flexible mixed-precision choices rather than uniformly preserving both layers at 8-bit.

  • Accuracy: 0.2% accuracy drop is observed for 4-bit quantization in both the first and last layers.At 2-bit quantization, the last fully connected layer is far more important than the first.
  • Model size: The first layer has minor model-size impact, whereas the last layer greatly affects memory footprint because it contains many weight parameters.Input images can still impose substantial memory requirements even when the first layer is small.
  • Latency: Latency sensitivity depends on architecture: the first layer dominates in ResNet-18, while the last layer matters more in MobileNetV2 and RegNet-600MF.Feature-map size, FLOPs, and weight-memory size all affect latency.
  • Mixed precision: 53.3 ms latency and 4% higher accuracy result when all ResNet-18 layers use 4-bit quantization, compared with 59.8 ms for 2-bit quantization with 8-bit first and last layers.The result indicates potential benefits from mixed precision.

B.2 EFFECT OF DATA

Calibration data has little effect at 4-bit quantization but becomes more important at 2-bit, where distilled data remains inferior to ImageNet. Mixed precision improves task performance below 4-bit, while mobile CPU gains at extremely low bitwidths mainly come from reduced data movement.

  • Calibration data: 5% accuracy increases when the number of calibration data points grows for 2-bit quantization, whereas the improvement is trivial for 4-bit quantization.
  • Calibration data: Distilled data performs well for 4-bit quantization but retains a large accuracy margin versus ImageNet at 2-bit quantization.Increasing the number of distilled samples provides little additional accuracy, possibly because they share the same low-diversity objective.
  • Mixed precision: Below 4-bit, mixed precision achieves better task performance than unified-precision models on ResNet-18 and ResNet-50.The experiments use a mobile CPU latency lookup table and validate only these ResNet models because the low-bit GEMM implementation supports only normal convolution.

B.4.4 IMPLEMENTATION DETAILS

Classification calibration uses 1024 ImageNet images, while detection uses 256 MS COCO images. Block reconstruction is applied to detection backbones, with naive layer reconstruction used elsewhere.

  • Image classification: 1024 ImageNet training images are used for classification calibration at 224×224 resolution.Batch normalization is folded into convolution, BN statistics are frozen, and Adam learns weight rounding and activation ranges during block reconstruction.
  • Object detection: 256 MS COCO training images are used for object-detection calibration.Resolution is 800 with maximum size 1333 for ResNet-18 and ResNet-50, and 600 with maximum size 1000 for MobileNetV2.
  • Object detection: Detection block reconstruction is restricted to the backbone because components such as the Feature Pyramid Network lack the block structure.The remaining network uses naive layer reconstruction.
Loading 2102.05426v2…