Source-linked AI summary
A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification
Babak Rokh, Ali Azarpeyvand, Alireza Khanteymoori
TL;DR
Large DNN models create storage, computation, memory, and energy challenges on constrained devices, motivating an integrated examination of quantization. The paper surveys quantization methods and training for image classification, including approximation, hardware operations, layer sensitivity, and evaluation. It concludes that QAT generally achieves higher inference accuracy than PTQ, while low-bit quantization remains difficult in deeper networks and activation quantization is more challenging than weight quantization.
Problem
DNNs’ numerous parameters and computations make deployment on constrained hardware difficult, while existing surveys have only limited focus on quantization methods.
Method
The paper comprehensively surveys DNN quantization for image classification, covering approximation methods, quantized-network training, hardware operations, layer sensitivity, mixed precision, benchmarks, and state-of-the-art results.
Results
QAT methods generally achieve higher accuracy than PTQ methods during inference, while POT can efficiently cover full-precision value distributions and enhance accuracy.
Takeaways & Limitations
Quantization supports compression and lower-cost operations, with quantizing both weights and activations offering higher compression and lower-cost operations than quantizing weights alone.
Takeaways & Limitations
Quantization below 4 bits remains challenging, especially in deeper networks, and activation quantization is more difficult because activations have a wider range.
Abstract
from arXiv · showhide
Recent advancements in machine learning achieved by Deep Neural Networks (DNNs) have been significant. While demonstrating high accuracy, DNNs are associated with a huge number of parameters and computations, which leads to high memory usage and energy consumption. As a result, deploying DNNs on devices with constrained hardware resources poses significant challenges. To overcome this, various compression techniques have been widely employed to optimize DNN accelerators. A promising approach is quantization, in which the full-precision values are stored in low bit-width precision. Quantization not only reduces memory requirements but also replaces high-cost operations with low-cost ones. DNN quantization offers flexibility and efficiency in hardware design, making it a widely adopted technique in various methods. Since quantization has been extensively utilized in previous works, there is a need for an integrated report that provides an understanding, analysis, and comparison of different quantization approaches. Consequently, we present a comprehensive survey of quantization concepts and methods, with a focus on image classification. We describe clustering-based quantization methods and explore the use of a scale factor parameter for approximating full-precision values. Moreover, we thoroughly review the training of a quantized DNN, including the use of a straight-through estimator and quantization regularization. We explain the replacement of floating-point operations with low-cost bitwise operations in a quantized DNN and the sensitivity of different layers in quantization. Furthermore, we highlight the evaluation metrics for quantization methods and important benchmarks in the image classification task. We also present the accuracy of the state-of-the-art methods on CIFAR-10 and ImageNet.
1 INTRODUCTION
DNNs deliver high accuracy but impose substantial storage, computation, memory, and energy demands, motivating compression for resource-constrained deployment. This survey focuses on quantization for image classification, covering its methods, training, hardware effects, layer sensitivity, benchmarks, and reported results.
- Motivation: DNNs’ large parameter counts and heavy computations create storage, memory, and energy challenges for deployment on constrained hardware.Compression can reduce memory access and allow smaller models to reside in energy-efficient on-chip memory rather than off-chip DRAM.
- Compression approaches: Quantization approximates network components with low-bit-width precision, such as mapping 32-bit floating-point weights to 16-bit or 8-bit integers.It is one of several compression approaches, alongside pruning, low-rank approximation, and knowledge distillation.
- Survey scope: Compression methods can be combined or jointly optimized, and this work specifically concentrates on quantization methods for DCNN compression.The introduction distinguishes independent combinations such as pruning plus quantization from proposed joint optimization approaches.
- Quantization advantages: The survey reports that quantization achieves high compression with less accuracy reduction than other approaches and can contribute to controlling overfitting.Its hardware flexibility also reduces accelerator design cost and supports diverse DNN implementations.
- Quantization advantages: Quantization is flexible across DNN architectures and replaces costly floating-point operations with lower-cost hardware operations.In 1-bit quantization, a 32-bit floating-point multiplier can be replaced with an XNOR operator, reducing cost by 200 times on a Xilinx FPGA.
- Survey scope: The paper surveys quantization for image classification from multiple perspectives, including scale-factor and clustering-based approximation methods.It also reviews quantized-network training with straight-through estimators, effects on multiply-accumulate operations, layer sensitivity, mixed precision, and benchmark evaluation.
2 PRELIMINARY CONCEPTS OF NEURAL NETWORKS
The section introduces neural-network and CNN structure, emphasizing convolution and fully connected layers as major sources of computation and parameters. It then explains quantization as mapping full-precision values to low-bit discrete levels, along with its components, training approaches, and key trade-offs.
- Neural-network structure: Neural networks are acyclic graphs with input, hidden, and output layers, where hidden neurons use activation functions while output neurons generally do not.
- Convolutional neural networks: CNNs extract complex features by combining simpler features and are designed primarily for image-processing inputs.
- Convolutional neural networks: Convolution layers apply shared filters to input feature maps, producing output maps, while fully connected layers connect each neuron to all neurons in adjacent layers.Convolution and fully connected layers contain most DCNN computations and parameters, motivating their focus in accelerators and compression techniques.
- Quantization fundamentals: Quantization maps continuous full-precision values to discrete lower-bit quantization levels using a piecewise quantization function.Accurate quantization depends on selecting suitable levels and step sizes between successive levels.
- Quantized components: Weights, activations, and gradients are quantizable components, although biases and batch-normalization parameters are usually retained in full precision because they comprise few parameters.Quantizing both weights and activations enables low-cost MAC operations and reduces computational cost and memory usage.
- Quantization training: Quantized-network training is challenging because convergence commonly requires more iterations than full-precision training, while QAT generally achieves higher accuracy than PTQ.PTQ methods may retrain quantized networks once or repeatedly to reach acceptable accuracy.
4 TRAINING OF QUANTIZED NEURAL NETWORK
Quantized-network training must address unavailable or zero gradients from discrete, piecewise quantization functions. The survey reviews STE-based gradient estimation, estimator design, and training considerations including regularization.
- Training challenges: Quantization creates undefined or zero derivatives, preventing weight updates and making activation quantization especially challenging.These issues arise because quantized networks commonly use constant piecewise and discrete functions.
- Straight-Through Estimator: The straight-through estimator assigns surrogate gradients during backpropagation to train networks with non-differentiable quantized functions.For the Sign function, STE assigns a gradient value of 1 in the backward pass.
- Straight-Through Estimator: Hard tanh estimates the Sign function with gradient 1 on [-1, 1] and 0 outside that interval.It has been used as an STE in XNOR-Net and BNN.
- Straight-Through Estimator: The Bi-RealNet second-order estimator reduces the shaded difference from the Sign function to 2/3, while higher order reduces the difference further.The authors considered the second-order function acceptable because higher-order functions require more complex computations.
- Alternative estimators: EDE trades off identity-function coverage and hard-tanh accuracy by varying its parameters during training.It begins closer to the identity function and transitions toward hard tanh as epochs increase.
- Activation quantization: HWGQ estimator choice depends on network depth: Log-Tailed ReLU performs better in AlexNet, whereas Clipped ReLU performs better in deeper VGGNet-Variant and ResNet-18.Using Vanilla ReLU can produce inaccurate gradients and unstable learning, with greater impact in deeper networks.
- Training considerations: The survey also reviews learning rate, network structure, and regularization as training considerations for binary and ternary quantized networks.These techniques are discussed as requirements for training quantized neural networks.
5 OPERATIONS IN QUANTIZATION
Quantization replaces expensive floating-point MAC computations with integer, bit-shift, and bitwise operations. The survey describes these implementations for logarithmic, fixed-point, and binary operands, including opportunities to omit zero products.
- Quantized MAC operations: Quantization maps 32-bit floating-point values to lower bit-widths such as 8, 4, 2, or 1 bit, enabling more efficient integer or bitwise operations.Floating-point MACs are a major accelerator bottleneck in DCNNs.
- Convolutional operations: The convolutional MAC computes dot products between filters and input feature-map patches to produce output feature maps.The survey presents this dot-product operation as the convolution-layer computation being optimized.
- Logarithmic quantization: In logarithmic quantization, floating-point multiplications are replaced with bit-shift operations and integer additions.The survey describes methods that quantize activations alone or both weights and activations using base-2 logarithms.
- Fixed-point quantization: Fixed-point integer weights and activations allow MAC computation through bitwise operations instead of floating-point arithmetic.The relevant operands are represented as M-bit weights and N-bit activations.
- Binary quantization: Binary MAC replaces floating-point multiplication with XNOR and floating-point addition with bitcount for operands in {-1,1}.For binary values in {0,1}, XNOR is replaced by AND.
- Sparse computation: Ternary weight quantization leaves only 16% of forward-pass and 33% of backward-pass operations non-zero in ResNet-34 on ImageNet.Zero operands allow multiplication operations to be omitted, further reducing computation.
6 LAYERS IN QUANTIZATION
Quantization affects layers differently: first and last layers are most accuracy-sensitive, while mixed-precision methods assign bit-widths to balance accuracy and compression costs.
- Layer sensitivity: First and last layers cause more accuracy loss under quantization, although their effects on compression and speed differ.The first layer is sensitive because it receives raw input, while the last layer directly affects network outputs.
- Layer sensitivity: Quantizing the last fully connected layer saves more memory than quantizing the first layer because it typically contains more weights.Latency depends on the DCNN architecture; in some cases, quantizing the first layer provides greater speed-up.
- Layer sensitivity: Some methods retain the first and last layers in full precision or use higher precision for them than for hidden layers.Other approaches quantize these layers using a learnable scale layer to align the final output with the desired output range.
- Mixed-precision quantization: Mixed-precision quantization assigns different bit-widths to layers according to sensitivity while optimizing accuracy and compression rate.The search space grows exponentially with layers, with O(M^N) possible assignments for M precisions across N layers.
- Mixed-precision quantization: HAWQ selects lower precision for layers with smaller Hessian spectra and many parameters to increase compression.Other approaches use group-Lasso regularization, differentiable architecture search, proxy classifiers, or channel-wise precision assignment.
- Mixed-precision quantization: Mixed-precision quantization is more complex to implement in software and hardware than fixed-precision quantization.Its optimization must account for model cost based on parameters, FLOPs, and weight and activation bit-widths.
7 EVALUATION AND DISCUSSION
Quantization is evaluated through accuracy, efficiency, and resource metrics across standard image-classification benchmarks. Results show that carefully designed low-bit methods can approach or exceed full-precision accuracy, but aggressive bit reduction increases degradation and comparisons require methodological context.
- Evaluation metrics: Common evaluation metrics include accuracy, compression rate, memory consumption, FLOPs, energy consumption, and processing speed.Low-bit storage and simple operations can reduce memory use, computation, power, and processing time.
- Evaluation metrics: Quantization requires a trade-off between compression rate and model accuracy because information loss increases as bit-width decreases.Limited quantization levels and removal of larger weights contribute to accuracy degradation.
- Benchmark datasets: Image-classification benchmarks include MNIST, CIFAR-10, SVHN, and ImageNet, with CIFAR-10 used more frequently than MNIST and SVHN.ImageNet contains high-resolution natural images across 1000 categories and is considered a large-scale, challenging benchmark.
- Comparison caveats: Cross-paper accuracy comparisons require checking original methodologies because studies differ in training conditions and whether first and last layers are quantized.Reported full-precision accuracy is presented as minimum and maximum values because learning rates, initialization, frameworks, and layer treatment vary.
- CIFAR-10 results: On CIFAR-10, most low-bit methods achieve accuracy close to full precision, with some surpassing the full-precision model.DMBQ reaches 93.9% with 1-bit weights and 2-bit activations on VGGNet-Small; TTQ and ProxQant also exceed full precision on ResNet-44.
- ImageNet results: On ImageNet, 1-bit or 2-bit weight methods can approach full-precision accuracy when activations remain full precision.With both weights and activations quantized, some 4-bit or 3-bit methods exceed full precision, whereas accuracy falls significantly at 2 or 1 bit.
8 CONCLUSION AND FUTURE WORKS
The survey synthesizes quantization methods, training strategies, precision choices, and benchmark results for image classification. It concludes that quantizing weights and activations can improve compression and operation cost, while activation quantization, very low precision, and mixed-precision selection remain challenging.
- Quantizing both weights and activations offers higher compression and lower-cost operations than quantizing weights alone.
- Activation quantization is more difficult because activations have wide ranges, non-differentiable functions, backward-pass estimation issues, and inference-time variation.
- QAT methods generally achieve higher inference accuracy than PTQ methods, but quantized training commonly requires extra convergence iterations and adaptive learning strategies.
- Non-uniform quantization, especially POT quantization, better covers full-precision value distributions and can enhance accuracy by allocating levels to informative regions.
- Low-bit weights and activations can achieve high accuracy on large-scale datasets such as ImageNet, but precision below 4 bits remains challenging in deeper networks.
- Mixed-precision quantization faces exponential time complexity when searching for the optimum bit-width for each layer, motivating polynomial-time solutions.