PaperlayerView exact revision on arXiv

Source-linked AI summary

Pruning and Quantization for Deep Neural Network Acceleration: A Survey

Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, Xiaotong Zhang

arXiv:2101.09671v3cs.CVcs.AI

TL;DR

Complex DNNs challenge real-time deployment because they require substantial computation, energy, and resources. This survey reviews pruning and quantization, compares techniques and frameworks, and reports that quantization usually improves performance by 2–3× while reducing storage, with scope limitations beyond CNN classification.

  • Problem

    Complex DNN architectures require substantial computation and energy, making efficient real-time deployment difficult, especially under embedded resource constraints.

  • Method

    The paper surveys pruning and quantization, covering their methods, mathematical formulations, classifications, frameworks, hardware implementations, and practical guidance.

  • Results

    Quantization usually improves performance by 2–3× and dramatically reduces storage requirements, while pruning and quantization can independently or jointly accelerate inference.

  • Takeaways & Limitations

    Pruning removes redundant computations and quantization reduces datatype precision, providing complementary approaches for reducing storage requirements and accelerating inference.

  • Takeaways & Limitations

    Compression research is primarily focused on CNN classification, leaving applications such as object detection, speech recognition, and language translation for future study.

Abstract

from arXiv · show

Deep neural networks have been applied in many applications exhibiting extraordinary abilities in the field of computer vision. However, complex network architectures challenge efficient real-time deployment and require significant computation resources and energy costs. These challenges can be overcome through optimizations such as network compression. Network compression can often be realized with little loss of accuracy. In some cases accuracy may even improve. This paper provides a survey on two types of network compression: pruning and quantization. Pruning can be categorized as static if it is performed offline or dynamic if it is performed at run-time. We compare pruning techniques and describe criteria used to remove redundant computations. We discuss trade-offs in element-wise, channel-wise, shape-wise, filter-wise, layer-wise and even network-wise pruning. Quantization reduces computations by reducing the precision of the datatype. Weights, biases, and activations may be quantized typically to 8-bit integers although lower bit width implementations are also discussed including binary neural networks. Both pruning and quantization can be used independently or combined. We compare current techniques, analyze their strengths and weaknesses, present compressed network accuracy results on a number of frameworks, and provide practical guidance for compressing networks.

1. Introduction

The introduction frames CNN acceleration around network structure, optimization, and specialized hardware, with this survey focusing on pruning and quantization as compression techniques. It reviews their methods, classifications, comparisons, and practical use for reducing computation and model size.

  • Motivation: Over-parameterized DNNs contain redundant neurons that can often be removed with little or no accuracy loss, easing deployment constraints.Large networks create computational, training-time, bandwidth, and embedded-deployment challenges.
  • Acceleration scope: CNN acceleration can be considered across network structure, network optimization, and hardware accelerator design.The paper focuses primarily on network optimization while noting relevant hardware implementations.
  • Network structure: Network structure includes novel components, neural architecture search, and knowledge distillation for producing more efficient models.Examples include separable, inception, and residual blocks; NAS searches predefined spaces, while KD trains a student to imitate a teacher.
  • Network optimization: Network optimization includes convolution optimization, parameter factorization, pruning, and quantization.Factorization decomposes higher-rank tensors into lower-rank tensors, while convolution preserves spatial structure and typically uses fewer coefficients than fully connected layers.
  • Compression techniques: Pruning removes parameters that do not affect accuracy, whereas quantization replaces datatypes with reduced-width representations such as FP32 with INT8.Both techniques are described as network-compression methods, and quantization can also use encoded values to preserve information.
  • Paper contributions: The survey reviews pruning and quantization methods, classifies pruning as static or dynamic, compares quantization techniques and frameworks, and provides practical guidance.It also discusses mathematical formulations, SOTA methods, accuracy effects, and quantized results for common networks.

2. Convolutional Neural Network

CNNs use convolution-based feature extraction and are deployed through training and inference, but model size and computation constrain deployment. The section introduces convolution operations, efficient structures, and model comparisons relevant to inference acceleration.

  • CNNs extract features with convolution operations and are commonly applied to visual tasks, while also supporting language, speech, recommendation, malware, and sensor applications.
  • CNN deployment consists of training followed by inference, with inference constrained by memory, bandwidth, and processing limits on edge, mobile, and embedded devices.Fine-tuning retrains a previously trained model and can help recover accuracy in pruned or quantized networks.
  • 2.3. Convolution Operations: Standard convolution can be transformed into a GEMM computation by flattening 3D tensors into 2D matrices for parallel matrix multiplication.The resulting matrices combine input features with convolutional kernels to produce activations.
  • 2.7. Parameters: GoogLeNet achieves 69.78% top-1 accuracy with 7 million parameters, while MobileNet achieves 70% top-1 accuracy with 4.2 million parameters and 1.14 GFLOPs.

3. Pruning

Pruning removes redundant parameters or neurons to reduce network size and deployment costs. The survey organizes pruning by execution timing, structure, element granularity, and mathematical formulation.

  • Pruning removes parameters that do not significantly affect network accuracy and can reduce memory size and bandwidth requirements.It was developed to reduce trained networks for constrained environments such as embedded systems.
  • Static pruning is performed offline before inference, whereas dynamic pruning is performed during runtime.
  • Pruning may be categorized as structured or unstructured, by neuron or connection type, and by static or dynamic execution.These categories can overlap.
  • Pruning granularity ranges from individual elements to rows, columns, filters, and layers, producing different sparsity patterns.Element-wise pruning typically has the smallest sparsity impact and produces an unstructured model.
  • The pruning function produces a pruned network configuration and pruned weights while the resulting performance loss is typically measured by classification accuracy.

3.1. Static Pruning

Static pruning removes neurons or parameters offline after training and may include fine-tuning or retraining. The survey covers magnitude, penalty, structured, recoverable, and adaptive sparsity methods with associated accuracy and hardware trade-offs.

  • Static pruning selects parameters, removes them offline, and may fine-tune or retrain the resulting network to recover performance.
  • Pruning Criteria: Element-wise pruning can create sparse matrices that execute inefficiently on instruction-set processors and often require specialized hardware for acceleration.Structured methods such as Group LASSO remove groups of neurons while preserving network organization.
  • Pruning combined with Tuning or Retraining: Recoverable pruning can splice pruned weights back during training; on AlexNet it achieved 17.7× reduction with no accuracy loss and reduced retraining iterations to 14.58% of Deep compression.The resulting asymmetric network still complicates hardware implementation.
  • Pruning combined with Tuning or Retraining: Soft Filter Pruning achieved 29.8% speed-up on ResNet-50 with 1.54% ILSVRC-2012 top-1 accuracy loss.
  • Automatically computed layer-wise sparsity produced 20× model-size compression, 5× computing reduction, and less than 0.1% accuracy loss on VGG.Excessive layer-wise sparsity can cause divergence and require substantial retuning.
  • 17.5× compression reduced VGG theoretical execution time to 15.56% of the unpruned network, while a related method retained ResNet accuracy with 5% to 10% of original weights.
  • 73.59% of VGG-16 compute operations were pruned with 2.39% ILSVRC2012 top-1 loss, while ResNet-50 pruned 65.80% with 3.10% accuracy loss.

3.2. Dynamic Pruning

Dynamic pruning selects layers, channels, or neurons at runtime based on changing inputs, potentially reducing computation, bandwidth, and power. Its benefits depend on decision mechanisms, runtime overhead, and accuracy trade-offs.

  • Dynamic pruning principles: Dynamic pruning chooses runtime-excluded layers, channels, or neurons according to changing input data.Unlike static pruning, it can adapt computation to each input and typically avoids runtime fine-tuning or retraining.
  • Decision mechanisms: Decision systems may use attached connections, learned connection characteristics, side networks, or reinforcement learning to determine what to prune.Stopping criteria can skip layers, select data paths, or terminate computation early.
  • System trade-offs: Dynamic pruning trades runtime decision overhead against reduced computation, bandwidth, power, and possible accuracy loss.The pruning criteria themselves require additional compute, bandwidth, and power.
  • Representative results: 5.3× speed-up on CIFAR-10 was achieved by a reinforcement-learning conditional-computing approach without accuracy loss.The method divides the network into blocks and learns block-selection policies as a reinforcement-learning problem.
  • Adaptive architectures: Adaptive and slim networks expose accuracy–efficiency choices through early exits, learned computation graphs, or selectable widths.Reported examples include confidence-threshold exits, RL-selected residual blocks, and switchable network widths across hardware platforms.
  • Representative results: 20% average speed-up on ResNet-101 for ILSVRC-2012 was achieved by BlockDrop without accuracy loss.Its policy network makes an image-specific block-participation decision once when an image is loaded.
  • Representative results: 2.3× to 5.9× execution-time reduction for VGG-16 incurred 2.32% to 4.89% top-5 accuracy loss under Runtime Neural Pruning.RNP uses an RNN-based side network trained with reinforcement learning to predict unnecessary feature maps.
  • Representative results: 5× computation speed-up increased VGG-16 top-5 error by 4.77% with Dynamic Channel Pruning, compared with 4.89% for RNP.DCP prunes channels whose T-Weighting score falls below a threshold; the threshold is found through iterative training.

3.3. Comparisons

Pruning results are difficult to compare because methods differ in training assumptions and pruning structure. Evidence indicates that the pruned architecture and pruning form can matter more than inherited weight values.

  • Comparison challenges: Pruning techniques are diverse and difficult to compare, motivating unified benchmark frameworks such as Shrinkbench.Shrinkbench is designed to provide pruning performance comparisons.
  • Training assumptions: Whether pruned models should retain pretrained weights is ambiguous, with some work showing they can be trained from scratch.This raises the possibility that the resulting architecture, rather than the inherited weights, is central to performance.
  • Pruning structure: Structured pruning can be trained from scratch, whereas unstructured pruning requires fine-tuning to restore accuracy.The cited comparison distinguishes the effects of pruning form and reports that simple magnitude pruning can outperform alternatives at matched computational complexity.
  • Reported comparison: A magnitude-pruned ResNet-50 achieved higher accuracy than state-of-the-art methods at the same computational complexity.This result is reported alongside the conclusion that pruning form matters for recovering or maintaining accuracy.

4. Quantization

Quantization compresses neural networks by replacing high-precision representations with lower-width values, reducing storage and potentially improving inference efficiency. The survey covers quantization categories, evolution, benefits, and practical speedup limits.

  • Quantization foundations: Quantization replaces continuous signals or high-precision datatypes with discrete or lower-width representations, such as FP32 to INT8.Clustering and parameter sharing are also included within the survey’s quantization definition.
  • Motivation: Reducing FP32 parameters to lower bit representations can reduce bandwidth, energy, and on-chip area.The motivation is that FP32 precision exceeds what many networks require.
  • Quantization evolution: Weight quantization, clustering, and sharing reduced storage requirements nearly 4×, including a reduction from 27MB to 6.9MB.The surveyed evolution then includes post-training quantization and quantization-aware training.
  • Accuracy and calibration: KL-divergence-calibrated quantization can accelerate many well-known models without accuracy loss.Fine-tuning can also be applied with this approach.
  • Taxonomy: Quantization research is organized around quantization-aware training versus post-training quantization, layer-wise versus channel-wise grouping, and N-bit widths.These categories distinguish training involvement, grouping location, and parameter precision.
  • Practical limits: INT8 inference typically provides about 2–3× speedup in TensorFlow-Lite and TensorRT rather than the expected 4× over FP32.With larger batch sizes, TensorRT achieves 3–4× acceleration; quantization and dequantization add overhead.

4.1. Quantization Algebra

The quantization algebra formalizes scaling, clamping, zero-point adjustment, rounding, convolution, and dequantization for features, weights, and outputs. It also describes implementation choices affecting range, offsets, and downstream activation placement.

  • General quantization framework: Equation 12 maps floating-point values to quantized values using a scale, clamp function, zero-point, and rounding function.The zero-point adjusts the true zero in some asymmetric quantization schemes.
  • Scaling methods: The min-max method derives quantization bounds from parameter minima and maxima, with representable range determined by bit-width.The zero-point is typically non-zero in this method.
  • Scaling methods: The max-abs method uses a symmetric bound with zero-point zero, reducing offset-convolution overhead while narrowing dynamic range.The narrower range is especially noticeable for ReLU-activated data, whose values are positive.
  • Quantized operands: Quantization applies to input features, weights, and biases, with separate feature and weight scales in the min-max formulation.The real-valued and quantized forms are distinguished by subscripts r and q.
  • Integer convolution: Integer quantized convolution preserves the convolution form while operating on quantized features and weights.Third-party libraries support this form of integer convolution acceleration.
  • Dequantization: Dequantization converts quantized convolution outputs back to floating point using feature and weight scales.This is useful for layers that process floating-point tensors.
  • Layer composition: Consecutive layers can compute with quantized parameters, allowing dequantization to be merged into one operation for the next layer.The next layer receives a quantized feature with its associated scale.
  • Implementation choices: Activation placement after quantized, dequantized, or re-quantized outputs can produce different numerical outcomes because these outputs have different precision.Biases are often retained in 32-bit precision while weights use 8-bit precision because biases consume little memory.

4.2. Quantization Methodology

Quantization spans moderate-precision representations, power-of-two and binary formats, ternary schemes, weight sharing, and combined compression methods, with accuracy and hardware trade-offs varying across techniques.

  • Moderate-Precision Quantization: 16-bit and 8-bit quantization can preserve accuracy with minimal or no loss, while mixed-precision training may sometimes improve performance.8-bit inference caused 1.5% accuracy loss on ResNet-50 in one study, whereas another reported lossless performance without retraining.
  • Logarithmic Quantization: Power-of-two quantization replaces multiplications with shifts and can achieve substantial compression with limited accuracy loss.INQ obtained 71× compression with 0.52% top-1 accuracy loss on ILSVRC-2012 with AlexNet.
  • Logarithmic Quantization: Shift-based networks can substantially reduce hardware cost, but pretrained fine-tuning is important because training from scratch is difficult on larger datasets.ShiftCNN achieved 75% power saving and up to 1090× clock cycle speed-up, while DeepShift reduced ResNet-18 accuracy loss from 4.48% to 1.09% through pretrained tuning.
  • Plus-minus Quantization: Binary and ternary networks simplify arithmetic and reduce model size, but their accuracy varies widely with dataset, architecture, and scaling or thresholding strategies.Binary networks can reduce network size by up to 32×, while TTQ exceeded full-precision AlexNet accuracy by 0.3%.
  • Plus-minus Quantization: Fine-grained and architecture-aware low-bit methods can approach full-precision or compact-network accuracy while retaining substantial computational savings.MeliusNet-59 reached 70.7% top-1 accuracy with 0.532 BFLOPs, and NAS-designed binary models reached 69.65% top-1 accuracy with 2.8× speed-up.
  • Other Approaches to Quantization: Weight sharing, hashing, pruning, and Huffman coding provide additional compression routes beyond direct precision reduction.Deep Compression achieved 49× compression on VGG-16 with no accuracy loss on ILSVRC-2012.
  • Quantization-aware Training: Quantization introduces irreversible information loss, especially in binary and shallow networks, but retraining can recover some accuracy.Binary weights and activations caused 29.10% accuracy loss on ResNet-34 and 24.20% on GoogLeNet in one reported comparison.

4.3. Quantization Deployment

The section surveys quantization deployment across software frameworks, libraries, hardware, and training settings. Implementations use calibration, low-precision arithmetic, optimized kernels, and heterogeneous platforms to balance efficiency and accuracy.

  • Deployment scope: Quantization deployment spans libraries, frameworks, specialized hardware, and compilers, with most listed frameworks using INT8 precision.Table 2 defines QAT, PTQ, and zero-point offsets; the section compares libraries, hardware implementations, and DNN compilers.
  • Frameworks: TensorRT converts pretrained floating-point models into reusable INT8 or FP16 models using profiling, layer fusion, memory management, and concurrent operations.For INT8 inference, convolution results accumulate in INT32 before dequantization.
  • Calibration: KL-divergence calibration selects the scale minimizing divergence, and GoogLeNet on ImageNet lost 0.36% top-1 accuracy after calibration with 125 images.The method searches a predefined scale range and evaluates each candidate.
  • Quantized arithmetic: Gemmlowp uses asymmetric quantization with feature and weight scales, while its pipeline reduces multi-add operations for quantized output production.Its zero-point corrections are integrated into a three-stage pipeline.
  • Efficient kernels: QNNPACK’s Panel Dot product computes multiple dot products in parallel and improves MobileNet convolution performance by about 1.41× 2.23× on mobile devices.The method loads multiple rows and columns rather than only two operands per multiply-accumulate operation.
  • Hardware platforms: FINN achieved 12.36 million images per second on MNIST with 4.17% accuracy loss on a Zynq-7000 SoC using binarized layers.Other deployments combine binary weights and 3-bit activations with 8-bit first and last layers.

4.4. Quantization Reduces Over-fitting

The section reports cases where quantization improves accuracy rather than merely reducing computation. Several low-bit models outperform their full-precision or reference counterparts on standard image-classification benchmarks.

  • 3-bit weights improve VGG-16 top-1 accuracy by 1.1% over its full-precision counterpart.
  • 2-bit weights with 8-bit activations reduce AlexNet’s reference top-1 error by 1.0%.
  • ResNet-34 reaches 74.52% top-1 accuracy with 4-bit weights and activations, versus 73.59% for the 32-bit version.

5. Summary

The survey summarizes pruning and quantization as complementary compression approaches whose effectiveness depends on structure, precision, training, hardware, and deployment constraints. It reports practical recommendations while noting accuracy trade-offs and scope limitations.

  • Pruning: Pruning includes static and dynamic techniques, with element-, channel-, shape-, filter-, and layer-wise choices offering different compression and acceleration benefits.Element-wise pruning improves storage, while filter-wise and layer-wise pruning can reduce computational complexity.
  • Pruning: Pruning can sometimes produce incremental accuracy gains, but the survey states that changing to a better architecture more reliably realizes accuracy improvements.Separable blocks are cited as an example of improved accuracy with reduced computational complexity.
  • Practical guidance: Compression should be matched to the deployment environment, target device, speed-compression trade-offs, and accuracy requirements.
  • Pruning: Dynamic pruning may improve accuracy and preserve higher network capacity, while layer-varying pruning ratios are preferred over uniform pruning.
  • Quantization: Quantization usually loses accuracy from information loss, especially in compact networks, although asymmetric, calibration-based, and quantization-aware methods can improve accuracy.Non-uniform data distributions may further deteriorate quantization performance.
  • Quantization: 8-bit quantization is widely applied as a trade-off between accuracy and compression, with minimal accuracy loss especially when quantization-aware training is enabled.Binarized networks can also achieve reasonable accuracy with specialized hardware.

6. Future Work

Future work targets automated compression search, broader neural-network applications, hardware-aware co-design, and methods that coordinate multiple optimization techniques. These directions address accuracy, generality, and system-efficiency gaps.

  • Automatic Compression: Automatic quantization and pruning could search encoding, compression, sparsity, and accuracy trade-offs without human intervention.Suggested search methods include random search and genetic search.
  • Compression on Other Types of Neural Networks: Compression research is primarily focused on CNN classification, leaving object detection, speech recognition, and language translation comparatively underexplored.
  • Hardware Adaptation: Hardware implementations can limit pruning effectiveness, motivating hardware-software co-design for better system efficiency.Element-wise pruning may only slightly reduce computation or bandwidth on GPU im2col-gemm, while shape-wise pruning may not suit dedicated CNN accelerators.
  • Global Methods: Future methods could coordinate pruning with tensor factorization and combine different compression types and levels.

7. Conclusions

Network compression addresses deployment costs in complex deep networks through pruning and quantization. The paper emphasizes their distinct mechanisms, levels of application, and potential use independently or together.

  • Network compression can reduce deployment computation and energy costs with little accuracy loss, and accuracy may sometimes improve.
  • Pruning removes redundant computations using criteria such as weight magnitude or l_p-norms, and can operate from individual elements through entire networks.
  • Quantization reduces computation by lowering datatype precision, typically converting weights, biases, and activations to 8-bit integers.
  • Very low-bit quantization, including binary networks, improves performance usually by 2-3x and reduces storage, but complicates retraining and gradient computation.
  • Pruning and quantization can be applied independently or combined to reduce storage requirements and accelerate inference.

8. Quantization Performance Results

Table 4 organizes quantization performance on ILSVRC2012 by weight and activation bit widths. The supplied table text identifies the comparison dimensions but does not report performance values.

  • Table 4 compares quantized network performance on ILSVRC2012 across different weight and activation bit widths.
  • The table labels weight width as W and activation, also called feature, width as A.
  • Top-1 and Top-5 are listed as performance columns, alongside model and deployment fields.
Loading 2101.09671v3…