PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Model Compression and Acceleration for Deep Neural Networks

Yu Cheng, Duo Wang, Pan Zhou, Tao Zhang

arXiv:1710.09282v9cs.LGcs.CV

TL;DR

Deep neural networks are accurate but computationally expensive and memory intensive, limiting deployment under strict resource or latency constraints. This paper surveys compression and acceleration techniques, organizing them into four categories and analyzing their performance, applications, advantages, drawbacks, benchmarks, and future challenges. It concludes that method choice depends on application requirements and that no single approach is universally best.

  • Problem

    DNN models can be computationally expensive and memory intensive, limiting deployment on low-memory devices and in latency-sensitive applications.

  • Method

    The paper reviews compression and acceleration methods across parameter pruning and quantization, low-rank factorization, transferred or compact filters, and knowledge distillation.

  • Results

    The survey synthesizes recent methods, evaluation metrics, datasets, benchmark efforts, advantages, drawbacks, and practical method-selection considerations.

  • Takeaways & Limitations

    There are no golden criteria for selecting the best approach; the appropriate compression method depends on application and requirements.

Abstract

from arXiv · show

Deep neural networks (DNNs) have recently achieved great success in many visual recognition tasks. However, existing deep neural network models are computationally expensive and memory intensive, hindering their deployment in devices with low memory resources or in applications with strict latency requirements. Therefore, a natural thought is to perform model compression and acceleration in deep networks without significantly decreasing the model performance. During the past five years, tremendous progress has been made in this area. In this paper, we review the recent techniques for compacting and accelerating DNN models. In general, these techniques are divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation. Methods of parameter pruning and quantization are described first, after that the other techniques are introduced. For each category, we also provide insightful analysis about the performance, related applications, advantages, and drawbacks. Then we go through some very recent successful methods, for example, dynamic capacity networks and stochastic depths networks. After that, we survey the evaluation matrices, the main datasets used for evaluating the model performance, and recent benchmark efforts. Finally, we conclude this paper, discuss remaining the challenges and possible directions for future work.

I. INTRODUCTION

Deep neural networks achieve strong accuracy but require substantial parameters, memory, computation, and training time, motivating compression and acceleration for resource-constrained deployment. The survey organizes approaches into four categories and compares their applicability and complementarity.

  • Motivation: DNNs with millions or billions of parameters can require substantial training time and computational resources.Examples include a 60-million-parameter ImageNet model and fully connected architectures with billions of parameters.
  • Motivation: Reducing storage and computational cost is critical for real-time applications and devices limited by memory, CPU, energy, or bandwidth.The paper highlights portable devices, distributed systems, embedded devices, and FPGAs as deployment targets.
  • Survey organization: The survey divides compression and acceleration methods into parameter pruning and quantization, low-rank factorization, transferred or compact convolutional filters, and knowledge distillation.These categories target redundancy, informative parameter estimation, specialized convolutional structures, and related compact-model strategies.
  • Survey organization: Pruning and quantization, low-rank factorization, and knowledge distillation support models with fully connected layers, whereas transferred or compact filters target convolutional layers.The methods therefore differ in architectural coverage.
  • Survey organization: Most approaches are complementary and can be combined, such as transferred layers with pruning and quantization or quantization with low-rank approximations.The paper notes that combined strategies can achieve further compression.

II. PARAMETER PRUNING AND QUANTIZATION

Parameter pruning and quantization reduce DNN complexity by removing or simplifying parameters, using methods ranging from low-bit representations and binarization to pruning and parameter sharing. The survey also identifies accuracy, convergence, sensitivity-setting, and efficiency trade-offs.

  • Quantization and binarization: Quantization reduces the number of bits used for each weight, with 8-bit representations providing speed-up with minimal accuracy loss in one reported study.A separate 16-bit fixed-point approach reduced memory usage and floating-point operations with little classification-accuracy loss.
  • Quantization and binarization: The method in combines pruning, weight-sharing quantization, and Huffman coding to compress weights and the codebook.It learns connectivity, prunes small-weight connections, and retrains the remaining sparse network before coding.
  • Quantization and binarization: Binary-weight networks directly learn binary weights or activations during training, but their accuracy can decline substantially on large CNNs such as GoogleNet.Existing binarization schemes also use simple matrix approximations and may ignore binarization-related accuracy loss.
  • Network pruning: Pruning methods remove redundant connections or neurons using magnitude, Hessian-based criteria, data-free redundancy analysis, parameter sharing, or sparsity constraints.Sparsity constraints may use l0, l1, group sparsity, or group-wise convolutional-filter pruning.
  • Network pruning: Pruning can require longer convergence, manual layer-sensitivity settings, and fine-tuning, while often reducing model size without improving training or inference time.These issues make pruning cumbersome for some applications.

C. Designing Structural Matrix

Structural matrices replace large dense parameter matrices with compact representations that reduce storage and accelerate matrix operations. Circulant and Adaptive Fastfood transforms illustrate this approach, while structural bias and matrix selection remain challenges.

  • Motivation: Fully connected layers can bottleneck memory because dense m × n matrices require mn parameters and O(mn) matrix-vector computation.Structured matrices aim to reduce memory while accelerating inference, training, and gradient computation through fast multiplication.
  • Circulant projections: Circulant projections reduce memory for a d-dimensional layer from O(d^2) to O(d) and enable FFT-based computation in O(d log d) time.The circulant structure represents the matrix using a vector of parameters and supports Fast Fourier Transform acceleration.
  • Adaptive Fastfood: Adaptive Fastfood reparameterizes fully connected layers, reducing storage from O(nd) to O(n) and computation from O(nd) to O(n log d).Its transform uses random diagonal matrices, a random permutation matrix, and a Walsh-Hadamard matrix.
  • Extensions: Structured efficient linear layers extend the parsimony idea to block and multi-level Toeplitz-like matrix classes related to multidimensional convolution.The survey describes these structures as extensions of the broader structured-matrix approach.
  • Drawbacks: Structural constraints can hurt performance by introducing model bias, and selecting an appropriate structural matrix lacks a theoretical derivation method.These issues limit how straightforward structural-matrix compression is in practice.

III. LOW-RANK APPROXIMATION AND SPARSITY

Low-rank methods exploit structure in convolutional tensors and fully connected matrices to compress and accelerate DNNs. They offer practical reductions but involve decomposition, layerwise optimization, and potentially extensive retraining.

  • Motivation: Reducing convolutional layers can improve compression and overall speed because convolution operations contribute most computations in deep DNNs.Convolution kernels can be treated as 3D tensors with exploitable sparsity, while fully connected layers can be treated as matrices or tensors with low-rank structure.
  • Convolutional layers: 2× speedup for a single convolutional layer was achieved with a 1% drop in classification accuracy using low-rank approximation and clustering schemes.The reported result applies to some simple DNN models and concerns one convolutional layer.
  • Convolutional layers: Low-rank convolution methods include separable filters, approximation, clustering, CP decomposition, and BN-based decomposition of convolutional kernel tensors.These approaches estimate compact representations of convolutional filters, sometimes by decomposing layers sequentially and fine-tuning subsequent layers.
  • Convolutional layers: CP decomposition may lack a best rank-K approximation because the optimization is ill-posed, whereas BN decomposition always has a decomposition.Both CP and BN Low-rank schemes can train CNNs from scratch, but their decomposition properties differ.
  • Fully connected layers: Low-rank factorization also applies to fully connected layers through low-rank parameterization and truncated SVD of final weight layers.The surveyed applications include dynamic parameters, acoustic modeling, and compact multi-task deep learning architectures.
  • Limitations: Layerwise low-rank methods require computationally expensive decomposition, cannot perform global compression across layers, and often need extensive retraining for convergence.The survey notes that different layers hold different information, making global parameter compression important.

IV. TRANSFERRED/COMPACT CONVOLUTIONAL FILTERS

Transferred and compact convolutional-filter methods reduce parameters or computation by generating filters from transformed base filters or compact blocks. They can preserve classification accuracy in some architectures, but their effectiveness depends on architecture and transfer assumptions.

  • Transferred convolutional filters: Transferred-filter methods exploit convolutional weight sharing and translation-related structure to build many filters from a smaller set of base filters.Spatial-domain transforms include negation, multi-bias nonlinearities, rotations, and flips.
  • Transferred convolutional filters: The negation transform achieves a 2× compression rate across convolutional layers and can improve classification accuracy by acting as a regularizer.It constrains filters through pair-wise positive-negative relationships to reduce redundancy.
  • Transferred convolutional filters: Rotation-based transforms using fixed 90°, 180°, and 270° angles or angles learned from data achieve good classification performance.The learned-angle formulation generalizes the fixed-angle transformation.
  • Evaluation: On CIFAR-10 and CIFAR-100 with VGGNet-16 as baseline, transferred-filter methods reduce parameters with little or no classification-accuracy drop.The comparison reports Top-5 error.
  • Limitations: Transform-constrained filters perform competitively in wide or flat architectures such as VGGNet and AlexNet, but not in thin or deep architectures such as ResNet.The transfer assumptions can also make results unstable in some situations.
  • Compact convolutional filters: Compact-filter approaches directly reduce convolutional computation by replacing over-parameterized filters with compact blocks, including 1 × 1 convolutions.The section cites decomposing 3 × 3 convolutions into two 1 × 1 convolutions and SqueezeNet’s replacement of 3 × 3 convolutions.

V. KNOWLEDGE DISTILLATION

Knowledge distillation compresses larger or ensemble networks into smaller student models by transferring output or hidden-layer knowledge. Across several benchmark datasets, distilled models match or outperform teacher performance with fewer parameters and multiplications, although applicability and competitiveness remain limited.

  • Knowledge distillation: Knowledge distillation trains a smaller student to reproduce knowledge from a larger teacher, extending earlier ensemble-compression ideas to deep and wide networks.The original approach used pseudo-data labeled by strong classifiers, while later distillation targeted shallower deep models.
  • Student-teacher training: The student-teacher framework penalizes the student using softened teacher outputs while also training it to predict classification labels.One framework compresses an ensemble of teacher networks into a student of similar depth.
  • Results: On MNIST, CIFAR-10, CIFAR-100, SVHN, and AFLW, distilled models match or outperform teacher performance with notably fewer parameters and multiplications.These results are reported across the benchmark datasets surveyed in the section.
  • Extensions: Extensions transfer knowledge through online approximation of Monte Carlo teachers or higher hidden-layer neurons rather than only softened label probabilities.Hidden-layer representations are described as preserving comparable information more compactly.
  • Limitations: Knowledge distillation applies only to tasks with softmax loss and generally delivers less competitive performance than other compression approaches.These constraints limit its usage despite its ability to make deeper models shallower.

VI. OTHER TYPES OF APPROACHES

Other approaches reduce computation by adapting network capacity, selectively executing layers or regions, replacing pooling or convolution operations, and using alternative network structures. These methods target inference cost, parameter count, or convolutional overheads through input-dependent or operator-level changes.

  • Dynamic computation: Attention-like mechanisms and dynamic networks reduce computation by selectively focusing on task-relevant regions or executing input-dependent subsets of neurons.Dynamic capacity networks combine low-capacity and larger subnetworks.
  • Architecture changes: Replacing fully connected layers with global average pooling reduces parameters, although architectures using this idea have not fully optimized internal computing-resource utilization.GoogleNet and Network in Network are cited as examples achieving strong benchmark results with this design.
  • Stochastic depth: Stochastic depth trains very deep residual networks by randomly dropping layers during training and bypassing them with identity functions.Related methods learn which layers to execute during inference to reduce total computation.
  • Operator acceleration: FFT-based convolution, Winograd convolution, stochastic spatial sampling pooling, and detail-preserving pooling reduce convolutional or pooling overheads without necessarily reducing memory storage.These methods target computation rather than memory storage.

VII. BENCHMARKS, EVALUATION AND DATABASES

The paper surveys benchmark models and evaluation criteria for compression and acceleration, emphasizing that parameter reduction and computational speed are related but application-dependent.

  • Benchmark models: AlexNet, LeNets, All-CNN-nets, NIN, VGG, and ResNet are commonly used baseline architectures for evaluating compression methods.LeNet-300-100 is fully connected, whereas LeNet-5 combines convolutional and fully connected layers.
  • Evaluation criteria: Compression rate and speedup rate are the standard criteria for evaluating model compression and acceleration.Compression rate compares original and compressed parameter counts, while speedup rate compares their running times.
  • Evaluation criteria: Index space saving is another measurement used in several compression studies.The paper denotes the space-saving rate as β(M, M ∗).
  • Interpretation: Smaller models often compute faster, but parameter size and computational time can differ across CNN designs and applications.Fully connected layers may contain most parameters, while image classification may be dominated by floating-point operations in convolutional layers.

VIII. CHALLENGES AND FUTURE WORK

The paper offers application-dependent guidance for selecting compression techniques, while noting that no single approach is best across all settings.

  • General suggestions: There are no golden criteria for selecting the best compression approach; the choice depends on application requirements.The paper frames method selection as application-specific rather than universally ranked.
  • Method selection: Pruning and quantization or low-rank factorization suit compacting pretrained networks, while low-rank and transferred filters support end-to-end solutions.The recommendation distinguishes pretrained-model compression from end-to-end design.
  • Method selection: Transferred convolutional filters may benefit domain-specific applications such as medical-image classification when human priors match image transformations.The paper gives organ-image rotation as an example of a useful prior.
  • Method selection: Pruning and quantization generally provide reasonable compression without hurting accuracy, making them suitable when stable model performance is required.This recommendation is explicitly tied to applications prioritizing stable performance.
  • Method selection: Knowledge distillation can provide robust student models for small or medium datasets by transferring knowledge from a teacher model.The paper presents distillation as useful when datasets are not large and efficiency improvement is needed.
  • Method selection: Because the techniques are orthogonal, combining methods can maximize gains and address models containing both convolutional and fully connected layers.One example combines low-rank compression for convolutional layers with pruning for fully connected layers.

B. Technique Challenges

The paper identifies unresolved challenges in configuring compressed networks, adapting methods to hardware, pruning channels, controlling priors, improving distillation, and interpreting compression decisions.

  • Configuration: Current methods often rely on well-designed CNNs with limited freedom to change architectures and hyper-parameters, restricting configuration for more complicated tasks.Future compressed models need more plausible configuration strategies.
  • Hardware constraints: Mobile, robotic, and self-driving platforms remain constrained by limited computational resources, motivating hardware-specific compression methods.The challenge concerns making full use of restricted platform computation.
  • Channel pruning: Channel pruning reduces feature-map width without special implementation, but removing channels can substantially alter the next layer’s input.This creates a configuration challenge despite its efficiency benefits.
  • Human priors: Human priors in structural matrices and transferred filters can significantly affect model performance and stability, requiring control over their impact.The paper identifies prior-induced effects as a challenge for these methods.
  • Knowledge distillation: Knowledge distillation avoids requiring special hardware or implementations, but further work is needed to improve its performance.The paper specifically calls for continued development of KD-based approaches.
  • Interpretability: The black-box nature of compression remains a barrier because the reasons for pruning particular neurons or connections are unclear.The paper identifies knowledge interpretability as an important open challenge.

C. Possible Future Directions

Proposed future directions include automated configuration, hardware-aware design, improved channel pruning and distillation, generalized spatial transformations, and broader applications beyond image classification.

  • Possible Future Directions: Neural architecture search and reinforcement learning can help automate hyper-parameter configuration and explore compressed-model design spaces.The cited framework learns how to exploit problem structure, while reinforcement learning samples design spaces efficiently.
  • Possible Future Directions: Hardware-aware methods can incorporate accelerator feedback or learn architectures optimized for target tasks and hardware platforms.HAQ is cited as an example of including hardware feedback in the design loop.
  • Possible Future Directions: Training-based and iterative two-step methods are suggested to address channel-pruning configuration, although training from scratch can be costly for deep CNNs.These methods impose sparsity during training or iteratively prune channels.
  • Possible Future Directions: Knowledge distillation may improve by transferring neuron selectivity or using contrastive loss instead of KL divergence.The paper also suggests selecting task-essential neurons as transferable knowledge.
  • Possible Future Directions: Convolutional-filter and structural-matrix methods could be generalized from predefined spatial transformations to broader families of spatial transformations.The proposed direction targets the limitations imposed by fixed human-designed transformations.
  • Applications: Future work should extend compact models to larger deep-network applications, including video, vision-language systems, and GANs.The paper notes existing adaptation beyond image classification and calls for more work in these areas.
Loading 1710.09282v9…