PaperlayerView exact revision on arXiv

Source-linked AI summary

Methods for Pruning Deep Neural Networks

Sunil Vadera, Salem Ameen

arXiv:2011.00241v2cs.LGcs.NE

TL;DR

Deep networks contain many parameters, creating a need for smaller, more efficient models that preserve accuracy. This paper surveys pruning methods, categorises over 150 studies, and collates reported results for cross-architecture and cross-data-set comparison. The comparison finds significant reductions for AlexNet, ResNet, and VGG, but no single best method and greater difficulty pruning ResNet.

  • Problem

    Deep networks can contain millions of parameters, motivating the search for smaller, more efficient networks without compromising accuracy.

  • Method

    The paper surveys over 150 pruning studies, focuses on magnitude, clustering, and sensitivity methods, and collates published results across architectures and data sets.

  • Results

    The comparison suggests significant reductions for AlexNet, ResNet, and VGG, with no single method best and ResNet harder to prune than the other architectures.

  • Takeaways & Limitations

    Combining pruning approaches may provide further compression, as shown by combinations involving Lasso regression, factorization, magnitude pruning, clustering, and Huffman coding.

  • Takeaways & Limitations

    Taylor-based feature-map pruning approaches are limited by skip connections and increased memory requirements when assessing feature-map changes.

Abstract

from arXiv · show

This paper presents a survey of methods for pruning deep neural networks. It begins by categorising over 150 studies based on the underlying approach used and then focuses on three categories: methods that use magnitude based pruning, methods that utilise clustering to identify redundancy, and methods that use sensitivity analysis to assess the effect of pruning. Some of the key influencing studies within these categories are presented to highlight the underlying approaches and results achieved. Most studies present results which are distributed in the literature as new architectures, algorithms and data sets have developed with time, making comparison across different studied difficult. The paper therefore provides a resource for the community that can be used to quickly compare the results from many different methods on a variety of data sets, and a range of architectures, including AlexNet, ResNet, DenseNet and VGG. The resource is illustrated by comparing the results published for pruning AlexNet and ResNet50 on ImageNet and ResNet56 and VGG16 on the CIFAR10 data to reveal which pruning methods work well in terms of retaining accuracy whilst achieving good compression rates. The paper concludes by identifying some promising directions for future research.

1 INTRODUCTION

The introduction frames pruning as a way to reduce the size and computational demands of parameter-heavy deep networks while preserving accuracy. It surveys pruning approaches and supplies a resource for comparing reported results across architectures and data sets.

  • ResNet50 has more than 25 million parameters and VGG16 more than 138 million weights, motivating smaller networks without compromising accuracy.
  • Its resource brings reported results together so researchers can compare methods across different architectures and data sets.
  • The survey identified over 150 pruning studies and categorised them into eight groups based on their underlying approaches.
  • The paper focuses on magnitude-based pruning, similarity and clustering, and sensitivity analysis, while directing readers to other surveys for additional categories.
  • Sections 3 to 5 present representative methods, Section 6 compares published pruning results, and Section 7 identifies insights and future research directions.

2 BACKGROUND

The background explains how convolutional networks transform inputs into features and classifications, and why their growing parameter counts motivate pruning. It also defines the measures used to evaluate pruning methods.

  • Convolutional filters apply learned weights to input feature maps, while pooling reduces feature-map size and fully connected layers classify the resulting features.
  • LeNet-5 has 60K parameters in 5 layers, whereas AlexNet has over 60M parameters in 8 layers.
  • The large parameter counts of modern models require greater computational resources and inhibit deployment in embedded systems.
  • Pruning studies evaluate architectures including ResNet, VGG, and DenseNet on data sets including ImageNet, CIFAR, and SVHN.
  • Reported measures include Top-1 and Top-5 accuracy, compression rate, and FLOPS required for classification.

3 MAGNITUDE BASED PRUNING

Magnitude-based pruning removes weights, nodes, or filters using magnitude or effects on subsequent layers. Influential studies show that iterative and dynamic procedures can achieve substantial compression while retaining accuracy.

  • Magnitude-based methods prune weights, nodes, and filters according to magnitude or the effect filters have on the next layer.
  • Han et al. prune weights below a threshold, fine-tune the network, and repeat until accuracy begins to deteriorate.
  • A factor-of-12 reduction in weights was achieved on LeNet networks without compromising accuracy; AlexNet and VGG16 achieved factors of 9 and 12 respectively.
  • L1 regularisation was better immediately after pruning, whereas L2 was better when the pruned model was fine-tuned.
  • Earlier layers were more sensitive to pruning, and iterative pruning outperformed removing the required proportion in one cycle.
  • Dynamic Network Surgery allows previously pruned weights to be reinstated and reduced AlexNet weights by a factor of over 17 on ImageNet.

3.2 The Lottery Ticket Hypothesis

The Lottery Ticket Hypothesis proposes that trained networks contain smaller subnetworks that can match the original network when suitably initialized. Studies examine how pruning strategy, initialization, learning rate, optimizer, task, and architecture affect these subnetworks.

  • The Lottery Ticket Hypothesis states that a trained network contains a subnetwork that can be trained to be at least as accurately as the original using no more training epochs.
  • Iterative pruning found a subnetwork comprising 3.6% of the original LeNet-300-100 network that performed just as well, while learning faster than the original.
  • Iterative pruning generally produced more accurate smaller networks than one-shot pruning, and random reinitialization learned more slowly than retaining original weight initializations.
  • Winning-lottery identification in larger VGG and ResNet models depended on learning rate, with lower rates performing better, and global pruning outperformed layer-by-layer pruning.
  • Later work found that initial-weight resetting performs poorly for larger networks, while weights from a later training iteration can help identify good ImageNet initializations; random retraining works only for moderate pruning up to 30%.
  • Experiments suggest winning tickets are generally optimizer independent, while transfer across tasks and results on NLP and reinforcement-learning tasks remain active questions.

3.3 Pruning Feature Maps and Filters

Feature-map and filter pruning removes structured components using data-dependent or data-independent criteria, often combining pruning with fine-tuning. Reported studies reduce parameters or computation while generally limiting accuracy loss, but comparisons depend on the pruning setup.

  • Structured pruning of filters and channels benefits from existing toolkit optimizations, unlike sparse weight pruning, which may require specialist libraries or hardware.
  • Feature-map and filter methods divide into data-dependent approaches based on output variation and data-independent approaches based on filter or channel properties such as zero proportions.
  • Layer-wise pruning with one or two fine-tuning epochs balances the computational cost of tuning after every layer against ignoring earlier layers' knock-on effects.
  • More than 16-fold compression on VGG16 incurred a 1.56% accuracy reduction, but the reported compression advantage included replacing fully connected layers with average pooling.
  • 64% of VGG16 parameters were pruned through layer sensitivity analysis without significantly compromising accuracy, with substantial pruning in layers containing smaller filters.
  • Filter sketching reduced parameters without significant accuracy loss relative to small-filter pruning and optimization-based channel pruning across CIFAR10 and ImageNet models.
  • Geometric-median pruning achieved 53.5% FLOPS reduction on ResNet-50, compared with 36.7% for ThiNet, 41% for Soft filter pruning, and 44% for NISP, with similar accuracy drops.
  • ThiNet reduced parameters by about 94% with a 1% Top-1 accuracy reduction, while other evaluations reported two-fold speed increases with accuracy or average-precision reductions of 1.4% and 0.4%.

4 PRUNING BASED ON SIMILARITY AND CLUSTERING

Similarity and clustering methods exploit redundant or similar weights and filters by grouping them and replacing each group with a representative. Evidence indicates greater similarity-based pruning opportunity in MLPs than CNNs, with clustering also evaluated on VGG and ResNet.

  • Similarity-based pruning is motivated by the possibility that duplicate or functionally similar weights and filters can be removed without affecting accuracy.
  • RoyChowdhury et al. grouped similar filters using inner products and replaced each group with its mean filter.
  • Similar weights and filters occurred more often in MLPs than CNNs, suggesting greater opportunity for similarity-based pruning in MLPs.
  • Ayinde et al. used agglomerative hierarchical clustering with inner-product similarity and replaced each cluster by a randomly selected filter in VGG16 and ResNet34 experiments.

5 SENSITIVITY ANALYSIS METHODS

Sensitivity analysis methods estimate how perturbing or removing weights, filters, channels, nodes, or feature maps affects loss or accuracy. The surveyed approaches range from node and channel importance measures to first- and second-order approximations, with experiments reporting substantial compression and competitive pruning outcomes.

  • Overview: Sensitivity analysis models the effect of perturbing or removing weights, filters, and channels on the loss function.This directly targets the pruning goal of removing components with the least effect on model accuracy.
  • Nodes and channels: Skeletonization assigns attentional strengths to nodes and uses their loss-related importance to identify pruning candidates.The approach estimates importance from the difference in loss when a node’s attentional strength is set to zero versus one, approximated using derivatives.
  • Nodes and channels: Using eight hidden nodes followed by skeletonization converged in all 4-bit multiplexor experiments and retained accuracy, whereas starting with four nodes failed in 17% of cases.The comparison was made with training limited to 1000 epochs.
  • Nodes and channels: SSS reduced VGG16 parameters by 30% on CIFAR10 without accuracy loss and reduced ResNet50 FLOPs by 15% with a 0.1% Top-1 accuracy reduction.SSS learns channel importance during training; unlike Network Slimming, it does not assume batch-normalization layers and can scale network blocks.
  • Weight sensitivity: OBS outperformed OBD and magnitude pruning on XOR for identifying removable weights and reduced NetTalk weights from 18000 to 1560.These results illustrate sensitivity-based pruning for fully connected networks.
  • Feature-map sensitivity: Taylor-based feature-map methods reported strong pruning results, including 37 GFLOPs reduced to 3 GFLOPs with a 2.6% accuracy reduction and better performance than several baselines.The first-order approach was reported to work especially well on ResNet-50 and ResNet-34, while EigenDamage outperformed adapted OBD, OBS, and Network Slimming on VGG and ResNet benchmarks.
  • Feature-map sensitivity: A follow-up Taylor formulation addressed limitations involving equal layer importance with skip connections and increased memory requirements from assessing feature-map changes.The alternative estimates squared loss due to removing a parameter and aggregates parameter contributions to obtain filter importance.

6 A Resource for Comparing Published Results

The paper provides a resource for comparing reported pruning results across architectures and data sets. Its examples compare compression and Top-1 accuracy changes for ImageNet and CIFAR10 models.

  • Resource: The resource collates reported performance for over 50 pruning methods across different architectures and data sets.It addresses the difficulty of comparing studies whose architectures, data sets, and algorithms evolved over time.
  • Compared cases: Figure 4 compares AlexNet and ResNet50 on ImageNet with ResNet56 and VGG16 on CIFAR10.The charts report reductions in Top-1 accuracy, FLOPs, and parameters where available.
  • AlexNet: 41% FLOPs reduction with a 0.46% increase in accuracy is reported for AOFP-B2 on AlexNet ImageNet.Dyn Surg, SSR-L, and NeST exceed 93% parameter reduction without accuracy loss.
  • ResNet50: 33% FLOPs reduction without affecting accuracy is reported for AOFP-C1 on ResNet50 ImageNet.KSE and ThinNet achieve 78% and 95% parameter reductions with accuracy declines of 0.64% and 0.84%, respectively.
  • CIFAR10 models: 60% and 28% FLOPs reductions without accuracy loss are reported for KSE and SFP-NFT on ResNet56 CIFAR10.For VGG16 CIFAR10, AOFP, PF EC, and NetSlimming report 75%, 63%, and 51% FLOPs reductions without accuracy reductions.
  • Cross-method observations: Performance varies with architecture and data, while accuracy compromises can produce larger reductions within methods but not consistently across methods.AOFP retains accuracy well in three of the four compared cases.

7 CONCLUSION AND FUTURE WORK

The conclusion synthesizes pruning research across major method families and emphasizes comparison resources, complementary compression strategies, and broader evaluation as future priorities.

  • Conclusion: The survey focuses on magnitude pruning, similarity and clustering, and sensitivity analysis methods.It summarizes the field’s progression from early pruning of small fully connected networks toward larger deep architectures.
  • Magnitude-based methods: Magnitude methods range from data-dependent relevance estimates to data-independent filter or channel contribution measures and optimization-based approximations.These approaches developed from removing small MLP weights to pruning filters and channels.
  • Sensitivity analysis: Sensitivity analysis approximates loss changes caused by network modifications, with Taylor approximations forming its primary research direction.OBD, OBS, EigenDamage, CCP, and first-order approaches make different assumptions or formulations to make pruning feasible.
  • Lottery Hypothesis: Lottery results include transfer-learning performance, NLP and LSTM applicability, and apparent independence from optimizer type.The survey also notes that fine-tuning affects lottery quality and that random pruning can perform well after fine-tuning.
  • Similarity and clustering: Similarity and clustering received the least research, although reported results include up to a 3-fold reduction on ResNet applied to ImageNet.The survey suggests developing a more theoretical understanding of functional equivalence among deep-network components.
  • Future work: Combining pruning approaches can produce additional compression, as shown by Lasso regression with factorization and by magnitude pruning with clustering and Huffman coding.The survey calls for comprehensive independent evaluation across broader data and architectures, including issues raised by the Lottery Hypothesis.

Appendix A: Categorisation of Studies on Pruning

Appendix A classifies existing pruning studies into eight categories and presents studies chronologically within at least the magnitude-based category.

  • Magnitude based pruning methods: Magnitude-based pruning studies are listed across publication periods from 1988–91 through later developments including 2018.The listed studies include Kruschke, Hanson and Pratt, Weigend, Han, Guo, Hu, Wen, Aghasi, He, Li, Liu, Luo, Wang, Zhu, Chen, Huang, and Lee.
  • Categorization: The appendix organizes studies by pruning approach and publication year.The supplied categorization excerpt specifically details the magnitude-based pruning grouping.
  • Magnitude based pruning methods: The magnitude-based category spans early methods and later algorithms associated with pruning deep-network structures.The excerpt records a progression from early references to studies published through 2018.

Appendix B: Summary of Data Sets used in Comparing Pruning Methods

Appendix B summarizes the data sets used to compare pruning methods, covering image classification, recognition, translation, detection, segmentation, and scene understanding tasks.

  • Digit and fashion data: MNIST and FashionMNIST contain 28x28 images, each with 60,000 training and 10,000 test examples.MNIST contains handwritten digits, while FashionMNIST contains fashion products in 10 categories.
  • Specialized tasks: The appendix includes specialized data sets for faces, flowers, scenes, translation, object detection, and semantic segmentation.Examples include CASIA-WebFace, LFW, Oxford-Flowers, Places365, WMT’14 En2De, PASCAL VOC, and CamVid.
  • Small-image classification: CIFAR-10 provides 32x32 color images in 10 classes with 50,000 training and 10,000 testing images.CIFAR-100 uses 100 classes with 500 training and 100 test images per class.
  • Large-scale visual recognition: ImageNet contains over 14M images organized into over 21K groups and supports large-scale visual-recognition benchmarking.TinyImageNet is a smaller 200-class subset with 500 training, 50 validation, and 50 testing images per class.
  • Recognition data: SVHN contains 600K 32x32 house-number images, including 73,257 training and 26,032 testing images.UCSD-Birds contains 11,788 images across 200 species, split into 5,994 training and 5,794 testing images.

Appendix C: Summary of Notation

The appendix defines notation for channels, weights, feature maps, pruning masks, loss functions, and norms used throughout the paper.

  • X, W, and Y denote input channels, filter weights, and output channels, respectively.
  • Y_i j denotes the output feature map produced by applying filter W_j,i to input channels X_i.
  • β is a binary mask in which 1 retains a feature map or filter and 0 removes it.
  • L denotes a loss function, while L0, L1, and L2 denote nonzero counts, sums of absolute values, and Euclidean norms.
  • ∥W∥n indicates a selected norm, and ∥W∥F denotes the Frobenius norm applied as Euclidean distance to matrix elements.
Loading 2011.00241v2…