PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

Hongrong Cheng, Miao Zhang, Javen Qinfeng Shi

arXiv:2308.06767v2cs.LGcs.CV

TL;DR

Deep neural networks are increasingly too large and resource-intensive for efficient deployment, while up-to-date comprehensive pruning reviews remain scarce. This survey organizes pruning research into a taxonomy, compares methods across settings, and provides resources and recommendations for future work.

  • Problem

    Growing DNN model size increases computational and storage demands, while up-to-date comprehensive reviews of pruning methods are scarce.

  • Method

    The survey reviews over 300 pruning papers, introduces a taxonomy, compares eight contrast settings, and curates datasets, networks, and evaluations.

  • Results

    The survey provides comparative experiments and analysis covering pruning advances such as large language models and different supervision levels.

  • Takeaways & Limitations

    The survey offers recommendations for selecting pruning methods under different application requirements and identifies promising future research directions.

Abstract

from arXiv · show

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning and explore emerging topics, including pruning for large language models, large multimodal models, post-training pruning, and different supervision levels for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. To facilitate future research, we build a curated collection of datasets, networks, and evaluations on different applications. Finally, we provide valuable recommendations on selecting pruning methods and prospect several promising research directions. We build a repository at https://github.com/hrcheng1066/awesome-pruning.

1 INTRODUCTION

Deep neural networks deliver strong performance across diverse applications but impose substantial computational, memory, energy, and bandwidth demands that hinder deployment on resource-constrained devices. This survey addresses these challenges through a comprehensive pruning review, taxonomy, comparative analysis, resources, recommendations, and future directions.

  • Motivation: DNNs’ parameter counts and computational costs increase processing time and memory demands, challenging deployment on resource-constrained devices.These constraints include limited CPU, GPU, memory, energy, and bandwidth.
  • Motivation: Autonomous driving, field rescue, and bushfire prevention require accurate, fast, and compact models, while DNN complexity can impede edge deployment.The paper also notes growing interest in compression with the popularity of large language models.
  • Motivation: Neural network pruning is one of several compression techniques proposed to create lightweight models and reduce memory usage.Other techniques include low-rank factorization, quantization, knowledge distillation, and neural architecture search.
  • Survey Contributions: The survey reviews pruning methods from small and medium models to large models and introduces a taxonomy covering representative pruning-method classes.It aims to provide a comprehensive overview for diverse readers and recommendations for selecting methods under different requirements.
  • Survey Contributions: The authors compare eight pairs of pruning contrast settings, examine advances such as LLM pruning and supervision levels, and provide experiments with discussions.They also summarize applications, benchmark datasets, networks, evaluations, recommendations, and promising future research directions.

2 TAXONOMY

The taxonomy organizes deep neural network pruning around three critical questions: whether acceleration is universal or specific, when pruning occurs, and what pruning is based on.

  • 2 TAXONOMY: The survey frames pruning taxonomy around three questions: acceleration type, pruning time, and the basis for pruning.The passage identifies these as whether acceleration is universal or specific, whether pruning occurs before, during, or after training, and whether pruning is based on spec…
  • 2 TAXONOMY: Universal acceleration operates independently of specialized hardware or software, whereas specific acceleration relies on specialized hardware or software.
  • 2 TAXONOMY: Pruning can occur before, during, or after training, corresponding to static pruning or dynamic, run-time pruning.

A Comprehensive Comparative Analysis

The survey organizes pruning methods around three primary questions: hardware/software dependence of speedup, pruning timing relative to training, and whether pruning uses criteria or learning. Different combinations of these aspects define the characteristics of pruning algorithms and motivate the survey’s taxonomy.

  • Speedup dependence: Pruning speedup is classified as unstructured, semi-structured, or structured according to its dependence on specific hardware or software.Only structured pruning provides universal acceleration without special hardware or software; unstructured and semi-structured pruning require such support.
  • Pruning timing: Static pruning is categorized as before training, during training, or after training, while dynamic pruning generates subnetworks at run-time for each input.These categories distinguish the arrangement between pruning weights and training weights.
  • Pruning mechanism: Pruning can use criteria such as magnitude, norm, or loss change, or learn sparsity through regularization training or dynamic sparse training.Both criteria-based and learning-based approaches aim to determine which network components to prune.
  • Taxonomy: Different combinations of speedup dependence, pruning timing, and pruning mechanism form diverse pruning methods and the survey’s taxonomy.These three aspects determine the main characteristics of a pruning algorithm.

3 SPECIFIC OR UNIVERSAL SPEEDUP

Pruning methods are categorized by their speedup scope: unstructured and semi-structured pruning provide specific speedup, whereas structured pruning provides universal speedup. Unstructured pruning operates at individual-weight granularity, while structured and semi-structured methods remove regular groups or patterns to improve deployability and flexibility.

  • Specific or Universal Speedup: Unstructured and semi-structured pruning correspond to specific speedup, whereas structured pruning corresponds to universal speedup.
  • Unstructured Pruning: Unstructured pruning is the finest-grained approach, applying binary masks or directly zeroing individual weights.Small or medium models commonly use fixed binary masks, while large models such as LLMs commonly set pruned weights directly to zero.
  • Structured Pruning: Structured pruning removes entire filters, channels, attention heads, or layers to rebuild narrower regular networks that directly accelerate inference and reduce model size.It does not require specialized sparse-convolution hardware or software.
  • Semi-Structured Pruning: Semi-structured pruning introduces regular patterns, such as stripes or 2:4 and 4:8 sparsity, to combine higher accuracy with structural regularity.It is finer-grained than coarse-grained structured pruning and targets flexibility when pruning rates are high.

4 WHEN TO PRUNE

This section categorizes pruning by when it occurs into pruning before, during, and after training, alongside runtime pruning. It explains how these pipelines differ in computational cost, training procedure, and the mechanisms used to discover sparse subnetworks.

  • Pruning Before Training: Pruning Before Training (PBT) prunes randomly initialized weights before training, then trains the resulting sparse network to convergence.PBT avoids the cost of pre-training and can reduce both training and inference time.
  • Pruning During Training: Pruning During Training (PDT) jointly updates weights and pruning masks, producing a subnetwork during training without necessarily requiring retraining.PDT includes dense-to-sparse and sparse-to-sparse training schemes.
  • Pipeline Comparison: Among the three static pruning pipelines, PBT is usually the cheapest and PAT is the most expensive.The section also distinguishes these static pipelines from runtime pruning.
  • Pruning During Training: Dynamic sparse training repeatedly prunes unimportant weights and regrows new ones to search for better sparse architectures.Dynamic sparse training significantly improves gradient flow in sparse networks, which otherwise have poor gradient flow at initialization.
  • Pruning After Training: Pruning After Training (PAT) applies pruning to a pre-trained dense model, followed optionally by retraining or fine-tuning to recover performance.PAT may use one-shot or iterative pruning and is especially applied to large models such as LLMs and diffusion models.

5 PRUNING CRITERIA

Pruning criteria evaluate the importance of weights, filters, or neurons from complementary perspectives, chiefly magnitude, norm, saliency or sensitivity, and loss change. These criteria support unstructured or structured pruning and can be paired with different pruning schedules and Taylor-based approximations.

  • Magnitude: Magnitude-based pruning removes the lowest-magnitude weights based on the assumption that smaller absolute values have less influence on network output.Magnitude criteria can combine weight and activation magnitude, as in Wanda.
  • Magnitude: Magnitude criteria apply to unstructured or structured pruning and can be combined with global or local, one-shot or iterative schedules.Examples include magnitude-based iterative global pruning methods.
  • Norm: Norm-based criteria evaluate importance using the l_p norm, with smaller norms making weights or filters more likely to be pruned.Norm-based importance can also be optimized with norm-based sparsity regularization.
  • Saliency and sensitivity: Saliency and sensitivity criteria measure importance through loss-related effects, derivatives, or channel-scale distributions, with higher sensitivity indicating greater importance.LeCun et al. define saliency as the loss change induced by pruning a weight, while Zhao et al. treat batch-normalization channel saliency through γ distributions.
  • Loss change: Loss-change criteria estimate the effect of removing a weight, filter, or neuron, most commonly with first-order Taylor expansion and also with second-order approximations.Second-order approximations include gradient and Hessian terms while neglecting higher-order terms.

6 LEARN TO PRUNE

The section surveys learning-based pruning methods that jointly learn sparsity, predict pruned-network weights, generate compressed networks with graph neural networks, or search configurations through reinforcement learning.

  • Sparsity regularization: Sparsity-regularization methods jointly learn network weights and masks by minimizing a loss augmented with a regularization term.Scaling factors for weights, channels, or filters are trained jointly, and their magnitudes serve as importance scores.
  • Sparsity regularization: Channel pruning can minimize feature-map reconstruction error, using LASSO regression and a greedy strategy to select unimportant channels.The formulation uses the Frobenius norm, with dimensions representing samples, input channels, retained channels, and output channels.
  • Meta-learning: Meta-learning methods train a meta network such as PruningNet to predict weights for pruned networks from layerwise channel-count encodings.Evolutionary search selects a pruned network under constraints because the encoding-vector search space is large.
  • Graph neural networks: GraphPruning uses a graph aggregator and fully connected layers to generate weights for a pruned network, which is then fully trained.The graph aggregator extracts high-level node features, while each fully connected layer generates weights for a corresponding pruned layer.
  • Reinforcement learning: AutoML pruning methods use reinforcement learning to search compression configurations automatically, with AMC employing a Q-learning-based DDPG agent.AMC defines rewards that incentivize reducing FLOPs or the number of network parameters, based on error being inversely proportional to log(FLOPs) or log(#Param).

7 A COMPREHENSIVE COMPARATIVE ANALYSIS

The survey compares pruning methods across eight contrast settings, including pruning structure, timing, data usage, initialization, layer-wise density, and supervision. Results generally favor unstructured, iterative, data-driven, and fine-tuning-based approaches, while showing that benefits depend on the pruning method and setting.

  • Comparative framework: The analysis evaluates eight pruning contrasts, including different layer-wise densities and supervision levels, while controlling functions within contrast settings.Additional comparisons across methods are provided in Appendix C.
  • Unstructured vs. structured pruning: At the same prune ratio, unstructured pruning generally outperforms semi-structured pruning, which outperforms structured pruning.Unstructured methods remove weights anywhere, whereas structured methods prune entire filters, channels, neurons, or layers to enable network compression and accelerated inference.
  • One-shot vs. iterative pruning: Iterative pruning generally achieves better accuracy than one-shot pruning, although one-shot pruning has negligible pruning cost.One-shot methods score and prune once, while iterative methods repeat score-prune-update cycles until reaching the target ratio.
  • Data-free vs. data-driven pruning: SynFlow and SNIP are similarly effective, while SynFlow significantly outperforms GraSP, indicating that PBT effectiveness is not strictly dependent on data usage.The comparison includes data-free methods Random, Magnitude, and SynFlow, alongside data-driven methods SNIP and GraSP.
  • Initialization and post-pruning training: For PAT pruning, pre-trained weights can facilitate better subnetworks, whereas for PBT pruning with GraSP, pre-training does not necessarily improve Top-1 accuracy.The survey also reports that fine-tuning generally outperforms training from scratch after pruning.
  • Sparsity allocation and supervision: The survey further compares uniform and non-uniform layer sparsity and reports that self-supervised, unsupervised, and supervised pruning can produce different accuracy outcomes.Uniform methods allocate the same prune ratio across layers, while non-uniform methods assign varying sparsity rates; unsupervised pruning usually fails to preserve original-model accuracy.

8 FUSION OF PRUNING AND OTHER COMPRESSION TECHNIQUES

The survey reviews combining pruning with quantization, tensor decomposition, knowledge distillation, and neural architecture search. These techniques offer more compression choices and can complement one another to improve performance and pruning ratios.

  • Pruning & Quantization: Pruning combined with quantization reduces weight or activation precision alongside redundant connections to obtain more compact, accelerated models.Quantization reduces model size and memory footprint with only a minor performance drop; CLIP-Q jointly applies both during fine-tuning.
  • Pruning & Tensor Decomposition: Tensor decomposition exploits low-rank weight structure while preserving convolutional output dimensions, and can be jointly learned with channel or filter sparsity.CC simultaneously learns sparsity and low rankness, while Hinge fuses filter pruning and decomposition under one formulation.
  • Pruning & NAS: NAS automates architecture discovery for pruning by searching network depth, width, filter types, pruning schemes, and layerwise rates.NPAS performs compiler-aware joint pruning and NAS for CNNs, while TAS searches network depth and width.
  • Pruning & Knowledge Distillation: Knowledge distillation can follow pruning or use a pruned teacher, directing distillation toward relevant regions or making the teacher more transferable.Some methods prune unimportant channels before distillation, while others prune the teacher network first and then distill its knowledge.
  • Pruning & Multi-compression Techniques: Some methods fuse pruning with multiple techniques, including pruning, quantization, and KD; pruning, NAS, and KD; or pruning, matrix factorization, and KD.Examples include GS for GAN compression, Joint-DetNAS for image translation, and LadaBERT for BERT compression in natural language understanding.

9 SUGGESTIONS AND FUTURE DIRECTIONS

The section recommends matching pruning methods to hardware, resources, supervision, and deployment requirements, and identifies theory, techniques, applications, and evaluation as four priorities for future development.

  • Recommendations: Structured pruning is more suitable than unstructured pruning for acceleration without specialized sparse hardware or software.Most software frameworks and hardware cannot accelerate sparse-matrix computation.
  • Recommendations: Use iterative PAT when pruning resources are sufficient, but one-shot PBT or post-training pruning when pruning and inference resources are limited, especially for LLMs.Iterative PAT typically minimizes performance impact at the same prune ratio.
  • Recommendations: Choose supervised pruning with abundant labels, semi-supervised or transfer pruning with few labels, and self-supervised, unsupervised, or transfer pruning without labels.
  • Future Directions: Future pruning research should address theoretical prune-ratio bounds, economical architecture learning, broader learning contexts, complex applications, and standardized evaluation.The survey groups future directions into theories, techniques, applications, and evaluation; it also highlights incomparable results caused by differing methods, architectures, tasks, and settings.

10 CONCLUSION · APPENDIX A TERMS AND NOTATIONS

The survey reviews deep neural network pruning across four organizing scopes and provides terminology clarifications for interpreting pruning methods and evaluations. Its appendix defines core pruning, sparsity, execution, retraining, and computational-cost concepts, while noting that some terms vary across prior works.

  • 10 CONCLUSION: The survey organizes pruning research around universal or specific speedup, when to prune, how to prune, and its fusion with other compression techniques.The reviewed speedup settings include unstructured, structured, and semi-structured pruning, while pruning timing covers before, during, and after training and run-time pruning.
  • APPENDIX A TERMS AND NOTATIONS: Prune ratio is the percentage of weights or structures removed from a dense network and complements keep ratio.It may be pre-defined or learning-decided.
  • APPENDIX A TERMS AND NOTATIONS: Compression ratio is defined inconsistently: some works use original-to-preserved weights, whereas another uses preserved-to-original weights.With 10% of weights preserved, the former definition gives 10 and the latter gives 10%.
  • APPENDIX A TERMS AND NOTATIONS: Sparsity ratio denotes the portion of zero weights or structures after pruning and is equivalent to compression ratio under one cited definition.The appendix notes that terms such as compression ratio have different definitions across prior works.
  • APPENDIX A TERMS AND NOTATIONS: One-shot pruning scores once before pruning to a target ratio, whereas iterative pruning repeatedly scores, prunes, and retrains across multiple rounds.Each score-prune-retrain round is one iteration in iterative pruning.
  • APPENDIX A TERMS AND NOTATIONS: Local pruning removes a percentage from each subdivided subset, while global pruning removes structures across the network until a target prune ratio is reached.Dynamic pruning generates input-specific subnetworks, whereas static pruning shares one pruned model across samples with fixed capacities.
  • APPENDIX A TERMS AND NOTATIONS: The lottery ticket hypothesis states that a randomly initialized dense network contains a trainable sparse subnetwork using the original weights, called a winning ticket when it matches trained-network performance.Winning tickets are evaluated after training for T epochs under a non-trivial prune ratio.
  • APPENDIX A TERMS AND NOTATIONS: The appendix distinguishes layer collapse, rewinding, fine-tuning, training from scratch, FLOPs, and MACs as structural, training, and computational concepts in pruning.Layer collapse removes all weights in a layer; weight rewinding restores earlier values W_t where t << T; FLOPs and MACs proxy theoretical computational consumption.

APPENDIX B EXPERIMENTAL SETTINGS · APPENDIX C MORE COMPARISON RESULTS · C.1 More Results for Unstructured vs. Structured Pruning

The appendices specify evaluation settings for image-classification and language-model pruning, provide implementation details and additional comparison tables, and show that unstructured pruning generally outperforms structured pruning at equal prune ratios.

  • APPENDIX B EXPERIMENTAL SETTINGS: Image-classification pruning is evaluated on CIFAR-10/100 or ImageNet using Top-1 accuracy with VGG-16, ResNet-32/152, or DeiT-Tiny.CIFAR-10/100 contain 50K training and 10K test images; ImageNet has over 1.28 million training and 50K validation images across 1,000 classes.
  • APPENDIX B EXPERIMENTAL SETTINGS: LLaMA-7B pruning is assessed on WikiText2 and PTB using zero-shot language-generation perplexity, plus classification and multiple-choice reasoning tasks on four common-sense datasets.The experiments use one NVIDIA A100 GPU with 40 GB of memory.
  • APPENDIX B EXPERIMENTAL SETTINGS: The appendix identifies publicly available implementations for LLM-Pruner, SparseGPT, and Wanda through their respective GitHub repositories.These repositories are listed as implementation resources for the evaluated LLM pruning methods.
  • APPENDIX B EXPERIMENTAL SETTINGS: SparseGPT uses blocksize 128 for iterative pruning in Section 7.2, while later experiments specify pretrained, retrained, and randomly initialized model variants.DeiT-Tiny is pretrained on ImageNet and retrained on CIFAR-10; ResNet-152 variants are randomly initialized or pretrained on CIFAR-100.
  • APPENDIX C MORE COMPARISON RESULTS: Additional appendix tables compare pruning methods across unstructured-versus-structured settings and CNN or Transformer models of small, medium, and large sizes.Tables 13–176 expand the comparison material discussed in Section 7 of the main text.
  • C.1 More Results for Unstructured vs. Structured Pruning: Using the same LLM-surgeon scoring method, unstructured pruning consistently outperforms structured pruning across models and five prune ratios.These Table 11 results are sourced from [164].
  • C.1 More Results for Unstructured vs. Structured Pruning: Unstructured pruning typically reaches prune ratios above 70% or 80%, whereas structured pruning generally remains below 50%.The comparison spans Tables 13–17 and reports typical rather than universal ratios.

C.2 More Results for One-shot vs. Iterative Pruning … D.6 Natural Language Processing

The survey compares pruning across multiple contrast settings and applications, finding that performance depends on pruning strategy, data and weight initialization, while extending coverage from computer vision to NLP and language models.

  • C.2 More Results for One-shot vs. Iterative Pruning: At 90% pruning on ResNet-50/ImageNet, iterative-SNIP reduces Top-1 accuracy loss to 11.90 versus SNIP’s 14.10; at 95%, losses are 30.90 and 31.30.On LLaMA-7B, iterative LoRAPruner also outperforms one-shot LLM-Pruner under similar equivalent prune ratios.
  • C.3 More Results for Data-free vs. Data-driven Pruning: 15.41 Top-1 accuracy loss at a 95.60% prune ratio lets data-free NTK-SAP outperform data-driven SNIP and Grasp, whose losses are 35.51 and 16.47.The passage states that data usage is not strictly required for effective PBT pruning, whereas it appears crucial for PAT methods.
  • C.4 More Results for Pruning on Initialized vs. Pre-trained Weights: Pre-trained weights are generally crucial for non-PBT methods to find effective subnetworks, although pruning pre-trained weights does not guarantee improved Top-1 accuracy for PBT methods.Most surveyed methods, especially those targeting Transformer-based models, use pre-trained rather than randomly initialized weights.
  • C.5 More Results for Global vs. Local Pruning: At 50% parameter pruning for LLaMA-7B, global Bonsai achieves WikiText2 perplexity 10.92 and BoolQ and WinoGrande results of 67.22 and 61.64, outperforming local LLM-pruner and LoRAPruner by large margins.The survey reports that global pruning is more common than local pruning across Tables 13–17.
  • C.6 More Results for Training from Scratch vs. Fine-tuning: Most surveyed methods fine-tune pruned networks rather than training from scratch, with only a few, such as DMCP, choosing scratch training.CP compares fine-tuning and scratch training under the same pruning ratio, but the supplied passage truncates the quantitative result.
  • APPENDIX D PRUNING FOR SPECIFIC APPLICATIONS; D.1 Image Classification; D.2 Object Detection: Pruning research spans image classification, object detection, image style translation, adversarial robustness, and other computer-vision tasks, with datasets, networks, and evaluation metrics summarized in Table 18.Image classification commonly uses ImageNet, whereas object detection is more challenging because of complex architectures and larger input sizes; only a few works study it.
  • D.3 Image Style Translation: GAN pruning differs from image-classification pruning because GANs use distinct generator–discriminator structures and high-dimensional outputs, while style-transfer results commonly use FCN-scores and FID.FCN-scores include pixel accuracy, class accuracy, and class IoU; the passage states that larger FID indicates better transfer results.
  • D.4 Adversarial Robustness; D.5 Other CV Tasks; D.6 Natural Language Processing: Only a few recent works jointly study pruning and robustness, alongside broader coverage of other CV tasks and NLP models including pruned LSTMs, Transformer attention, and pre-trained BERT.In NLP, GLUE uses task-specific metrics, BLEU evaluates machine translation, and lower perplexity indicates a better pruned model; Transformer-based LLMs such as LLaMA and OPT are dominant.

D.7 Vision-and-Language Tasks · D.8 Audio and Speech Processing

The survey highlights pruning evidence and evaluation practices across vision-and-language and audio-and-speech tasks. In VL, relaxed winning tickets retain 99% of full accuracy at 50%-70% parameter pruning, while speech work studies gradual pruning and lottery-ticket extensions, evaluated with WER and CER.

  • D.7 Vision-and-Language Tasks: Vision-and-language pruning covers Visual Question Answering and image-text retrieval, with UNITER among the widely employed models.The survey identifies VL as a common multimodal research area and notes that Table 18 lists models, datasets, and metrics for these tasks.
  • D.7 Vision-and-Language Tasks: 50%-70% parameter pruning in UNITER yields relaxed winning tickets matching 99% of full accuracy.Gan et al. pioneered lottery-ticket investigations for VL tasks and reported this result.
  • D.8 Audio and Speech Processing: Speech recognition is a major audio-and-speech task, and lightweight recognition is indispensable on mobile devices.The survey discusses pruning Deep Speech 2 to validate a gradual pruning scheme.
  • D.8 Audio and Speech Processing: Lottery-ticket research in speech recognition extends to Deep Speech 2, while PARP prunes pre-trained wav2vec 2.0 and XLSR-53.Ding et al. investigate winning tickets, and Nrang et al. use Deep Speech 2 to validate gradual pruning.
  • D.8 Audio and Speech Processing: WER is the standard speech-recognition accuracy metric, defined as WER = (S + I + D)/(S + I + C), with lower values indicating better accuracy.S, I, D, and C denote substitutions, insertions, deletions, and correct words, respectively.
  • D.8 Audio and Speech Processing: WMT’14/16 denotes the translation-task resource referenced by the survey.The passage supplies the corresponding website: https://www.statmt.org/wmt14/translation-task.html.
  • D.8 Audio and Speech Processing: Detailed information about GLUE tasks is available in Table 1 of reference.This is provided as a cross-reference for the task information.
  • D.8 Audio and Speech Processing: CER uses the exact WER definition but counts characters instead of words.WER counts words, whereas CER counts characters.
Loading 2308.06767v2…