PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Pruning for Deep Convolutional Neural Networks: A survey

Yang He, Lingao Xiao

arXiv:2303.00566v2cs.CV

TL;DR

CNNs achieve strong performance but their millions of parameters and high computing demands challenge deployment on resource-limited hardware. This paper surveys structured pruning methods and organizes extensive evidence to clarify pruning techniques and CNN design.

  • Problem

    CNN architectures contain millions of parameters and require substantial computing power, making deployment on resource-limited hardware challenging.

  • Method

    The paper provides a comprehensive survey of more than 200 structured pruning papers and summarizes over 800 experiment results from more than 100 papers across three datasets.

  • Results

    The survey organizes pruning methods and reports into a comparative resource covering 21 tables across CIFAR-10, CIFAR-100, and ImageNet-1K.

  • Takeaways & Limitations

    The survey offers a broad reference for understanding structured pruning methods and CNN properties relevant to efficient model design.

  • Takeaways & Limitations

    Second-order Taylor expansion methods require computationally prohibitive second-degree derivatives, while first-order methods provide less information.

Abstract

from arXiv · show

The remarkable performance of deep Convolutional neural networks (CNNs) is generally attributed to their deeper and wider architectures, which can come with significant computational costs. Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. In contrast to weight pruning, which results in unstructured models, structured pruning provides the benefit of realistic acceleration by producing models that are friendly to hardware implementation. The special requirements of structured pruning have led to the discovery of numerous new challenges and the development of innovative solutions. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning. While discussing structured pruning algorithms, we briefly introduce the unstructured pruning counterpart to emphasize their differences. Furthermore, we provide insights into potential research opportunities in the field of structured pruning. A curated list of neural network pruning papers can be found at https://github.com/he-y/Awesome-Pruning . A dedicated website offering a more interactive comparison of structured pruning methods can be found at: https://huggingface.co/spaces/he-yang/Structured-Pruning-Survey .

1 INTRODUCTION

Deep CNNs achieve strong performance but their scale creates deployment costs, motivating compression and especially structured pruning. The survey organizes prior work and expands coverage of structured pruning methods and research directions.

  • Motivation: Deep CNNs contain millions of parameters and require substantial computing power, challenging deployment on resource-limited hardware.Model compression targets parameter count, computational cost, and memory consumption.
  • Compression: Pruning removes network components to create sparse models, aiming to reduce parameters without significantly harming performance.The survey places pruning among compression techniques including quantization, decomposition, and knowledge distillation.
  • Structured versus Unstructured Pruning: Structured pruning removes entire filters and can accelerate models on standard hardware, whereas unstructured pruning removes individual weights and often needs specialized support.The distinction is tied to unstructured versus structured sparsity and hardware-efficient libraries such as BLAS.
  • CNN Structure: CNNs impose locality and weight-sharing inductive biases, motivating renewed analysis of their structure alongside Transformer-based models.The survey frames structured pruning as a way to understand CNNs and inform efficient design.
  • Survey Scope: The survey covers more than 200 structured pruning papers and compares them with related compression surveys.Existing surveys often emphasize unstructured pruning or cover only a small number of structured-pruning studies.
  • Organization: Its taxonomy groups structured pruning methods into categories, while a subsequent section discusses potential future directions.Methods spanning multiple categories are assigned to the category considered most appropriate for the survey.

2 METHODS

The survey presents structured pruning as filter-level model compression and organizes methods by how they rank filters, regularize networks, execute dynamically, search architectures, and target applications or research topics.

  • Preliminaries: Structured pruning removes filters or channels to reduce convolutional-network complexity while preserving a hardware-friendly structure.The preliminaries describe convolutional weights as collections of three-dimensional filters and identify filter-set cardinality as a sparsity target.
  • 2.1 Weight-Dependent: Weight-dependent criteria rank filters from their weights, with filter norm and filter correlation forming two main subcategories.These criteria do not require input data and generally have lower computational cost than activation-based methods.
  • 2.1.1 Filter Norm: Filter-norm methods use ℓp values to identify filters for pruning; PFEC uses ℓ1-norm, while SFP reports slightly better empirical performance from ℓ2-norm.The survey also describes FPGM, which identifies redundant filters through relationships and geometric-median proximity rather than assuming smaller norms are always less important.
  • 2.1.2 Filter Correlation: Correlation-based methods assess relationships among filters, including cross-layer importance, layer redundancy, normalization, and cost-aware budget control.COP uses Pearson correlation and cost-aware regularization, while SRR repeatedly targets the most redundant layer using graph properties.
  • Regularization and Optimization: The taxonomy also covers regularization on batch-normalization parameters, extra parameters, or filters; Taylor and variational-Bayesian approaches; and other optimization methods.The listed categories include first- and second-order Taylor expansion, variational Bayesian methods, SGD, ADMM, Bayesian optimization, and straight-through methods.
  • Dynamic Execution and Search: Dynamic pruning is divided into methods operating during training and methods operating during inference.The survey separately lists neural architecture search, reinforcement-learning, gradient-based, evolutionary-based, lottery-ticket, joint-compression, and special-granularity approaches.

2.2 Activation-Based

Activation-based pruning uses feature maps rather than weights to decide which channels or filters to remove. The survey groups these methods by whether they use current-layer, adjacent-layer, or all-layer information.

  • Overview: Activation-based methods use activation maps for pruning decisions, and removing activation channels is equivalent to removing filters.Pruning a layer also affects the next layer through its feature maps.
  • Current Layer: Current-layer methods score channels using reconstruction error, activation-map rank, channel independence, or post-activation statistics.Examples include CP, HRank, CHIP, APoZ, and DropNet.
  • Current Layer: CP formulates layer-wise pruning as sparse activation-map reconstruction and solves channel selection through LASSO regression with fixed weights.The procedure alternates between selecting channels to prune and minimizing reconstruction error.
  • Current Layer: HRank uses the average rank of activation maps from a small input set, while CHIP measures channel independence through nuclear-norm changes.HRank obtains rank with singular-value decomposition, and CHIP treats greater channel independence as greater importance.
  • Adjacent Layers: Adjacent-layer methods exploit dependencies between neighboring layers, such as approximating next-layer activations with subsets of current-layer activations.ThiNet uses a greedy subset-selection process, while AOFP introduces damage isolation and a multi-path scoring framework.
  • Dynamic Inference: Some dynamic methods use previous-layer activation information during inference, including global-average-pooling scores in RNP and FBS.These methods are discussed with dynamic pruning during inference because their pruning decisions depend on runtime inputs.
  • All Layers: All-layer methods evaluate broader effects of filter removal by propagating importance or discrimination information through the network.NISP propagates scores backward from a Final Response Layer, while DCP adds discrimination-aware losses to intermediate layers.

2.3 Regularization

Regularization induces structured sparsity by making channel, filter, or other structure-selection parameters identify what to prune. The survey covers BN-based penalties, extra trainable gates, and filter-level regularization strategies.

  • BN parameters and extra gates: Regularizers can be applied to BN parameters, whose values indicate pruning decisions for channels or filters.Extra learnable gates provide an alternative when batch normalization is unavailable.
  • BN parameters: Network Slimming jointly trains weights and BN scaling parameters with channel-level ℓ1 regularization, then prunes channels with near-zero γ.A subgradient descent method optimizes the non-smooth penalty.
  • BN parameters: Other BN-based methods modify sparsity behavior by polarizing γ, using ISTA with γ-W rescaling, or considering BN shifts and ReLU effects.PR separates important and unimportant scaling factors, RSNLI prunes channels with γ equal to zero, and SCP treats large negative β with large γ as unimportant.
  • Extra parameters: Extra-parameter methods place trainable scaling factors or gates after structures and remove structures whose learned gates fall below a threshold or become zero.SSS applies this approach to neurons, groups, or residual blocks, while GDP-Guo drives gates toward zero or values close to one.
  • Extra parameters: Discrete Model Compression samples subnetworks with stochastic binary gates, allowing channels with nonzero parameters to be sampled and updated with a Straight-Through Estimator.The stochastic gates produce different subnetworks during training despite their non-differentiability.
  • Extra parameters: SCOP controls real and knockoff features with complementary scaling factors and defines filter importance as I = θ − ˜θ, pruning filters with small scores.The method uses knockoff counterparts to reduce the influence of potential factors such as input data on importance ranking.

2.4 Optimization Tools

Optimization tools estimate structural importance, impose sparsity, or approximate difficult second-order information to guide structured pruning. The survey contrasts first- and second-order Taylor methods with Bayesian approaches and related optimization strategies.

  • Taylor Expansion: Taylor Expansion estimates the loss change caused by pruning filters or channels by evaluating the loss around zeroed weights.First-order terms use gradients, while second-order terms additionally use the Hessian.
  • Taylor Expansion: First-order Taylor expansion is faster and requires no additional memory, whereas second-order expansion provides more information but is computationally prohibitive.The first-order form can be obtained from backpropagation; the second-order form requires second derivatives.
  • Taylor Expansion: Mol-19 computes filter importance from weights and gradients, avoiding activation storage while accepting a slightly higher accuracy drop than second-order expansion.Its structural set S can represent a convolutional filter, with importance derived from individual weights and their gradients.
  • Second-order Taylor: Second-order structured pruning approximates the Hessian because storing and computing it is challenging for deep CNNs with millions of parameters.CCP uses first-order derivatives to avoid additional storage, while ED uses K-FAC and a Kronecker-Factored eigenspace to address filter correlations.
  • Bayesian methods: Bayesian pruning models channel importance probabilistically, using variational distributions, sparsity-inducing priors, or inter-layer dependencies to identify redundant channels.Variational Bayesian methods approximate the posterior because evidence integrals and exact posterior optimization can be intractable.

2.5 Dynamic Pruning

Dynamic pruning changes masks during training or inference rather than relying only on permanently removed structures. Training-time methods preserve recovery opportunities, while inference-time methods adapt computation to each input.

  • Training-time dynamic pruning: Training-time dynamic pruning maintains a changing mask, whereas hard pruning permanently removes weights using a fixed mask.Soft pruning can recover improper decisions later during training.
  • Training-time dynamic pruning: Soft Filter Pruning generates masks from filter ℓ2-norms each epoch and sets selected filters to zero without permanently removing them.Previously soft-pruned filters can be updated in later epochs.
  • Training-time dynamic pruning: GDP-Lin uses first-order Taylor expansion for global filter importance and updates its binary mask every e iterations to accelerate convergence.The update interval decreases because frequently changing masks may not guide pruning effectively.
  • Recovery and regrowth: Some dynamic methods combine sparse and dense models or prune-and-regrow channels to use feedback and explore nonredundant structures.DPF updates a dense model with gradients from the sparse model, while CHEX regrows channels through orthogonal projection.
  • Inference-time dynamic pruning: Inference-time methods estimate input-specific importance with reinforcement learning, auxiliary predictors, spatial-channel gates, or self-supervised masks.RNP models pruning as a Markov decision process, while DDG separately gates spatial and channel features.

2.6 NAS-Based Pruning

NAS-based pruning automates the selection of sparse structures and pruning ratios that are otherwise manually specified. The survey groups approaches into reinforcement-learning, gradient-based, and evolutionary searches.

  • Search strategies: NAS-based pruning searches sparse subnetworks or pruning ratios through reinforcement learning, gradient-based optimization, or evolutionary algorithms.These categories operate over actions such as pruning ratios, differentiable architecture parameters, or candidate structures.
  • Reinforcement-learning methods: AMC uses deep deterministic policy gradients to select continuous layer-wise pruning ratios from layer-dependent states such as FLOPs.Its reward is modified to support accuracy-guaranteed pruning.
  • Reinforcement-learning methods: AGMC replaces AMC’s manually selected fixed states with graph encoder-decoder representations learned from computational graphs.The computational graph supplies richer environmental information to the reinforcement-learning agent.
  • Gradient-based methods: Gradient-based methods make pruning or sparsity allocation differentiable through Markov transitions, probabilistic pruning, hypernetworks, or reparameterization.DMCP parameterizes transitions and budget loss, while DSA optimizes continuous sparsity under a budget constraint.
  • Evolutionary methods: Evolutionary methods search channel configurations using generated or restricted candidates, with CCEP decomposing the network by layer to reduce the search space.CCEP evaluates candidates using accuracy and FLOPs and retains top-k individuals iteratively.

2.7 Extensions

The survey extends structured pruning through lottery-ticket methods, joint compression, and finer pruning granularities. These extensions address training schedules, coupled compression objectives, and alternatives to filter-level sparsity.

  • Lottery Ticket Hypothesis: Learning-rate rewinding consistently outperforms fine-tuning and outperforms or matches weight rewinding across the reported scenarios.The method rewinds the learning-rate schedule without rewinding weight values.
  • Lottery Ticket Hypothesis: RVNP reports that fine-tuning performs no better than pruning from scratch and that structured lottery tickets fail on large-scale datasets.This re-evaluates the value of conventional pruning and lottery-ticket extensions.
  • Early pruning: Early-pruning methods identify promising subnetworks or pruning times using mask similarity, meta-gradients, gradient-flow preservation, or architecture stability.EarlyCroP targets the lazy kernel regime, while PaT uses an early pruning indicator based on subnetwork structure similarity.
  • Joint compression: Joint compression methods combine pruning with NAS, quantization, or decomposition because sequentially applying techniques with different objectives can be sub-optimal.APQ jointly searches structures and mixed-precision quantization under latency or energy constraints.
  • Special granularities: Structured sparsity can target groups, stripes, kernels, patterns, blocks, or entire layers rather than only filters and channels.SDN evaluates layers with classifier probes before removing unimportant layers, while other granularities organize weights across output, input, or kernel dimensions.

3 FUTURE DIRECTIONS

Future work extends structured pruning beyond current mechanisms and CNN settings toward theory, adaptive training, broader tasks and architectures, and practical deployment objectives.

  • Pruning theory: Pruning theory can draw on synaptic flow, signal propagation, graph theory, model interpretations, loss landscapes, stability, entropy, and Lottery Ticket Hypothesis analyses.
  • Pruning mechanism: Structured pruning may adopt alternatives to the training-pruning-retraining pipeline, including Lottery Ticket extensions, single-shot pruning, and joint dense-sparse training.
  • Pruning rate and domain: Future strategies include learning layer-wise pruning ratios and using frequency-domain representations to guide pruning.
  • Pruning domain: Pruning has been applied to super-resolution, person re-identification, medical imaging diagnosis, face attribute classification, and ensemble learning.
  • Applications: Structured pruning can support federated learning by reducing communication costs and continual learning by reusing pruned filters without deteriorating previous-task performance.
  • Broader directions: Emerging directions include dataset compression, pruning other neural networks, CNN-based Transformers, Transformer-based architectures, hardware-aware pruning, energy-aware pruning, and robustness-aware design.

Supplementary for Structured Pruning for Deep

The supplementary section is identified only as a survey of Convolutional Neural Networks.

  • The supplementary material is titled “Convolutional Neural Networks: A survey.”

1 RELATED WORKS

Related work distinguishes pruning from other compression approaches and contrasts unstructured pruning with structured-pruning-focused surveys.

  • Unstructured Pruning: Unstructured pruning removes weights using saliency, iterative pruning with regularization and retraining, and connection splicing to restore wrongly pruned weights.
  • Quantization: Quantization approximates 32-bit floating-point weights with lower-precision representations such as 8-bit integers to reduce computation and storage costs.
  • Decomposition: Decomposition factorizes matrices or tensors into products of low-rank tensors, using methods including SVD, QR, Tucker, CP, and tensor networks.
  • Knowledge Distillation: Knowledge distillation transfers knowledge from larger to smaller networks through offline, online, or self-distillation, and can combine with pruning for few-shot compression.
  • Neural Architecture Search: Neural architecture search seeks network designs with reduced human intervention through choices of search space, search strategy, and related search components.

2 EXPERIMENTS

The experiments consolidate results from structured-pruning studies into indexed comparison tables, using multiple accuracy, FLOPs, and parameter metrics while accounting for differing baselines.

  • Experiment overview: 800+ experiment results from 100+ papers are organized into 21 tables across CIFAR-10, CIFAR-100, and ImageNet-1K.
  • Metrics: The comparison reports nine metrics covering baseline and post-pruning accuracy, FLOPs, and parameters, plus their respective drops.
  • Comparison settings: Tables account for differing baseline and fine-tuning settings across methods and are sorted by the FLOPs of pruned models.
  • Tables: Table 1 summarizes experiments across 21 tables, while Table 2 indexes those tables.

3 OUR WEBSITE

The website provides an interactive platform for comparing structured pruning methods and querying results by dataset, model, method category, and target constraints. Matching methods can be inspected with associated bibliographic and implementation information.

  • The website offers an interactive and dynamic platform for visualizing and comparing structured pruning experiment results.It is presented as a supplement to the survey’s comparative analysis.
  • Users can query methods by selecting a dataset, model, pruning category, and targets for accuracy, pruned FLOPs, and model size.The example uses ImageNet-1K, ResNet-50, regularization-based pruning, accuracy above 75%, pruned FLOPs above 40%, and model size below 30M.
  • The example query narrows the results to five data points from three methods: GBN, SCOP, and OTO.
  • Selecting a method name generates its paper title, link, code, venues, and BibTeX information.SCOP is given as an example of a selectable method.
Loading 2303.00566v2…