Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

1 INTRODUCTION

• Overparameterized models ease training but impose substantial memory and computation costs, motivating sparsification for efficient inference, regularization, and training.The motivation is especially strong for mobile, battery-driven, cost-conscious, and increasingly expensive large-model workloads.Source: §1 · p. 2
• Sparsification can reduce model size by 10-100x with corresponding theoretical gains in computational, storage, and energy efficiency, without significant accuracy loss.These benefits depend on realizing speedups through efficient hardware implementations.Source: §1 · pp. 2–3

1.1.1 Document structure.

• The survey organizes sparsification by which network elements are removed or added, when changes occur, and how sparse training and hardware-efficient execution are achieved.It covers removal, growth, ephemeral per-example sparsity, convolutional and transformer architectures, software, specialized hardware, observations, and research challenges.Source: §1 · p. 3, §1.1.1 · p. 5

1.1 Overview of Model Compression Techniques

• The paper focuses on sparsification among six compression strategies, alongside down-sizing, operator factorization, value quantization, value compression, and parameter sharing.Sparsification preserves high-dimensional feature spaces while reducing representational complexity by zeroing subsets of parameters.Source: §1.1 · pp. 3–4

1.2 Background and Notation

• Deep learning training alternates forward evaluation, loss computation, backward gradient propagation, and weight updates, with generalization assessed on unseen examples.Networks are parameterized graphs of layers implementing nonlinear functions, and architecture design is traditionally separated from weight training.Source: §1.2 · p. 5

1.2.1 The Deterministic Formulation for Training. Í𝑁

• The deterministic formulation represents training as minimizing empirical loss over d parameters using stochastic gradients and learning-rule updates.The Jacobian captures first-order sensitivities, while the Hessian captures local second-order loss curvature.Source: §1.2.1 · pp. 6–7

1.2.2 The Probabilistic Formulation.

• The probabilistic formulation models inputs and labels as distributions, connects loss minimization with negative log-likelihood, and uses Fisher information to approximate Hessian curvature when model predictions are accurate.The empirical Fisher replaces the model distribution with the empirical training distribution; Bayesian pruning then approximates an otherwise intractable posterior using simplifying assumptions.Source: §1.2.2 · pp. 7–8, §1.2.3 · p. 8

1.2.3 The Bayesian Formulation.

• Variational inference approximates the posterior p(w|S) with q_ϕ(w) and optimizes ϕ by maximizing a variational lower bound whose terms represent expected log-likelihood and prior regularization.For complex models, reparametrization tricks enable unbiased, differentiable, minibatch-based Monte Carlo estimators of the expected log-likelihood.Source: §1.2.3 · pp. 8–9
• Gaussian dropout has a Bayesian interpretation: multiplicative Gaussian noise corresponds to sampling each weight from a parameterized normal distribution, with the weight treated as a random variable.Gaussian Dropout training is equivalent to stochastic optimization of expected log-likelihood under a log-uniform weight prior when one weight sample is drawn per minibatch.Source: §1.2.3 · p. 9
• Variational Dropout makes dropout rates trainable variational parameters at layer, neuron, or weight granularity, enabling effective DNN sparsification as shown by Molchanov et al. [2017].Sparse Variational Dropout optimizes the posterior parameters through stochastic variational inference and uses additive-noise reparameterization to reduce high-variance gradients for large dropout rates.Source: §1.2.3 · pp. 9–10

1.2.4 Convolutional Layers as Designed Sparsity.

• Convolutional operators impose designed sparsity by connecting neurons only within local regions determined by kernel size, strides, padding, and dilation, rather than connecting every input-output pair.The resulting locally connected operators can specialize filters across spatial regions, and convolution itself can be viewed as a sparse fully connected layer.Source: §1.2.4 · p. 10, Figure 3 · p. 10, §1.2.3 · p. 10
• Weight sharing sparsifies convolutional operators further while providing translational equivariance by reusing local filters across output locations and channels.A typical convolution combines C_in input channels with C_out output features using a unique set of C_out filters for each input channel.Source: §1.2.4 · p. 10

2.1 Generalization

• Moderate sparsity can improve generalization or preserve accuracy, whereas extreme sparsity degrades quality; computational performance usually rises with sparsity until storage and control overheads dominate.These trends form an Occam’s hill for accuracy and a generally increasing performance curve with diminishing returns at extreme sparsities.Source: §2.1 · p. 11

2.2 Performance and model storage

• Sparsification reduces operations and memory, but storage choices depend on sparsity, structure, access patterns, and hardware; bitmap, delta, offset, and dimension-aware formats are efficient in different regimes.In the example with n_c = n_r = 10^4 and n = 10^8, bitmap indexing is best at 10–70% sparsity, while delta encoding is best above 80% when d̂ < 1000; no sparse scheme helps below 10% sparsity.Source: §2.2 · pp. 11–12

2.3 What can be sparsified?

• Model sparsity removes weights, neurons, filters, or attention heads, with structured removal producing smaller dense models and unstructured weight removal offering fine-grained pruning but requiring index overheads.Ephemeral sparsity instead changes activations, gradients, or computation paths per example, including ReLU, dropout, top-k selection, delayed updates, and conditional computation.Source: §2.3 · pp. 12–13

2.4.1 Sparsify after training.

• Post-training sparsification provides a strong quality baseline by pruning a converged dense model and fine-tuning it, while pruning during training can be cheaper and reduce overfitting but is more brittle and initially requires dense-model memory.Some schedules gradually prune to correct premature-pruning errors, whereas fully sparse training enables very high-dimensional models that cannot fit in dense form.Source: §2.4.1 · p. 14, §2.4.2 · p. 14, §2.4.3 · p. 15

2.4.2 Sparsify during training.

• During-training schedules improve results through adaptive pruning, alternating dense and sparse phases, or early structure adaptation, because important connections are often identified early and later rarely re-added.Iterative hard thresholding alternates top-k pruning and sparse fine-tuning with dense training to regularize the network while recovering better representations; Prechelt [1997] reports improved generalization for well-tuned static-dynamic schedules.Source: §2.4.2 · p. 15

2.4.3 Sparse training.

• Sparse training may be dynamic, with pruning and regrowth during optimization, or fixed before training; dynamic schemes can vary the parameter budget, while fixed structured sparsity can train smaller models from scratch but unstructured ImageNet pruning still benefits from fine-tuning.NeST uses seed, growth, and pruning stages, SET combines magnitude pruning with random regrowth, and Liu et al. [2019b] report state-of-the-art neuron pruning but failure of from-scratch weight pruning on ImageNet.Source: §2.4.3 · p. 16
• At extreme sparsity, preserving gradient flow is essential: SNIP can create bottlenecks or layer collapse, whereas GraSP improves upon SNIP in very sparse regimes and iterative alternatives reduce complexity while retaining competitive quality.de Jorge et al. [2020] report a 5x performance improvement over GraSP with similar quality, although the method remains less accurate than pruning fully trained ResNets.Source: §2.4.3 · pp. 16–17

2.4.5 Sparsify for transfer learning and fine tuning

• Transfer-learning sparsification can produce task-specific sparse models: iterative magnitude pruning yields 40–90% sparsity in fine-tuned BERT weights, while moderately sparse image models transfer with negligible loss or a small gain.Fine-tuning uses gradient-based or magnitude-based pruning to capture changes from pretrained weights to task-specific weights; Chen et al. [2020] find pretrained masked-language-modeling structure transfers more broadly.Source: §2.4.5 · p. 18, §2.4.4 · p. 18

2.4.6 General Sparse Deep Learning Schedules.

• Iterative schedules initialize structure and weights, train, prune or regrow elements, and optionally retrain, with loops enabling repeated sparsification and weight resets.Retraining is often skipped but generally improves model accuracy; iterative schedules commonly apply multiple fine-tuning periods.Source: §2.4.6 · pp. 18–19
• Fine-tuning can level differences between pruning methods: WoodFisher’s immediate advantage largely disappears between pruning steps, yet it ultimately exceeds magnitude pruning by 0.4% Top-1 validation accuracy.The gradual schedule increases sparsity from 5% to 95% with fixed fine-tuning periods and a longer final period.Source: §2.4.6 · p. 19, Figure 9 · p. 20
• Structural sparsity updates often benefit from delays that reduce noise and amortize data-structure rearrangement, whereas higher update frequency improves training with ephemeral weight and activation sparsity.Update frequency is a crucial hyperparameter related to minibatch size and gradient noise, while other sparsification hyperparameters may follow schedules.Source: §2.4.6 · pp. 19–20

2.5 Ensembles

• With a fixed parameter budget, training 2–3 smaller ensemble models and combining their outputs can improve image-recognition performance over one model [Collins and Kohli 2014].Sparsification enables this tradeoff by allowing multiple models within the same limited parameter and compute budget.Source: §2.5 · p. 20

3 SELECTING CANDIDATES FOR REMOVAL

• Importance-guided selection generally achieves the best highly sparse compressed models, whereas exhaustive leave-some-out evaluation is precise but requires training n^k candidate networks and random removal can also work in some settings.Comparative studies identify no clear universal winner, so method efficacy depends on architecture, hyperparameters, learning-rate schedule, and task.Source: §3 · pp. 20–21

3.1 Structured vs. unstructured element removal

• Structured sparsity lowers index-storage and processing costs through patterns such as strides or blocks, but restricting removal patterns can produce worse models by reducing sparsification flexibility.Unstructured sparsity requires storing nonzero offsets and handling sparse structure explicitly, whereas structured removal supports simpler dense operations.Source: §3.1 · p. 21
• Structured removal can eliminate whole neurons, convolutional filters, transformer heads, channels, kernels, or strided connections while preserving compact representations and efficient matrix operations.Strided sparsity stores an offset, stride, and nonzero elements; convolutional sparsity can also group entries in im2col matrices or restrict filter connectivity.Source: §3.1 · pp. 21–22

3.2 Data-free selection based on magnitude

• Magnitude pruning removes weights with the smallest absolute values and remains a simple, effective method for individual weights or structured groups, often combined with retraining or gradually increasing sparsity.The same criterion can aggregate |W_i| over blocks, rows, columns, or other groups, and has a theoretical justification under some assumptions.Source: §3.2 · p. 22
• Magnitude pruning removes weights near zero, and thresholds can be fixed globally or per layer, learned during SGD, or selected by reinforcement learning for resource or accuracy targets.Kusupati et al. [2020] use learnable layer thresholds through thresholded weights, while He et al. [2019a] use a DDPG agent.Source: §3.2 · pp. 22–23
• During sparse training, magnitude pruning can maintain approximately constant connection density, and sign-flip pruning fixes a weight to zero when SGD would reverse its sign.Magnitude pruning was popularized for modern neural-network compression by Han et al. [2016b] and extended to filters, recurrent networks, sparse training, and lottery-ticket methods.Source: §3.2 · p. 23

3.2.1 Other Data-free methods.

• Data-free pruning methods can operate before or after data-driven pruning, but more precise data-dependent schemes can perform better at high sparsity and may reduce the expensive retraining needed to recover accuracy.Magnitude pruning can provide state-of-the-art results, while coreset pruning offers a controllable sparsity–approximation-error tradeoff and showed improved accuracy at 90% sparsity on very small example networks.Source: §3.2.1 · p. 23

3.3 Data-driven selection based on input or output sensitivity

• Sensitivity-based pruning removes elements whose outputs change little across training examples, treating low variation as evidence of low relevance to the network function.These methods directly use examples to select elements while maintaining or improving prediction accuracy during sparsification.Source: §3.3 · p. 23
• Output-similarity methods remove redundant neurons or approximate one neuron with others, then adjust biases or outgoing weights to minimize dataset-wide output changes.Linear-system and least-squares formulations can update consuming weights without hyperparameter tuning, and related methods remove features or individual weights.Source: §3.3 · pp. 23–24
• The same sensitivity principle extends to convolutional filters, using minibatch output impact, final-response error, converged filter similarity, or geometric proximity to select filters for pruning.These approaches include greedy optimization, multi-layer information flow, centripetal SGD, and geometric-median criteria.Source: §3.3 · p. 24
• Input-feature pruning summarizes each feature’s output sensitivity across examples, prunes low-scoring inputs, and retrains the network, optionally repeating the procedure.Sensitivity can be aggregated by mean square, absolute sum, or maximum after evaluating each output’s derivative with respect to each input.Source: §3.3 · p. 25

3.3.1 Selection based on activity and correlation.

• Activity and correlation criteria remove neurons or connections with redundant, low-energy, mostly zero, or weakly correlated behavior; correlated-weight pruning improved generalization and sparsification in convolutional face recognition.Similar-output neurons can be fused, while APoZ targets ReLU units that are zero for many examples and activation magnitude can provide higher fidelity.Source: §3.3 · p. 24, §3.3.1 · p. 25
• These methods mainly minimize changes to input-output behavior, so pruning a low-accuracy network does not itself improve its accuracy.This limitation motivates incorporating the training loss directly into pruning to improve the resulting model.Source: §3.3.1 · pp. 25–26

Gradient-based selection

• Gradients computed during ordinary optimization can be reused with gating elements to select arbitrary weights, neurons, or filters for removal.Gradient computations are generally cheap, enabling importance estimation alongside normal training.Source: §3.4 · p. 26

Gradient and movement criteria

• Accumulated weight changes identify parameters changed least during training, while squared gradient-weight products approximate neuron or filter importance [Golub et al. 2019; Karnin 1990; Molchanov et al. 2019].With an all-zero starting state, this criterion becomes equivalent to absolute-magnitude sparsification.Source: §3.4 · p. 26

Gating methods

• Binary gating gradients enable iterative neuron pruning, stopping after a large relevance-gradient jump; related methods gate weights using absolute gradients [Mozer and Smolensky 1988; Lee et al. 2019; Xiao et al. 2019].The gating approach requires separate backpropagation for weights and gate perturbations.Source: §3.4 · p. 26

Discrete gating

• Tri-state ReLU learns binary-like gating and layer-removal parameters, using regularization, rounding, and first-order gradient-weight products to select weights or structurally prune filters [Srinivas and Babu 2016; Srinivas et al. 2016; You et al. 2019].A shared layer parameter can remove the whole layer, while the traditional ReLU is recovered for d = 0 and w = 1.Source: §3.4 · pp. 26–27

Transfer learning and dynamic selection

• During transfer learning, gradient magnitude prunes feature maps, whereas movement pruning uses accumulated task-specific direction toward or away from zero to identify less important weights [Molchanov et al. 2017; Sanh et al. 2020].Global sparse momentum instead updates important weights normally and gradually zeros unimportant weights through decay, with rare re-selection that decreases during training [Ding et al. 2019b].Source: §3.4 · p. 27

3.5.1 Pruning as an optimization task.

• Second-order pruning removes the weight whose perturbation minimizes the quadratic loss increase, using a saliency measure based on the inverse Hessian diagonal; OBS additionally updates the remaining weights.The derivation assumes a locally optimized model so the first-order gradient term is negligible, and solves the constrained minimization with Lagrange multipliers.Source: §3.5.1 · p. 28

3.5.2 Magnitude pruning as a special case.

• Magnitude pruning is a special case when the Hessian is approximately diagonal with uniformly distributed diagonal entries, but that structural assumption may not hold in practice.Under this approximation, the second-order saliency is proportional to the squared weight magnitude.Source: §3.5.2 · p. 28, §3.5.3 · p. 28

3.5.3 Discussion of assumptions and guarantees.

• The framework assumes a well-trained model, invertible and locally constant Hessian, small perturbations, and negligible correlations among simultaneously removed weights.Practical schemes address some limitations through gradual pruning or Hessian re-estimation, while removing multiple weights requires approximating their interactions.Source: §3.5.3 · pp. 28–29

3.5.4 A Simple Illustration.

• In a two-variable example, magnitude pruning yields L_MAG(0, −0.3) = 0.045, OBD yields L_OBD(0.1, 0) = 0.02, and OBS updates the remaining weight to reach L_OBS(0, 0) = 0.The example illustrates that OBS can account for non-diagonal curvature and compensate for a pruned weight by adjusting the others.Source: §3.5.4 · p. 29, Figure 12 · p. 29

3.5.5 Large-scale pruning based on second-order information.

• Second-order pruning at deep-network scale replaces exact Hessian inversion with layerwise, Fisher, low-rank, block-diagonal, or K-FAC approximations.L-OBS uses neuron-structured Hessian blocks; empirical-Fisher methods rely on classification and well-optimized output distributions, while K-FAC makes inversion efficient through Kronecker structure but requires additional manipulation for convolutional and recurrent layers.Source: §3.5.4 · p. 30, §3.5.5 · pp. 30, 32
• Singh and Alistarh [2020] report that block-diagonal second-order pruning predicts local loss more accurately than magnitude, diagonal Fisher, and K-FAC methods, achieving state-of-the-art accuracy for one-shot and gradual unstructured pruning.Accuracy improves further when the first-order gradient term is included and the Hessian is re-estimated along the pruning direction.Source: §3.5.5 · pp. 30–31

3.6 Selection based on regularization of the loss during training

• Regularization adds penalty terms to the loss to guide sparsity and reduce model complexity, but can introduce local minima and requires delicate, sometimes layer-wise tuning.Weight decay is a standard regularizer that can be combined with magnitude pruning.Source: §3.6 · pp. 32–33
• L0 penalties directly count nonzero weights but are nondifferentiable and NP-complete to optimize, motivating stochastic gates, straight-through estimators, continuous sigmoid approximations, learned thresholds, top-k pruning, and polarization.These methods trade optimization tractability against sampling noise, instability, limited weight reanimation, or difficult hyperparameter tuning; sigmoid approaches use temperature schedules, while learned thresholds can be optimized through SGD.Source: §3.6 · pp. 33–34

3.6.3 Grouped regularization.

• Grouped regularization extends lasso to predefined groups, enabling structured pruning of neurons, filters, channels, layers, and residual connections; sparse group lasso also promotes sparsity within groups.Group definitions determine which network structures are removed together, while scaling-factor methods can be folded into batch normalization and pruned by magnitude.Source: §3.6.3 · pp. 36–37

3.6.4 Other regularization techniques.

• Other regularizers reduce neuron energy, prune low-sensitivity parameters, or jointly gate quantization widths, but L1 and L2 regularization can fail with batch normalization because rescaling cancels their pruning effect.Sensitivity-based pruning is applied gradually and is suggested to start from a pretrained network.Source: §3.6.5 · p. 37, §3.6.4 · p. 37

Variational dropout

• Sparse Variational Dropout prunes weights with large α, often in one shot without fine-tuning, but requires doubled parameters and can be difficult to train from scratch.A warm-up period or pretrained initialization can improve training; ImageNet-scale results achieve high sparsity but show substantial variance in accuracy and sparsity.Source: §3.7 · p. 38
• For CNNs, Sparse Variational Dropout behaves similarly to global magnitude pruning, whereas Transformer variances change more substantially and distinguish pruned from retained weights.Despite this behavior, variational dropout can prune many weights immediately after training with only a small test-accuracy drop, suggesting regularization beyond final variance values.Source: §3.7 · pp. 38–39

Structured Bayesian pruning

• Structured Bayesian pruning shares variational parameters across neurons or filters and thresholds low-signal-to-noise groups, producing structured sparsity and demonstrated acceleration on MNIST and CIFAR-10.A truncated log-normal posterior preserves input signs and permits a closed-form KL-divergence term.Source: §3.7 · p. 39

Bayesian pruning with hierarchical priors

• Hierarchical Bayesian pruning uses shared scale variables within neurons or feature maps to encourage structured sparsity, while posterior uncertainty determines fixed-point precision for quantized networks.The method combines variational inference with minimum description length through sparsity-inducing hierarchical priors.Source: §3.7 · p. 39

Bayesian pruning for recurrent neural networks

• Bayesian pruning extends to recurrent networks by sparsifying individual LSTM weights, neurons, gates, or parameters using Sparse VD, hierarchical variational inference, or automatic relevance determination.These methods were evaluated on text classification and language-modeling problems, with ARD pruning weights whose prior variances fall below a threshold.Source: §3.7 · pp. 39–40

Related methods

• Other variational approaches prune through soft weight sharing with Gaussian-mixture priors or through information bottlenecks and sensitivity-variance criteria.Soft weight sharing begins from a pretrained network, while information-bottleneck methods optimize inter-layer mutual information to identify redundant neurons.Source: §3.7 · p. 40

3.8 Other selection schemes

• Genetic algorithms evolve pruned networks through mutation, reproduction, and crossover, rewarding parameter reduction and generalization but requiring impractical ensemble training costs.Populations contain separately trained pruned networks, making the approach unsuitable for modern compute-intensive training.Source: §3.8.1 · p. 40

3.8.2 Sampling-based pruning with guarantees.

• Empirical-sensitivity sampling selects incoming edges for pruning and provides guarantees on sparsity and generalization, with extensions to structured neuron and CNN-filter pruning.The method estimates edge importance from a data subset, while later work guarantees the pruned network’s size and final output.Source: §3.8.2 · p. 40

3.8.3 Diversity and quantized networks.

• Diversity networks use Determinantal Point Processes to retain diverse neurons while fusing similar ones, starting from fully trained networks without fine-tuning.The selection occurs within each layer.Source: §3.8.3 · p. 40
• Guerra et al. [2020] select filters for pruning in binary or quantized networks using rounding error between quantized and full-precision weights.The rounding-error metric repurposes the approximation function already used by quantized neural networks.Source: §3.8.3 · p. 40
• Tang et al. [2021] generate label-independent knockoff features and prune filters highly sensitive to knockoffs but relatively insensitive to real inputs.Knockoffs match the training-set distribution while remaining independent of the example’s label; pruning uses comparative filter-sensitivity scaling factors.Source: §3.8.3 · pp. 40–41

Practical layer budgets

• Choosing layer-wise sparsity is therefore an important practical consideration for sparsification.The choice concerns whether sparsity parameters are set per operator or globally for the whole model.Source: §3.9 · p. 41

Parameter allocation

• Global sparsity selection can work well for RNN pruning, but heterogeneous layer types and network positions often require different sparsity levels per layer [See et al. 2016; Mocanu et al. 2018].Per-layer tuning can improve flexibility but introduces potentially complex hyperparameters.Source: §3.9 · p. 41

Automatic allocation

• Automatic budgeting links sparsity to layer properties or redistributes parameters during training using saliency, such as loss gradients after dropping small-magnitude weights [Evci et al. 2020; Mostafa and Wang 2019].These approaches reduce the need to tune separate layer-wise hyperparameters manually.Source: §3.9 · p. 41

Practical layer budgets

• Many global schemes assign more parameters to earlier ResNet-50 layers than later ones, and practitioners often disable pruning of the first layers because this empirically improves accuracy.Figure 16 compares sparsity distributions across layers for multiple sparsification methods.Source: Figure 16 · p. 41, §3.9 · p. 41

Landscape of surveyed work

• Nearly 50% of surveyed papers focus on weight sparsification, closely followed by neuron sparsification, while other structured schemes and inputs remain a minority.The overview classifies papers by candidate element, including neurons, weights, convolutional filters, transformer heads, transformer hidden dimensions, and inputs.Source: §3.10 · pp. 41–42

Inference and training

• More than 60% of papers target inference, whereas training-focused sparsification is gaining popularity and largely concentrates on weights.Inference studies mainly prune neurons, weights, or filters.Source: §3.10 · p. 42, Figure 17 · p. 42

Method–objective patterns

• Inference is dominated by regularization followed closely by magnitude pruning, while training focuses on magnitude and the overall landscape is led by magnitude-based weight pruning followed by sensitivity-based neuron pruning.Among weight-sparsification schemes, simple magnitude pruning is followed by first- and second-order methods.Source: §3.10 · pp. 42–43, Figure 17 · p. 42

Selection methods

• Magnitude pruning and regularization together account for 50% of works; magnitude pruning most often targets weights, while regularization is applied across element types.Filters, blocks, and heads are grouped into a single structured category in this comparison.Source: §3.10 · pp. 42–43, Figure 18 · p. 42

4 DYNAMIC PRUNING: NETWORK REGROWTH DURING TRAINING

• Regrowth maintains approximately constant model size while traversing possible architectures, but selecting new weights is difficult because all absent weights share the same zero value.Gradient or Hessian information can guide selection, but increases memory and computation overhead.Source: §4 · p. 43

4.1 Random or uniform regrowth

• Random, uniform, and layer-wise regrowth provide simple alternatives, including random architectural walks, preferential allocation to retained layers, and width-multiplier growth.Recomputing pruned random weights on demand can preserve their training influence without retaining them at inference.Source: §4.1 · p. 43

4.2 Based on gradient information

• Gradient-based regrowth adds weights with large or accumulated gradients, offering informative architecture search but requiring substantial dense-gradient memory and computation.Maintaining pruned weights during training can improve final accuracy, while layer-wise or top-(k + d) schemes reduce gradient-storage demands.Source: §4.2 · pp. 43–44

4.3 Locality-based and greedy regrowth

• Locality-based and greedy methods exploit neuron distance or estimated benefit, adding connections with locality structure or neurons and filters that maximize loss reduction or minimize sensitivity.Greedy schemes offer theoretical guarantees but generally provide limited sparsification.Source: §4.3 · p. 44

5.1 Sparsifying neuron activations

• ReLU activations are naturally sparse, with some convolutional layers reaching up to 90% sparsity, substantially exceeding the 50% expected from the activation structure [Glorot et al. 2011b; Rhu et al. 2018].Hoyer regularization can produce higher activation sparsity with lower accuracy loss than L1 regularization, while thresholding and sparse kernels can yield CPU inference speedups.Source: §5.1 · p. 45

5.2 Dropout techniques for training

• Dropout randomly zeros neurons during training to reduce co-adaptation and improve generalization, while inference uses the full network; its activation sparsity increases with dropout probability p [Hinton et al. 2012; Srivastava et al. 2014a].Extensions drop weights, feature maps, spatial regions, recurrent updates, or entire layers and paths.Source: §5.2 · p. 46

5.3.1 Magnitude-based gradient sparsification.

• Gradient sparsification reaches up to 99.9% sparsity, making gradients more amenable to sparsification than weights or activations.Magnitude-based methods typically remove smaller gradients and use error feedback to mitigate compression error; Fig. 19 summarizes their components and attainable sparsity.Source: §5.3 · p. 47, Figure 19 · p. 47, §5.3.1 · p. 48

5.3.2 Variance-based gradient sparsification.

• Variance-aware gradient sparsification addresses the convergence cost of increased gradient variance by selecting or rescaling updates under sparsity or variance budgets, matching or outperforming threshold sparsification on CIFAR-10 and ImageNet.Rand-k and related methods preserve unbiasedness, while Tsuzuku et al. [2018] use within-mini-batch gradient variance as the sparsification criterion.Source: §5.3.2 · p. 48

5.3.4 Convergence of sparsified gradient methods.

• Sparsified gradient methods converge at roughly standard SGD rates when error feedback is used, although guarantees differ by objective, distribution setting, and compression granularity.Layer-wise compression is theoretically better in general, but entire-model sparsification can outperform it in some practical cases.Source: §5.3.4 · p. 49

5.4 Errors and optimizer state

• Top-k sparsification of back-propagated errors reduces computation and can produce 95% gradient sparsity, whereas sparse optimizer states remain largely unexplored.meProp reapplies top-k selection after each fully connected layer; sparse optimizer states are expected to improve training memory efficiency.Source: §5.4 · p. 50

5.5 Dynamic networks with conditional computation

• Ephemeral sparsity also arises from input-dependent routing, where gates or switching mechanisms select only parts of dynamic networks and thereby increase capacity without increasing computational cost.Runtime pruning can select channels per input using reinforcement learning, while product-key networks are noted as related but outside the overview’s scope.Source: §5.5 · p. 50
• Switch Transformers improve pretraining speed by up to 7x over dense models while supporting up to trillion-parameter models through sparse mixture-of-experts routing.Their best configuration uses k = 1 expert and a load-balancing loss; Lepikhin et al. [2020] trained a 600-billion-parameter transformer with k = 2.Source: §5.5 · p. 50

6.1 Sparsifying convolutional neural networks

• CNN results show increasing accuracy and compression over time: magnitude pruning dominates below 90% sparsity, whereas regularization and first- or second-order methods perform best above 90%.The comparison covers the reviewed CNN pruning strategies and their sparsity–accuracy tradeoffs.Source: §6.1 · p. 51, Figure 20 · p. 52

6.1.1 CNN architecture evolution.

• CNN architecture evolution reduces parameters through Inception modules, separable convolutions, depth-wise convolutions, and NAS-based designs such as EfficientNet.These approaches are presented as handmade or searched sparse formulations of convolutional computation.Source: §6.1.1 · p. 52

6.1.2 CNN sparsification.

• Sparse training can outperform dense training at very high sparsity: fully sparse methods exceed 95%-sparse CNN baselines, while a 90%-sparse ResNet-50 reaches 72.3% accuracy with 2.7×–5.6× fewer computations.Parameter redistribution and cosine schedules are important, particularly for larger layers.Source: §6.1.2 · pp. 53–54
• Well-tuned magnitude pruning is competitive on ResNet-50, reaching 76.52% accuracy at 80% sparsity and 75.2% at 90% after 100 epochs, while variational dropout and L0 pruning can cost up to 3× more time and 2× more memory.Variational dropout performs best only above 95% sparsity, while tuned magnitude pruning remains close and is judged the most practical method.Source: §6.1.2 · p. 54
• At 50–80% fewer ResNet-50 inference operations, sparsified models retain accuracy with potential speedups of up to 5×.Fig. 21 summarizes flop counts and accuracy for state-of-the-art ResNet-50 pruning methods on ImageNet.Source: §6.1.2 · p. 54, Figure 21 · p. 54

6.2 Sparsifying transformer networks

• For transformers, structured head pruning works at low sparsity, unstructured magnitude pruning is strongest at 40–80%, and movement pruning is the reported high-accuracy method above 90%.Transformer pruning has relatively few results at very high sparsity or with large accuracy degradation, partly because many studies preserve near-original accuracy.Source: §6.2 · p. 56

6.2.1 Structured sparsification.

• Structured transformer pruning removes 80% of attention heads for only 0.15 BLEU loss, while broader component pruning reaches 40–75% sparsity with about a 5-point F1 loss.Pruning too many heads can be detrimental to accuracy for some tasks, while attention accounts for only about one-third of BERT’s parameters.Source: §6.2.1 · p. 56

6.2.2 Unstructured sparsification.

• Movement pruning targets parameters that shrink during fine-tuning rather than parameters with small magnitude, achieving higher performance than magnitude- or L0-based pruning at very high sparsity such as 97%.Distillation can further improve movement pruning for transfer learning.Source: §6.2.2 · p. 57

7 SPEEDING UP SPARSE MODELS

• Unstructured sparse models can run slower than dense models on current hardware, while some structured patterns map directly to optimized dense operations.90% sparse workloads were slower than dense computation on GPUs, and 89% sparse AlexNet was 25% slower on CPUs; structured neuron, filter, or head sparsity can use libraries such as cuDNN.Source: §7 · p. 58

7.1 Algorithmic and software support for sparse models

• Efficient sparse execution depends on matching sparsity levels and representations to hardware, because both insufficient and excessive sparsity can increase overhead or underutilize compute.CPU performance is highest in a workload-dependent sparsity range; excessive sparsity raises index-storage overhead and shifts execution toward memory-bandwidth and data-movement bottlenecks.Source: §7.1 · p. 59

7.1.1 Structured sparsity.

• Structured formats reduce representation overhead by storing offsets for blocks or fixed patterns rather than individual nonzeros.Blocked formats reduce storage overhead by a factor of the block size, while block-balanced and strided formats further constrain nonzero placement for more compact or balanced execution.Source: §7.1.1 · pp. 59–60, Figure 23 · p. 60

7.1.2 Tuned block sparsity in practice.

• 50% sparsity on ARM and less than 10% on Intel CPUs can suffice for speedups with SIMD-aligned block and neuron pruning, while larger blocks reduce accuracy at 60–77% sparsity.Scalpel lowers the speedup threshold from 70% to 50% on ARM and from 50% to less than 10% on Intel; magnitude-based pruning shows worsening top-5 accuracy as block size increases.Source: §7.1.2 · pp. 60–61

7.2.1 Inference accelerators.

• Sparse accelerators exploit weight and activation sparsity through specialized dataflows, achieving substantial inference gains across CNN and recurrent workloads.EIE reports 13x speed and 3,400x energy efficiency versus a Titan X GPU, while other accelerators achieve 2–15x speedups and 2–10x lower energy consumption.Source: §7.2.1 · pp. 61–62
• Predicting output sparsity can improve inference by avoiding computation of expected zeros.SparseNN improves performance by up to 70% while halving power consumption, and low-rank mask prediction can be 20x faster than evaluating the full layer for sufficiently small rank.Source: §7.2.1 · p. 62

7.2.2 Training accelerators.

• Sparse training requires dedicated designs because inference formats and sparsity patterns do not transfer efficiently through activation storage and backward computation.Procrustes exploits structural weight sparsity and is up to 4x faster and 3.36x more energy efficient than dense training accelerators; another design accelerates training by up to 40%.Source: §7.2.2 · p. 63

7.2.3 Overview of accelerators for sparse deep learning.

• Accelerator architectures are specialized by operation, layer type, storage format, load balancing, and reuse, with most supporting inference and only some explicitly supporting training.Table 1 summarizes these design choices, while generic bitmap-based architectures perform best for relatively dense sparse operations and future higher sparsity may favor different accelerator classes.Source: §7.2.2 · p. 63, §7.2.3 · p. 63, Table 1 · p. 64

8 DISCUSSION

• Sparse models generally outperform dense models at equal parameter budgets, but pruning is most effective for overparameterized architectures, making parameter efficiency essential when reporting sparsity [Elsen et al. 2019; Lee et al. 2020a; Hoefler and Belli 2015].Switching to a better architecture may sometimes be more efficient than pruning.Source: §8 · pp. 64–65

8.1 Relation to Biological Brains

• Biological inspiration can guide innovation, but silicon’s structural and computational constraints mean biologically implausible mechanisms may be more efficient; brain-inspired pruning remains uncompetitive on large complex tasks.Weight sharing illustrates how engineering refinement can outperform biological fidelity.Source: §8.1 · p. 65

8.2 Permutation Groups and Information Loss

• Permutation symmetries create exponentially many equivalent dense networks, suggesting sparsification may retain substantial information content and relate to robustness in high-dimensional spaces [Changpinyo et al. 2017; Ahmad and Scheinkman 2019].Under stated conditions, a network with hidden-layer sizes n_1,...,n_l has at least Π_i=1^l n_i! equivalent networks.Source: §8.2 · p. 65

8.3 Sparse subnetworks for training and lottery tickets

• Lottery-ticket findings indicate that sparse subnetworks can train effectively from suitable initial structures, signs, or even fixed random weights, while larger-network evidence supports iterative sparse training more than rewinding [Frankle and Carbin 2019; Zhou et al. 2020; Pensia et al. 2020].A trained supermask reached 93.5% MNIST and 65.4% CIFAR-10 accuracy at around 50% sparsity without changing random initial weights; rewinding has not been proven effective [Renda et al. 2020].Source: §8.3 · p. 66, §8.3.1 · p. 66, §8.3.2 · pp. 66–67

8.4 Structured vs. unstructured pruning

• Structured pruning delivers better modern-device performance despite lower accuracy per element than unstructured pruning, whereas random initialization pruning works better for neurons and filters than weights, motivating different schemes.The most efficient granularity remains dependent on future architectures and the overhead of storing fine-grained indices.Source: §8.4 · p. 67

8.5 Optimization algorithms during model training

• Sparsification restricts optimization to lower-dimensional subspaces and can increase loss-function Lipschitz constants, while iterative pruning acts as guided nonlinear search that may escape local minima; pruning-induced accuracy drops can also predict better generalization [Evci et al. 2020; Bartoldson et al. 2020].Momentum may help but can require more iterations, and carefully tuned magnitude-pruning schedules may flatten the loss landscape.Source: §8.5 · pp. 67–68

8.6 Emerging Benchmarks

• Comparisons require standardized tasks and level playing fields; the paper recommends ResNet-50 on ImageNet and BERT on GLUE, while noting that MNIST alone is insufficient and global magnitude pruning is a useful baseline [Blalock et al. 2020; Singh and Alistarh 2020].Achieved sparsity at high accuracy correlates with literature attention, so architecture prunability cannot be inferred without extensive controlled experiments.Source: §8.6 · p. 68

8.7 Parameter Efficiency / 8.8 Generalization and biases

• Hardness-normalized parameter efficiency shows many pruned networks exceed dense baselines, but pruning can expose compression thresholds, while its effects on class accuracy, distribution shifts, and demographic subgroups demand finer-grained evaluation.A pruned ResNet-50 achieved 66% validation accuracy with ≈281,000 parameters, and pruning to 95% sparsity increased blond-hair identification false-positive rates by 49.54% for men versus 6.32% for others.Source: §8.7 · pp. 69, 71, §8.8 · p. 71

• Sparse training must enable gigantic models that exceed dense hardware budgets without sacrificing accuracy, while multi-objective pruning must optimize goals such as energy under memory constraints.These questions concern training feasibility and simultaneous optimization of multiple deployment objectives.Source: §9 · p. 72
• Future work must clarify structured versus unstructured tradeoffs, co-design hardware with pruned models, and determine how neural architecture search compares with pruning for efficient networks.The section asks how structural bias affects accuracy and model size, and how cost, accuracy, and structured sparsity should be balanced.Source: §9 · p. 72
• Theory and sparse representations remain open: researchers must relate sparsity to learning dynamics and generalization, characterize representational power, and rigorously define parameter efficiency.Source: §9 · p. 72
• Pruning methods must generalize from MLPs and CNNs to transformers and other architectures, while one-shot data-free methods must approach the accuracy of data-dependent approaches.Source: §9 · p. 72
• Robust sparse models and sparsification approaches must address fairness and bias while preventing adversarial attacks; brain-related challenges are omitted because the work emphasizes engineering.Source: §9 · p. 72

• 10–100x theoretical efficiency gains and increasing pruning opportunities in larger networks motivate hardware–software co-design for next-generation deep learning architectures.The authors argue that training extremely large models with sparse methods could provide many opportunities for substantial speedups.Source: §10 · p. 73
• Very high sparsity (>99%) and sparse training of large models in high-dimensional spaces remain promising areas for future breakthroughs.Source: §10 · p. 73