PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, Alexandra Peste

arXiv:2102.00554v1cs.LGcs.AIcs.ARcs.CVcs.NE

TL;DR

Deep learning’s overparameterized models impose substantial memory and computation costs, motivating sparsification. This survey synthesizes methods for pruning, growth, training, and hardware exploitation across the field. It reports 10-100x model-size reductions and theoretical efficiency gains, while identifying practical and scope limitations.

  • Problem

    Overparameterized dense models impose substantial memory and computation costs, while the field lacks a consistent basis for comparing diverse sparsification methods.

  • Method

    The paper surveys and tutorials sparsification methods for removing and adding network elements, training sparse models, and exploiting sparsity in software and hardware.

  • Results

    10-100x model-size reduction is reported, with corresponding theoretical gains in computational, storage, and energy efficiency; sparse models often outperform dense models at equal parameter budgets.

  • Takeaways & Limitations

    Sparsity offers a route toward more efficient inference and training, particularly as models grow and hardware-software co-design can exploit sparse computation.

  • Takeaways & Limitations

    Pruning criteria based on an identity-like Hessian rely on a strong structural assumption that may not hold in practice.

Abstract

from arXiv · show

The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, if not better than, the original dense networks. Sparsity can reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field.

1 INTRODUCTION

Deep learning’s reliance on overparameterized dense models creates substantial memory and computation costs, motivating sparsification as a way to reduce models while preserving accuracy. This survey organizes sparsification techniques, training strategies, evaluation challenges, and practical acceleration methods.

  • Motivation: Overparameterized dense models increase memory and computation costs, especially for mobile, battery-driven, and cost-conscious inference settings.Sparse representations can reduce storage and often computational effort, while sparsification may also act as regularization.
  • Motivation: Sparse models can reduce model size by 10-100x while providing corresponding theoretical gains in computational, storage, and energy efficiency.
  • Scope: The paper organizes sparsification by which network elements are changed, when sparsification occurs, and how elements are removed or added during training.It also discusses sparse training, connection regrowth, and performance results across architectures.
  • Evaluation: Different tasks, models, techniques, and evaluation settings produce incomparable results, making state-of-the-art comparisons and method rankings difficult.The authors therefore summarize qualitative design aspects before reviewing architectures and performance results.
  • Scope: The survey covers model compression techniques, including down-sizing, operator factorization, value quantization, and sparsification, with these methods reducing memory requirements.The paper focuses on sparsification while situating it within the broader compression landscape.
  • Broader context: The survey addresses sparsity’s historical development, biological analogies, theoretical foundations, and hardware-software mechanisms for exploiting sparse models.It notes that the literature expanded substantially after deep learning’s resurgence, while biological and CMOS constraints differ.

2 OVERVIEW OF SPARSITY IN DEEP LEARNING

Sparsity trades model density for improved generalization and reduced inference or training cost, but practical benefits depend on sparsity level, storage format, schedule, and hardware. The section surveys sparse representations, training strategies, and failure modes across the design space.

  • Overview: Sparsification serves two goals: improved generalization and improved inference or training performance.It removes nonessential elements to reduce computation and memory while potentially regularizing the model.
  • Generalization and performance: Test error can initially decrease as sparsity removes learned noise, then rise sharply at high sparsity in Occam’s hill.The corresponding performance curve often increases with sparsity after low-sparsity overheads, before diminishing returns at extreme sparsity.
  • Storage: Sparse storage requires indexing for arbitrary nonzero weights, with overhead ranging from n-bit bitmaps to m log(n)-bit coordinate schemes.The optimal format depends on sparsity, structure, weight size, and access pattern.
  • Storage: No sparse storage scheme helps below 10% sparsity; bitmaps work best from 10–70%, while delta encoding is best above 80% under the stated assumption.Offset and dimension-aware formats may help at very high sparsity, but their usefulness for deep models remains unclear.
  • Training schedules: Iterative hard thresholding alternates top-k magnitude pruning with sparse fine-tuning and dense retraining to regularize while relearning representations.Related dense-sparse-dense schedules report significantly higher generalization performance.
  • Sparse training: SNIP can create bottlenecks or layer collapse above 99% sparsity, whereas GraSP preserves gradient flow and improves upon SNIP in very sparse regimes.Iterative methods can improve performance over GraSP but may still underperform pruning of fully trained ResNets.

3 SELECTING CANDIDATES FOR REMOVAL

Candidate selection ranges from exhaustive or random removal to importance-based pruning, with structured sparsity improving hardware efficiency by constraining removable patterns. Magnitude pruning remains simple and effective, while data-free methods can require retraining to recover accuracy.

  • Selection principles: Exhaustive leave-some-out selection is precise but requires training networks for candidate removals, making it poorly scalable.Random removal is simpler and can work effectively in some settings, while importance metrics often perform best at high sparsity.
  • Comparisons: Comparative studies identify no clear winner because pruning efficacy depends on architecture, hyperparameters, learning-rate schedule, and task.The survey therefore emphasizes comprehensive coverage rather than a universal ranking.
  • Structured sparsity: Structured sparsity removes patterns such as neurons, filters, heads, channels, kernels, or blocks to reduce index overhead and simplify processing.Removing whole neurons or filters yields smaller dense computations, unlike arbitrary unstructured weight sparsity.
  • Magnitude pruning: Magnitude pruning removes weights or groups with the smallest absolute magnitude and can be applied to individual weights, blocks, rows, columns, or neurons.It is theoretically justified under some assumptions and remains effective across modern pruning settings.
  • Magnitude pruning: Magnitude pruning can remove weights with |w| ≤ 0.17 before retraining, concentrating removals around the zero-centered bulk of the weight distribution.Thresholds may be fixed by a weight budget, selected per layer, or learned during SGD.
  • Data-free methods: Data-free methods can be effective and state of the art, but more precise methods often perform better at high sparsity and data-free schemes may need expensive retraining.The trade-off is between inexpensive selection and recovery of accuracy close to the original model.

3.3 Data-driven selection based on input or output sensitivity

Data-driven sensitivity methods use training examples to identify elements whose removal minimally changes network behavior, using output variation, correlations, or spectral measures. Their pruning objective preserves input-output behavior rather than directly improving an already inaccurate model.

  • Sensitivity measures: Sensitivity-based selection evaluates how neuron, filter, weight, or input changes affect network outputs across examples.Low-sensitivity elements are removed and the network may then be retrained and pruned again.
  • Redundancy: Output redundancy can be exploited by removing neurons with little variation or merging neurons with similar or opposite activations.Biases and outgoing weights are adjusted to minimize the change caused by removal.
  • Filters: Filter-pruning methods can select filters by their minibatch output sensitivity or by minimizing error entering later classification layers.These methods formulate pruning as an optimization problem or use importance metrics tailored to downstream outputs.
  • Spectral methods: FAST estimates weight relevance from Fourier amplitudes and removes neurons contributing less than 5% of total output variance.Its simulation cost grows linearly with the number of weights, while interval-based variants approximate input distributions using bounds.
  • Correlation: Correlation-based pruning can preserve strongly correlated connections and drop weakly correlated ones, improving generalization in a fully trained face-recognition convolutional network.The reported benefit comes from refining an existing model rather than selecting solely by individual weight magnitude.
  • Limitation: Sensitivity methods minimize changes to input-output behavior, so they do not improve a network whose baseline accuracy is already low.This limitation follows from their objective rather than from a requirement that the model be dense.

3.4 Selection based on 1st order Taylor expansion of the training loss function

First-order Taylor methods reuse gradients or gating derivatives to estimate how removing parameters affects training loss. They support weight, neuron, filter, and layer selection, including transfer-learning and dynamic sparse-training variants.

  • Gradient-based selection: Gradient-based pruning reuses gradients computed during optimization to estimate parameter importance with low additional computational cost.Binary gating elements can extend the approach from weights to arbitrary removable components.
  • Importance estimates: First-order importance can be approximated from total training updates or squared gradient-weight products for weights, neurons, and filters.These measures prioritize elements that changed little or contributed little during learning.
  • Gating: Gating methods iteratively prune the least important neurons according to the loss derivative with respect to disabling them.The procedure can stop when the derivative exhibits a large jump, and analogous methods gate weights.
  • Gating: The Tri-state ReLU generalizes gating by using learnable binary parameters to control neuron activity and optionally remove an entire layer.Its regularizer and final rounding support discrete selection of the gating variables.
  • Redundancy: Jacobian factorization methods identify redundant weights or neurons by exploiting correlated gradients or hidden-node outputs.QR factorization is applied to the Jacobian or hidden-node outputs after training has progressed.
  • Training and transfer: Transfer-learning pruning uses gradient magnitude for feature-map removal, while movement pruning considers gradient direction during fine-tuning.Global sparse momentum periodically reselects important weights, though reselection becomes rare as training proceeds.

3.5 Selection based on 2nd order Taylor expansion of the training loss function

Second-order pruning selects weights by estimating their loss impact through Taylor expansions and Hessian information. The framework can improve pruning outcomes but relies on assumptions that complicate application to modern networks.

  • Pruning as an optimization task: Second-order pruning estimates each weight’s loss increase under a zeroing perturbation and removes the candidate with the lowest pruning statistic.At a local minimum, the first-order gradient term is ignored, leaving a quadratic Hessian-based expression.
  • Pruning as an optimization task: The saliency measure uses the inverse Hessian diagonal, while Optimal Brain Surgeon also computes compensating updates to the remaining weights.This differs from one-at-a-time removal without adjusting the other parameters.
  • Magnitude pruning as a special case: When the Hessian is proportional to the identity, the second-order statistic becomes proportional to squared weight magnitude, recovering magnitude pruning.The identity-Hessian assumption is strong and may not hold in practice.
  • Framework assumptions and limitations: OBD and OBS assume a well-trained model, an invertible and locally stable Hessian, and negligible correlations when pruning multiple weights together.Exact repeated Hessian recomputation for individual removals is infeasible for modern networks.
  • A simple illustration: In a two-variable illustration, OBS reaches L_OBS(0, 0) = 0, whereas magnitude pruning reaches L_MAG(0, −0.3) = 0.045 and OBD reaches L_OBD(0.1, 0) = 0.02.The example uses an SGD point (0.1, −0.3) with loss 0.065.
  • Large-scale pruning: Large-scale extensions approximate curvature layerwise or through structured blocks, and reported methods reduce single-step accuracy drops relative to other pruning methods.Further gains are reported from including the gradient term and re-estimating the Hessian along the pruning direction.

3.6 Selection based on regularization of the loss during training

Regularization-based sparsification adds penalties to training objectives to encourage sparse structures or weights. These methods are broadly flexible and practical, but their effectiveness depends strongly on penalty design and tuning.

  • Penalty-based sparsification: Regularization adds a penalty P(w) to the original loss, guiding optimization toward sparse models and potentially reducing complexity or improving generalization.Penalties can target weights, neurons, or computational metrics such as floating-point operations.
  • Limitations: Regularization can introduce additional local minima and requires delicate, sometimes layer-specific tuning of its penalty parameters.The balance between the ordinary loss and regularization term is central to successful sparsification.
  • L0 and discrete objectives: Direct L0 regularization counts nonzero weights but is difficult to optimize because its discrete objective is nondifferentiable and NP-complete.Differentiable stochastic gating variables provide one approximation strategy.
  • Estimating discrete functions: Straight-through estimators ignore the derivative of binary gating, while sigmoid relaxations use temperature β to trade approximation quality against trainability.Higher β more closely approximates the Heaviside step function but is harder to train.
  • Structured penalty designs: Top-k methods impose a hard parameter budget and retain only the k largest-magnitude weights, while polarization separates gating elements toward zero or away from zero.The cited top-k studies report better performance than other soft regularization schemes.
  • L1 and grouped regularization: L1 regularization encourages small weights but usually requires magnitude thresholding to produce exact zeros; group lasso instead zeros predefined groups together.Group lasso reduces to the original lasso when every group has size one.

3.7 Variational selection schemes

Variational selection schemes treat sparsification probabilistically, using learned distributions or sparsity-inducing priors to identify removable parameters. They support both unstructured and structured pruning, with behavior varying across architectures.

  • Variational dropout: Sparse VD models each weight with a variational Gaussian distribution and uses learned parameters to identify weights associated with large dropout rates for pruning.The method is motivated by variational dropout and sparse Bayesian learning.
  • Empirical behavior: For CNNs, Sparse VD can behave similarly to global magnitude pruning, while Transformer networks show more distinctive changes in the variational log-variances.This comparison concerns converged variational parameters and the weights eventually pruned.
  • Empirical behavior: Variational dropout can prune many weights immediately after training with a small test-accuracy drop, unlike magnitude pruning, which requires fine-tuning to recover performance.The authors associate this behavior with a regularization effect that is not always reflected in final weight variances.
  • Structured Bayesian pruning: Structured Bayesian pruning modifies the approximate posterior to produce structured sparsity, which is more desirable than unstructured sparsity when inference acceleration is the goal.The cited approach uses a truncated log-normal approximate posterior.
  • Bayesian pruning: Bayesian pruning uses hierarchical sparsity-inducing priors and variational inference to compress neural networks under a model-complexity and data-misfit objective.The minimum description length principle connects the objective to communicating models with fewer bits.
  • Recurrent networks: Bayesian pruning has also been applied to recurrent networks, including pruning individual LSTM weights or sparsifying recurrent neurons and gates.Reported applications include text classification and language modeling.

3.8 Other selection schemes

Other selection schemes choose removable network elements using evolutionary search, sensitivity estimates, diversity objectives, quantization error, or knockoff features. These approaches target different notions of redundancy and importance.

  • Evolutionary methods: Genetic pruning evolves populations of pruned networks through mutation, reproduction, and crossover, rewarding smaller models and improved generalization.The paper notes that this approach is not practical for modern large networks.
  • Sensitivity-based methods: Empirical-sensitivity methods estimate the relative importance of incoming edges from a data subset and use the resulting distribution to form sparse weight matrices.The proposed algorithm includes theoretical guarantees tied to the sparsity level.
  • Diversity-based methods: Diversity networks use Determinantal Point Processes to select diverse neurons and fuse similar neurons without requiring fine-tuning.The method starts from fully trained networks.
  • Quantization-based methods: Quantized-network pruning uses the distance between quantized and full-precision weights, interpreted as rounding error, to select filters for removal.The approach applies to binary or quantized networks.
  • Knockoff-based methods: Knockoff-feature pruning compares filter sensitivities to real inputs and independently generated inputs, removing features more sensitive to knockoffs than to real examples.The knockoffs are designed to match the training distribution while being independent of the label.

3.9 Parameter budgets between different layers

Parameter budgets can be assigned globally or per layer, with later methods adapting budgets automatically during training. Across ResNet-50 methods, global schemes commonly retain more parameters in earlier layers than later ones.

  • Sparsification hyperparameters may be selected per layer/operator or globally across the model.
  • Earlier methods often used global selection, but heterogeneous layer types motivated treating layers differently.
  • Later approaches automatically determine layer budgets from layer properties or redistribute them during training using saliency metrics.
  • Global schemes that balance parameters across layers often assign more parameters to earlier layers than later ones.
  • Tuning sparsity across layers is an important practical consideration, and practitioners sometimes disable pruning in the first layers for higher accuracy.

3.10 Literature overview

The literature overview classifies sparsification papers by candidate element, selection method, and training or inference use. Weight and neuron sparsification dominate, with magnitude pruning especially prominent and inference receiving more attention than training.

  • The survey classifies papers by removed candidate element, selection method, and whether sparsification targets training or inference.Candidates include neurons, weights, filters, transformer heads, hidden dimensions, and inputs.
  • Nearly 50% of surveyed papers focus on weight sparsification, closely followed by neuron sparsification.
  • More than 60% of papers focus on inference, while training-focused sparsification has recently gained popularity.Inference work commonly prunes neurons, weights, or filters; training work largely focuses on weights.
  • Inference is dominated by regularization and magnitude pruning, whereas training focuses primarily on magnitude-based methods.
  • 50% of works focus on either magnitude pruning or regularization, with magnitude pruning most often applied to weights.The broader classification again shows weight pruning by magnitude followed by sensitivity-based neuron pruning.

4 DYNAMIC PRUNING: NETWORK REGROWTH DURING TRAINING

Dynamic pruning removes and regrows network elements during training, navigating possible architectures while maintaining approximately constant model size. Regrowth strategies range from random activation to gradient-, layer-, and loss-based selection, each with different overheads or constraints.

  • Fully-sparse training removes elements and re-adds others to keep the model approximately the same size.This process resembles architecture search through a space of possible model architectures.
  • Randomly activating new weights creates a random walk in model space and can eventually produce power-law graphs through preferential attachment.
  • Layer-wise regrowth can improve accuracy by preferentially adding parameters to layers that retain more parameters after pruning.
  • Uniformly growing filters or neurons with a width multiplier has been effective within iterative grow/prune methodologies.
  • Gradient-based regrowth identifies important absent weights but substantially increases memory and computation overhead.Keeping dense weight copies or computing gradients for zero weights can reduce some benefits of sparse computation.
  • Layer-by-layer gradient computation reduces storage overhead but may lower accuracy because instantaneous gradients are noisy.

5 EPHEMERAL SPARSIFICATION APPROACHES

Ephemeral sparsification exploits transient zeros in activations or gradients during training and inference. ReLU activations can be highly sparse, regularization can increase activation sparsity, and gradient sparsification can reach very high sparsity for distributed training.

  • 5.1 Sparsifying neuron activations: ReLU layers naturally produce sparse activations, with some layers reaching up to 90% sparsity.This exceeds the approximately 50% sparsity intuitively expected from random inputs.
  • 5.1 Sparsifying neuron activations: Activation sparsity can reduce memory consumption during training and support compressed activation maps for computational gains.
  • 5.1 Sparsifying neuron activations: L1 regularization increased activation sparsity by up to 60% relative to naturally occurring sparsity on ImageNet CNNs.
  • 5.1 Sparsifying neuron activations: Hoyer regularization provided higher activation sparsity with lower accuracy loss than L1 regularization.
  • 5.2 Dropout: Dropout randomly zeros neurons during training to prevent co-adaptation and improve generalization.Variants instead drop weights, feature maps, or other structured components.
  • 5.3 Gradient sparsification: Gradient sparsification primarily compresses distributed-training communication by removing small gradients, with Lin et al. achieving up to 99.9% sparsity.The survey reports gradients as more amenable to sparsity than weights or activations.

6 SPARSE DEEP LEARNING ARCHITECTURES

Pruning methods for CNNs and transformer networks increasingly achieve high sparsity while preserving accuracy, but effective strategies depend on sparsity level, architecture, and training procedure. Iterative, adaptive, and first-order methods often outperform simpler approaches at extreme sparsity, while structured pruning remains constrained by which components dominate model size.

  • CNN architectures: CNN pruning has jointly improved compression and accuracy over time, with magnitude-based methods dominating reviewed work.At 0–90% sparsity, iterative magnitude pruning performs relatively well; beyond 90%, regularization, first-order, and second-order methods offer better sparsity–accuracy tradeoffs.
  • CNN architectures: 35× and 49× reductions in AlexNet and VGG-16 size preserved accuracy through magnitude pruning, quantization, weight sharing, and Huffman coding.The models were sparsified by more than 90% with manually tuned layerwise sparsity, while convolutional layers were pruned less aggressively.
  • CNN architectures: 74% sparsification improved DeepID2+ accuracy by 33%, while up to 88% sparsification retained the same accuracy.Dense-trained, then sparsified networks outperformed fully sparse training in this study.
  • CNN architectures: 90% sparse ResNet-50 reached 72.3% accuracy while reducing required computations by 2.7×–5.6×.A fully sparse training schedule also achieved similar performance with a 95% sparse VGG on CIFAR-10.
  • CNN architectures: Inference studies estimate 50–80% operation savings for sparsified ResNet-50 models, with potential speedups of up to 5× without significant accuracy loss.The estimate concerns computational intensity measured by inference flop count.
  • Transformer architectures: For transformers, structured head pruning performs well at low sparsity but degrades when important heads are removed, whereas unstructured magnitude pruning performs well at 40–80% sparsity.At above 90% sparsity, movement pruning is the reported method achieving high accuracy on MNLI.

7 SPEEDING UP SPARSE MODELS

Sparse models do not automatically run faster: practical gains depend on sparsity structure, storage overhead, hardware support, and specialized implementations. Structured formats and dedicated accelerators can convert sparsity into substantial inference and training speedups, but excessive or unsupported sparsity can reduce performance.

  • Current frameworks and hardware do not always execute sparse networks faster than dense networks, especially with unstructured sparsity.Some structured patterns map directly to optimized dense operations, whereas other formats require framework support.
  • Sparse representations trade index-management overhead against computation reduction, creating an intermediate CPU sparsity range with the best performance.Too little sparsity increases management overhead, while too much increases index-storage costs and underutilizes compute units.
  • Structured formats reduce storage overhead by encoding blocks or fixed patterns instead of individual non-zero offsets.Blocked formats can reduce storage overhead by a factor of B; Ampere uses bitmaps for blocks of four at 50% sparsity, while strided formats are more compact but constrained.
  • Mobile inference implementations exploit weight and block sparsity by optimizing caches, data movement, and SIMD-aligned computation.XNNPACK targets 70–95% sparsity, while Scalpel aligns weight blocks with SIMD width and lowers the sparsity needed for speedups on ARM and Intel CPUs.
  • Balanced block and neuron sparsity improve parallel execution by reducing load imbalance, although larger blocks can degrade top-5 accuracy at 60–77% sparsity.Block-balanced pruning keeps blocks at the same sparsity ratio, while fixed neuron degree creates balanced row and column sparsity.
  • Specialized accelerators exploit weight, activation, or predicted output sparsity to improve inference and training efficiency.Reported accelerator designs achieve 2–15x speedups and 2–10x lower energy, while sparse training designs report up to 4x speedup, 3.36x energy efficiency, or 40% training acceleration.

8 DISCUSSION

The discussion synthesizes evidence that sparse networks can be highly parameter-efficient, while emphasizing that pruning outcomes depend on architecture, training strategy, hardware, and evaluation scope. It also identifies unresolved theoretical questions and methodological limits that complicate comparisons.

  • Empirical observations: Sparse models generally outperform dense models at the same parameter budget, but reported pruning rates should account for over-parameterization through parameter efficiency.Some studies also report sparse models outperforming larger dense models; switching architectures may sometimes be more efficient than pruning.
  • Training and pruning strategies: Winning-ticket results show that subnetworks can reach accuracy comparable to the original network, although findings vary with network scale, initialization, and rewinding strategy.For larger networks, early optimization or learning-rate rewinding can matter, while original initialization alone often does not improve final accuracy.
  • Hardware and structure: Unstructured pruning often preserves better accuracy per element, whereas structured pruning provides substantially higher computational performance on modern devices.The practical choice therefore depends on whether the priority is parameter-level accuracy or hardware-executable speed.
  • Training and pruning strategies: Sparse training is sensitive to network structure and optimization settings, and sparsification can increase the loss function’s Lipschitz constant, complicating optimization.Momentum may help but can require more iterations; pruning steps may also help escape local minima under well-tuned schedules.
  • Benchmarks and limitations: Comparisons remain difficult because experimental setups vary widely and achieved sparsity correlates with the attention different architectures have received.The authors recommend standardized benchmarks and reporting meaningful tasks such as CIFAR-100 and ImageNet alongside calibration tasks such as MNIST.

9 CHALLENGES AND OPEN QUESTIONS

The field faces open questions spanning sparse training, model structure, hardware co-design, evaluation objectives, theory, representations, method transfer, data-free sparsity, and fairness. These challenges concern how to preserve accuracy and efficiency while extending sparsity across architectures and deployment settings.

  • Sparse training: Sparse training must determine whether gigantic models can be trained within hardware budgets without accuracy loss.
  • Structure and hardware: Structured-sparsity research must clarify accuracy, performance, and model-size tradeoffs while co-designing hardware and pruned models.
  • Optimization objectives: Multi-objective pruning must optimize competing goals such as minimizing energy consumption for a fixed memory size.
  • Theory and representations: Open theoretical questions connect sparsity with learning dynamics, generalization, and the representational power of sparse networks.
  • Generalization and robustness: The field must test whether pruning methods generalize across architectures, datasets, and settings, including data-free methods and fairness concerns.
  • Scope: The survey focuses on engineering aspects of sparsity, while biological analogies primarily serve as inspiration rather than a separate set of research challenges.

10 CONCLUSIONS AND OUTLOOK

The authors conclude that sparsity already offers substantial theoretical efficiency gains and that larger networks may provide greater pruning opportunities. They identify very high sparsity and sparse training of large models as promising directions for future systems.

  • 10–100x theoretical efficiency improvement is already possible through sparsity.
  • Larger networks appear to offer more opportunity for pruning, suggesting that compression potential may continue as architectures grow.
  • Very high sparsity (>99%) and sparse training of large models in high-dimensional spaces remain promising areas for future breakthroughs.
Loading 2102.00554v1…