What is the State of Neural Network Pruning?

Davis Blalock; Jose Javier Gonzalez Ortiz; Jonathan Frankle; John Guttag

Source-linked AI summary

What is the State of Neural Network Pruning?

Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, John Guttag

arXiv:2003.03033v1Mar 6, 2020cs.LGstat.ML

TL;DR

Large neural networks create resource demands, while pruning research lacks standardized benchmarks and metrics for reliable comparison. The paper surveys the literature, analyzes pruning practices and models, and introduces ShrinkBench for standardized evaluation. It finds that pruning can substantially compress networks without reducing accuracy, but comparisons remain constrained by inconsistent experimental practices and imperfect evaluation proxies.

Problem
Inconsistent benchmarks, metrics, baselines, and experimental settings make it difficult to compare pruning techniques or assess progress in the field.
Method
The paper conducts a meta-analysis of 81 pruning papers, analyzes existing practices and hundreds of models, and introduces ShrinkBench for standardized evaluation.
Results
Pruning can substantially compress networks without reducing accuracy, while many methods outperform random pruning for large amounts of pruning.
Takeaways & Limitations
Standardized best practices and ShrinkBench are proposed to make pruning evaluations more comparable and help answer questions about relative technique efficacy.
Takeaways & Limitations
The meta-analysis necessarily focuses on parameter and FLOP counts and image-classification accuracy, which are imperfect proxies for deployment outcomes and broader tasks.

Abstract

from arXiv · show

Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.

1 INTRODUCTION

Deep neural networks often require substantial computation and memory, making deployment difficult in resource-constrained environments. This paper surveys pruning research and finds that inconsistent evaluation practices prevent reliable comparisons, motivating standardized evaluation through ShrinkBench.

Deep neural networks’ computation and memory requirements increase infrastructure costs and hinder deployment on mobile phones and smart devices.
Neural network pruning removes parameters from an existing accurate network to produce a smaller network with similar accuracy.
The study surveys 81 recent pruning papers to investigate accuracy-efficiency tradeoffs, architecture- and dataset-specific strategies, and high-level design choices.
Magnitude-based pruning substantially compresses networks without reducing accuracy, while many pruning methods outperform random pruning.
Inconsistent methodologies and sparse comparisons make the literature’s motivating questions impossible to answer reliably.A quarter of papers compare with no other pruning method, and half compare with at most one other method.
The paper recommends best practices and introduces ShrinkBench, an open-source library providing standardized pruning primitives, models, datasets, and training routines.

2 OVERVIEW OF PRUNING

Pruning typically trains a network, scores and removes parameters, then fine-tunes the remaining model, often repeating this cycle. Its benefits must be evaluated along an efficiency-quality tradeoff, but common metrics are imperfect proxies for deployment outcomes.

2.1 Definitions: Neural network pruning maps a model f(x; W) to a new model f(x; M ⊙W ′), where a binary mask removes or zeroes selected parameters.
2.2 High-Level Algorithm: The standard pruning cycle trains a network to convergence, scores parameters or structures, prunes them, and fine-tunes the result to recover accuracy.
2.2 High-Level Algorithm: Pruning methods vary in sparsity structure, scoring, scheduling, and fine-tuning, including unstructured parameter removal and structured removal of neurons, filters, or channels.
2.3 Goals of Pruning: Pruning goals include reducing storage, computation, or energy, and each goal favors different design choices and evaluation metrics.
2.4 Evaluating Pruning: Pruning increases efficiency while typically decreasing quality, so methods are best characterized by efficiency-quality curves rather than single pruned models.
2.4 Evaluating Pruning: Parameter and FLOP counts are loose proxies for real-world latency, memory usage, and power consumption, while image classification covers only one class of tasks.

3 LESSONS FROM THE LITERATURE

The surveyed literature contains repeated evidence that pruning can compress models with little accuracy loss and can outperform random pruning, while sparse models may outperform dense ones at equal parameter counts. Results also suggest that pruning improves an architecture’s tradeoff, but switching to a better architecture often helps more.

The meta-analysis aggregates results from 81 papers selected through literature searches and citation-based tracing of comparisons between pruning papers.
3.2 How Effective is Pruning?: Pruning can significantly compress models with little or no accuracy loss, and small amounts of compression can sometimes increase accuracy.
3.2 How Effective is Pruning?: For large amounts of pruning, many pruning methods outperform random pruning, although this does not always hold for small amounts.
3.2 How Effective is Pruning?: Sparse models tend to outperform dense models with the same number of parameters, and pruned models can sometimes exceed the accuracy of their original dense models.
3.3 Pruning vs Architecture Changes: Reported ImageNet results compare pruned-model accuracy and compression or speedup with unpruned architectures plotted as family-level curves.
3.3 Pruning vs Architecture Changes: Pruning can improve an architecture’s time-or-space versus accuracy tradeoff, but switching to a better architecture generally helps more.

4 MISSING CONTROLLED COMPARISONS

Pruning research rarely supports controlled comparisons because papers use fragmented datasets, architectures, metrics, and operating points. This fragmentation, together with uncontrolled experimental variables, makes it difficult to determine which methods advance the state of the art.

Omission of Comparison: Pruning papers rarely make direct, controlled comparisons, leaving many methods untested against one another.More than a fourth of the corpus compares with no prior pruning method, another fourth compares with only one, and nearly all compare with three or fewer.
Dataset and Architecture Fragmentation: 81 papers used 49 datasets, 132 architectures, and 195 dataset–architecture combinations, while the most common combination appeared in only 22 papers.Three of the six most common combinations involved MNIST, whose results may be unlikely to generalize to other image-classification datasets.
Metrics Fragmentation: Reported results use varied metrics and operating points, making methods difficult to compare and revealing few consistent trends beyond an efficiency–accuracy tradeoff.Methods from later years do not consistently outperform earlier methods, and nearby results are often incomparable because papers omit standard deviations or other measures of central tendency.
Incomplete Characterization of Results: Most papers evaluate at most three dataset–architecture pairs and often use a single point to characterize each efficiency–accuracy curve.This incomplete characterization, combined with fragmented experimental choices, leaves different methods rarely directly comparable.
Confounding Variables: Meaningful comparisons remain difficult even with matched datasets, models, metrics, and operating points because few studies reuse the compared methods’ code and average across random variation.The cited practice is described as exceptionally rare, with Gale et al. (2019) and Liu et al. (2019) identified as arguable examples.
Confounding Variables: Fine-tuning choices can produce nearly as much variability as different pruning methods, while initial-model differences remain a confounder even when pruning methods are identical.Reported accuracy changes are often less than 1%, so small confounding effects can substantially affect which method appears superior.

5 FURTHER BARRIERS TO COMPARISON

Beyond fragmented experimental choices, pruning papers often describe architectures and metrics ambiguously. These presentation practices make reported results difficult to reproduce and compare even when nominally similar experiments are used.

Architecture Ambiguity: Architecture names such as “ResNet-50” may not identify a unique network because multiple ResNet variants exist.VGG-16 reports are especially ambiguous because papers often use custom variants with altered fully connected layers, dropout, or batch normalization.
Architecture Ambiguity: Some papers fail to specify their model precisely, and at least one reported Lenet-5-Caffe architecture conflicts with the available model files and documentation.Other examples include an unresolved circular reference and a nonexistent “VGG-S” model name in the Caffe Model Zoo.
Metric Definitions: “Pruned%” can mean either the fraction of parameters or FLOPs remaining or the fraction removed, while “compression ratio” is also widely misused.The same metric label can therefore encode different quantities across papers.
Metric Definitions: Reported “speedup” may denote wall time, a FLOPs ratio, a more complex formula, or an unspecified quantity.Even FLOPs are measured inconsistently, with up to a factor of four variation reported for the same architecture and dataset.

6 SUMMARY AND RECOMMENDATIONS

Existing pruning studies often lack the standardized experimental choices needed for reliable comparison. The authors recommend broader, more rigorous reporting and controls to enable direct evaluation of pruning methods.

Studies often use too few dataset–architecture combinations and too few operating points to characterize efficiency–accuracy tradeoffs.
Existing work makes pruning methods difficult or impossible to compare because setups, metrics, baselines, and confounding controls are inconsistent.
Researchers should specify architectures, datasets, and metrics; use at least three modern dataset–architecture pairs; and report compression ratio and theoretical speedup.
Whenever feasible, report means and sample standard deviations rather than one-off measurements.
Comparisons should use identical libraries, data loading, and other code to the greatest extent possible.
Reviewers should demand substantially greater rigor from papers claiming improved neural-network pruning methods.

7 SHRINKBENCH

ShrinkBench provides standardized tools for implementing and evaluating pruning methods across models, datasets, and experimental conditions. Its evaluations expose how metrics, models, pruning amounts, random seeds, and initial weights can alter apparent method rankings.

ShrinkBench standardizes training, pruning, fine-tuning, metric computation, and plotting with pretrained models and datasets.
The framework supports arbitrary scoring functions, layer allocations, and sparsity structures through pruning callbacks and standardized evaluation routines.
Baselines: The authors implemented global and layerwise magnitude, gradient-magnitude, and random pruning as baseline heuristics.
Evaluation: Over 800 networks were pruned across varying datasets, architectures, compression ratios, initial weights, and random seeds.
Avoiding pruning pitfalls: Reporting only parameter or FLOP reduction can change the relative ranking of pruning methods because metric correlations differ by method.
Avoiding pruning pitfalls: Method performance is not constant across models, datasets, and pruning amounts; Global Gradient reverses its ranking relative to other methods across CIFAR-VGG and ResNet-56.
Avoiding pruning pitfalls: At compression 16, different random seeds produced accuracies of 0.88 versus 0.61 for Global Gradient because of minibatch-selection randomness.
Avoiding pruning pitfalls: Different initial models produce different pruning tradeoff curves and can make one method appear better even when the initial model is controlled.

8 CONCLUSION

The paper finds that inconsistent baselines and experimental settings prevent confident comparisons of pruning techniques. It proposes best practices and ShrinkBench to support standardized evaluations and more comparable results.

Missing baselines and inconsistent experimental settings make it impossible to assess the state of the art or confidently compare pruning techniques.
The paper offers best practices and ShrinkBench to help future pruning research produce more comparable results.

A CORPUS AND DATA CLEANING

The corpus combines literature searches and conference proceedings, standardizes comparable metrics where possible, and acknowledges scope and extraction limitations.

81 papers were selected through cited-paper searches and reviews of recent NeurIPS, ICML, CVPR, ECCV, and ICLR proceedings.
Reported Top-1 error rates were converted to Top-1 accuracies, and fractions of parameters pruned were converted to compression ratios when possible.Size metrics cannot be converted into speedup metrics because parameter computation depends on layer placement.
The analysis focuses on model size reduction and theoretical speedup rather than metrics such as power, memory bandwidth, or fine-tuning time.The authors identify this metric selection as a limitation.
The authors report that reading hundreds of pages of dense technical content may have introduced isolated errors.They invite corrections and direct readers to the arXiv version for the most up-to-date revision.

B CHECKLIST FOR EVALUATING A PRUNING METHOD

The checklist emphasizes reproducible, well-contextualized pruning evaluations with broad compression results, explicit baselines, implementation detail, and appropriately bounded claims.

For all pruning results presented: Methods should be contextualized against magnitude pruning, recent techniques, and pre-2010s techniques.
For any pruning technique proposed: Claims about a technique should be restricted to the experiments actually presented, such as CIFAR-10, ResNets, or image-classification tasks.
For all results: Results should span compression ratios, include raw accuracy, report repeated runs with error bars, and provide FLOP counts when efficiency claims are made.
For all pruning results presented: Pruning results should compare against random pruning, magnitude pruning, and other relevant state-of-the-art techniques.The checklist also calls for matching layerwise proportions where applicable and describing how comparisons were produced.
For any pruning technique proposed: Pruning algorithms, subroutines, hyperparameters, comparison procedures, architectures, datasets, and training settings should be described well enough for reimplementation.The checklist additionally asks for source-code links and descriptions of relevant differences from cited architectures.

C EXPERIMENTAL SETUP

ShrinkBench fixes dependency random seeds and uses specified pruning and fine-tuning procedures, including repeated CIFAR-10 runs and early stopping.

ShrinkBench fixes random seeds for PyTorch, NumPy, and Python to support reproducibility.
Three independent runs with different random seeds were performed for every CIFAR-10 experiment.The authors observed variance across methods relying on randomness.
Pruning starts from pretrained weights, followed by fine-tuning with early stopping when validation accuracy repeatedly decreases.
CIFAR-10: CIFAR-10 fine-tuning uses 30 epochs, Adam, an initial learning rate of 3 × 10−4, and a fixed learning-rate schedule.
ImageNet: ImageNet fine-tuning uses batch size 256, 20 epochs, SGD with Nesterov momentum of 0.9, an initial learning rate of 1 × 10−3, and a fixed schedule.

D ADDITIONAL RESULTS

ShrinkBench reports results across four CIFAR-10 architectures and one ImageNet architecture, using accuracy-versus-compression and accuracy-versus-theoretical-speedup views.

ShrinkBench results cover CIFAR-VGG, ResNet-20, ResNet-56, and ResNet-110 on CIFAR-10, plus ResNet-18 on ImageNet.The CIFAR-10 plots include standard deviations across three random runs as error bars.
Accuracy across compression levels: Figures 9, 11, 13, 15, and 17 plot accuracy across several compression levels for CIFAR-VGG, ResNet-20, ResNet-56, ResNet-110, and ResNet-18, respectively.
Accuracy versus theoretical speedup: Figures 10, 12, 14, 16, and 18 plot accuracy against theoretical speedup for CIFAR-VGG, ResNet-20, ResNet-56, ResNet-110, and ResNet-18, respectively.