PaperlayerView exact revision on arXiv

Source-linked AI summary

The State of Sparsity in Deep Neural Networks

Trevor Gale, Erich Elsen, Sara Hooker

arXiv:1902.09574v1cs.LGstat.ML

TL;DR

Large-scale sparsification lacks representative benchmarks, and prior work disagrees about whether learned sparse architectures can be retrained from scratch. This paper evaluates competing methods and retraining claims on Transformer and ResNet-50, finding that complex methods are inconsistent at scale, magnitude pruning is comparable or better, and learned sparse architectures do not recover the same performance from scratch.

  • Problem

    The field lacks standardized large-scale benchmarks, while prior studies disagree about whether pruned sparse architectures can match optimized models when retrained from scratch.

  • Method

    The paper evaluates three sparsification techniques and random pruning on Transformer and ResNet-50, and replicates scratch and lottery-ticket experiments across sparsity levels.

  • Results

    Complex sparsification techniques perform inconsistently on large-scale tasks, while magnitude pruning achieves comparable or better results; scratch-trained learned sparse architectures do not match jointly sparsified models.

  • Takeaways & Limitations

    The results support large-scale benchmarks and provide open-source code, checkpoints, and hyperparameter results as baselines for future sparsification research.

  • Takeaways & Limitations

    The study cannot exclude techniques performing consistently well on other architectures and datasets beyond Transformer and ResNet-50.

Abstract

from arXiv · show

We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.

1. Introduction

Large neural networks create substantial training, deployment, storage, and computation costs, motivating sparsity as a compression strategy. The paper addresses inconsistent evidence by evaluating sparsification methods at realistic scale.

  • State-of-the-art models can contain tens of millions of parameters and require billions of floating-point operations per prediction.
  • Sparsity sets some parameters exactly to zero, allowing multiplications to be skipped and models to be stored compactly.
  • The sparsity literature lacks standardized, representative large-scale benchmarks, making relative merits of competing approaches difficult to assess.
  • Prior studies disagree about whether sparse architectures can be retrained from scratch with random initialization or require the original learned initialization.
  • The paper evaluates variational dropout, l0 regularization, and magnitude pruning on Transformer and ResNet-50, finding inconsistent large-scale performance for complex methods and comparable or better results from magnitude pruning.

2. Sparsity in Neural Networks

Prior sparsification research spans simple magnitude-based pruning, theoretically motivated methods, optimization-based approaches, and biologically inspired techniques. A central design choice is whether sparsity is unstructured or imposed on groups of weights.

  • Magnitude pruning removes small weights and has achieved high compression with limited accuracy loss and reduced computational complexity.
  • Bayesian and information-theoretic methods provide theoretical motivation while achieving high compression rates.
  • Second-order and first-order loss approximations seek to preserve model quality while reducing the computational cost of sparsification.
  • Reinforcement learning and biologically inspired approaches have also been used to prune weights or filters and induce sparse structures.
  • Unstructured sparsity offers flexibility but is harder to parallelize and support in software, creating a trade-off with structured sparsity between compression and practicality.

3. Evaluating Sparsification Techniques at Scale

The paper compares three sparsification methods and a random-pruning baseline across large-scale Transformer and ResNet-50 tasks. The evaluation combines extensive tuning with common training-step budgets.

  • The study evaluates magnitude pruning, variational dropout, l0 regularization, and random pruning on ImageNet ResNet-50 and WMT 2014 English-to-German Transformer models.
  • Magnitude pruning removes low-magnitude weights according to a gradual schedule and permits masked weights to reactivate during training.
  • Variational dropout learns per-parameter dropout rates so weights with high dropout rates can be removed after training.
  • l0 regularization uses stochastic hard-concrete gates to optimize a differentiable approximation of the number of non-zero weights.
  • Random pruning follows the same sparsity schedule as magnitude pruning but selects weights randomly and prevents their reactivation.
  • More than 4000 experiments were conducted with constant training steps and extensive hyperparameter tuning across techniques and models.

4. Sparse Neural Machine Translation

On the Transformer, the methods show broadly consistent relative performance across sparsity levels. Magnitude pruning is strongest at high sparsity and is faster under the shared training-step budget.

  • l0 regularization and variational dropout lead at low-to-mid sparsity, while magnitude pruning performs best for highly sparse Transformer models.
  • At 90% global sparsity, l0 regularization and variational dropout learn non-uniform layer distributions, whereas magnitude pruning uses a uniform distribution.
  • Despite learning non-uniform sparsity distributions, l0 regularization and variational dropout do not significantly outperform magnitude pruning and are inferior at high sparsity.
  • With equal training steps, magnitude-pruned Transformers train 1.24x faster than l0-regularized models and 1.65x faster than variational-dropout models.

5. Sparse Image Classification

On ResNet-50, sparsification methods vary substantially in effectiveness: variational dropout performs especially well at high sparsity, while l0 regularization fails to produce useful sparse models. Modified magnitude pruning can outperform variational dropout across most sparsity levels with lower resource requirements.

  • ResNet-50 Results & Analysis: l0 regularization failed to produce sparse ResNet-50 models without significantly damaging model quality.Across hundreds of experiments, models either retained full performance without sparsification or approached random-guessing performance after sparsification.
  • ResNet-50 Results & Analysis: Variational dropout maintained over 70% top-1 accuracy with less than 4% of standard ResNet-50 parameters.Its strongest results occurred in the high-sparsity range.
  • ResNet-50 Results & Analysis: Variational dropout reduced sparsity in ResNet-50’s first and last layers, learning a non-uniform layerwise distribution.Magnitude and random pruning instead used constant sparsity across layers; the first convolution contains only .037% of model parameters.
  • ResNet-50 Results & Analysis: Variational dropout required more resources, using over 2x the memory of magnitude pruning and 9.75 hours versus 12.50 hours on different accelerator counts.The reported comparison used 32 accelerators for variational dropout and 8 for magnitude pruning.
  • Pushing the Limits of Magnitude Pruning: Modified magnitude pruning outperformed variational dropout at all but the highest sparsity levels while using fewer resources.The modifications altered layerwise sparsity and increased training time.
  • Pushing the Limits of Magnitude Pruning: At 80% sparsity, modified magnitude pruning achieved 76.52% top-1 accuracy, .17% below the baseline and .41% above He et al. (2018).The result was reported as a new state-of-the-art sparsity-accuracy trade-off for ImageNet ResNet-50.

6. Sparsification as Architecture Search

At large scale, learned sparse architectures could not reproduce models trained with sparsification during optimization. Across Transformer and ResNet-50 experiments, this gap increased with sparsity, and the lottery-ticket phenomenon was not replicated.

  • Context and scope: The prior studies mainly used convolutional networks and small benchmark datasets, limiting evidence about large-scale and non-convolutional tasks.The cited prior ImageNet study examined only 30% and 60% sparsity and could not reproduce full accuracy for unstructured weight sparsity.
  • Experimental framework: The study repeats scratch and lottery-ticket experiments on Transformer and ResNet-50 across sparsity levels from 50% to 98%.The experiments compare learned sparse architectures with tuned magnitude-pruning models.
  • Experimental framework: The scratch experiments reinitialize final magnitude-pruning masks and train them normally or with up to twice as many training steps.The longer-training variant adjusts training to use approximately the same number of FLOPs as optimization with sparsification.
  • Scratch results: Across all experiments, training from scratch with a learned sparse architecture did not match the performance of models sparsified during optimization.This result holds for both Transformer and ResNet-50.
  • Scratch results: Doubling training steps improved scratch results but remained insufficient, while the performance gap from magnitude pruning increased as sparsity rose.Augmented initialization provided no benefit in either model.
  • Lottery-ticket results: The lottery-ticket experiments did not replicate Frankle and Carbin’s reported phenomenon at the scale and task complexity studied here.The authors identify model and task scale as a likely contributor to the discrepancy.

7. Limitations of This Study

The study’s conclusions are bounded by incomplete hyperparameter coverage and evaluation on only two selected architectures and datasets. Other settings may yield different results.

  • Hyperparameter exploration: Thousands of hyperparameter configurations were explored, but untested settings could allow some techniques to outperform those evaluated.The authors cannot eliminate this possibility because the space of settings is much larger than the practical search budget.
  • Neural architectures and datasets: Transformer and ResNet-50 represent diverse large-scale tasks, but the study cannot exclude consistently strong performance on other architectures or datasets.The authors call for broader evaluation in future work.

8. Conclusion

The paper finds that complex sparsification methods perform inconsistently at large scale, while simple heuristics can match or exceed them. It also challenges claims that pruned sparse architectures can be trained from scratch to match jointly optimized sparse models.

  • Three state-of-the-art sparsification techniques were extensively evaluated on two large-scale learning tasks.The techniques were variational dropout, l0 regularization, and magnitude pruning.
  • Complex techniques that perform well on small datasets behave inconsistently at scale, whereas simple heuristics achieve comparable or better results with less computation.
  • The study provides counterexamples to theories claiming pruned sparse topologies can be trained from scratch to match models sparsified during optimization.
  • Variational dropout approximates Bayesian inference by optimizing a parameterized posterior using a variational lower bound with a KL-divergence regularizer.
  • Variational dropout learns weight means and variances, then removes weights with the highest learned dropout parameters after training.
  • l0 regularization induces sparsity by reparameterizing weights with stochastic hard-concrete gates and directly optimizing the expected number of non-zero weights.

B. Variational Dropout Implementation Verification

The variational dropout implementation was reproduced on MNIST models and achieved results close to, and under some thresholds better than, the original report.

  • The implementation was verified on LeNet-300-100 and LeNet-5-Caffe using the original paper’s released hyperparameters.
  • 97.52% global sparsity and 98.42% test accuracy were achieved on LeNet-300-100.The original results reported 98.57% sparsity and 98.08% accuracy.
  • 98.5% global sparsity with 98.40% test accuracy was obtained by lowering the log α threshold to 2.0.
  • 99.1% global sparsity with 98.13% test accuracy was obtained at a log α threshold of 0.1, exceeding the originally published sparsity and accuracy.
  • 99.29% global sparsity and 99.26% test accuracy were achieved on LeNet-5-Caffe.Lowering the threshold to 2.0 produced 99.5% sparsity with 99.25% accuracy.

C. l0 Regularization Implementation Verification

The l0 regularization implementation was checked against a Wide ResNet experiment and produced accuracy close to baseline alongside comparable training-time FLOPs reduction.

  • The weight-level implementation was not directly comparable to the original group-sparsity method, so verification used Wide ResNet on CIFAR-10.
  • The implementation applied l0 regularization to the first convolutional layer in residual blocks using hard-concrete reparameterization settings aligned with the original experiment.
  • 95.34% test accuracy and a training-time FLOPs reduction comparable to Louizos et al. were achieved with an l0-norm weight of .0003.The baseline WRN-28-10 accuracy was 95.45%.
  • Errors were identified in the original WRN-28-10 FLOP calculations, and the authors contacted the original researchers to clarify the results.

D. Sparse Transformer Experiments

The sparse Transformer experiments varied pruning, regularization, and training schedules across many target sparsity levels. High sparsity exposed numerical instability and poor performance in some configurations.

  • Magnitude pruning experiments covered seven target sparsities from 50% through 98%, two pruning frequencies, and three regularization settings.
  • Variational dropout required extensive tuning of the KL-divergence coefficient, with weights in the range [ .1/N ] producing the target sparsity range.
  • The experiments compared constant, linear, and cubic regularizer-weight schedules for variational dropout and l0 regularization.
  • l0-norm weights in the range [ 1/N ] produced models in the target sparsity range, using fixed hard-concrete parameters and initialized log α of 2.197.
  • Random-pruning experiments explored early pruning schedules and the same hyperparameter combinations used for magnitude pruning.
  • At 98% target sparsity, only one tested configuration completed training without NaNs, while sparse first layers caused instability and low test performance.

E.3. Variational Dropout Details

The experiments examine sparsification methods and training schedules for ResNet-50, including variational dropout, l0 regularization, random pruning, and scratch training. Results show strong sensitivity to initialization for variational dropout, failure of l0 regularization, and schedule-dependent outcomes for extended training.

  • Variational Dropout: ResNet-50 variational dropout was highly sensitive to log σ2 initialization: -10 missed baseline accuracy, -20 produced no sparsity, and -15 produced good results.The authors contrast this sensitivity with their Transformer experience.
  • Variational Dropout: Across log α thresholds, ResNet-50 showed consistent accuracy and nearly identical sparsity; log α = 0.5 slightly outperformed the standard threshold of 3.0.The reported main-text results use a threshold of 0.5.
  • l0 Regularization: All ResNet-50 l0-regularization experiments failed to exceed random-guessing test performance, with severe accuracy loss when the l0-norm loss began to drop.Additional training and parameterization tweaks did not improve the results.
  • Experimental Coverage: The released data include results for all explored learning-rate schemes and l0-regularization hyperparameter combinations.These releases cover the tested configurations rather than only the main-text results.
  • Random Pruning: Random-pruning experiments varied pruning start and end times, frequencies of 2k, 4k, and 8k, and the use of label smoothing.Starting points were 0, 8k, and 20k; ending points were 40k, 68k, and 76k.
  • Scratch-b Training: For scratch-b ResNet-50 training, uniformly doubling the duration of all five learning-rate regions produced the best results by a wide margin.Four learning-rate schemes were explored for the extended 2x training time.
Loading 1902.09574v1…