PaperlayerView exact revision on arXiv

Source-linked AI summary

Pruning Neural Networks at Initialization: Why are We Missing the Mark?

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin

arXiv:2009.08576v2cs.LGcs.CVcs.NEstat.ML

TL;DR

Pruning at initialization promises lower training costs, but existing methods remain below post-training magnitude pruning. The paper evaluates these methods and finds that their subnetworks are insensitive to within-layer mask shuffling and weight reinitialization, highlighting unresolved challenges for initialization-time pruning.

  • Problem

    Researchers seek to prune before or early in training to reduce training costs, but it remains unclear how initialization-time methods compare with baselines and post-training pruning.

  • Method

    The paper evaluates SNIP, GraSP, SynFlow, and magnitude pruning at initialization against random and post-training baselines, including mask-shuffling and weight-reinitialization ablations.

  • Results

    Initialization-time methods generally outperform random pruning but remain below post-training magnitude pruning, while their accuracy is preserved when masks are shuffled within layers or weights are reinitialized.

  • Takeaways & Limitations

    Existing initialization-time methods require trading reduced training cost for some accuracy, motivating tradeoff-free approaches to training-cost reduction.

  • Takeaways & Limitations

    It remains unclear whether insensitivity to ablations reflects these methods specifically or broader limitations of pruning at initialization.

Abstract

from arXiv · show

Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.

1 INTRODUCTION

Pruning at initialization could reduce training costs, but existing methods are evaluated against baselines and generally remain below post-training magnitude pruning. Their accuracy is also insensitive to which weights are pruned or reinitialized, raising broader questions about initialization-time pruning.

  • Motivation: Pruning early or before training could reduce the computational and environmental costs of training.The motivation includes reducing the cost of finding pruned networks and enabling exploration at larger scales.
  • Motivation: Lottery-ticket results suggest that subnetworks found early in training can reach full accuracy at sparsities comparable to post-training pruning.This motivates searching for subnetworks that remain sparse throughout much or all of training.
  • Findings: SNIP, GraSP, SynFlow, and magnitude pruning generally outperform random pruning, but no single initialization-time method is best across all settings.SNIP performs consistently well, magnitude pruning is surprisingly effective, and GraSP and SynFlow become more competitive with improvements.
  • Findings: Magnitude pruning after training outperforms the initialization-time methods, motivating investigation of broader limits on pruning at initialization.The paper compares methods near sparsities where post-training magnitude pruning matches full accuracy.
  • Ablations: The produced subnetworks retain accuracy when pruning masks are shuffled within layers or unpruned weights are randomly reinitialized.These ablations suggest that the methods identify layer-level pruning proportions rather than specific connections or weight values.
  • Open questions: These behaviors may reflect properties of initialization-time subnetworks or broader limits on achieving high accuracy while pruning before training.The paper frames this as an open question rather than a settled explanation.

2 RELATED WORK

Pruning research expanded from inference efficiency to training efficiency, while lottery-ticket work and new initialization-time methods motivated efficient discovery of trainable subnetworks.

  • Research trajectory: Pruning research initially focused on inference efficiency, then increasingly addressed training efficiency through gradual pruning during training.This shift reflects interest in reducing computational costs beyond inference.
  • Initialization-time pruning: Lottery-ticket studies and related work proposed subnetworks before or early in training that can reach high or full accuracy.SNIP, GraSP, SynFlow, and NTT are identified as initialization-time pruning methods.

3 METHODS

The paper defines pruning through scores and masks, compares early-pruning heuristics with random and post-training benchmarks, and evaluates them across image-classification models and sparsity regimes.

  • Framework: Pruning assigns binary masks to layer weights, with sparsity defined as the fraction of weights removed.A subnetwork uses the element-wise product of weights and masks.
  • Framework: Pruning methods score weights and convert those scores into masks at a target overall sparsity, either in one shot or iteratively.The scoring and removal operations are separated in the paper’s formalization.
  • Early-pruning methods: Random pruning assigns uniform random scores and removes the lowest-scoring weights, serving as a naive baseline for early pruning.It empirically prunes each layer to approximately the target sparsity.
  • Early-pruning methods: Magnitude pruning scores weights by absolute value, while SNIP scores |gℓ⊙wℓ| from sampled data and gradients.SNIP is applied in one iteration; magnitude pruning is used as an additional early-pruning comparison.
  • Early-pruning methods: GraSP uses Hessian-gradient products to score weights, whereas SynFlow uses data-independent synaptic-strength scores and prunes iteratively.SynFlow replaces weights with their absolute values and is designed to avoid layer collapse.
  • Benchmarks: Post-training magnitude pruning and lottery-ticket rewinding provide benchmarks for accuracy attainable with later information.The experiments use one-shot pruning for fair comparison, then retrain with the full learning-rate schedule repeated from the start.
  • Evaluation: Experiments cover image classification on CIFAR-10, TinyImageNet, and ImageNet using ResNet-20, VGG-16, ResNet-18, and ResNet-50.The study repeats experiments five times on CIFAR-10 and three times on TinyImageNet and ImageNet.

4 PRUNING AT INITIALIZATION

At initialization, early-pruning methods generally beat random pruning but vary by architecture and sparsity; none matches post-training magnitude pruning across accuracy or full-accuracy sparsity.

  • Matching sparsities: At initialization, SNIP, SynFlow, and magnitude pruning dominate depending on the network at matching sparsities.On VGG-16, SynFlow leads until 91.4% sparsity before SNIP overtakes it; on ResNet-18, SNIP dominates after 79.0% sparsity.
  • Matching sparsities: SNIP and SynFlow reach full accuracy on VGG-16 until 59% sparsity and on ResNet-18 until 89.3% and 79% sparsity, respectively.Random pruning reaches 20% in both cases, while post-training magnitude pruning reaches 93.1% and 96.5%.
  • Extreme sparsities: At extreme sparsities, GraSP overtakes other methods on ResNet-20 at 98.2% sparsity, while SynFlow performs best on ResNet-50.The ordering changes across ResNet architectures, unlike the reported VGG-16 pattern.
  • Overall comparison: No single initialization-time method is state of the art across all settings and sparsities.SNIP is consistently strong, SynFlow is frequently competitive, magnitude is effective, and GraSP improves at extreme sparsities.
  • Overall comparison: Initialization-time methods generally outperform random pruning but cannot match post-training magnitude pruning in accuracy or the sparsities where full accuracy is retained.The comparison is made at matching sparsities and the lowest extreme sparsities.

5 ABLATIONS AT INITIALIZATION

The ablations show that initialization-time pruning methods mainly determine how much to prune in each layer, rather than which individual weights to remove. This insensitivity contrasts with post-training pruning and raises questions about the heuristics’ mechanisms and performance ceiling.

  • Randomly shuffling: Layerwise shuffling preserves or improves accuracy for initialization-time pruning, indicating that pruning proportions—not individual connections—carry the useful information.This contrasts with post-training magnitude pruning, whose performance falls when its mask is shuffled.
  • Magnitude pruning: Magnitude pruning at initialization can perform competitively because layerwise initialization variances alone provide sufficient information for its shuffled-mask performance.These variances come from He initialization and depend on fan-in or fan-out.
  • SynFlow: SynFlow prunes entire convolutional channels unusually often, with neuron collapse affecting 31%, 52%, 69%, and 29% on ResNet-20, VGG-16, ResNet-18, and ResNet-50.Layerwise shuffling reduces these rates to 1%, 0%, 3.5%, and 13%, respectively.
  • SynFlow: SynFlow’s path-based scores can create neuron collapse because pruning one connection reduces other weights’ path contributions, making them more likely to be pruned next.The analysis also identifies heavy pruning of skip-connection weights, which participate in fewer paths.
  • Reinitialization: Reinitializing unpruned weights preserves accuracy for all early pruning techniques, unlike magnitude pruning after training and lottery ticket rewinding.The early methods are insensitive to whether they use the original or newly sampled initialization.
  • Inversion: GraSP’s inverted scores leave accuracy unchanged, and pruning its lowest-magnitude scores can improve accuracy, challenging the premise of its gradient-flow heuristic.By contrast, magnitude pruning, SNIP, and SynFlow lose accuracy when their purportedly most important weights are pruned.
  • Implications: The ablations leave unclear whether SNIP, GraSP, and SynFlow succeed because of their proposed hypotheses, while suggesting that initialization-time methods may remain in a lower performance stratum.The paper recommends using these and other ablations to test whether pruning heuristics behave according to their claimed justifications.

6 PRUNING AFTER INITIALIZATION

Pruning with magnitude, SNIP, and SynFlow after training improves accuracy compared with pruning at initialization, showing the initialization setting—not the methods alone—drives the performance gap. As training progresses, these methods also become more sensitive to pruning ablations.

  • 6 PRUNING AFTER INITIALIZATION: Pruning later in training distinguishes method-specific behavior from behavior specific to pruning at initialization.The experiment trains for k iterations, prunes with each technique, and then repeats the full learning-rate schedule.
  • 6 PRUNING AFTER INITIALIZATION: Random pruning reaches the same accuracy regardless of when it occurs, providing a control for the effects of later pruning.This control indicates that later pruning alone does not explain the accuracy improvements of the other methods.
  • 6 PRUNING AFTER INITIALIZATION: Magnitude, SNIP, and SynFlow improve as training progresses, with magnitude and SNIP approaching LTR and SynFlow close behind.The authors conclude that the performance gap is due to applying these methods at initialization rather than being intrinsic to the methods.
  • 6 PRUNING AFTER INITIALIZATION: Lottery-ticket subnetworks initially outperform the pruning methods, with magnitude pruning matching LTR only after 25K, 26K, and 36K iterations on ResNet-20, VGG-16, and ResNet-18.This pattern suggests that pruning may remain difficult for a substantial period after initialization.

7 DISCUSSION

The discussion finds that initialization-time pruning beats random pruning but remains below post-training magnitude pruning, while different methods show similar behavior under ablations. The authors frame this as a tradeoff between training cost and accuracy and identify unresolved limits and evaluation challenges.

  • State of the art: All initialization-time pruning methods surpass random pruning at some or all matching sparsities, and some retain full accuracy at non-trivial sparsities.The result varies across settings and sparsities.
  • State of the art: No single initialization-time method is best across networks, datasets, and sparsities.Each early pruning method achieves the highest accuracy in at least one setting.
  • State of the art: All initialization-time methods reach lower accuracy than magnitude pruning after training.This comparison motivates broader questions about the attainable performance of pruning at initialization.
  • The challenge of pruning at initialization: Methods using different signals—magnitudes, gradients, Hessian information, data, or no data—nevertheless reach similar accuracy and respond similarly to ablations.GraSP is the exception when comparing improvement after initialization.
  • Ablations: None of the examined initialization-time methods is sensitive to which weights are pruned or to their specific values, unlike post-training pruning methods.The authors leave open whether this reflects the methods or initialization-time pruning more generally.
  • Why is it challenging?: The authors do not identify why initialization-time pruning struggles, leaving optimization-phase effects and the generality of the challenge for future work.They specifically suggest that the initial phase of training may make selecting specific weights difficult.
  • Looking ahead: Reducing training cost with current initialization-time methods requires sacrificing some accuracy, motivating tradeoff-free pruning methods.The paper focuses on training-cost reduction rather than inference-cost reduction.
  • Measuring progress: Future evaluations should weigh sparsity benefits against accuracy decreases and compare overall training cost using measures such as area under the curve, FLOPs, time, or energy.The comparison becomes harder when methods prune at different sparsities and training steps.

CONTENTS OF THE APPENDICES

The appendices document experimental settings, method implementations, replications, and supplementary analyses for initialization-time pruning. They include network and dataset coverage, ablations, method variants, and comparisons between reported and reimplemented SNIP results.

  • Contents of the appendices: The appendices include plots for random pruning, magnitude pruning at initialization and after training, and lottery-ticket rewinding.These plots are provided as baseline comparisons.
  • Supplementary experiments: Supplementary experiments analyze shuffling and reinitialization after training, effective sparsity from disconnected weights, and comparisons using modified architectures.These analyses extend the main ablation and architecture evaluations.
  • Method variants: The GraSP appendix evaluates pruning the lowest-score, highest-score, and lowest-magnitude-score weights.These are the three GraSP variants studied.
  • Networks and datasets: The experiments cover LeNet-300-100 on MNIST, ResNet-20 and VGG-16 on CIFAR-10, ResNet-18 on TinyImageNet, and ResNet-50 on ImageNet.A modified ResNet-18 and modified TinyImageNet are included to match the SynFlow setup.
  • SNIP replication: SNIP scores weights using the absolute derivative of the loss with respect to a virtual parameter and prunes the lowest-scoring weights.The implementation rewrites the score as the gradient of a weight multiplied by the weight value.
  • SNIP replication: The SNIP replication uses ten randomly selected examples from each class and does not reinitialize the network after pruning.SNIP computation is performed on CPU before training the pruned network on TPU.
  • Results: Replication results average five random-seed replicates and more closely match the original SNIP results after pruning than before pruning.The authors note that differing unpruned-network accuracies make the comparison difficult to interpret definitively.
  • Results: The reimplemented unpruned networks can exceed the original SNIP accuracies, including a 1.9-percentage-point higher VGG-16 accuracy and 1.3–1.7-point higher wide-ResNet accuracies.The paper attributes some expected differences to implementation variation, especially for VGG-style CIFAR-10 networks.

B.5 RESULTS FROM GRASP PAPER

This section describes the GraSP replication setup, its gradient-flow-based scoring rationale, and comparisons with the original reported results.

  • Comparison basis: The study compares reported and reimplemented SNIP performance using the networks and datasets examined in the GraSP paper.The comparison is presented in Table 3, while Table 4 lists the evaluated networks and datasets.
  • GraSP method: GraSP aims to preserve gradient flow by pruning weights according to their estimated effect on the loss change after the first training step.The analysis defines ΔL using a learning-rate step and approximates pruning effects through a Hessian-gradient product.
  • GraSP method: GraSP interprets lower scores as more important and therefore prunes weights with the highest scores to maximize post-pruning gradient flow.The score sign indicates whether removing weights reduces or increases gradient flow.
  • Experimental setup: The replication follows the authors’ PyTorch implementation, scores parameters using sampled examples, and does not reinitialize after pruning.The replication uses ten randomly sampled examples per class, including 10,000 ImageNet examples.
  • Experimental setup: The ImageNet scoring batch is uncertain in the original implementation, which appears to use different sample counts across configuration files and code.The configurations suggest 1,000 examples, whereas the implementation uses 19,200 examples with uncertain class coverage.

C.4 RESULTS

The GraSP replication broadly matches the reported behavior, while SynFlow replication agrees in some settings but exposes substantial configuration sensitivity, especially for ResNet-18.

  • GraSP replication: The GraSP replication averages results across five random seeds and broadly matches the authors’ implementation despite modest baseline accuracy differences.VGG-19 is 0.7 percentage points lower, WRN-32-2 differs by 0.3 points, and ImageNet ResNet-50 is higher by 0.5 points.
  • GraSP replication: GraSP pruned-network differences generally track unpruned-network differences, but replicated ImageNet ResNet-50 accuracy is 0.6 to 1.1 percentage points lower.The authors attribute the ResNet-50 discrepancy potentially to different scoring examples, batch size, and learning rate.
  • SynFlow method: SynFlow iteratively removes the lowest-scoring remaining weights while progressively increasing sparsity toward a target.It uses 100 iterations in the replication and scores weights from an all-ones forward pass, summed logits, and weight gradients.
  • SynFlow method: SynFlow’s design seeks maximal critical compression by maintaining network connectivity until the most extreme feasible sparsity.The stated sufficient properties are iterative pruning, positive scores, and conservative layer-wise score sums.
  • SynFlow replication: VGG-11 and VGG-16 CIFAR-10 results are identical across implementations, but ResNet-18 replication required different hyperparameters and revealed an unknown implementation difference.On TinyImageNet, accuracy changed from 58.8% to 64%; on CIFAR-10, it changed from 95% to 93.6%.

D.4 RESULTS

This section reports SynFlow comparisons and connects its score function to local preservation of a ReLU network’s path norm.

  • Results: SynFlow is compared with random pruning, magnitude pruning at initialization, SNIP, and GraSP across sparsities from 0% to 99.9%.The replication samples additional logarithmically spaced sparsities beyond the original 13 settings.
  • Results: SynFlow’s ResNet-18 replication differs from the original in method ordering and extreme-sparsity performance, likely because the model or training configuration differs.The authors also identify a batch-normalization evaluation-mode bug in the original GraSP implementation as a possible source of differences.
  • Path norm connection: SynFlow’s weight score is identical to the local path-norm sensitivity expression, so its highest-scoring weight has the smallest infinitesimal effect on the path norm.The connection is established after making weights positive and evaluating the network on an all-ones input.
  • Path norm connection: The path norm sums products of connection weights over all input-to-output paths and can serve as a capacity measure for ReLU networks.It is a group norm associated with generalization bounds and is invariant to certain balanced weight rescalings.
  • Path norm connection: Removing a connection is modeled by deleting all paths through that connection, with the resulting path-norm decrease approximated by a first-order Taylor expansion.This supplies the local preservation interpretation for SynFlow pruning.
  • Baselines: The baseline comparison includes random and magnitude pruning at initialization, magnitude pruning after training, and lottery-ticket rewinding.Rewinding accuracy depends on the iteration used to restore the pruned network state.

F.1 ABLATIONS AT THE END OF TRAINING

Ablation results show that pruning at initialization is largely insensitive to weight shuffling or reinitialization, whereas pruning later in training becomes more sensitive to these changes.

  • Magnitude: Magnitude pruning after training loses accuracy when subnetworks are shuffled or reinitialized, confirming its dependence on learned weights.On ResNet-20, the ablations match full accuracy only through 49% sparsity versus 73.8% for the unmodified network.
  • SNIP: SNIP’s post-training ablations show network-dependent differences, indicating that its robustness is related to when pruning is applied rather than inherent.The unmodified network is slightly better on ResNet-20, similar to shuffling on VGG-16, and more distinct on ResNet-18.
  • GraSP: GraSP’s post-training ablations generally have limited effects, with shuffling sometimes improving accuracy at lower sparsities and declining at extreme sparsities.The strongest sensitivity appears on VGG-16 and ResNet-18, while ResNet-20 variants perform the same.
  • SynFlow: SynFlow ablations affect accuracy differently by network: shuffling improves ResNet-20 but decreases accuracy at higher sparsities on VGG-16 and ResNet-18.Reinitialization lowers accuracy in all three post-training networks described in the later-training analysis.
  • Pruning time: Magnitude pruning becomes ablation-sensitive early in training, while unmodified performance continues improving after the ablated variants plateau.Sensitivity begins after iteration 5,000 for ResNet-20, 10,000 for VGG-16, and 4,000 for ResNet-18.
  • Summary: Overall, initialization-pruned subnetworks maintain or improve accuracy under shuffling and reinitialization, unlike subnetworks pruned after initialization.The result supports difficulty in making initialization pruning connection-specific or initialization-sensitive.

J LAYERWISE PRUNING PROPORTIONS

Early pruning methods produce substantially different per-layer pruning proportions yet achieve similar accuracy, while performance remains limited across networks and settings.

  • Layerwise proportions: Different layerwise proportions yield similar accuracy: performance gaps stay below 1, 1.5, 2.5, and 1 percentage point on ResNet-20, VGG-16, ResNet-18, and ResNet-50.SynFlow prunes ResNet-20 early layers below 30% sparsity, whereas GraSP exceeds 60%; SNIP and SynFlow prune later layers more heavily.
  • Layerwise proportions: GraSP most closely matches magnitude pruning’s layerwise proportions on ResNets but is not the best-performing method at the highest matching sparsity.
  • Layerwise proportions: The diversity of layerwise proportions alongside universally limited performance suggests that choosing pruning fractions per layer is insufficient to match benchmark performance.
  • Skip connections: SynFlow prunes ResNet skip connections more heavily than other methods, while other methods preserve these layers at similar or lower sparsities than adjacent layers.The relevant skip connections are 1x1 convolutions used when ResNets downsample activation maps.
  • The output layer: All methods except random pruning prune output layers less than other layers, because these few weights directly control network outputs.
  • Additional settings: On modified TinyImageNet, magnitude pruning at initialization performs best at lower sparsities, GraSP at higher sparsities, and SynFlow matches no better than random pruning.Magnitude pruning after training matches only at much lower sparsities in this higher-accuracy benchmark.
  • Additional settings: Random shuffling maintains or improves accuracy across methods, while random reinitialization also maintains performance in the modified TinyImageNet experiment.SynFlow shows dramatic improvements after shuffling, and only magnitude pruning degrades when the initialization distribution is changed to fixed variance.
  • Additional settings: SynFlow’s neuron collapse is reduced by random shuffling, linking its strong shuffling gains with a change in network connectivity behavior.
Loading 2009.08576v2…