PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu

arXiv:2106.10404v4cs.LGcs.CV

TL;DR

During-training pruning is less explored than post-training and before-training pruning, despite its potential for efficiency and performance. This paper studies pruning plasticity and introduces GraNet, which uses zero-cost neuroregeneration; on ResNet-50/ImageNet, its sparse-to-sparse version outperforms various dense-to-sparse methods without extending training time.

  • Problem

    During-training pruning is less explored, and a performance gap remains between it and full dense training despite its potential efficiency and performance.

    Source: §1 · p. 2

  • Method

    The paper studies pruning plasticity and proposes GraNet, a gradual pruning method that regenerates connections without increasing the parameter count.

    Source: §1 · pp. 2–3

  • Results

    GraNet advances dense-to-sparse and sparse-to-sparse training, with its ResNet-50/ImageNet sparse-to-sparse version outperforming various dense-to-sparse methods without extending training time.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    Pruning plasticity can be leveraged to boost sparse-training performance, and its observed effects generalize to final performance after continued training.

    Source: §1 · p. 3

Abstract

from arXiv · show

Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization). The former method suffers from an extremely large computation cost and the latter usually struggles with insufficient performance. In comparison, during-training pruning, a class of pruning methods that simultaneously enjoys the training/inference efficiency and the comparable performance, temporarily, has been less explored. To better understand during-training pruning, we quantitatively study the effect of pruning throughout training from the perspective of pruning plasticity (the ability of the pruned networks to recover the original performance). Pruning plasticity can help explain several other empirical observations about neural network pruning in literature. We further find that pruning plasticity can be substantially improved by injecting a brain-inspired mechanism called neuroregeneration, i.e., to regenerate the same number of connections as pruned. We design a novel gradual magnitude pruning (GMP) method, named gradual pruning with zero-cost neuroregeneration (\textbf{GraNet}), that advances state of the art. Perhaps most impressively, its sparse-to-sparse version for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods with ResNet-50 on ImageNet without extending the training time. We release all codes in https://github.com/Shiweiliuiiiiiii/GraNet.

1 Introduction

Post-training pruning can incur substantial retraining costs, while before-training methods may fall short of magnitude-pruning performance; during-training pruning offers early sparsity benefits but remains less understood. The paper studies pruning plasticity, examines neuroregeneration, and introduces GraNet, reporting state-of-the-art results in dense-to-sparse and sparse-to-sparse training.

1 Introduction

  • The paper frames post-training pruning as costly because it typically requires dense pretraining and repeated retraining, while before-training methods may not match post-training magnitude-pruning performance.During-training pruning can gain sparsity-related acceleration early while using training information, but its dynamics and performance gap relative to dense training are less explored.Source: §1 · pp. 1–2
  • The study measures pruning plasticity by testing whether pruned models recover original performance after short continued training, and considers both dense-to-sparse and sparse-to-sparse training.It proposes parameter-efficient connection regeneration during gradual pruning.Source: §1 · p. 2
  • Recovery depends on pruning rate: low-rate pruning such as 0.2 is readily recoverable in both training regimes, whereas removing too many parameters at once causes accuracy drops.The authors connect this pattern to iterative magnitude pruning, which typically repeats small pruning steps.Source: §1 · p. 2
  • Pruning plasticity declines as the learning rate falls, with recovery dropping significantly after the second learning-rate decay; the authors relate this to pruning schedules and rewinding results.They connect the finding to ending gradual magnitude pruning before the second decay, decreasing pruning rates in dynamic sparse training, and rewinding outperforming fine-tuning.Source: §1 · p. 2
  • Regenerating as many connections as were pruned markedly improves pruning plasticity, but adds memory and computational overhead that GraNet aims to avoid.The paper states that it provides an affirmative answer to whether neuroregeneration can be made efficient during training without extra costs.Source: §1 · p. 3
  • GraNet establishes state-of-the-art performance in dense-to-sparse and sparse-to-sparse training, with sparse-to-sparse gains over dense-to-sparse methods on ResNet-50/ImageNet without extended training time.The learned subnetworks are also reported as more accurate than those from an existing gradual pruning method.Source: §1 · p. 3

2 Related Work

Related work distinguishes pruning after training, during training, and before training by when sparsity is identified and how the network is trained. It also contrasts dense-to-sparse methods with dynamic sparse training, which evolves a sparse network's topology during training.

2 Related Work

  • Post-training pruning derives sparse networks from pretrained models using criteria such as weight magnitude, gradients, Hessians, or Taylor expansion, and commonly requires repeated pruning and retraining cycles.Low-rank decomposition can induce structured sparsity in channels or filters.Source: §2 · p. 3
  • Gradual magnitude pruning progressively sparsifies a dense network during training, alongside approaches using L0/L1 regularization or jointly trainable sparsity heuristics.These approaches are classified as dense-to-sparse training because they begin with a dense network.Source: §2 · p. 3
  • Dynamic sparse training starts from a randomly initialized sparse network and jointly optimizes its topology and weights, but typically trails dense-to-sparse accuracy without extended training.The passage points to surveys for further details.Source: §2 · p. 3
  • Before-training pruning seeks a trainable sparse network before main training, while cited analysis finds existing initialization methods identify layer-wise sparsity ratios rather than indispensable weight values and positions.The authors state that both mask positions and weight values matter for GraNet.Source: §2 · p. 3

3 Methodology for Pruning Plasticity

The section defines pruning plasticity as short-term recovery after pruning and studies it through controlled pruning and regeneration experiments. It reports that learning-rate decay limits recovery, while connection regeneration improves it.

3 Methodology for Pruning Plasticity

  • The experiments focus on pruning during standard training, excluding before-training and post-training pruning.Source: §3 · p. 3

3.1 Metrics

  • Pruning plasticity is the test-accuracy change after k continued-training epochs at the current learning rate; final performance gap compares the pruned model after the remaining schedule with the unpruned model.The fixed current learning rate is used to isolate pruning effects from learning-rate decay.Source: §3.1 · p. 4

3.2 Architectures and Datasets

  • Plasticity tests use VGG-19 and ResNet-20 on CIFAR-10; main-body experiments report ResNet-20 across four initial sparsities and four pruning rates.The initial sparsities and pruning rates are 0, 0.5, 0.9, and 0.98.Source: §3.2 · p. 4, §3.4 · p. 5

3.3 How to Prune, and How to Regenerate

  • The study tests one-shot magnitude pruning with zeroed weights, layer-wise or global sparsity, and gradient regeneration of as many largest-gradient connections as were pruned.Structured pruning removes groups or whole units; unstructured pruning retains individual weights and is discussed as a route to extreme sparsity.Source: §3.3 · pp. 4–5

3.4 Experimental Results

  • At learning rate 0.1, models recover or exceed their original performance except at pruning rate 0.98; plasticity falls after learning-rate decays, while regeneration helps, especially when over-pruned.Even with regeneration, pruning after a learning-rate drop still degrades plasticity.Source: Figure 2 · p. 5, §3.4 · p. 5
  • Mild pruning benefits the final performance of extreme-sparse models, regeneration further improves recovery, and the performance gap grows as the learning rate drops.The final-performance comparison is measured after continued training through the remaining learning-rate schedule.Source: §3.4 · pp. 5–6, Figure 3 · p. 6

4 Gradual Pruning with Zero-Cost Neuroregeneration

GraNet regenerates important connections during training in a parameter-efficient fashion, avoiding the parameter-count increase of naive regeneration.

4 Gradual Pruning with Zero-Cost Neuroregeneration

  • Naive regeneration increases parameter count, whereas GraNet regenerates important connections during training in a parameter-efficient fashion.The method is proposed as gradual pruning with zero-cost neuroregeneration (GraNet).Source: §4 · p. 6

4.1 Gradual Pruning

  • GraNet gradually prunes from initial to target sparsity over n iterations; it uses global pruning, while conventional methods leave pruned weights unupdated and unlikely to reactivate.The authors select global pruning because it generally performs better than uniform pruning.Source: §4.1 · p. 6

4.2 Zero-Cost Neuroregeneration

  • GraNet removes the smallest-magnitude connections and regenerates the same number using gradient magnitude, without increasing the connection count.GraNet can start dense or sparse; in its sparse-to-sparse setting it begins denser than fixed-sparsity methods and gradually prunes toward the target.Source: §4.2 · pp. 6–7

4.3 Experimental Results

  • On CIFAR-100, GraNet (si = 0) beats GMP at all sparsities and SIS by a large margin; on CIFAR-10, it improves over GMP except at 95% sparsity.For sparse-to-sparse training, GraNet (si = 0.5) beats RigL by 1.40% and 2.22% with VGG-19 on CIFAR-10 and CIFAR-100, respectively, at sparsity 0.98.Source: §4.3 · p. 7, Table 2 · p. 8
  • On ImageNet ResNet-50, GraNet (si = 0.5) reaches 76% test accuracy at sparsity 0.8 without extending training time, outperforming RigL by 0.9% at sparsity 0.8 and 1.5% at 0.9.GraNet (si = 0) also outperforms GMP with half the training FLOPs and has the highest accuracy among dense-to-sparse methods at sparsity 0.9.Source: Table 3 · p. 8, §4.3 · p. 9

4.4 Effect of the Initial Sparsity

  • With ResNet-50 on ImageNet and final sparsity fixed at 0.9, initial sparsities up to 0.5 yield higher accuracy and fewer feedforward FLOPs, while training FLOPs remain robust.The lower feedforward FLOPs balance the overhead from starting with a denser model.Source: §4.4 · p. 9, Table 4 · p. 9

4.5 Performance of GraNet at Extreme Sparsities

  • At extreme sparsities, GraNet uses lower or equal training FLOPs than RigL at 0.8 and 0.9, but more at 0.95 and 0.965, with accuracy gains above 2%.Gradual pruning lasts 30 epochs, followed by 70 epochs on the improved distribution to amortize early overhead.Source: §4.5 · pp. 9–10, Table 5 · p. 9

4.6 Ablation Study of Random Reinitialization

  • Randomly reinitializing GraNet’s pruned network with the same mask substantially reduces performance; retraining performance is higher for GraNet than GMP.This supports the finding that both connectivity and weight values matter for GraNet’s learned subnetworks.Source: §4.6 · p. 10, Figure 4 · p. 10

5 Conclusion, and Reflection of Broader Impacts

The paper studies during-training pruning through pruning plasticity and introduces GraNet, which performs cost-free connection regeneration. GraNet advances the state of the art in dense-to-sparse and sparse-to-sparse training.

5 Conclusion, and Reflection of Broader Impacts

  • The paper quantitatively studies during-training pruning through pruning plasticity and proposes a method based on its findings and neuroregeneration.Source: §5 · pp. 10–11
  • GraNet performs cost-free connection regeneration during training and advances the state of the art in both dense-to-sparse and sparse-to-sparse training.The paper identifies reducing ML energy costs and CO2 emissions as a potential broader impact of reducing training and inference resources.Source: §5 · p. 11
  • The authors identify reduced ML energy costs and CO2 emissions as a potential impact of reducing training and inference resources.Source: §5 · p. 11

A Remaining Experimental Results of Pruning Plasticity

This appendix reports pruning-plasticity experiments across structured and unstructured pruning settings on CIFAR-10. The figures compare recovery with and without connection regeneration and show final performance gaps.

A Remaining Experimental Results of Pruning Plasticity

  • Structured-pruning experiments pre-train ResNet-20 and VGG-19 at four sparsities, then remove filters at four pruning rates to measure plasticity.Filter pruning uses absolute weight sums for pruning and absolute gradient sums for regenerating zero filters.Source: A Remaining Experimental Results of… · p. 16

A.1 ResNet-20 on CIFAR-10 with Structured Filter Pruning

  • Figures 5–6 show ResNet-20 on CIFAR-10: Figure 5 tracks pruning plasticity during 30 epochs of continued training, with and without connection regeneration, using uniform pruning; Figure 6 reports the final gap between unpruned and pruned models.Source: Figure 5 · p. 16, Figure 6 · p. 16

A.2 VGG-19 on CIFAR-10 with Structured Filter Pruning

  • Figures 7–8 report structured-pruning results for VGG-19 on CIFAR-10: Figure 7 shows pruning plasticity during 30 epochs of continued training with and without connection regeneration using uniform pruning, while Figure 8 shows the final performance gap between unpruned and pruned models.Source: Figure 7 · p. 17, Figure 8 · p. 17

A.3 ResNet-20 on CIFAR-10 with Unstructured Uniform Pruning

  • Figures 9–10 show ResNet-20 unstructured uniform-pruning plasticity with and without regeneration and the final performance gap.Source: Figure 9 · p. 18, Figure 10 · p. 18

A.4 VGG-19 on CIFAR-10 with Unstructured Global Pruning

A.5 VGG-19 on CIFAR-10 with Unstructured Uniform Pruning

B Implementation Details of GraNet

The appendix provides GraNet's pseudocode and implementation details, including the initialization setting that distinguishes dense-to-sparse from sparse-to-sparse training and a table of experiment hyperparameters.

B Implementation Details of GraNet

B.1 Algorithm

  • GraNet initializes weights at sparsity si; dense-to-sparse training sets si = 0, while sparse-to-sparse training sets si > 0.The algorithm trains with SGD and applies gradual pruning and zero-cost neuroregeneration at scheduled steps between t0 and tf.Source: §B.1 · p. 21

B.2 Hyperparameters

  • Table 7 lists GraNet's experiment hyperparameters, including learning rate, batch size, epochs, sparsity, pruning schedule, and regeneration ratios.Source: §B.2 · p. 21, Table 7 · p. 21

B.3 Implementation

The implementation builds on existing open-source repositories and uses a specified approximation for training FLOPs.

B.3 Implementation

  • The implementation modifies the Sparse Momentum repository and adds VGG-19 with batch normalization from the GraSP repository.Source: §B.3 · p. 21
  • ResNet-50 ImageNet inference FLOPs are calculated using code modified from the rethinking-network-pruning repository.Source: §B.3 · p. 21
  • Training FLOPs follow RigL's approximation, with backward-pass FLOPs estimated at around twice the forward-pass FLOPs.Source: §B.3 · p. 21

C Implementation Details of GMP

The appendix gives GMP’s gradual-pruning schedule and reports its comparison with GraNet under matched CIFAR-10/100 hyperparameters. It also notes that GraNet requires fewer training FLOPs than GMP’s reported result, while offering a possible explanation.

C.1 Algorithm

  • GMP gradually increases sparsity during training until the target is reached, applying a pruning rate at each iteration.The pseudocode initializes weights at the initial sparsity, trains with SGD, and prunes between t0 and tf every ΔT steps.Source: §C.1 · p. 22

C.2 Hyperparameters

  • For the CIFAR-10/100 comparison, the reproduced GMP uses the same hyperparameters as GraNet, leaving Zero-Cost Neuroregeneration as their only difference.The authors reproduce GMP with their own implementation for this comparison.Source: §C.2 · p. 22
  • GraNet requires fewer training FLOPs than GMP’s reported result; the authors guess GMP pruned late, making GraNet’s shorter dense-training period a possible explanation.The specific GMP hyperparameters used for the reported result were not shared.Source: §C.2 · p. 22

C.3 Implementation

The appendix compares reproduced GMP and GraNet, including uniform-pruning results and implementation differences. Its reported CIFAR-100 comparison favors GraNet consistently under uniform pruning, while its own ImageNet GMP implementation performs worse than the published result.

C.3 Implementation

  • The authors’ ResNet-50 ImageNet GMP implementation performs much worse than the results in [13]; their ImageNet comparison uses results obtained directly from [13].Their GMP reproduction for a fair comparison with GraNet was limited to CIFAR-10/100.Source: §C.3 · p. 22, Table 8 · p. 22
  • Unlike the original GMP implementation, the authors’ version zeros pruned weights and prunes the first and last layers.The original implementation retained pruned weight values behind zero masks, used uniform pruning, kept the first layer dense, and capped last-layer sparsity at 0.8.Source: §C.3 · pp. 22–23
  • With uniform pruning, GraNet consistently outperforms GMP on CIFAR-100, while the CIFAR-10 results are unclear.The authors also report that uniform pruning generally performs worse than global pruning.Source: §C.3 · p. 23, Table 9 · p. 23

D ResNet-50 Learnt Budgets and Backbone Sparsities

Table 10 summarizes final sparsity budgets for methods producing 90%-sparse ResNet-50 models on ImageNet-1K. It distinguishes backbone budgets from the last fully connected layer and defines two method labels.

D ResNet-50 Learnt Budgets and Backbone Sparsities

E FLOPs Dynamics During Training with ResNet-50 on ImageNet

Figure 15 tracks single-sample inference FLOPs for GraNet with ResNet-50 on ImageNet during training. It contrasts runs with different initial sparsities and describes the trajectory of the initially dense run.

E FLOPs Dynamics During Training with ResNet-50 on ImageNet

Related papers

Loading 2106.10404v4…