PaperlayerView exact revision on arXiv

Source-linked AI summary

When to Prune? A Policy towards Early Structural Pruning

Maying Shen, Pavlo Molchanov, Hongxu Yin, Jose M. Alvarez

arXiv:2110.12007v1cs.CVcs.LG

TL;DR

Conventional post-training pruning leaves the heavy cost of dense training in place, while pruning at initialization can reduce training cost but suffers performance gaps. This paper uses early dense training and EPI to find when subnetworks stabilize and pruning can begin. It reports that pruning-aware training outperforms pruning-at-initialization alternatives and reduces training time across pruning methods and ratios.

  • Problem

    Conventional pruning leaves dense training largely untouched, while pruning at initialization seeks training savings but suffers noticeable accuracy loss.

    Source: §1 · p. 1

  • Method

    The method uses initial dense training to guide architecture and EPI, based on consecutive-epoch subnetwork similarity, to trigger pruning when structure stabilizes.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Pruning-aware training outperforms pruning-at-initialization alternatives by a large margin and works with magnitude- and gradient-based pruning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EPI identifies an early point for pruning across pruning algorithms and ratios, with reduced accuracy drop and training time.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Pruning enables appealing reductions in network memory footprint and time complexity. Conventional post-training pruning techniques lean towards efficient inference while overlooking the heavy computation for training. Recent exploration of pre-training pruning at initialization hints on training cost reduction via pruning, but suffers noticeable performance degradation. We attempt to combine the benefits of both directions and propose a policy that prunes as early as possible during training without hurting performance. Instead of pruning at initialization, our method exploits initial dense training for few epochs to quickly guide the architecture, while constantly evaluating dominant sub-networks via neuron importance ranking. This unveils dominant sub-networks whose structures turn stable, allowing conventional pruning to be pushed earlier into the training. To do this early, we further introduce an Early Pruning Indicator (EPI) that relies on sub-network architectural similarity and quickly triggers pruning when the sub-network's architecture stabilizes. Through extensive experiments on ImageNet, we show that EPI empowers a quick tracking of early training epochs suitable for pruning, offering same efficacy as an otherwise ``oracle'' grid-search that scans through epochs and requires orders of magnitude more compute. Our method yields $1.4\%$ top-1 accuracy boost over state-of-the-art pruning counterparts, cuts down training cost on GPU by $2.4\times$, hence offers a new efficiency-accuracy boundary for network pruning during training.

1. Introduction

Conventional pruning leaves dense training largely untouched, while pruning at initialization can reduce training cost but suffers accuracy loss. The paper proposes pruning-aware training, using early dense training and EPI to select an early pruning point as dominant subnetworks stabilize.

1. Introduction

  • CNN complexity raises memory-footprint and compute demands, making deployment on resource-constrained devices challenging.The introduction situates pruning among techniques for compressing models.Source: §1 · p. 1
  • Conventional pruning trains a dense model, prunes it, then fine-tunes; the latter stages can nearly double training time, motivating pruning during initial training.The paper frames avoiding costly additional fine-tuning as a motivation for pruning during training.Source: §1 · pp. 1–2
  • Pruning-at-initialization methods reduce training cost by training only sparse networks, but their reliability remains unsatisfactory and they suffer noticeable accuracy gaps.The introduction describes this as a limitation of approaches that identify subnetworks before training.Source: §1 · p. 2
  • Pruning-aware training uses initial dense training to guide architecture, then prunes during training to save compute, avoid extra fine-tuning, and suppress accuracy loss.The policy aims to identify the best subnetwork as early as possible rather than waiting until training ends.Source: §1 · p. 2
  • Across magnitude- and gradient-based criteria, dominant subnetworks emerge within a few epochs and their architectures stabilize thereafter.The paper's all-epoch trimming analysis finds that early pruning can yield different final architectures, while the dominant architecture stabilizes later.Source: §1 · p. 2
  • EPI measures structural similarity between subnetworks pruned at consecutive epochs to automatically identify an early point for pruning.It is designed to use weights and gradients already available during training, avoiding a lengthy grid search for starting epochs.Source: §1 · p. 2

2. Related Work

Related work distinguishes pruning by when it occurs and describes trade-offs between training efficiency and final accuracy. The paper positions its approach among in-training subnetwork-selection methods and motivates a structural similarity measure for deciding when to prune.

2. Related Work

  • Train-prune-finetune methods recover performance after pruning pretrained networks, but do not improve training efficiency and can nearly double training time.Prior methods focus on removing redundant connections while limiting output perturbation [30] [31], [18] [27].Source: §2 · pp. 2–3
  • Pruning-at-initialization methods train sparse subnetworks for efficiency, but their reliability remains unsatisfactory because of performance gaps [9] [24] [7] [43] [41] [10].The section identifies SNIP [24], FORCE [7], GraSP [43], and SynFlow [41] as examples.Source: §2 · p. 3
  • In-training pruning methods use sparsity regularization [3] [13] [28] or saliency-based sub-ticket selection [2], but often lack an automatic pruning start point.Some methods learn subnetworks during training [3] [28], while Frankle et al. [10] study the need for initial training iterations.Source: §2 · p. 3
  • Existing network-similarity measures compare representations of fully trained models, while structural comparison here focuses on remaining-neuron counts across layers.Spearman’s and Kendall’s tau correlations use neuron rankings and all neurons, yielding the same value across pruning ratios.Source: §2 · p. 3
  • Frankle et al. [10] relate network instability to pruned-model accuracy, but their measure requires completed training and cannot signal when to prune during training.Their work finds that iterative magnitude pruning performs best after some initial training.Source: §2 · p. 3

3. Method

PaT combines initial dense training with iterative structural pruning, using neuron importance to guide which neurons to remove. EPI tracks dominant sub-network stability to identify when pruning can begin.

3.1. Objective Function

  • The objective learns network parameters under a target filter-sparsity ratio, while neuron importance can be ranked by weight magnitude or gradient-based criteria.The gradient-based criterion estimates the loss change from removing a neuron; for batch-normalized networks, pruning is applied to batch-normalization layers.Source: §3.1 · pp. 3–4

3. Method

  • PaT trains densely, prunes neurons with the lowest averaged importance scores, then continues training only the remaining neurons.Pruning is iterative and can be completed within one training epoch.Source: §3.1 · pp. 3–4

3.2. Towards Early Pruning

  • PaT delays pruning until dense training has provided guidance, then trains the pruned network for the remainder of the schedule.The approach seeks to reduce training compute while avoiding the accuracy loss associated with pruning at initialization.Source: §3.2 · p. 4

3.3. Early Pruning Indicator (EPI)

  • EPI measures stability by comparing layer-wise neuron-count distributions in dominant sub-networks across recent epochs.It averages structural similarity across the selected past epochs; the indicator ranges from 0 to 1, with higher values indicating greater structural stability.Source: §3.3 · pp. 4–5
  • When EPI reaches a threshold, PaT treats the sub-network as reliable and starts pruning.The method selects the top-k neurons by importance at each epoch and forms the layer-wise structure used for comparison.Source: §3.3 · p. 5

4. Experiments

Experiments on ImageNet evaluate early pruning across ResNet34, ResNet50, and MobileNetV1, including EPI-guided policy comparisons, training speed, and ablations. The results compare pruning during training with oracle and prior-method baselines and examine when architectures stabilize.

4.1. Understanding Early Pruning Epochs

  • Pruning after a few epochs consistently performs better than pruning at initialization, while pruning late can leave too little training time for recovery.For magnitude-based pruning, pruning 50% of ResNet50 neurons at epoch 0 can make the network untrainable.Source: §4.1 · pp. 5–6

4.3. Training Speedup

  • Pruning at initialization can be inefficient because it removes many neurons from deeper layers, yielding slower models and larger accuracy drops, especially at high prune ratios.The reported comparison concerns structural pruning policies on ResNet50.Source: §4.3 · p. 7, Figure 5 · p. 7
  • At a 50% pruning ratio, EPI reduces ResNet50 training cost to $154 versus about $364 for train-prune-finetune, a 2.4× reduction.The method achieves up to 30% training-time reduction; the dense baseline takes around 9 hours on 8 NVIDIA Tesla V100 GPUs.Source: §4.3 · p. 7

4.4. Comparisons with the State-of-the-art

  • Compared with GPWP [2], the method trades lower accuracy for higher FLOP reduction; it improves on PruneTrain [28], while LFPC reduces 2% more FLOPs but loses 0.67% more accuracy and requires extra training time.The comparison reports 10% filter pruning for ResNet34 and 40% for ResNet50.Source: §4.4 · p. 7, Table 2 · p. 7

4.5. Ablation Studies

  • Across ResNet architectures and pruning criteria, EPI stability rises rapidly early in training and continues increasing more gradually later.EPI scores decrease as pruning ratios increase.Source: §4.5 · p. 8, Figure 6 · p. 8
  • EPI-guided pruning outperforms pruning pre-trained weights at equal epochs while saving compute, and equal-time finetuning improves results further.The comparison uses 90 epochs of pretrained weights plus 90 finetuning epochs, versus 180 total epochs for EPI at equal epochs.Source: §4.5 · p. 8, Table 3 · p. 8

5. Conclusions

The paper introduces EPI to identify when pruning can begin during training while keeping accuracy loss low. Experiments report reduced training time and accuracy drop.

  • EPI measures sub-network structure stability and automatically identifies an early pruning point during training.Source: §5 · p. 8
  • Experiments across pruning algorithms and ratios show reduced accuracy drop when pruning a network.Source: §5 · p. 8
  • The approach also produces a significant reduction in training time.Source: §5 · p. 8

Appendices

The passage only introduces the following sections as providing further experimental details.

  • The appendix preface directs readers to subsequent sections for additional experimental details.Source: Appendices · p. 11

A. Additional EPI-guided pruning results

Additional results report that EPI-guided pruning approaches oracle performance while beginning pruning earlier; magnitude-based EPI also outperforms heuristic pruning.

A. Additional EPI-guided pruning results

B. Stability analysis for MobileNetV1

MobileNetV1’s sub-network architecture changes substantially early in training, then gradually converges. The EPI indicator rises rapidly early and continues rising more slowly for both magnitude- and gradient-based pruning.

B. Stability analysis for MobileNetV1

  • The analysis aims to show that MobileNetV1’s sub-network architecture varies significantly during early training.It follows the stability-analysis goal described for the main paper.Source: B. Stability analysis for MobileNetV1 · p. 11
  • Table 5 reports top-1 accuracy under different magnitude-based pruning policies, with EPI entries pairing accuracy and pruning epoch.The table also summarizes average accuracy drop relative to the oracle method; “–” indicates pruning produced a non-trainable network.Source: B. Stability analysis for MobileNetV1 · p. 11
  • EPI rises rapidly early in training and continues increasing steadily later for MobileNetV1 under both magnitude- and gradient-based pruning.The stability analysis computes EPI for different prune ratios while training the model to convergence.Source: B. Stability analysis for MobileNetV1 · p. 11, Figure 7 · p. 11

C. Comparison to other similarity criteria

The comparison finds that ranking-correlation and instability measures do not distinguish pruning ratios and lack early-stage discriminative power. EPI instead focuses on architecture changes and offers better insight into network stability.

C. Comparison to other similarity criteria

  • Ranking-correlation measures require all neurons and cannot distinguish pruning ratios, so a correlation threshold implies the same pruning epoch for every ratio.The figure compares Kendall and Spearman correlations between consecutive epochs.Source: Figure 9 · p. 12, C. Comparison to other similarity… · p. 12
  • Other similarity measures lack enough early-stage discriminative power, whereas EPI’s architecture focus provides better insight into network stability.The comparison is reported for ResNet50; the authors refer to Fig. 6 in the main paper for the EPI result.Source: C. Comparison to other similarity… · p. 12
  • The instability measure focuses on identical architectures trained with different SGD noise and can only be computed after training ends.The figure description also notes that the measure involves prune ratio, while its curves are not distinguishable across ratios.Source: C. Comparison to other similarity… · p. 12, Figure 8 · p. 12

D. EPI-guided Pruning on Object Detection

For SSD300-RN34 object detection, EPI-guided pruning is evaluated on Pascal VOC at 9% and 40% pruning ratios. Its selected epochs yield mAP drops of 0.589% and 0.212% relative to grid search, respectively.

D. EPI-guided Pruning on Object Detection

  • The detection experiment uses SSD with a ResNet34 backbone and 300 × 300 inputs on the union of Pascal VOC 2007 and 2012.Training lasts 800 epochs, with a per-GPU batch size of 128.Source: D. EPI-guided Pruning on Object… · p. 12
  • Pruning too early causes large accuracy drops, while delaying pruning can achieve accuracy on par with the 76.8% mAP unpruned upper bound, especially at a lower pruning ratio.The comparison tests magnitude-based pruning at 9% and 40% against grid-search pruning with 50 epochs.Source: D. EPI-guided Pruning on Object… · p. 12, Figure 10 · p. 12
  • 0.589% and 0.212% mAP drops versus grid search result from EPI-guided pruning at 9% and 40% ratios, respectively.EPI triggers pruning at epochs 96 and 255, respectively; Fig. 10 presents the pruning results and EPI curves.Source: D. EPI-guided Pruning on Object… · p. 12, Figure 10 · p. 12
  • For detection, EPI similarity is computed across the past 50 epochs, and its value rises rapidly early before increasing gradually.The experiment uses the magnitude universal threshold τ = 0.983.Source: D. EPI-guided Pruning on Object… · p. 12, Figure 10 · p. 12

Related papers

Loading 2110.12007v1…