PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Value of Network Pruning

Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, Trevor Darrell

arXiv:1810.05270v2cs.LGcs.CVstat.ML

TL;DR

Network pruning commonly trains a large model before pruning and fine-tuning, but the value of that inherited model and its weights is uncertain. The paper evaluates scratch training against fine-tuning across structured pruning settings and finds that scratch-trained pruned models are often comparable or better, shifting attention toward the pruned architecture itself.

  • Problem

    The paper examines whether training a large over-parameterized model and inheriting its selected weights are necessary for obtaining efficient pruned models.

  • Method

    The paper empirically compares fine-tuning inherited pruned models with training pruned architectures from random initialization across multiple datasets, architectures, and pruning methods.

  • Results

    For the structured pruning methods examined, scratch training is comparable to or better than fine-tuning, while unstructured results vary with dataset, sparsity, and evaluation setting.

  • Takeaways & Limitations

    The results suggest that structured pruning may provide value through identifying efficient architectures and should be evaluated against scratch-trained architecture baselines.

  • Takeaways & Limitations

    The conventional pruning pipeline can remain faster when a pretrained large model and little training budget are available, or when multiple model sizes are needed.

Abstract

from arXiv · show

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization.

1 INTRODUCTION

The paper questions the standard three-stage pruning pipeline and finds that, for structured pruning, directly training smaller models can match or exceed fine-tuning inherited weights. It argues that the discovered architecture may matter more than the preserved weights.

  • Motivation: The conventional pipeline trains a large model, prunes it using a criterion, and fine-tunes the resulting smaller model.This procedure is motivated by the presumed value of over-parameterized representations and selected important weights.
  • Main observations: For predefined structured targets, training the small target model from random initialization can match or outperform the full pruning pipeline.In this setting, the large-model training stage is not necessary to obtain the target architecture’s performance.
  • Main observations: For automatically discovered structured targets, training from scratch can also match or exceed fine-tuning, suggesting that the architecture matters more than inherited weights.The large model may still be needed to discover the target architecture.
  • Implications: The findings suggest that automatic pruning can function as implicit architecture search by identifying efficient layer structures.For channel pruning, this includes searching the number of channels in each layer.

2 BACKGROUND

The background places pruning within efforts to reduce the computational and memory demands of over-parameterized deep networks. It distinguishes individual-weight pruning from structured pruning, which removes channels or layers while preserving convolutional structure.

  • Efficiency motivation: Deep networks can impose prohibitive model-size, memory, computation, and power costs in resource-constrained settings.Large models may be difficult to store or run in real time on embedded systems.
  • Pruning approaches: Individual-weight pruning removes selected weights, including through Hessian-based criteria and magnitude-based pruning.Magnitude-based pruning also appears in the Deep Compression pipeline.
  • Pruning approaches: Structured pruning removes channels or layers while preserving the original convolutional structure.This allows efficiency benefits without dedicated hardware or libraries.
  • Structured pruning: Channel pruning is a widely used structured approach because it is fine-grained while remaining compatible with conventional deep-learning frameworks.Methods use heuristics, sparsity constraints, scaling factors, or feature-reconstruction objectives to select channels.
  • Related observations: The paper relates its analysis to evidence that random channel pruning can rival sophisticated criteria and to the Lottery Ticket Hypothesis for unstructured pruning.These studies motivate examining what pruning algorithms contribute beyond selecting parameters.

3 METHODOLOGY

The study compares scratch training with fine-tuning for predefined and automatically determined pruned architectures across standard datasets, networks, and pruning methods. It controls training comparisons through epoch- and computation-matched scratch baselines.

  • Experimental design: The methodology trains small target models from scratch and compares them with models fine-tuned from inherited weights.The comparison covers predefined and automatic structured pruning, plus a magnitude-based unstructured method.
  • Target architectures: Pruned architectures are classified as predefined when humans set layerwise pruning ratios and automatic when algorithms determine structures by globally comparing importance across layers.Unstructured pruning is also treated as automatic because zero positions are determined during training and pruning.
  • Evaluation scope: The evaluation spans CIFAR-10, CIFAR-100, and ImageNet with VGG, ResNet, and DenseNet architectures and multiple pruning methods.The benchmark includes four predefined and two automatic structured pruning methods.
  • Training budget: Scratch-E matches fine-tuning epochs, whereas Scratch-B matches the computation budget and proportionally extends the learning-rate schedule when needed.The experiments generally find Scratch-E sufficient, with Scratch-B used when comparable accuracy requires additional training.
  • Implementation: The implementation follows original public training setups when available and re-implements simpler pruning procedures otherwise.Standard hyperparameters, augmentation, SGD with Nesterov momentum, and stepwise learning-rate decay are used.

4 EXPERIMENTS

Across predefined and automatically discovered structured pruning methods, training pruned architectures from scratch generally matches or exceeds fine-tuning inherited weights. Unstructured pruning shows a different pattern, with fine-tuning often advantageous, especially at high sparsity or on ImageNet.

  • Experimental design: The experiments compare scratch training with inherited-weight fine-tuning across predefined and automatic structured pruning, plus magnitude-based unstructured pruning.The structured methods include filter, channel, feature-reconstruction, and residual-block pruning.
  • Predefined structured pruning: Across L1-norm filter pruning, ThiNet, and feature reconstruction, scratch-trained structured models generally match or outperform fine-tuned models.ThiNet is an exception only for aggressively pruned VGG-Tiny, where the training budget is sharply reduced.
  • Automatic structured pruning: Network Slimming produces scratch-trained models at least as accurate as fine-tuned models across all evaluated networks.Scratch-B outperforms fine-tuned models in 8 of 10 experiments, while Scratch-E is usually within the standard deviation.
  • Automatic structured pruning: Sparse Structure Selection shows that scratch-trained ResNet-41, ResNet-32, and ResNet-26 models outperform inherited-weight alternatives.Scratch-B is better than both comparison models for every evaluated pruned architecture, while Scratch-E wins on average.
  • Unstructured magnitude-based pruning: For unstructured pruning, scratch training is competitive on CIFAR at prune ratios up to 80%, but fine-tuning can win at 95% sparsity and generally performs better on ImageNet.The authors associate this difference with the greater difficulty of directly training highly sparse networks and with dataset scale or complexity.

5 NETWORK PRUNING AS ARCHITECTURE SEARCH

The paper evaluates pruning as architecture search by comparing pruning-derived and uniformly pruned architectures trained from scratch. Automatically discovered structures can improve parameter efficiency, especially when redundancy is uneven, and their design patterns can sometimes transfer across models and datasets.

  • Architecture-search setup: Pruning-derived architectures are compared with uniformly pruned architectures by training both from random initialization, isolating architecture from inherited weights.This comparison tests whether pruning’s value lies in discovering efficient structures rather than transferring parameters.
  • Parameter efficiency: Network Slimming architectures are more parameter efficient than uniformly channel-pruned architectures on the evaluated VGG networks.All architectures are trained from random initialization for the same number of epochs.
  • Parameter efficiency: Pruned channel and weight architectures exhibit consistent sparsity patterns, suggesting that automatic pruning can identify task-relevant redundancy.The paper reports this pattern as evidence for the efficiency of automatic pruning on the evaluated architectures.
  • Architecture-dependent results: On PreResNet and DenseNet, pruning-derived architectures are not significantly more efficient than uniformly pruned ones, with near-uniform stage sparsity as a possible explanation.The contrast with VGG is associated with VGG’s more imbalanced redundancy across layer stages.
  • Generalizable design principles: Guided Pruning and Guided Sparsification construct new architectures from average channel counts or sparsity patterns and perform on par with the corresponding pruned architectures.The guidance patterns are extracted from pruned models but the resulting architectures are trained from scratch.
  • Generalizable design principles: Patterns transferred from VGG-16 on CIFAR-10 to VGG-19 on CIFAR-100 can outperform uniform pruning or sparsification, although they may be slightly worse than directly pruned architectures.This indicates that some efficient design patterns can be reused across a model variant and dataset.

6 EXPERIMENTS ON THE LOTTERY TICKET HYPOTHESIS (FRANKLE & CARBIN, 2019)

The paper revisits the Lottery Ticket Hypothesis by comparing winning-ticket and random initializations under different pruning types and learning rates. The apparent advantage of winning tickets depends on using a small learning rate, which produces lower accuracy than the widely used large rate, while structured pruning shows no improvement.

  • Hypothesis and evaluation: The Lottery Ticket Hypothesis claims that a subnetwork and its original initialization form a winning ticket that can train effectively in isolation.The original initialization is hypothesized to be necessary for competitive performance relative to random reinitialization.
  • Hypothesis and evaluation: The evaluation compares original and randomly reinitialized weights for iterative and one-shot unstructured pruning, using initial learning rates 0.1 and 0.01.The experiments also include structured filter pruning and use stepwise decay with momentum SGD.
  • Unstructured pruning: For unstructured pruning, winning-ticket initialization improves over random initialization only with the small initial learning rate 0.01.At the larger learning rate, the reported advantage does not appear.
  • Structured pruning: For structured pruning, winning-ticket initialization is only on par with random initialization at both tested learning rates.The structured result is reported for L1-norm-based filter pruning.
  • Interpretation: The small learning rate 0.01 yields lower accuracy than the widely used large learning rate 0.1, limiting the practical value of its winning-ticket improvement.The paper attributes the apparently contradictory findings mainly to learning-rate differences in the unstructured CIFAR setting.

7 DISCUSSION AND CONCLUSION

The discussion argues that training predefined target models from scratch can match conventional pruning outcomes while reducing training and implementation burdens. It also emphasizes fair comparisons against uniformly pruned baselines and identifies settings where conventional pruning remains faster.

  • Benefits: Training predefined target models from scratch can use less GPU memory and may be faster because the model is smaller.This is presented as a benefit over training the original large model.
  • Benefits: Scratch training avoids implementing pruning criteria and procedures that may require layer-wise fine-tuning or architecture-specific customization.The paper presents this as a practical simplification of the conventional pipeline.
  • Benefits: Scratch training also avoids tuning additional hyperparameters introduced by the pruning procedure.This reduces the extra tuning burden associated with conventional pruning.
  • Evaluation guidance: Structured pruning methods should be compared with uniformly pruned models trained from scratch to establish whether they identify more efficient architectures.If uniform baselines are not worse, the pruning pipeline can also be skipped.
  • Scope boundary: Conventional pruning can remain faster when a pretrained large model and little training budget are available, or when multiple model sizes are needed.It is also useful when the desirable model size is unknown and different pruning ratios can be applied.

A RESULTS ON SOFT FILTER PRUNING (HE ET AL., 2018A)

The Soft Filter Pruning evaluation extends the paper’s observation beyond the main structured-pruning experiments. Scratch-trained models outperform pruned models most of the time, with Scratch-B doing so in nearly all cases.

  • Method: Soft Filter Pruning updates pruned filters during training, allowing pruned weights to recover.The method can operate from either random initialization or a pretrained model.
  • Results: Scratch-E outperforms pruned models most of the time, while Scratch-B outperforms them in nearly all evaluated cases.The comparison is reported both without pretrained models and with pretrained models.

B TRANSFER LEARNING TO OBJECT DETECTION

Transfer-learning experiments on PASCAL VOC detection show that scratch-trained pruned models can outperform fine-tuned models, including when pruning is performed at different stages. Additional experiments indicate that more fine-tuning or aggressive pruning does not reverse this pattern.

  • Transfer to detection: Scratch-trained pruned models can surpass fine-tuned models when transferring to object detection.The experiment evaluates L1-norm filter-pruned ResNet-34-A and ResNet-34-B backbones on PASCAL VOC with Faster-RCNN.
  • Transfer to detection: Prune-C outperforms Prune-D, despite pruning being performed before transfer for Prune-C and after transfer for Prune-D.The authors hypothesize that earlier pruning reduces susceptibility to bad local minima caused by inheriting weights from the large model.
  • Transfer to detection: Scratch-E/B further improves performance by training the small model from scratch during the classification stage before transfer to detection.Scratch-E/B denotes training the small classification model, then transferring it to detection.
  • Aggressive pruning: Under aggressive pruning, scratch training outperforms fine-tuning by an even larger margin.The paper reports this pattern for Network Slimming, L1-norm filter pruning, and unstructured pruning.
  • Fine-tuning duration: Fine-tuning for more epochs produces negligible accuracy increases or small decreases, while Scratch-E remains on par with sufficiently fine-tuned models.The comparison uses L1-norm filter pruning; Scratch-E models are trained for 160 epochs.

E EXTENDING THE STANDARD TRAINING SCHEDULE

Extending CIFAR training from 160 to 300 epochs does not change the comparison: scratch-trained models remain at least as accurate as fine-tuned models.

  • Extended schedule: Scratch-trained models remain at least on par with fine-tuned models after extending CIFAR training from 160 to 300 epochs.The experiment uses L1-norm based filter pruning and an extended standard training schedule.

F WEIGHT DISTRIBUTIONS

Weight-distribution comparisons on VGG-16 and CIFAR-10 examine unpruned, fine-tuned, and scratch-trained models for structured and unstructured pruning. For unstructured pruning, fine-tuned and scratch-trained models have markedly different distributions.

  • Visualization setup: Figure 8 compares weight distributions for unpruned, fine-tuned, and scratch-trained VGG-16 models on CIFAR-10.The comparison covers Network Slimming and unstructured pruning.
  • Unstructured pruning: For unstructured pruning, the fine-tuned model has nearly no close-to-zero weights, unlike the scratch-trained model.The authors identify this distributional difference as a potential reason scratch training can sometimes underperform fine-tuning.

G MORE SPARSITY PATTERNS FOR PRUNED ARCHITECTURES

Additional sparsity-pattern analyses examine pruned architectures across several models and datasets. The reported patterns are generally close to uniform for a given prune ratio, except that later VGG-16 stages tend to contain more redundancy.

  • Additional sparsity analyses: Sparsity patterns are analyzed for PreResNet-164, PreResNet-110, DenseNet-40, and VGG-16 across pruning ratios.The analyses cover Network Slimming and unstructured pruning on CIFAR-10 and CIFAR-100.
  • PreResNet-164: For a given prune ratio, PreResNet-164 channel sparsity is close to uniform across stages.The values represent the ratio of channels kept.
  • Unstructured patterns: For a given prune ratio, unstructured sparsity in PreResNet-110 and DenseNet-40 is close to uniform across stages.The values represent the ratio of weights kept in 3×3 kernels.
  • VGG-16: In VGG-16, later stages tend to have more redundancy than earlier stages at each prune ratio.The table reports the ratio of channels kept for Network Slimming on CIFAR-10.
Loading 1810.05270v2…