Source-linked AI summary
Pruning from Scratch
Yulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou, Hang Su, Bo Zhang, Xiaolin Hu
TL;DR
Network pruning traditionally relies on pre-training to identify structures, but it is unclear whether those weights are necessary and may constrain the search. The paper learns pruned structures directly from frozen random weights using channel gates and resource-constrained configuration, achieving comparable or better accuracy with much faster searches. These results support pruning from scratch as an efficient alternative, while the method remains scoped to pruning convolutional channels without changing the original layer topology.
Problem
Traditional pruning learns structures from pre-trained models, creating a cumbersome optimization step and potentially restricting the diversity of structures that can be discovered.
Method
The method learns channel importance with scalar gates on randomly initialized frozen weights, then uses binary search to select channel configurations under resource constraints.
Results
On CIFAR10 and ImageNet, the method provides at least 10× and 100× search speedups while achieving comparable or better accuracy than traditional pruning methods.
Takeaways & Limitations
Pruning from scratch removes the pre-training burden and explores a larger structure space that can contain better pruned model architectures.
Takeaways & Limitations
The method prunes channels in convolutional layers without changing the original layer connection topology.
Abstract
from arXiv · showhide
Network pruning is an important research field aiming at reducing computational costs of neural networks. Conventional approaches follow a fixed paradigm which first trains a large and redundant network, and then determines which units (e.g., channels) are less important and thus can be removed. In this work, we find that pre-training an over-parameterized model is not necessary for obtaining the target pruned structure. In fact, a fully-trained over-parameterized model will reduce the search space for the pruned structure. We empirically show that more diverse pruned structures can be directly pruned from randomly initialized weights, including potential models with better performance. Therefore, we propose a novel network pruning pipeline which allows pruning from scratch. In the experiments for compressing classification models on CIFAR10 and ImageNet datasets, our approach not only greatly reduces the pre-training burden of traditional pruning methods, but also achieves similar or even higher accuracy under the same computation budgets. Our results facilitate the community to rethink the effectiveness of existing techniques used for network pruning.
1. Introduction
Traditional pruning pre-trains a large model, learns a pruned structure, and fine-tunes inherited weights. This paper argues that structure learning can instead begin from random weights, enabling faster searches for diverse structures without sacrificing performance.
- Network pruning removes redundant parameters or structures to reduce model size and inference latency, with structured channel pruning favored for GPU deployment.
- Traditional pruning uses pre-training, pruning, and fine-tuning, although pruned models can also be trained from scratch after their structures are obtained.
- The paper asks whether pruned structures must be learned from pre-trained weights, since extracting them from well-trained models requires cumbersome weight optimization.
- Pre-trained weights tend to produce homogeneous structures, whereas random weights discover more diverse and potentially better-performing pruned structures.
- The proposed pipeline learns channel importance with scalar gates on frozen random weights, then uses binary search to configure channels under resource constraints.It reports at least 10× and 100× search speedups on CIFAR10 and ImageNet, respectively, with comparable or better accuracy than traditional methods.
2. Related Work
Related work progresses from unstructured weight removal toward structured channel pruning and compressed-architecture search. Existing approaches use channel criteria, reconstruction, regularization, reinforcement learning, or supernet-based search, often with substantial search cost.
- Unstructured pruning removes individual weights but generally requires custom inference engines for runtime acceleration on general-purpose GPUs.
- Structured pruning emphasizes channel removal because it supports practical deployment and acceleration on general-purpose hardware.
- Representative methods rank channels using weight norms, reconstruction error, LASSO-based importance, or automatically learned layer compression ratios.
- Neural architecture search also discovers compressed structures by trimming supernets or searching internal cell connections, but these methods require substantial training time.
3. Rethinking Pruning with Pre-Training
The experiments test whether pre-training determines useful pruned structures. They find that early weight updates rapidly make structures homogeneous, while random initialization yields more diverse structures that remain competitively accurate when trained from scratch.
- The common pruning procedure associates channel gates with a pre-trained model and optimizes channel importance using sparsity regularization.
- The study saves checkpoints across training epochs and learns channel importance from each checkpoint to measure how pre-training affects the resulting structures.
- Pruned structures are represented by vectors of layerwise pruning ratios, and their similarity is measured with correlation coefficients across five random-seed experiments.
- Random-weight pruning produces diverse structures, whereas after only ten epochs of weight updates, structures become almost homogeneous; nearby checkpoints yield highly similar structures.
- The shrinking structure space during pre-training may limit potential performance, while random initialization permits broader exploration.
- Randomly obtained structures consistently reach comparable accuracy to structures from pre-trained weights and sometimes achieve higher accuracy, including for ResNet20.
- Because pre-trained-weight structures offer little final-performance advantage despite costly optimization, pruning can begin directly from random initialization.
4. Our Solution: Pruning from Scratch
Pruning from scratch learns a pruned architecture from randomly initialized weights, without updating those weights during channel-importance learning, then selects a structure under a target computation budget.
- Pruning from scratch obtains the pruned structure directly from randomly initialized weights rather than pre-trained weights.
- The network is represented as f(x; W, α), with pruning focused primarily on each layer’s channel number while preserving the original connection topology.
- Scalar gates modulate layer outputs channel-wise, so near-zero values suppress channels and create a pruning effect.
- Channel importance is optimized with sparsity regularization, while the random weights remain fixed and pre-training is not used.
- Among possible gate configurations, the method selects gates below the target sparsity ratio with maximum validation accuracy.
- A global gate threshold is determined by binary search until the generated architecture satisfies the target FLOPS constraint.
5. Experiments
Experiments on CIFAR10 and ImageNet evaluate pruning from scratch against established methods under matched computation budgets. The method achieves competitive accuracy, reduces structure-search cost, and discovers diverse architectures.
- Experimental setup: Experiments use separate validation sets while learning channel gates, then train the selected pruned models from scratch.CIFAR10 validation uses 5,000 training images; ImageNet validation uses 50,000 images.
- Experimental setup: The evaluation compares accuracy drops or top-1 accuracy under the same FLOPS constraints against conventional and reinforcement-learning pruning methods.Comparisons include uniform shrinkage or expansion, ThiNet, CP, L1-norm pruning, Network Slimming, SFP, Rethink, AMC, and NetAdapt.
- CIFAR10 results: Our method achieves less performance drop than state-of-the-art methods across CIFAR10 architectures and can outperform the baseline on large models such as ResNet110 and VGGNets.It also consistently outperforms Rethink under the same budget training scheme.
- ImageNet results: On ImageNet, the method outperforms uniform expansion and other pruning strategies across MobileNetV1, MobileNet-V2, and ResNet50.It achieves comparable or better performance than the original full-model design and similar or faster CPU inference speed than other pruned models.
- Lottery Ticket comparison: The pruning pipeline achieves higher accuracy than the Lottery Ticket Hypothesis alternatives in all reported CIFAR10 cases.The authors report no necessity for re-initializing pruned models with the original full-model weights in this structured-pruning setting.
- Search cost: 0.12 hours is the reported search time for pruning ResNet56 on CIFAR10, compared with 2.3 hours for Network Slimming and 1.0 hours for AMC.For ResNet50 on ImageNet, the pipeline takes 2.8 hours to obtain the structure from randomly initialized weights.
6. Ablation Study
Ablations examine channel expansion, pruning ratio, sparsity ratio, and structural variation across random seeds. The method remains robust across these settings while exposing diverse pruned structures.
- Channel expansion rate: At 50% FLOPS reduction, excessively large channel expansion rates degrade pruned-model performance.The authors attribute this trend to an enlarged search space, whereas preset reduced capacity can make efficient structures easier to find.
- Pruning ratio: The method remains robust under different pruning ratios, achieving comparable prediction performance even when a large portion of FLOPS is removed.The models use the budget training scheme with predefined sparsity ratio r = 1 − pruning ratio.
- Sparsity ratio: Final pruned-model accuracy is not very sensitive to the sparsity ratio, although a small sparsity level may negatively affect performance.These experiments require 50% FLOPS reduction relative to the original full models.
7. Discussion and Conclusions
The discussion argues that pre-training can constrain structural search, whereas pruning from random weights enables faster exploration of larger and more diverse structure spaces. The reported experiments support this pipeline’s effectiveness and efficiency.
- Conclusions: Pruning from scratch eliminates cumbersome pre-training and searches the pruned structure directly from randomly initialized weights.The authors report this pipeline as efficient and effective across various models and datasets.
- Conclusions: Random initialization allows exploration of a larger structure space, which can help search for better pruned model structures.The authors contrast this with the original network size limiting the available pruned structures.
- Discussion: Pre-trained weights reduce the search space, and possible pruned structures become stable and limited even after short pre-training.The authors suggest structure learning may converge faster than weight learning.
A. Effects of Pre-training on Pruning
The pre-training analysis compares structures learned from random weights with structures obtained from checkpoints during pre-training. Random-weight pruning yields more diverse structures, while nearby checkpoints produce more similar ones.
- Analysis setup: The analysis uses correlation matrices, layer-wise channel counts, multiple random seeds, and checkpoints across pre-training to compare pruned structures.All displayed pruned models reduce 50% FLOPS on CIFAR10 in the ResNet20 and ResNet56 analyses.
- Structural diversity: Structures learned from random weights are unlike the structures obtained from pre-trained weights and show more diverse correlation coefficients.The same phenomena are reported for ResNet20 and ResNet50, alongside the VGG16 observations.
- Pre-training effect: Structures based on checkpoints from nearby epochs are more similar, with high within-run correlation coefficients.This pattern indicates increasing structural homogeneity during pre-training in the reported experiments.
- Architecture-specific observation: Residual-network structures from different random seeds are less similar than VGG16 structures because only residual-branch layers are pruned.Fixed backbone channel counts leave greater freedom in the pruned residual layers.