Source-linked AI summary
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang-Ting Cheng, Jian Sun
TL;DR
Channel pruning requires selecting effective layer-wise structures while managing computational cost, manual tuning, and channel dependencies. MetaPruning learns a PruningNet that generates weights for sampled structures and uses evolutionary search under constraints without search-time finetuning. It reports higher accuracy than uniform and state-of-the-art pruning baselines across MobileNet V1/V2 and ResNet.
Problem
Finding effective layer-wise channel numbers is computationally prohibitive, and channel dependencies make pruning—especially shortcut channels—non-trivial.
Method
MetaPruning stochastically trains a PruningNet to generate weights for varied pruned structures, then uses evolutionary search to select structures under FLOPs or latency constraints.
Results
MetaPruning achieves higher accuracy than uniform pruning baselines and other state-of-the-art channel-pruning methods across MobileNet V1/V2 and ResNet.
Takeaways & Limitations
A single trained PruningNet supports efficient searches for pruned networks under different constraints with little human participation.
Takeaways & Limitations
Latency estimation assumes independent layer execution and uses a lookup table measured on the Titan Xp GPU target device.
Abstract
from arXiv · showhide
In this paper, we propose a novel meta learning approach for automatic channel pruning of very deep neural networks. We first train a PruningNet, a kind of meta network, which is able to generate weight parameters for any pruned structure given the target network. We use a simple stochastic structure sampling method for training the PruningNet. Then, we apply an evolutionary procedure to search for good-performing pruned networks. The search is highly efficient because the weights are directly generated by the trained PruningNet and we do not need any finetuning at search time. With a single PruningNet trained for the target network, we can search for various Pruned Networks under different constraints with little human participation. Compared to the state-of-the-art pruning methods, we have demonstrated superior performances on MobileNet V1/V2 and ResNet. Codes are available on https://github.com/liuzechun/MetaPruning.
1. Introduction
MetaPruning addresses the computational and manual burden of channel-pruning structure selection by learning a PruningNet that generates weights for sampled structures, then searching efficiently under constraints. It reports higher accuracy than baselines and prior pruning methods across MobileNet and ResNet models.
- Motivation: Conventional channel pruning typically trains an over-parameterized network, removes weights or channels, and then finetunes or retrains the pruned network.Iterative layer-wise pruning with finetuning or weight reconstruction is commonly used to retain accuracy.
- Method: After training, evolutionary search evaluates many candidate structures using PruningNet-generated weights, without finetuning or retraining at search time.The search can incorporate constraints such as computation FLOPs or hardware latency.
- Motivation: Exhaustively searching layer-wise channel numbers is computationally prohibitive because even a 10-layer network with 32 channels per layer has 32^10 possible combinations.The paper frames pruning as finding good structures rather than merely retaining inherited weights.
- Method: MetaPruning trains a PruningNet that maps randomly sampled network encoding vectors to weights for corresponding pruned network structures.Each encoding vector specifies the number of channels in each layer.
- Results: At the same FLOPs, accuracy is 2.2%-6.6% higher than MobileNet V1, 0.7%-3.7% higher than MobileNet V2, and 0.6%-1.4% higher than ResNet-50.At the same latency, accuracy is 2.1%-9.0% higher than MobileNet V1 and 1.2%-9.9% higher than MobileNet V2.
- Contribution: MetaPruning reduces manual tuning and can optimize directly for desired metrics while pruning shortcut channels in ResNet-like structures.The authors contrast this with conventional pruning and AutoML methods that require different forms of manual or hyperparameter tuning.
2. Related Works
Related work spans traditional channel pruning, AutoML pruning, meta-learning-based weight prediction, and neural architecture search. MetaPruning combines structure search and weight prediction to reduce manual effort while addressing channel dependencies.
- Pruning: Traditional channel pruning removes entire filters, but layer compression ratios are often manually set using heuristics and may reach sub-optimal solutions.These methods can operate iteratively or use data-driven sparsity constraints.
- AutoML: AutoML pruning reduces manual hyperparameter tuning by iteratively pruning channels using reinforcement learning or an automatic feedback loop.The related methods account for real-time inference latency on multiple devices.
- Meta Learning: Meta-learning uses observations across learning tasks, and weight prediction refers to generating a network's weights with another neural network.The paper draws on this weight-prediction mechanism for pruning.
- Neural Architecture Search: Neural architecture search explores network structures with reinforcement learning, genetic algorithms, or gradient-based approaches.One-shot search jointly trains multiple operation choices in an over-parameterized network and searches for a high-accuracy path.
- Connection to NAS: MetaPruning's stochastic training randomizes channel-count vectors, generates matching weights, and crops them to fit candidate pruned networks.This lets one PruningNet learn weights for multiple structures rather than training each candidate independently.
3. Methodology
MetaPruning formulates channel pruning as finding layer-wise widths that minimize trained-network loss under resource constraints. A PruningNet generates weights for sampled structures, enabling efficient evolutionary search without evaluating every configuration.
- MetaPruning searches layer-wise channel widths that minimize pruned-network loss while satisfying FLOPs or latency constraints.
- The method targets shortcut dependencies by considering the overall pruned structure rather than pruning channels independently layer by layer.
- A PruningNet takes a network encoding vector of layer-wise channel counts and generates weights for the corresponding pruned network.
- The generated weight matrix is cropped to match the pruned network's input and output channels, and gradients update the PruningNet through differentiable reshape and convolution operations.
- Stochastic structure sampling trains the PruningNet across varied encodings, after which validation performance is estimated and evolutionary search selects structures under soft or hard constraints.
4. Experimental Results
The experiments evaluate MetaPruning on established convolutional networks and compare its search procedure with uniform and state-of-the-art channel-pruning baselines.
- Experimental design: The evaluation applies MetaPruning to MobileNet V1, MobileNet V2, and ResNet, then compares uniform-pruning baselines with state-of-the-art channel-pruning methods.
- Evolutionary search: Evolutionary search uses PruningNet inference to evaluate candidate genes, retains the top candidates, and creates new candidates through mutation and crossover under constraints.
4.1. Experiment settings
MetaPruning is designed to make large-scale ImageNet experiments feasible through a short PruningNet training stage and rapid evolutionary search without search-time finetuning.
- ImageNet 2012 experiments are feasible because the proposed MetaPruning procedure is computationally efficient.
- PruningNet training from scratch with stochastic structure sampling takes 14 epochs, and longer training provides little additional final accuracy gain.
- During evolutionary search, PruningNet directly predicts weights for candidate networks, eliminating finetuning or retraining at search time.
- Search evaluates pruned-network performance using a 50,000-image sub-validation set, while the remaining training images form the sub-training set.
- For shortcut-based networks, pruning preserves equal block input and output channels while pruning middle channels.
4.2. MetaPruning on MobileNets and ResNet
The paper adapts PruningNet architectures to MobileNet V1, MobileNet V2, and ResNet while encoding channel widths and respecting shortcut-induced dependencies.
- MobileNet V1: MobileNet V1 uses one two-layer fully connected PruningNet block per convolution layer because it has no shortcuts.
- MobileNet V1: For MobileNet V1, the encoding vector specifies each layer's channel count and is decoded into input and output compression ratios for weight generation.
- MobileNet V1: Stochastic sampling generates different output-channel encodings, and the corresponding generated matrices are cropped before training.
- MobileNet V2: MobileNet V2 encodes both stage output channels and each block's middle channels so shortcut-connected inputs and outputs remain matched.
- ResNet: ResNet adopts a similar PruningNet design to MobileNet V2 because both use shortcut-connected structures, differing in convolution types and block organization.
4.3. Comparisons with state-of-the-arts
MetaPruning is evaluated against uniform, traditional, and AutoML pruning baselines under FLOPs and latency constraints. It reports higher accuracy while reducing manual tuning and supporting precise constraint matching.
- 6.6% higher accuracy than the baseline 0.25× MobileNet V1 is obtained with MetaPruning’s learned pruning scheme.
- 3.7% accuracy boost is achieved on MobileNet V2 when the model size is as small as 43M FLOPs.MetaPruning also prunes shortcut channels, unlike previous methods that prune only middle channels.
- MetaPruning produces superior results to AMC and NetAdapt while precisely meeting FLOPs constraints.The comparison is reported for state-of-the-art AutoML pruning methods.
- MetaPruning can generate multiple pruned structures with different accuracy-speed trade-offs after one PruningNet training run.The PruningNet is trained using one-fourth as many epochs as normally training the target network.
4.4. Pruned result visualization
Visualization shows that MetaPruning learns nonuniform channel allocations across MobileNet stages. It retains more channels near downsampling and later stages while pruning more aggressively in earlier stages under tighter budgets.
- The visualization investigates whether MetaPruning learns meaningful pruning schemes rather than relying only on manually designed heuristics.The paper reports that the learned structures contain interesting patterns.
- MetaPruning keeps more channels at downsampling layers in MobileNet V1 and prunes middle channels less when MobileNet V2 blocks shrink feature maps.The observed peaks are attributed to compensating for resolution degradation during stride-2 downsampling.
- 145M-FLOPs MobileNet V2 keeps similar last-stage channel counts to the 300M model but prunes more channels in early stages.This comparison concerns jointly pruned shortcut and middle channels.
- At 45M FLOPs, MobileNet V2 approaches its maximum pruning ratio and must prune later-stage channels.Accuracy degradation from 145M to 45M FLOPs is reported as more severe than from 300M to 145M FLOPs.
4.5. Ablation study
The ablation compares PruningNet with and without weight prediction. Weight prediction substantially improves the accuracy of uniformly pruned networks and the searched low-FLOPs model.
- PruningNet without weight prediction achieves 10% lower accuracy than PruningNet with weight prediction on uniformly pruned MobileNet V1 networks.The comparison evaluates uniformly pruned layers across ratios from 0.25 to 1.
- 55.3% top1 accuracy is achieved without weight prediction for a searched MobileNet V1 below 45M FLOPs.This is 1.9% lower than the pruned network obtained with weight prediction.
- Weight prediction uses the PruningNet’s fully connected layers instead of directly cropping a shared weight matrix for different channel widths.The ablation removes these two fully connected layers to construct the comparison model.
5. Conclusion
MetaPruning reports higher accuracy than uniform baselines and other channel-pruning methods while flexibly handling constraints and ResNet-like architectures. The complete pipeline is also highly efficient.
- MetaPruning achieves higher accuracy than uniform pruning baselines and other traditional and AutoML-based channel-pruning methods.
- The method flexibly optimizes channel pruning for different constraints without introducing extra hyperparameters.
- MetaPruning effectively handles ResNet-like architectures.
- The entire channel-pruning pipeline is highly efficient.