Source-linked AI summary
Channel Pruning via Automatic Structure Search
Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, Yonghong Tian
TL;DR
Channel pruning must reduce CNN resource demands, but existing methods often select channels using rule-of-thumb criteria rather than optimizing the pruned structure. ABCPruner shrinks the structure space and searches it automatically with an artificial bee colony algorithm. Experiments report effective compression and accuracy that is better or comparable to full models across popular CNNs.
Problem
Existing channel-pruning methods commonly select channels by importance or rule-of-thumb designs, while the channel number in each layer determines the pruned structure.
Method
ABCPruner limits preserved channels to a predefined percentage space and uses the artificial bee colony algorithm to automatically optimize the channel number in each layer.
Results
Extensive experiments on popular CNNs demonstrate ABCPruner’s efficacy over traditional channel-pruning methods and an automatic-based counterpart.
Takeaways & Limitations
ABCPruner can compress popular CNNs while maintaining better or at least comparable accuracy against the full model.
Takeaways & Limitations
Fitness evaluation requires training the pruned model on the training set, which is time-consuming and infeasible when the training set is large.
Abstract
from arXiv · showhide
Channel pruning is among the predominant approaches to compress deep neural networks. To this end, most existing pruning methods focus on selecting channels (filters) by importance/optimization or regularization based on rule-of-thumb designs, which defects in sub-optimal pruning. In this paper, we propose a new channel pruning method based on artificial bee colony algorithm (ABC), dubbed as ABCPruner, which aims to efficiently find optimal pruned structure, i.e., channel number in each layer, rather than selecting "important" channels as previous works did. To solve the intractably huge combinations of pruned structure for deep networks, we first propose to shrink the combinations where the preserved channels are limited to a specific space, thus the combinations of pruned structure can be significantly reduced. And then, we formulate the search of optimal pruned structure as an optimization problem and integrate the ABC algorithm to solve it in an automatic manner to lessen human interference. ABCPruner has been demonstrated to be more effective, which also enables the fine-tuning to be conducted efficiently in an end-to-end manner. The source codes can be available at https://github.com/lmbxmu/ABCPruner.
1 Introduction
ABCPruner reframes channel pruning as automatically searching for the channel count in each layer rather than selecting important channels. It shrinks the search space and uses the artificial bee colony algorithm to identify a pruned structure for efficient fine-tuning.
- Channel pruning removes entire filters but is challenging because changing one layer alters the next layer’s input.
- Existing rule-of-thumb channel-selection methods can produce sub-optimal structures, motivating direct optimization of channel numbers per layer.
- ABCPruner initializes candidate structures, evaluates fitness using briefly trained assigned filters, updates structures with ABC, and retains the best structure for fine-tuning.
- (10α)^L feasible structures replace Q^L possibilities by preserving channels from {0.1c_j, 0.2c_j, ..., αc_j} in each layer.
- Unlike the prior automatic method, ABCPruner uses a one-stage process without a separately trained PruningNet.
2 Related Work
Related work contrasts irregular weight pruning with hardware-friendly channel pruning and describes automatic pruning as an emerging alternative to human-guided hyper-parameter selection.
- Weight pruning creates sparse, irregular networks, whereas channel pruning removes entire filters and is better supported by general-purpose hardware and BLAS libraries.
- Traditional channel-pruning methods estimate filter importance using measures such as l_p-norms or activation sparsity.
- Automatic pruning methods reduce reliance on human hyper-parameter analysis through Q-value methods, reinforcement learning, or automatic feedback loops.
3 The Proposed ABCPruner
ABCPruner formulates channel pruning as automatic search for the channel number in each layer, constrains the candidate space, and optimizes structures with an artificial bee colony algorithm. Candidate structures are evaluated using briefly trained pruned models, while the best structure receives longer fine-tuning.
- 3.1 Optimal Pruned Structure: ABCPruner searches for the optimal channel number in each layer instead of selecting channels by rule-of-thumb importance criteria.The pruned structure is represented as C′ = (c′_1, c′_2, ..., c′_L), with c′_j ≤ c_j.
- 3.1 Optimal Pruned Structure: The optimization selects a pruned structure whose model, trained or fine-tuned on Ttrain, achieves the best accuracy on Ttest.The pruned weights W′ are updated directly in the pruned model.
- 3.3 Automatic Structure Search: Fitness-guided candidate selection progressively favors better pruned structures, while direct weight updates make the implementation one-stage rather than dependent on a separately trained PruningNet.The one-stage implementation differs from the two-stage scheme described for MetaPruning.
- 3.2 Combination Shrinkage: Constraining each layer to preserve channels from {0.1c_i, 0.2c_i, ..., αc_i} reduces the structure combinations from a potentially huge space to (10α)^L.α is shared across layers and may take values from 10% through 100%; when αc_i < 1, one channel is preserved.
- 3.3 Automatic Structure Search: The artificial bee colony algorithm automatically searches the remaining structures, reducing reliance on manual structure design or exhaustive enumeration.It initializes n candidate structures, evaluates fitness, updates candidates through employed and onlooker bees, and reinitializes stagnant candidates with scout bees.
- 3.3 Automatic Structure Search: Each candidate structure is initialized with randomly selected filters from the pretrained model and trained for a small number of epochs to estimate fitness.The structure with the best fitness is then fine-tuned for more epochs.
4 Experiments
Experiments evaluate ABCPruner across representative CNNs on CIFAR-10 and ILSVRC-2012, measuring compression, accuracy, and training efficiency. Results show substantial reductions with comparable accuracy, while deeper networks often permit higher pruning rates and the method uses fewer training epochs than alternatives.
- CIFAR-10: ABCPruner removes 61.20% of VGGNet channels, 73.68% of FLOPs, and 88.68% of parameters while achieving 93.08% accuracy on CIFAR-10.The reported accuracy is slightly better than the baseline model.
- CIFAR-10: ABCPruner reduces GoogLeNet channels by 22.19%, FLOPs by 66.56%, and parameters by 60.14%, with only a 0.21% accuracy drop.
- CIFAR-10: For ResNet-56 versus ResNet-110, channel reduction rises from 27.07% to 33.28%, FLOPs reduction from 54.13% to 65.05%, and parameter reduction from 54.20% to 67.41%.The paper attributes the larger reductions to the deeper network being more over-parameterized.
- ILSVRC-2012: On ILSVRC-2012, accuracy drops are larger than on CIFAR-10, while deeper ResNets obtain higher pruning rates and smaller accuracy drops.The paper relates the dataset difference to ILSVRC-2012's 1,000 categories and the network-depth trend to greater redundancy in deeper models.
- In-depth Analysis: Layer-wise pruning on VGGNet-16 removes over 20% of channels from most layers, with pruning rates differing across layers under ABCPruner-80%.ABCPruner-80% allows at most 80% of channels to be preserved in each layer.
- Comparison with Other Methods: ABCPruner requires 102 training epochs, including 12 for structure search and 90 for fine-tuning, fewer than the compared methods.Importance-based methods are reported to require 196 epochs, while other methods use differing retraining or fine-tuning schedules.
5 Conclusion
ABCPruner automatically searches for optimal channel configurations by reducing the structure space and applying an artificial bee colony algorithm. Experiments on popular CNNs demonstrate efficacy over traditional and automatic channel-pruning methods.
- ABCPruner searches automatically for the optimal pruned structure, defined by the channel number in each layer.
- The method first shrinks the combinations of possible pruned structures to enable efficient search.
- ABCPruner integrates an artificial bee colony algorithm to perform automatic optimal-structure search.
- Extensive experiments on popular CNNs demonstrate efficacy over traditional channel-pruning methods and an automatic-based counterpart.