Source-linked AI summary
Channel Pruning for Accelerating Very Deep Neural Networks
Yihui He, Xiangyu Zhang, Jian Sun
TL;DR
Very deep CNNs are accurate but expensive to run, and prior channel-pruning approaches face challenges in handling changed layer inputs and achieving practical speed-ups. The paper proposes inference-time pruning that alternates LASSO channel selection with least-squares reconstruction, extending it across layers and branches. It reports 5× VGG-16 acceleration with a 0.3% error increase and 2× acceleration for ResNet-50 and Xception-50 with 1.4% and 1.0% accuracy loss, respectively.
Problem
Inference-time channel pruning must reduce computation without substantially changing following-layer inputs, while prior approaches have limited speed-up or costly training requirements.
Method
The method alternates LASSO-based representative-channel selection with linear least-squares feature-map reconstruction, accounting for accumulated layer-wise error and extending to multi-branch networks.
Results
5× acceleration with a 0.3% error increase is reported for VGG-16, while 2× acceleration yields 1.4% and 1.0% accuracy loss for ResNet-50 and Xception-50, respectively.
Takeaways & Limitations
The reduced CNNs maintain accuracy while providing inference-efficient networks using off-the-shelf libraries across VGG and ResNet-like architectures.
Takeaways & Limitations
The approach is currently limited to inference time, with training-time integration left for future work.
Abstract
from arXiv · showhide
In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks.Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We further generalize this algorithm to multi-layer and multi-branch cases. Our method reduces the accumulated error and enhance the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5x speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, Xception and suffers only 1.4%, 1.0% accuracy loss under 2x speed-up respectively, which is significant. Code has been made publicly available.
1. Introduction
The paper targets inference-time acceleration of very deep CNNs through channel pruning, which reduces feature-map width while preserving outputs through reconstruction. Its iterative method selects representative channels with LASSO and reconstructs outputs with least squares, achieving substantial speed-ups across several architectures.
- Motivation: Structured simplification methods include tensor factorization, sparse connection, and channel pruning, with this work focusing on the last approach.Tensor factorization adds computation and does not reduce feature-map width, while sparse connection can be implementation-dependent.
- Motivation: Channel pruning directly reduces feature-map width, producing a thinner network that is efficient on CPU and GPU without special implementation.
- Motivation: Inference-time pruning is challenging because removing channels changes the next layer's input, while prior methods reported limited speed-up or costly training procedures.Very deep CNN results on ImageNet were rarely reported for training-based approaches.
- Method: The proposed inference-time method exploits redundancy across feature maps rather than analyzing individual filter weights.
- Method: The algorithm alternates LASSO-based channel selection with linear least-squares reconstruction, approximating the network layer by layer while accounting for accumulated error.The approach is also discussed for multi-branch networks such as ResNet and Xception.
- Results: 5× acceleration with a 0.3% error increase is achieved for VGG-16 when channel pruning is combined with tensor factorization.The reported results also include 4× acceleration with a 1.0% increase in top-5 error for VGG-16 and 2× speed-up on ResNet-50 and Xception-50.
2. Related Work
Prior CNN acceleration research includes optimized implementations, quantization, and structured simplification. Within structured simplification, channel pruning offers a direct route to thinner networks, but existing training-based and inference-time approaches face cost, scalability, or speed-up limitations.
- Acceleration categories: CNN acceleration methods include optimized implementations, quantization, and structured simplification.
- Structured simplification: Sparse connection removes neuron or channel connections, but its practical speed-up depends strongly on implementation.One cited method accelerates fully connected layers by up to 50×.
- Structured simplification: Tensor factorization decomposes weights into several pieces and can factorize convolutional layers into combinations such as 3 × 3 and 1 × 1.
- Channel pruning: Training-based channel-pruning methods regularize networks, but they are costly and have rarely been evaluated on very deep networks with large datasets.
- Channel pruning: Inference-time channel pruning remains challenging, while data-free methods have reported limited speed-up or require long retraining and repeated evaluation.One cited approach uses over 100 random trials, making evaluation on very deep models and large datasets infeasible.
3. Approach
The approach prunes channels by alternating LASSO-based selection with least-squares reconstruction, then extends this procedure across layers and multi-branch residual networks.
- Single-layer pruning: Channel pruning reduces feature-map width by removing selected input channels and corresponding filter weights.The retained-channel count c′ is set according to the desired speed-up ratio.
- Single-layer pruning: The algorithm alternates representative-channel selection by LASSO regression with output reconstruction by linear least squares.The procedure fixes filters while selecting channels, then fixes channel coefficients while reconstructing the filters.
- Single-layer pruning: The ℓ0 channel-selection problem is relaxed to ℓ1 regularization because exact optimization is NP-hard.Increasing λ produces more zero coefficients and can yield a higher speed-up ratio, while normalized filter weights avoid trivial solutions.
- Whole-model pruning: Sequential whole-model pruning uses original-model output feature maps as targets so accumulated error is accounted for layer by layer.The method obtains input volumes from the current pruned feature map and output volumes from the un-pruned model.
- Pruning Multi-Branch Networks: Residual-block pruning samples shared input channels for the first residual convolution and reconstructs the combined shortcut-plus-residual output at the last layer.These adaptations address the shared shortcut input and the parameter-free shortcut branch.
- Pruning Multi-Branch Networks: Filter-wise pruning independently selects representative input channels for each residual-branch filter and improves top-5 accuracy by 0.5% for 2× ResNet-50 in the reported setting.The improvement applies to the first layer of each residual branch without pruning the shared shortcut input channels.
4. Experiment
The evaluation uses the proposed approach on VGG Nets, ResNet, and Xception across ImageNet, CIFAR-10, and PASCAL VOC 2007.
- Evaluation setup: The approach is evaluated on VGG Nets, ResNet, and Xception using ImageNet, CIFAR-10, and PASCAL VOC 2007.Batch normalization is merged into convolutional weights before evaluation, with Caffe and scikit-learn used for network evaluation and solver implementation.
4.1. Experiments with VGG-16
Experiments on VGG-16 show that feature-map-based channel selection consistently reduces pruning error across layers and speed-up ratios, while whole-model pruning and combined acceleration methods achieve substantial speed-ups with limited accuracy degradation.
- Single Layer Pruning: Channel selection based on feature-map correlations consistently outperforms first-k and max-response baselines across layers and speed-up ratios.Error increases as the speed-up ratio increases, and max-response selection can perform worse than selecting the first k channels.
- Whole Model Pruning: Shallower VGG-16 layers contain more redundancy, so whole-model pruning removes channels more aggressively there than in deeper layers.The remaining-channel ratio for shallow layers conv1_x to conv3_x versus deep layers conv4_x is 1 : 1.5, while conv5_x is not pruned.
- Whole Model Pruning: 2× speed-up without accuracy loss and 4× speed-up with a 1.0% drop are achieved after fine-tuning the pruned model.These whole-model VGG-16 results are reported after fine-tuning.
- Comparisons with Training from Scratch: The fine-tuned pruned model outperforms counterparts trained from scratch under matched computational complexity and architecture.The comparison evaluates both scratch-trained counterparts and normal-setting networks with the same computational complexity and architecture.
- Acceleration for Detection: For Faster R-CNN, the 4× accelerated model reduces runtime from 220ms to 94ms per image, while the 2× model incurs a 0.4% mAP drop.Convolutional layers contribute about 64% of Faster R-CNN runtime, and mmAP shows no large degradation for higher localization requirements.
4.2. Experiments with Residual Architecture Nets
The experiments extend channel pruning to residual and Xception-style multi-branch networks, using architecture-aware pruning choices and multi-branch enhancement. The method achieves strong accuracy retention across ImageNet and CIFAR-10, including 2× acceleration results for Xception-50 and ResNet-56.
- Motivation: Residual and Xception networks are harder to prune because spatial 1 × 1 convolutions are difficult to factorize and residual designs already target efficiency.These constraints motivate the multi-branch pruning experiments.
- ResNet Pruning: Residual-block pruning keeps 70% of channels in sensitive blocks and 30% elsewhere, while pruning branch2a more aggressively than branch2b and branch2c.The branch2a:branch2b:branch2c preservation ratio is 2:4:3 within the retained channels.
- ResNet Pruning: 4.0% improvement is obtained with multi-branch enhancement by accounting for shortcut-error accumulation and reducing entry feature-map width through sampling.The reported comparison is from Table 6.
- Xception Pruning: Batch Normalization layers must be maintained during Xception pruning because otherwise fine-tuning the pruned model becomes nontrivial.The paper attributes this difficulty to instability in the Xception block.
- Xception Pruning: 1.0% increase of error is reported for Xception-50 under 2× acceleration after fine-tuning.Without fine-tuning, top-5 error is 100%, while 20 training epochs produce a 4.3% increased error; modern networks have less redundancy by design.
- CIFAR-10: 2× speed-up on ResNet-56 is competitive with scratch-trained pruning before fine-tuning and significantly better than Filter pruning and scratch-trained pruning afterward.The experiments use CIFAR-10 with a 92.8% reproduced baseline accuracy and leave the final 8 × 8 stage unchanged.
5. Conclusion
The paper presents inference-time channel pruning for very deep networks, producing efficient reduced CNNs while maintaining accuracy and using off-the-shelf libraries. It demonstrates speed-ups and accuracy retention across VGG and ResNet-like networks, while leaving training-time acceleration for future work.
- Conclusion: Inference-time channel pruning produces inference-efficient reduced CNNs while maintaining accuracy and requiring only off-the-shelf libraries.The conclusion frames this as the paper’s central outcome.
- Conclusion: Compelling speed-ups and accuracy are demonstrated for VGG Net and ResNet-like networks on ImageNet, CIFAR-10, and PASCAL VOC.The conclusion summarizes results across architectures and datasets.
- Future Work: Training-time integration remains future work, as the current approach is designed for inference time only.The authors identify possible acceleration of the training procedure as a future direction.