Source-linked AI summary
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
Jian-Hao Luo, Jianxin Wu, Weiyao Lin
TL;DR
Deep CNNs are costly to deploy because of their large computational and storage demands. ThiNet uses structure-preserving filter-level pruning guided by next-layer statistics, achieving substantial compression and acceleration with limited accuracy loss. The method also supports very small models and transfer-learning applications, while ResNet acceleration remains constrained by non-tensor layers and pruning restrictions.
Problem
Deep CNNs impose substantial computational and storage costs that challenge deployment on resource-constrained devices.
Method
ThiNet formulates filter pruning as an optimization problem and removes unimportant filters using statistics from the next layer while preserving the network structure.
Results
3.31× FLOPs reduction and 16.63× compression on VGG-16 produce only a 0.52% top-5 accuracy drop, while ResNet-50 achieves 2.26× FLOPs and 2.06× parameter reductions with roughly 1% top-5 accuracy drop.
Takeaways & Limitations
ThiNet provides a unified structure-preserving approach for CNN acceleration and compression, including transfer learning and a 5.05MB VGG-16 model preserving AlexNet-level accuracy.
Takeaways & Limitations
On ResNet-50, non-tensor layers account for more than 40% of GPU inference time, and the experiments leave block outputs and projection shortcuts unchanged.
Abstract
from arXiv · showhide
We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this strategy, which has advanced the state-of-the-art. We also show the performance of ThiNet on ILSVRC-12 benchmark. ThiNet achieves 3.31$\times$ FLOPs reduction and 16.63$\times$ compression on VGG-16, with only 0.52$\%$ top-5 accuracy drop. Similar experiments with ResNet-50 reveal that even for a compact network, ThiNet can also reduce more than half of the parameters and FLOPs, at the cost of roughly 1$\%$ top-5 accuracy drop. Moreover, the original VGG-16 model can be further pruned into a very small model with only 5.05MB model size, preserving AlexNet level accuracy but showing much stronger generalization ability.
1. Introduction
ThiNet addresses the deployment cost of deep CNNs by pruning whole filters while preserving the original network structure. It identifies filters using next-layer statistics and reports substantial compression and acceleration with limited accuracy loss.
- VGG-16 has 138.34 million parameters, exceeds 500MB storage, and requires 30.94 billion FLOPs per image, challenging small devices.
- Filter-level pruning removes whole unimportant filters while preserving network structure, enabling support by off-the-shelf deep learning libraries.It can also reduce intermediate activation memory, not only model parameters.
- ThiNet determines filter importance from statistics of the next layer’s outputs rather than the filter’s own layer.This optimization-based criterion is presented as the method’s distinguishing insight.
- 3.31× FLOPs reduction and 16.63× compression on VGG-16 incur only a 0.52% top-5 accuracy drop.
- ThiNet achieves the best trade-off between model size and accuracy in transfer learning on small-scale datasets.
- 5.05MB VGG-16 preserves AlexNet-level accuracy while showing stronger generalization ability on transfer learning.
2. Related work
Prior compression methods include unstructured pruning, structured sparsity, filter-importance criteria, and complementary quantization or low-rank approaches. ThiNet is situated among these methods as a structure-preserving filter-pruning framework.
- Deep models are heavily over-parameterized, motivating reduction of model size after training.
- Iterative threshold pruning removes small-weight connections but produces non-structured sparsity that reduces universality and practical flexibility.
- Group-wise sparsity regularizes weight groups, filters, channels, shapes, or depths, but can destroy the original network structure and require dedicated libraries.
- Filter-importance methods have used weight magnitude or post-ReLU activation sparsity to identify removable structures.
- Quantization and low-rank approximation are complementary strategies that can be combined with filter pruning for further improvement.
3. ThiNet
ThiNet prunes filters by selecting and removing weak input channels in the next layer, preserving the network’s structure while reducing model size and computation. It formulates channel selection as reconstruction-error minimization, uses a greedy solver with fine-tuning, and adapts pruning to architectural constraints.
- Framework of ThiNet: ThiNet evaluates layer i’s filters using statistics from layer i + 1, selecting input channels whose removal minimally affects the next layer’s output.Each selected input channel corresponds to a filter in the preceding layer, so channel and filter pruning are coupled.
- Framework of ThiNet: The pruned network retains the original structure but has fewer filters and channels, enabling support by off-the-shelf libraries and reducing parameters and intermediate activations.The framework targets simultaneous acceleration and compression during training and inference.
- Collecting training examples: ThiNet samples output elements from layer i + 2 and pairs them with corresponding post-ReLU sliding-window channel vectors to construct training examples for channel selection.Different images, channels, and spatial locations provide additional instances of the sampled variables.
- Channel selection: Channel selection minimizes reconstruction error subject to preserving a predefined fraction r of the C channels, equivalently removing C × (1 − r) channels.The complementary removed-channel formulation is faster because the removed set is usually smaller than the preserved set.
- Channel selection: The NP-hard selection problem is approximated greedily by adding one channel at a time, choosing the addition with the smallest current objective value.The authors report that this suboptimal solution is compensated by fine-tuning and performs better and faster than their tested sparse-coding alternative.
- Pruning strategy: After selection, least-squares channel weighting supplies scaling factors that improve initialization for fine-tuning, while ResNet pruning must respect channel consistency within residual blocks.For ResNet, the first two layers are preferred because most parameters are located there and later residual layers have structural restrictions.
4. Experiments
Experiments show that ThiNet consistently improves the accuracy–compression trade-off across VGG-16, ResNet-50, and domain-specific transfer tasks, while revealing practical deployment constraints.
- 4.1. Different filter selection criteria: ThiNet achieves consistently higher accuracy than alternative channel-selection methods on CUB-200, especially after applying least-squares reconstruction.The method remains stronger at high compression rates, while least squares improves fine-tuning initialization.
- 4.1. Different filter selection criteria: Random selection can perform well at moderate compression but becomes unstable and produces very low accuracy when all layers are heavily compressed.The authors therefore conclude that random selection is not applicable in practice.
- 4.1. Different filter selection criteria: 70.75% top-1 accuracy is achieved after removing 60% of conv1-1 filters, compared with 40.99% for weight-sum selection and 51.26% for random selection before fine-tuning.The comparison shows that kernel-weight magnitude is poorly aligned with classification accuracy, and fine-tuning cannot fully recover its loss.
- 4.2. VGG-16 on ImageNet: 5.05MB is sufficient for ThiNet-Tiny to retain AlexNet-level accuracy while matching SqueezeNet’s model-complexity level.Although ThiNet-Tiny uses more FLOPs, its simpler structure makes it faster than SqueezeNet in practice.
- 4.4. Domain adaptation ability of the pruned model: ThiNet’s unchanged network structure supports transfer learning, and ImageNet-pruned models provide the best trade-off between model size and accuracy on CUB-200 and Indoor-67.ThiNet-Tiny achieves 3%–8% higher classification accuracy than AlexNet on domain-specific tasks with 50× fewer parameters.
5. Conclusion
ThiNet is presented as a unified framework for CNN acceleration and compression, with filter-level pruning outperforming existing methods. The authors identify further ResNet pruning and broader vision-task evaluation as future directions.
- ThiNet provides a unified framework for CNN model acceleration and compression.
- The proposed filter-level pruning method shows significant improvements over existing methods.
- Future work includes pruning ResNet projection shortcuts and exploring the pruned networks on additional vision tasks.