Source-linked AI summary
Learning to Prune Filters in Convolutional Neural Networks
Qiangui Huang, Kevin Zhou, Suya You, Ulrich Neumann
TL;DR
Large CNNs are computationally expensive and over-parameterized, motivating methods that simplify them while preserving performance. The paper trains try-and-learn agents to prune filters with a reward balancing accuracy and efficiency, and reports effective pruning across recognition and segmentation CNNs with controllable performance–scale tradeoffs.
Problem
Large CNNs have many redundant parameters and high computational costs, while data-driven filter pruning and easy control of the performance–scale tradeoff remain needed.
Method
The method trains pruning agents to remove filters from CNNs using a reward that combines validation performance with pruning efficiency.
Results
The method aggressively prunes filters while maintaining performance at a desired level across visual recognition and semantic segmentation CNNs.
Takeaways & Limitations
The approach provides automatic, data-driven pruning with control over the tradeoff between network performance and scale.
Abstract
from arXiv · showhide
Many state-of-the-art computer vision algorithms use large scale convolutional neural networks (CNNs) as basic building blocks. These CNNs are known for their huge number of parameters, high redundancy in weights, and tremendous computing resource consumptions. This paper presents a learning algorithm to simplify and speed up these CNNs. Specifically, we introduce a "try-and-learn" algorithm to train pruning agents that remove unnecessary CNN filters in a data-driven way. With the help of a novel reward function, our agents removes a significant number of filters in CNNs while maintaining performance at a desired level. Moreover, this method provides an easy control of the tradeoff between network performance and its scale. Per- formance of our algorithm is validated with comprehensive pruning experiments on several popular CNNs for visual recognition and semantic segmentation tasks.
1. Introduction
Large CNNs are computationally expensive and over-parameterized, while existing pruning approaches leave data-driven filter selection and easy performance–scale control unresolved. The paper proposes try-and-learn pruning agents that automatically remove redundant filters while maintaining a specified performance level.
- Large CNNs contain many parameters, are computationally expensive, and are often over-parameterized.
- Filter pruning provides compression and speedup without specially designed sparse software or hardware, unlike individual-weight pruning.
- Whether data-driven filter pruning outperforms hand-crafted criteria remains an open question.
- Existing methods lack easy control over the tradeoff between network performance and scale, often requiring substantial human effort to test alternatives.
- The method automatically produces compact models at a specified performance level without human intervention.
- The method trains pruning agents to make binary filter-removal decisions from filter weights using a reward that favors pruning while constraining performance loss.
2. Related Works
Related work includes low-rank approximation, sparsity, quantization, filter pruning, and reinforcement-learning approaches. The paper distinguishes its method through data-driven filter selection and automatic performance–scale control.
- Low-rank approximation: Low-rank approximation decomposes CNN filters or features into lightweight layers using matrix factorization.
- Increasing Sparsity: Weight-sparsity methods remove small-weight connections but generally require masks and specialized software or hardware for real speedup.
- Quantization and binarization: Quantization and binarization reduce storage or computation through encoding and bit-wise operations, and can also complement filter pruning.
- Pruning filters: Prior filter pruning used magnitude-based criteria, whereas this work uses data-driven pruning and supports automatic performance–scale control.
- Reinforcement learning: Earlier reinforcement-learning methods generated network architectures from scratch and required substantial computing resources.
3. Method
The method trains neural pruning agents layer by layer to select filters using a reward that balances validation performance and pruning efficiency. Policy-gradient updates learn actions that remove filters while respecting a specified performance-drop bound.
- Overall procedure: The method prunes a baseline CNN by training pruning agents for its convolutional layers.
- Agent actions: Each agent takes filter weights as input and outputs binary actions indicating whether individual filters are removed or kept.
- Reward: The learning objective is a reward on validation data formed by multiplying accuracy and efficiency terms.
- Accuracy term: The accuracy term penalizes performance drops exceeding bound b, which controls the tradeoff between network performance and scale.
- Efficiency term: The efficiency term rewards keeping fewer filters, with C(A_l) denoting the number of kept filters.
- Optimization: Because the objective is non-differentiable, the method estimates gradients with REINFORCE policy gradients and normalized sampled rewards.
- Network-wide pruning: Pruning proceeds from lower to higher layers, fine-tuning the entire network after each layer to compensate for performance loss.
4. Experiments
Experiments evaluate the pruning algorithm across recognition and segmentation networks, showing efficient, controllable removal of redundant filters with maintained or improved performance. Results also compare data-driven pruning with magnitude-based pruning and examine how pruning varies across architectures and layers.
- Experimental setup: Experiments cover VGG-16 and ResNet-18 on CIFAR-10, plus FCN-32s on Pascal VOC and SegNet on CamVid.Pruning agents use a common neural-network design protocol, with implementation and measurement details specified for recognition and segmentation experiments.
- VGG-16 on CIFAR-10: VGG-16 pruning shows that higher layers contain more unnecessary filters, with over 95% removed from some layers while relatively preserving performance.For a 64-filter layer, the agent converged in fewer than 200 epochs despite 2^64 possible decisions.
- Tradeoff control: Larger drop bounds generally produce higher pruning ratios, greater FLOP savings, higher speedups, and larger test-set accuracy drops.The final accuracy drop can differ from the specified bound because of the validation-to-test generalization gap.
- VGG-16 on CIFAR-10: At matched pruning ratios, the data-driven method finds better VGG-16 filter combinations than magnitude-based pruning, yielding smaller accuracy drops.The experiments also report that pruning all VGG-16 layers remains aggressive, although ratios are slightly smaller than in single-layer pruning.
- ResNet-18 on CIFAR-10: ResNet-18 is pruned less aggressively than VGG-16 overall, but the method still removes many redundant filters; within residual blocks, first convolutions are easier to prune than second convolutions.Larger drop bounds increase pruning ratios across ResNet-18 layers.
- Segmentation experiments: The method removes 63.7% of redundant FCN-32s filters and accelerates inference by 37.0% on GPU and 49.1% on CPU.The two large converted fully connected layers contain 87.8% of network parameters, with 51.6% and 68.7% of their filters removed.
- SegNet on CamVid: On SegNet, the method removes near 56.9% of baseline parameters, speeds inference by 42.4% on GPU and 53.0% on CPU, and increases global accuracy by 2.1%.Magnitude-based pruning at the same pruning ratio decreases global accuracy by 3.0%; the second half of SegNet loses more filters than the first half.
5. Conclusion
The paper introduces a try-and-learn algorithm for data-driven CNN filter pruning and controlling the tradeoff between network performance and scale. Experiments cover visual recognition and semantic segmentation networks, including FCN-32s and SegNet visualizations.
- Conclusion: The method frames filter pruning as a try-and-learn learning task using policy gradients and a novel reward function.The reward encourages high pruning ratios while keeping pruned-network performance above a specified level.
- Conclusion: Segmentation experiments include pruning results for CNNs and visualization of FCN-32s and SegNet predictions across accuracy-change groups.The visualizations distinguish samples with increased, unchanged, and decreased global accuracy.
- Conclusion: The algorithm is designed to prune redundant filters aggressively while maintaining performance at a desired level.Its reward function provides control over the tradeoff between network performance and model scale.
- Conclusion: The method is benchmarked on widely used CNNs for visual recognition and semantic segmentation.The conclusion identifies FCN-32s on Pascal VOC and SegNet on CamVid among the segmentation experiments.