Pruning Filters for Efficient ConvNets

Hao Li; Asim Kadav; Igor Durdanovic; Hanan Samet; Hans Peter Graf

Source-linked AI summary

Pruning Filters for Efficient ConvNets

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf

arXiv:1608.08710v3Aug 31, 2016cs.CVcs.LG

TL;DR

CNNs have high computation and inference costs, while weight pruning can leave irregular convolutional sparsity. This paper prunes low-importance filters and their feature maps, achieving about 30% FLOP reduction for VGGNet and deep ResNets on CIFAR-10 without significant accuracy loss.

Problem
High-capacity CNNs incur substantial inference costs, while weight pruning may not adequately reduce convolutional computation because it creates irregular sparsity.
Method
The method ranks filters by ℓ1-norm, physically removes low-magnitude filters and corresponding feature maps, then retrains the resulting dense network.
Results
About 30% FLOP reduction was achieved for VGGNet and deep ResNets on CIFAR-10 without significant loss in original accuracy.
Takeaways & Limitations
Structured filter pruning reduces CNN computation without introducing irregular sparsity and supports straightforward tuning for a target speedup.
Takeaways & Limitations
Sparse CNN libraries are often limited, and maintaining sparse data structures can add significant storage overhead.

Abstract

from arXiv · show

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

1 INTRODUCTION

Deep CNNs incur growing inference costs, while weight pruning can leave convolutional computation inefficient because of irregular sparsity. The paper therefore proposes structured filter pruning to reduce computation without requiring sparse libraries or specialized hardware.

Motivation: Deepening CNN architectures increase parameters and convolution operations, creating substantial inference costs, especially for embedded applications.
Limitations of weight pruning: Magnitude-based weight pruning can compress models without hurting overall accuracy, but it may not adequately reduce convolutional computation.
Limitations of sparse execution: Sparse CNN acceleration libraries are limited, while maintaining sparse data structures can add significant storage overhead for low-precision weights.
Proposed approach: The paper prunes filters from well-trained CNNs, exploiting redundancy among filters and feature channels to reduce computation through structured sparsity-free pruning.
Proposed approach: Removing whole filters directly correlates with acceleration and avoids reliance on sparse libraries or specialized hardware.

2 RELATED WORK

Prior work reduces CNN redundancy through saliency-based weight pruning, neuron or feature-map removal, low-rank and FFT-based convolution approximations, and sparse constraints. This work selects unimportant filters with the ℓ1-norm, physically prunes them, and fine-tunes without additional regularization or layer-wise meta-parameters.

Optimal Brain Damage and Optimal Brain Surgeon prune unimportant weights using saliency measures and second-order derivative information, respectively.
Low-rank matrix products and FFT-based convolutions reduce convolutional computation without changing the original number of filters.
Other methods remove redundant feature maps using particle-filter-selected weight masks or infrequent activation detection on sample inputs.
Sparse-constraint approaches learn compact CNNs through group-wise convolution-kernel pruning or group-sparse regularization on neurons.
This approach uses the ℓ1-norm to select unimportant filters, physically prunes them, and fine-tunes conventionally without additional regularization.
The method adds no layer-wise regularizer meta-parameters beyond the filter-pruning percentage, which directly relates to desired speedup, and supports stage-wise pruning with one rate per stage.

3 PRUNING FILTERS AND FEATURE MAPS

The method selects low-importance convolutional filters using their absolute-weight sums, then removes those filters, corresponding feature maps, and next-layer kernels. Pruning can be performed independently or greedily across layers, with retraining strategies adapted to layer sensitivity and network architecture.

Filter and feature-map structure: Each convolutional filter generates one output feature map, so pruning a filter removes its feature map and the corresponding next-layer kernels.This converts filter removal into a reduction of dense convolutional dimensions rather than irregular weight sparsity.
Filter importance: The method ranks filters by the sum of their absolute kernel weights, using this ℓ1-norm as an indicator of expected output-feature-map magnitude.Because filters have the same number of input channels, the sum also represents average kernel-weight magnitude.
Pruning procedure: Pruning removes the filters with the smallest sums, their feature maps, and the corresponding kernels in the next convolutional layer before rebuilding both affected kernel matrices.The remaining kernel weights are copied into the new model.
Sensitivity and retraining: Layer sensitivity determines pruning amounts and retraining strategy: one-shot pruning can recover accuracy in resilient layers, whereas sensitive or heavily pruned networks may require iterative pruning and retraining.Residual architectures also impose structural restrictions on which filters can be pruned.
Network-wide pruning: Greedy pruning accounts for filters removed in previous layers and produces higher-accuracy pruned networks than independent selection, especially when many filters are removed.The greedy method excludes kernels associated with previously pruned feature maps when calculating later filter sums.

4 EXPERIMENTS

Experiments evaluate filter pruning on VGG-16 and ResNets across CIFAR-10 and ImageNet, showing that pruning sensitivity varies by layer and residual stage. Retraining and ℓ1-norm-based selection support substantial pruning while preserving or improving accuracy relative to alternatives.

VGG-16 on CIFAR-10: VGG-16 on CIFAR-10 has 13 convolutional and 2 fully connected layers, with fully connected layers occupying a small parameter fraction.The small input size and fewer hidden units make fully connected pruning less consequential.
VGG-16 on CIFAR-10: 60% of filters can be dropped from each 512-feature-map convolutional layer without affecting accuracy, while retraining enables almost 90% removal.These layers operate on 4 × 4 or 2 × 2 feature maps, which may limit meaningful spatial connections.
ResNets on CIFAR-10: ResNet-56-pruned-A improves performance by pruning 10% filters while skipping sensitive layers 16, 20, 38 and 54.Deeper layers are more sensitive, so pruning rates differ across residual stages.
ResNet-34 on ImageNet: ResNet-34-pruned-C prunes the third stage with p3=20%, achieving 7.5% FLOP reduction with 0.75% loss in accuracy.The first layer of each residual block is more effective for reducing overall FLOPs than the second layer.
Pruning criteria: Smallest-filter pruning outperforms random-filter pruning for most layers and pruning ratios, including every layer at a 90% pruning ratio.Pruning filters with the largest ℓ1-norms causes accuracy to drop quickly as the pruning ratio increases.
Pruning criteria: Smallest-filter pruning outperforms four activation-based feature-map criteria, while σmean-std performs better or similarly to ℓ1-norm pruning up to a 60% pruning ratio.The activation statistics use the whole CIFAR-10 training set, N = 50,000.

5 CONCLUSIONS

The paper presents filter pruning based on relatively low weight magnitudes to reduce CNN computation without introducing irregular sparsity. It reports about 30% FLOP reduction for VGGNet on CIFAR-10 and deep ResNets without significant accuracy loss.

Method: Filter pruning targets CNN filters with relatively low weight magnitudes to reduce computation costs.The method is designed for modern CNNs with high training and inference costs.
Method: The approach reduces computation without introducing irregular sparsity.This distinguishes filter pruning from methods that produce irregular sparse connectivity.
Results: about 30% FLOP reduction is achieved for VGGNet on CIFAR-10 and deep ResNets without significant loss in original accuracy.The reported reduction applies to both model settings named in the passage.

6 APPENDIX

The appendix compares ℓ1- and ℓ2-norm filter-pruning criteria and explains how physically creating smaller dense models enables FLOP-based and wall-clock inference measurements. Experiments report inference-time reductions on CIFAR-10 and, for ResNet-34, ILSVRC 2012, with saved time close to FLOP reduction.

Norm comparison: ℓ1-norm pruning works slightly better than ℓ2-norm pruning for VGG-16 layer conv 2, with no significant difference in other layers.The comparison is shown for VGG-16 on CIFAR-10.
Implementation: Physical filter pruning creates a smaller model without masks or sparsity, preserving dense BLAS operations.The resulting model has fewer filters and copied weights.
Evaluation: Inference time is evaluated on CIFAR-10 test data and, for ResNet-34 only, ILSVRC 2012 validation data.The datasets contain 10,000 32 × 32 images and 50,000 224 × 224 images, respectively.
Evaluation: Saved inference time is close to the FLOP reduction for the evaluated pruned models.Evaluation uses Torch7 with a Titan X (Pascal) GPU, cuDNN v5.1, and mini-batches of 128.