Source-linked AI summary
An Entropy-based Pruning Method for CNN Compression
Jian-Hao Luo, Jianxin Wu
TL;DR
CNNs are difficult to deploy on resource-constrained devices because of their computational, storage, and activation-memory demands. The paper uses entropy-based filter pruning, fine-tuning, and a learning schedule to compress and accelerate CNNs. On ILSVRC-12, it reports substantial VGG-16 and ResNet-50 gains with about a 1% top-5 accuracy decrease.
Problem
CNN deployment is constrained by computational, storage, and intermediate-activation memory costs, especially on small devices and with large batch sizes.
Method
The framework evaluates filter importance with entropy-based channel selection, prunes weak filters, fine-tunes the model, and uses a learning schedule for iterative pruning.
Results
On ILSVRC-12, the method achieves 3.3× acceleration and 16.64× compression on VGG-16, and 1.54× acceleration and 1.47× compression on ResNet-50, with roughly 1% top-5 accuracy drops.
Takeaways & Limitations
The pruned model can run with current popular deep learning libraries without dedicated library support.
Takeaways & Limitations
Pruning too many layers at once can significantly damage accuracy, while fine-tuning the whole network after each pruning step requires many epochs for deep networks.
Abstract
from arXiv · showhide
This paper aims to simultaneously accelerate and compress off-the-shelf CNN models via filter pruning strategy. The importance of each filter is evaluated by the proposed entropy-based method first. Then several unimportant filters are discarded to get a smaller CNN model. Finally, fine-tuning is adopted to recover its generalization ability which is damaged during filter pruning. Our method can reduce the size of intermediate activations, which would dominate most memory footprint during model training stage but is less concerned in previous compression methods. Experiments on the ILSVRC-12 benchmark demonstrate the effectiveness of our method. Compared with previous filter importance evaluation criteria, our entropy-based method obtains better performance. We achieve 3.3x speed-up and 16.64x compression on VGG-16, 1.54x acceleration and 1.47x compression on ResNet-50, both with about 1% top-5 accuracy decrease.
1. Introduction
The paper targets CNN deployment constraints by pruning unimportant filters while reducing intermediate activations, training and inference costs, and model size. An entropy-based selection framework with fine-tuning and a learning schedule achieves substantial compression and acceleration on VGG-16 and ResNet-50.
- VGG-16 has 138.34 million parameters, exceeds 500MB storage, and requires 15.5 billion FLOPs per image, challenging deployment on small devices.
- Intermediate activations can dominate memory use, reaching 109.89MB per VGG-16 image and at least 12.25MB during forward inference.Reducing activations also accelerates inference and lowers memory consumption.
- The proposed framework prunes unimportant filters using an entropy-based channel-selection metric and fine-tunes the pruned model to recover discrimination ability.The approach is designed for both training and test stages.
- The method achieves 3.3× acceleration and 16.64× compression on VGG-16, with about a 1% top-5 accuracy drop.
- A learning schedule seeks a better trade-off between training speed and classification accuracy in the iterative pruning framework.
- The framework is supported by current popular deep learning libraries without dedicated libraries and aims to reduce intermediate-activation memory consumption.
2. Related work
Related work covers parameter-focused compression, pruning, quantization, compact architectures, knowledge distillation, and binary networks. The paper distinguishes its filter-pruning approach by targeting activation size and using entropy-based selection with different fine-tuning.
- Low-rank approximation: Low-rank approximation represents dense parameter matrices with smaller low-rank matrices to reduce memory footprint and computational complexity.Fast techniques such as the FFT can accelerate the resulting matrix-vector multiplication.
- Network pruning: Network pruning removes redundant connections or filters, but connection pruning may leave activation tensors large when batch size is high.Prior filter-pruning methods use data-driven neuron importance or absolute weight sums, whereas this paper uses a different channel-selection and fine-tuning strategy.
- Parameter quantization: Parameter quantization compresses models by quantizing parameter representations, including product quantization, hashing, and shared values within hash buckets.
- Designing more compact architecture: Compact architectures reduce parameters by replacing fully connected layers with average pooling, as in Network-In-Network, GoogLeNet, and ResNet.
- Knowledge distillation: Knowledge distillation trains a smaller student model to mimic knowledge from a cumbersome teacher model.The student can even outperform the teacher under the described approach.
- Binary network: Binary networks reduce model size by representing parameters with binary values and can accelerate multiplication using dedicated hardware such as XNOR gates.The cited work describes progress from small datasets to ImageNet-scale models.
3. Entropy-based pruning method
The method ranks filters by activation-channel entropy, removes low-information filters and corresponding next-layer channels, then fine-tunes the pruned network. It supports architecture-specific pruning strategies and an iterative learning schedule intended to balance accuracy recovery with training cost.
- Overall framework: Pruning removes weak filters and the corresponding input channels of the next layer, reducing parameters, running time, memory consumption, and activation sizes.The framework then fine-tunes the whole network because weak filters may still store useful knowledge and pruning can affect generalization.
- Entropy-based filters selection: Entropy measures activation-channel information to estimate filter importance: lower entropy indicates a less important channel that can be removed.The method collects channel outputs, bins their values, computes entropy, and uses the resulting score to rank filters.
- Entropy-based filters selection: Global average pooling converts each layer output from c × h × w into a 1 × c vector before entropy is computed across an evaluation set.The resulting matrix contains one score distribution per channel, enabling channel-wise entropy estimation.
- Pruning boundary: Filters can be pruned by thresholding entropy scores or by retaining a fixed number of top-ranked filters, avoiding a difficult threshold hyperparameter.The constant compression-rate strategy sorts filters in descending entropy order and preserves only the top k.
- Network-specific pruning strategies: For VGG-16, the method prunes the first ten convolutional layers because they contain more than 90% of FLOPs, while fully connected layers are replaced with global average pooling.This strategy targets computationally dominant layers and the model’s large parameter contribution from fully connected layers.
- Network-specific pruning strategies: For ResNet, only intermediate convolutional layers are pruned because channel consistency constrains the last layer of each residual block; this also reduces the next layer’s parameters.The strategy is illustrated for each residual block and avoids requiring projection shortcuts for mismatched sums.
- Learning schedule: Iterative pruning is preferred over pruning once because removing too many layers at once can cause unrecoverable accuracy loss, although iterative fine-tuning is time-consuming.The learning schedule reduces overall training time and is intended to prevent poor local minima early in fine-tuning.
4. Experiments
Experiments on VGG-16 and ResNet-50 evaluate pruning speed, compression, accuracy, learning schedules, baselines, and transfer to other recognition tasks. The results support entropy-based filter pruning with fine-tuning and a schedule designed to balance training speed and accuracy.
- VGG-16 on ImageNet: 3.3× speed-up is achieved for VGG-16 in both training and inference after pruning half the filters in each layer.With mini-batch size 50 on a K80 GPU, forward/backward time falls from 863.04ms/2268.74ms to 349.21ms/869.45ms.
- VGG-16 on ImageNet: The Pruned-GAP model is fine-tuned after replacing fully-connected layers with global average pooling, producing slightly lower classification accuracy than the original model.The authors motivate global average pooling as less prone to overfitting and report that further pruning could produce a smaller model.
- VGG-16 on ImageNet: Compared with APoZ and Taylor expansion baselines, the proposed method performs better on VGG-16 compression experiments.APoZ-1 degrades accuracy, APoZ-2 has little acceleration effect, and the Taylor method causes significant performance degradation while the proposed method can improve accuracy.
- Effectiveness analysis: 58.82%/81.91% Top-1/Top-5 accuracy results when the same layers are pruned at one time, substantially worse than iterative pruning.The comparison uses the same number of fine-tuning epochs, while the iterative strategy avoids pruning too many filters simultaneously.
- Effectiveness analysis: The learning schedule reduces training epochs dramatically while producing better classification accuracy than naive iterative pruning.The schedule is intended to balance training speed and accuracy and is reported as more efficient than the naive strategy.
5. Conclusion
The entropy-based framework accelerates and compresses CNNs during training and inference without dedicated libraries. Future work includes dynamic pruning, improved channel selection, and evaluation on more vision tasks.
- The framework simultaneously accelerates and compresses CNN models during training and inference.
- The method is designed for broad use with current deep learning libraries because it requires no dedicated library.
- Future work includes dynamic pruning and improved channel-selection methods.
- Evaluating pruned networks on additional vision tasks, including detection, segmentation, and depth estimation, remains an open direction.