Source-linked AI summary
HRank: Filter Pruning using High-Rank Feature Map
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, Ling Shao
TL;DR
Filter pruning still lacks practical and theoretical guidance for identifying important or redundant filters, while CNN computation and memory demands hinder deployment on resource-limited devices. HRank ranks feature maps to guide filter removal, and experiments report improved compression and acceleration over state-of-the-art methods without additional auxiliary constraints.
Problem
CNN computation and memory demands hinder deployment on edge devices, while filter pruning remains constrained by limited guidance on filter importance and redundancy.
Method
HRank determines filter importance from feature-map rank, removes lower-rank filters, and can freeze some high-rank filters during fine-tuning.
Results
HRank demonstrates improved acceleration and compression over state-of-the-art property-importance and adaptive-importance methods across various modern CNNs.
Takeaways & Limitations
HRank provides efficient pruning guidance without additional auxiliary constraints or retraining, while freezing high-rank filters can reduce fine-tuning cost with little performance compromise.
Takeaways & Limitations
Prior data-driven sparsity methods require over ten thousand input images for reasonable prediction, and further theoretical analysis of HRank remains future work.
Abstract
from arXiv · showhide
Neural network pruning offers a promising prospect to facilitate deploying deep neural networks on resource-limited devices. However, existing methods are still challenged by the training inefficiency and labor cost in pruning designs, due to missing theoretical guidance of non-salient network components. In this paper, we propose a novel filter pruning method by exploring the High Rank of feature maps (HRank). Our HRank is inspired by the discovery that the average rank of multiple feature maps generated by a single filter is always the same, regardless of the number of image batches CNNs receive. Based on HRank, we develop a method that is mathematically formulated to prune filters with low-rank feature maps. The principle behind our pruning is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced. Besides, we experimentally show that weights with high-rank feature maps contain more important information, such that even when a portion is not updated, very little damage would be done to the model performance. Without introducing any additional constraints, HRank leads to significant improvements over the state-of-the-arts in terms of FLOPs and parameters reduction, with similar accuracies. For example, with ResNet-110, we achieve a 58.2%-FLOPs reduction by removing 59.2% of the parameters, with only a small loss of 0.14% in top-1 accuracy on CIFAR-10. With Res-50, we achieve a 43.8%-FLOPs reduction by removing 36.7% of the parameters, with only a loss of 1.17% in the top-1 accuracy on ImageNet. The codes can be available at https://github.com/lmbxmu/HRank.
1. Introduction
HRank addresses the efficiency and labor costs of filter pruning by using feature-map rank as practical guidance for selecting filters. It reports that low-rank filters are less informative and that HRank improves compression and acceleration without additional constraints.
- 1. Introduction: Filter pruning seeks high compression and low accuracy compromise, but existing approaches face either inefficient compression or heavy retraining costs.Property-importance methods avoid modified losses but can be suboptimal, while adaptive-importance methods require joint retraining and additional machine and human labor.
- 1. Introduction: HRank estimates feature-map rank from convolutional outputs and uses it as the criterion for pruning filters.The framework obtains feature maps, estimates their ranks, prunes selected filters, and fine-tunes the remaining network.
- 1. Introduction: The average rank of feature maps generated by a single filter remains almost unchanged across the amount of data seen by the CNN.This observation suggests that rank can be estimated efficiently from a small portion of input images.
- 1. Introduction: Low-rank feature maps are mathematically shown to be less informative, so their filters can be removed first while high-rank filters are more important to preserve.The paper also reports experiments supporting the relationship between feature-map rank and filter importance.
- 1. Introduction: Extensive experiments demonstrate HRank’s efficiency and effectiveness for model compression and acceleration across representative CNNs and benchmarks.Experiments cover CIFAR-10 and ImageNet with VGGNet, GoogLeNet, ResNet, and DenseNet, and compare against property- and adaptive-importance pruning methods.
2. Related Work
Related work contrasts structured filter pruning with weight pruning and distinguishes property-based from adaptive importance approaches. The discussion motivates HRank as an attempt to address compression efficiency and retraining cost through better guidance on filter importance.
- 2. Related Work: Filter pruning removes entire filters, reducing storage and inference computation, whereas weight pruning removes entries from weight matrices.Structured filter removal can be integrated with BLAS libraries without specialized software or hardware support.
- 2. Related Work: Low-rank decomposition compresses convolutional operations by representing weight matrices as products of smaller matrices rather than removing filters.It is presented as an alternative compression strategy for over-parameterized neural networks.
- 2. Related Work: Existing filter-pruning methods face inefficient acceleration and compression or high machine and labor costs because practical and theoretical guidance on filter importance and redundancy is missing.HRank focuses on feature-map rank as such guidance.
3. The Proposed Method
HRank measures feature-map information using rank and prunes filters with the lowest average ranks. The method is tractable because rank expectations are stable across input batches, allowing offline estimation from relatively few images.
- HRank: Filter pruning is formulated to remove the least important filters, where importance is measured from the information content of their generated feature maps.The method defines the importance function on feature maps rather than directly on filter properties, incorporating both filter properties and input images.
- Notations: The pruning formulation ignores the computational cost of nonlinear operations such as pooling, batch normalization, ReLU, and dropout when calculating FLOPs and parameters.This simplification is used for ease of FLOPs and parameter calculation.
- HRank: HRank defines feature-map rank as an information measure, with higher-rank maps containing more information than lower-rank maps.A rank-r feature map can be decomposed into a lower-rank component and additional information.
- HRank: The average rank of feature maps generated by a filter is robust to input images, so a small batch can accurately estimate its expected rank.The observed rank variance across different images is negligible, making rank estimation less dependent on the input distribution.
- Tractability of Optimization: The method estimates average ranks offline, re-ranks filters in decreasing order, preserves the highest-ranked filters, and removes the lowest-ranked set before fine-tuning.The experiments use 500 images to estimate average ranks, compared with over ten thousand images used for feature-map sparsity estimation in prior work.
4. Experiments
Experiments evaluate HRank across mainstream CNN architectures on CIFAR-10 and ImageNet using accuracy, parameter reduction, and FLOPs reduction. Across these settings, HRank achieves strong compression and acceleration, while ablations support preserving high-rank filters.
- Experimental Settings: Experiments cover VGGNet, GoogLeNet, ResNet, and DenseNet on CIFAR-10, plus ResNet-50 on ImageNet.The evaluation uses top-1 accuracy and pruning rates on CIFAR-10, and top-1/top-5 accuracy on ImageNet.
- VGGNet: 53.5% parameter and 82.9% FLOPs reductions are achieved by HRank on VGG-16, exceeding L1 pruning reductions of 34.3% and 64.0%.HRank also reports higher accuracy than several adaptive methods, including SSS, Zhao et al., and GAL variants.
- GoogLeNet: 94.53% top-1 accuracy is achieved on GoogLeNet with around 55% FLOPs and parameter reductions, outperforming the cited L1, randomized, and adaptive baselines.The results indicate applicability to inception modules without additional constraints.
- ResNet-56/110: 58.2% FLOPs and 59.2% parameter reductions are achieved on ResNet-110 with only a 0.14% accuracy loss versus GAL-0.5.HRank also improves accuracy over the baseline and L1 under the reported ResNet-110 settings.
- DenseNet-40: Over 40.8% FLOPs are removed on DenseNet-40, while HRank reports higher accuracy and FLOPs reduction than Zhao et al. and GAL-0.05.The authors state that HRank can also be applied to networks with dense blocks.
- ResNet-50 on ImageNet: 2.30B FLOPs and 16.15M parameters are removed from ResNet-50 while retaining 74.98% top-1 and 92.33% top-5 accuracy on ImageNet.The passage reports that HRank surpasses its counterparts across accuracy and complexity measures in the evaluated settings.
- Ablation Study: 93.13% top-1 accuracy is retained when around 15%–20% of filters remain untrained during fine-tuning, versus 93.17% when all reserved filters are updated.With around 20%–25% of weights untrained, accuracy is 93.01%, a 0.11% drop.
5. Conclusions
HRank ranks filters by their feature maps and removes lower-rank filters, while freezing some high-rank filters during fine-tuning to reduce training cost with little performance compromise.
- HRank determines filter importance by observing the rank of feature maps generated by each filter.
- With no filters frozen, fine-tuning yields 93.17% top-1 precision.
- Freezing 15%–20% of filters during fine-tuning yields 93.13% top-1 precision.
- Figure 5 compares layer-wise percentages of pruned, trained, and untrained filters when assessing freezing effects.
- Lower-rank feature maps are identified as less important and targeted for removal.