Source-linked AI summary
CHIP: CHannel Independence-based Pruning for Compact Neural Networks
Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Zonouz, Bo Yuan
TL;DR
CNN compression requires reducing computation and storage, while existing filter-pruning methods often rely on intra-channel importance measures. CHIP introduces inter-channel Channel Independence to identify redundant filters, and reports accuracy gains alongside substantial storage and computation reductions across CIFAR-10 and ImageNet evaluations.
Problem
Existing filter-pruning methods predominantly assess filter importance using intra-channel information, motivating investigation of inter-channel information for efficient compression.
Method
CHIP uses Channel Independence, a cross-channel correlation metric, and studies its quantification, measurement scheme, sensitivity, and reliability for filter pruning.
Results
Across evaluated models and datasets, CHIP reports high pruning performance with preserved accuracy and substantial storage and computational reductions.
Takeaways & Limitations
Lower-independence feature maps are treated as containing redundant information, so their corresponding filters can be pruned without affecting model capacity.
Takeaways & Limitations
The paper notes that the near-equivalence between CHIP's pruned-filter indices and globally optimal methods has not been theoretically proven.
Abstract
from arXiv · showhide
Filter pruning has been widely used for neural network compression because of its enabled practical acceleration. To date, most of the existing filter pruning works explore the importance of filters via using intra-channel information. In this paper, starting from an inter-channel perspective, we propose to perform efficient filter pruning using Channel Independence, a metric that measures the correlations among different feature maps. The less independent feature map is interpreted as containing less useful information$/$knowledge, and hence its corresponding filter can be pruned without affecting model capacity. We systematically investigate the quantification metric, measuring scheme and sensitiveness$/$reliability of channel independence in the context of filter pruning. Our evaluation results for different models on various datasets show the superior performance of our approach. Notably, on CIFAR-10 dataset our solution can bring $0.90\%$ and $0.94\%$ accuracy increase over baseline ResNet-56 and ResNet-110 models, respectively, and meanwhile the model size and FLOPs are reduced by $42.8\%$ and $47.4\%$ (for ResNet-56) and $48.3\%$ and $52.1\%$ (for ResNet-110), respectively. On ImageNet dataset, our approach can achieve $40.8\%$ and $44.8\%$ storage and computation reductions, respectively, with $0.15\%$ accuracy increase over the baseline ResNet-50 model. The code is available at https://github.com/Eclipsess/CHIP_NeurIPS2021.
1 Introduction
Filter pruning offers hardware-friendly acceleration, but existing methods often assess filter importance using only intra-channel information. CHIP instead uses inter-channel Channel Independence to identify redundant feature maps and prune filters while preserving accuracy and reducing costs.
- Existing Filter Pruning Methods: Filter pruning removes entire filters, producing structured sparsity that CPUs and GPUs can exploit for practical acceleration.Weight pruning can achieve high compression but creates unstructured sparsity patterns that general-purpose hardware does not support well.
- Determining Importance: Intra-channel & Inter-channel Perspectives: Existing feature-guided methods commonly measure a feature map's importance using information from its own channel בלבד.This intra-channel perspective evaluates each feature map and corresponding filter independently of other channels.
- Benefits of Inter-channel Perspective: Inter-channel information can provide more stable measurements and identify unnecessary correlations among feature maps, supporting improved pruning and compression.Cross-channel features may reduce sensitivity to input data while capturing redundancy between channels.
- Technical Preview and Contributions: Channel Independence measures cross-channel feature-map correlations and treats lower independence as evidence that a filter's information is encoded elsewhere.The corresponding filter is therefore considered unimportant and removable without affecting model capacity.
- Technical Preview and Contributions: CHIP systematically studies channel-independence quantification, measurement complexity, sensitivity, and reliability, then applies it to efficient filter pruning.The proposed calculation scheme is described as low-cost, fine-grained, and robust.
- Technical Preview and Contributions: 0.90% and 0.94% accuracy increases accompanied 42.8% and 48.3% model-size reductions for baseline ResNet-56 and ResNet-110, respectively.The corresponding FLOPs reductions were 47.4% and 52.1%; ImageNet ResNet-50 achieved 40.8% storage and 44.8% computation reductions with a 0.15% accuracy increase.
2 Preliminaries
Network pruning can be formulated as constrained loss minimization, while feature-guided filter pruning selects important output feature maps and preserves their corresponding filters.
- Filter Pruning: A convolutional layer contains output filters whose dimensions depend on input channels, output channels, and kernel size.For layer l, the output-channel count is c_l, input-channel count is c_{l-1}, and kernel size is k_l.
- Filter Pruning: Network pruning minimizes model loss subject to preserving at most κ_l non-zero filters in each layer.The loss uses ground-truth labels, input data, and the CNN output function; the ℓ0-norm counts non-zero filters.
- Feature-guided Filter Pruning: Feature-guided pruning minimizes a loss defined over layer feature maps rather than directly optimizing the filters.This approach uses feature information from both filters and input data.
- Feature-guided Filter Pruning: After identifying κ_l important feature maps, pruning preserves their corresponding κ_l filters.The layer output is represented as a set of c_l feature maps, with one map associated with each channel.
3 The Proposed Method
The method measures filter importance from inter-channel relationships, using channel independence to identify feature maps whose information is replaceable. It uses nuclear-norm changes for efficient pruning, approximates combined independence through individual measurements, and examines reliability across input batches.
- Inter-channel filter importance: Channel independence measures correlations among multiple feature maps to determine filter importance from an inter-channel perspective.Lower independence indicates greater linear dependence on other channels and less unique information.
- Metric selection: Nuclear-norm change provides a finer-grained independence measure than rank change when a feature map is removed.Rank changes can remain nearly identical across removed maps, whereas nuclear-norm changes vary significantly.
- Pruning scheme: Brute-force evaluation of all m-row removals is computationally impractical, requiring more than 5 × 10^75 nuclear-norm calculations for one example.The example considers pruning 50% of filters from a 256-output-channel ResNet-50 layer.
- Pruning scheme: Individual independence measurements approximate combined independence while reducing complexity from O(C(N, κ)) to O(N).The paper reports that this approximation still achieves superior filter-pruning performance.
- Pruning scheme: The pruning scheme removes individual feature maps with the smallest nuclear-norm changes and prunes their corresponding filters.These selected maps are interpreted as less independent from the remaining feature maps.
- Reliability: Average channel independence is stable across input batches, supporting measurement with small batches of image samples.The paper also reports that additional learning-based mask adjustment does not easily improve accuracy or compression at the target settings.
4 Experiments
Experiments evaluate CHIP across CNN models and datasets, using averaged channel independence to select filters for pruning and subsequent fine-tuning. The approach reduces storage and computation while maintaining or improving accuracy.
- CHIP is evaluated on ResNet-56, ResNet-110, and VGG-16 using CIFAR-10, and on ResNet-50 using ImageNet.
- CHIP computes channel independence from feature maps across sampled inputs, averages the values, prunes filters with the smallest scores, and fine-tunes the resulting model.
- 0.90% accuracy increase accompanies 42.8% model size and 47.4% FLOPs reductions for ResNet-56 on CIFAR-10.
- 0.94% accuracy increase accompanies 48.3% model size and 52.1% FLOPs reductions for ResNet-110 on CIFAR-10.
- 81.6% model size and 58.1% FLOPs reductions are achieved for VGG-16 on CIFAR-10 with only a 0.1% accuracy drop.
- 40.8% storage and 44.8% computation reductions accompany a 0.15% accuracy increase for ResNet-50 on ImageNet.
5 Conclusion
CHIP uses channel independence, an inter-channel metric, to evaluate filter importance and achieve storage and computational reductions while preserving high model accuracy.
- CHIP evaluates filter importance with channel independence and reports significant storage and computational cost reductions while preserving high model accuracy.
Broader Impact
Network pruning helps resource-constrained devices run increasingly complex neural-network tasks with smaller models and faster inference.
- Pruning reduces memory size and accelerates inference, enabling IoT devices to execute complex projects with small and efficient models.
6 Additional Studies
Additional studies examine channel independence’s relation to feature-map importance, pruning trade-offs, input sensitivity, and possible mask adjustment. The studies support its consistency and show no added accuracy from further mask training, while theoretical optimality remains unproven.
- 6.1 Relationship between Channel Independence and Importance of Feature Map: A numerical example links lower channel independence with feature-map redundancy and identifies Row-2 as the least important row.
- 6.1 Relationship between Channel Independence and Importance of Feature Map: Channel independence assigns the smallest score to Row-2, matching the expectation based on its linear dependence and smaller l2-norm.
- 6.2 Accuracy-Pruning Rate Trade-off: CHIP’s ImageNet pruning curve examines how test accuracy changes with pruning ratio while reducing model parameters and FLOPs.
- 6.3 Comparison with Other Methods: Accuracy-pruning trade-off curves compare CHIP with SCOP and HRank for ResNet-50 on ImageNet.
- 6.4 Quantified Sensitiveness of Channel Independence to Input Data: Channel independence is reported as insensitive to input data because vectors computed from different batches are highly correlated.
- 6.5 Is Additional Adjustment of Importance Ranking Needed?: An additional learned mask-adjustment phase is proposed as a possible extension, but experiments find no extra accuracy increase and unchanged selected filters.
- 6.5 Is Additional Adjustment of Importance Ranking Needed?: The authors report that experimentally pruned-filter indices nearly match those from global-optimal methods, although this has not been proven theoretically.
7 Detailed Setting of κl and Pruning Ratios
This section reports the preserved-filter counts κl and pruning ratios for each layer across the evaluated CIFAR-10 and ImageNet models.
- κl and layer-wise pruning ratios are reported for ResNet-56, ResNet-110, and VGG-16 on CIFAR-10, and ResNet-50 on ImageNet.