Source-linked AI summary
SCOP: Scientific Control for Reliable Neural Network Pruning
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, Chang Xu
TL;DR
Filter pruning needs reliable importance estimates because existing hypotheses can be disturbed by irrelevant factors. SCOP introduces label-independent knockoff features as a scientific control, propagates them through the network, and prunes filters using competing scaling factors. The method reports higher performance at similar compression or acceleration ratios, including 57.8% parameter and 60.2% FLOP reductions for ResNet-101 with 0.01% top-1 accuracy loss on ImageNet.
Problem
Existing filter-pruning hypotheses can be affected by irrelevant factors, making reliable filter-importance estimation difficult.
Method
SCOP generates knockoff features with similar distributions but no label information, propagates them through network layers, and compares their scaling factors with real features.
Results
SCOP achieves higher performance at similar compression or acceleration ratios and reduces 57.8% of ResNet-101 parameters and 60.2% of FLOPs with 0.01% top-1 accuracy loss on ImageNet.
Takeaways & Limitations
The scientific-control design supports more reliable discovery of redundant filters for compact neural networks and edge-device deployment.
Takeaways & Limitations
The second-order knockoff treatment focuses on matching expectations and covariances rather than the full distributions.
Abstract
from arXiv · showhide
This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Existing pruning methods have developed various hypotheses to approximate the importance of filters to the network and then execute filter pruning accordingly. To increase the reliability of the results, we prefer to have a more rigorous research design by including a scientific control group as an essential part to minimize the effect of all factors except the association between the filter and expected network output. Acting as a control group, knockoff feature is generated to mimic the feature map produced by the network filter, but they are conditionally independent of the example label given the real feature map. We theoretically suggest that the knockoff condition can be approximately preserved given the information propagation of network layers. Besides the real feature map on an intermediate layer, the corresponding knockoff feature is brought in as another auxiliary input signal for the subsequent layers. Redundant filters can be discovered in the adversarial process of different features. Through experiments, we demonstrate the superiority of the proposed algorithm over state-of-the-art methods. For example, our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet. The code is available at https://github.com/huawei-noah/Pruning/tree/master/SCOP_NeurIPS2020.
1 Introduction
CNNs deliver strong computer-vision performance but are difficult to deploy on resource-constrained edge devices. SCOP targets unreliable filter-importance estimates by introducing a scientific control for pruning.
- CNNs are widely used in image classification, object detection, and video analysis but require substantial computing power and memory.
- Filter pruning removes entire redundant filters to build compact architectures while enabling practical acceleration.
- Existing pruning methods estimate filter importance using hypotheses such as norm, geometric median, or association with loss.
- Potential factors including channel dependence and input fluctuations can distort importance rankings and make pruning results unstable.
- SCOP introduces knockoff features as a control group to reduce irrelevant-factor disturbance and identify redundant filters more reliably.
2 Preliminaries and Motivation
The paper recasts filter pruning as feature selection: preserve intermediate features associated with labels and remove redundant ones. Knockoff counterparts provide a control that shares feature behavior while lacking label information.
- Filter pruning can be formulated as selecting a limited number of filters that minimize task loss under an ℓ0 constraint.
- Features produced by filters are used to assess importance because their association with labels directly reflects relevance to prediction.
- Interfeature dependence, input fluctuations, and method-specific factors complicate reliable discovery of label-related features.
- A knockoff counterpart has the same shape as the real feature and satisfies exchangeability and conditional independence properties.
- Real and knockoff features differ in that real features may associate with labels, whereas knockoffs are conditionally independent of labels given real features.
3 Approach
SCOP generates knockoff data, propagates approximate knockoff features through network layers, and jointly trains real and control scaling factors to expose redundant filters. Filters are ranked by the competition between real and knockoff features.
- 3.1 Knockoff Data and Features: Knockoff conditions are preserved across element-wise activations and approximately across linear transformations, enabling feature knockoffs throughout deep networks.The analysis uses exchangeability and conditional independence for activations, and first- and second-moment matching for linear layers.
- 3.1 Knockoff Data and Features: Only input knockoffs need to be generated: feeding real and knockoff data through the network derives corresponding features in all layers.
- 3.2 Filter Pruning with Scientific Control: SCOP feeds real and knockoff features into the next layer through adversarial selection with scaling factors constrained by βl + ˜βl = 1.The pretrained network parameters remain fixed while the scaling factors are optimized under label supervision.
- 3.2 Filter Pruning with Scientific Control: The real feature is treated as the treatment group and the knockoff feature as a control group that lacks label information.
- 3.2 Filter Pruning with Scientific Control: Filter importance is measured by Il = βl − ˜βl; filters whose real features cannot suppress knockoff counterparts should be pruned.For architectures with batch normalization, the statistic is additionally scaled by the absolute batch-normalization scale.
4 Experiments
Experiments on CIFAR-10 and ImageNet evaluate SCOP across pruning rates, architectures, comparisons, acceleration, control-group ablations, and feature visualizations. The results show competitive accuracy and substantial reductions in parameters, FLOPs, and errors, while supporting knockoffs as an effective control group.
- Experimental setup: SCOP is evaluated on CIFAR-10 and ImageNet using pruned ResNet models and MobileNetV2.The experiments use benchmark datasets and compare networks with different depths and architectures.
- Comparison on CIFAR-10: 56.0% FLOPs reduction yields 6.36% error with a 0.06% accuracy drop for ResNet-56 on CIFAR-10.This result is reported as better than other compared filter-pruning methods in the supplied passage.
- Comparison on CIFAR-10: 40.3% FLOPs reduction incurs only a 0.24% accuracy drop for MobileNetV2 on CIFAR-10.The result is reported even for the compact MobileNetV2 architecture.
- Comparison on ImageNet: On ImageNet, SCOP-B reaches 24.74% top-1 error with 54.6% FLOPs reduction on ResNet-50, compared with Taylor’s 25.50% error and 44.9% reduction.The passage reports lower errors and greater FLOPs reductions than several existing criteria and superiority over other compared methods.
- Varying pruning rate: Increasing pruning rate rapidly reduces parameters and FLOPs, while ResNet-56 accuracy drops negligibly even when 50% of parameters and FLOPs are pruned.The pruning-rate study examines accuracies together with parameter and FLOPs reductions on CIFAR-10.
- Ablation studies: Removing the knockoff control increases ResNet-56 test error from 6.36% to 6.83%, while noise and random-sample controls produce larger errors than knockoffs.The ablations attribute the difference to the control group's role in excavating redundant filters and its exchangeability and label-information properties.
5 Conclusion
SCOP improves pruning reliability by using knockoff features as a scientific control and removes filters that favor knockoff samples over real data. Experiments report strong compression with minimal accuracy loss.
- 5 Conclusion: SCOP introduces knockoff features as a control group to improve the reliability of neural network pruning.The knockoffs have a similar distribution to real features but contain no ground-truth label information.
- 5 Conclusion: Filters that attend more to knockoff samples than real data are removed to obtain compact neural networks.
- 5 Conclusion: 57.8% parameters and 60.2% FLOPs are reduced for ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet.
- 5 Conclusion: The authors plan to investigate scientific control design in other deep learning problems, including neural architecture search.
Broader Impact
Network pruning supports deployment of deep neural networks on edge devices by reducing their computational and memory requirements. This can make powerful models usable on devices with limited computing capability.
- Broader Impact: Network pruning accelerates inference and reduces memory requirements for deep neural networks.
- Broader Impact: Pruning promotes deployment of powerful models on edge devices such as mobile phones and wearable gadgets.The passage describes this benefit even for cheap devices with limited computer capability.
- Broader Impact: Pruning lowers the barrier to applying artificial intelligence on devices with limited computing capability.