Source-linked AI summary
Convolutional Neural Network Pruning with Structural Redundancy Reduction
Zi Wang, Chengcheng Li, Xiangyang Wang
TL;DR
CNN over-parameterization limits deployment on resource-constrained devices, while prevailing pruning methods focus on least-important filters. This paper models pruning as redundancy reduction, proposes structural redundancy reduction for layer-adaptive channel pruning, and reports improved state-of-the-art results on image classification tasks.
Problem
CNN over-parameterization prevents deployment on resource-limited devices, motivating methods that reduce computation and storage costs.
Method
The paper statistically models pruning through redundancy reduction and identifies structurally redundant layers using graph-based measures before pruning filters adaptively within selected layers.
Results
The approach improved state-of-the-art image-classification results; on CIFAR-10 ResNet20, it pruned 45.8% FLOPs while increasing test accuracy by 0.21%.
Takeaways & Limitations
Pruning filters in layers with the most structural redundancy can matter more than pruning the least-important filters across all layers.
Takeaways & Limitations
The theoretical analysis does not imply p_g ≈ p_η as n approaches infinity because it does not assume m/n approaches zero.
Abstract
from arXiv · showhide
Convolutional neural network (CNN) pruning has become one of the most successful network compression approaches in recent years. Existing works on network pruning usually focus on removing the least important filters in the network to achieve compact architectures. In this study, we claim that identifying structural redundancy plays a more essential role than finding unimportant filters, theoretically and empirically. We first statistically model the network pruning problem in a redundancy reduction perspective and find that pruning in the layer(s) with the most structural redundancy outperforms pruning the least important filters across all layers. Based on this finding, we then propose a network pruning approach that identifies structural redundancy of a CNN and prunes filters in the selected layer(s) with the most redundancy. Experiments on various benchmark network architectures and datasets show that our proposed approach significantly outperforms the previous state-of-the-art.
1. Introduction
CNNs achieve strong performance through increasing width and depth, but over-parameterization limits deployment on resource-constrained devices. This study addresses pruning by identifying structural redundancy rather than only ranking filters by importance.
- Over-parameterized CNNs are difficult to deploy on resource-limited devices such as mobile phones and robots.
- Channel pruning removes entire convolutional filters, enabling acceleration without specialized hardware and often achieving substantial pruning ratios with limited performance degradation.
- Existing channel-pruning methods commonly rank filters by importance or similarity and remove the least important filters across layers.
- Statistical analysis shows that pruning filters in the most redundant layer(s) can outperform pruning the least important filters across all layers.
- The proposed structural redundancy reduction approach builds layer-wise graphs, measures redundancy using ℓ-covering number and quotient space size, then prunes selected filters.
- 44.1% FLOPs reduction with only 0.37% top-1 accuracy loss is reported for the pruned ResNet50 on ImageNet.
2. Related work
Prior pruning research includes unstructured weight pruning and structured channel pruning. Recent studies increasingly suggest that the post-pruning network structure can matter as much as, or more than, selecting individually unimportant filters.
- Weight pruning creates unstructured sparsity that generally requires specialized software or hardware for acceleration.
- Channel pruning removes entire filters and avoids the specialized hardware requirement associated with unstructured pruning.
- Recent empirical studies report comparable or better performance from compact networks with random initialization and performance recovery after random pruning.
- This study extends those observations by theoretically showing that pruning in highly redundant layers can outperform globally pruning the least important filters.
3. A theoretic analysis of network pruning
The analysis models pruning as redundancy reduction and concludes that removing filters from the most redundant layer can outperform globally selecting the least important filters. This result motivates layer-adaptive pruning under explicit statistical assumptions.
- Statistical model: Layer redundancy is modeled by the number of filters, while each filter’s contribution is represented by a positive random variable.The contribution may be the absolute training-accuracy drop or training-loss change after pruning a filter.
- Main finding: If a layer has much higher redundancy, pruning filters there—randomly or selectively—outperforms pruning the least important filters across all layers.The claim is derived from the statistical model and treats the CNN as a black box without assuming a prior distribution for filter contributions.
- Statistical model: The model compares no pruning, random pruning, layer-wise least-important pruning, and globally least-important pruning through probability-based system-performance expressions.The two-layer setup uses m and n filters, with n much larger than m, and defines performance through threshold-exceeding events.
- Interpretation: The redundancy perspective explains why a layer with many filters can tolerate pruning while preserving its total contribution above the performance threshold.In the model, exceeding the threshold means no performance loss, and greater redundancy makes this more likely after pruning.
- Assumptions and scope: The conclusion relies on large-layer asymptotics, bounded filter-contribution variances, positive lower-bounded expected contributions, and a bounded number of correlated filter pairs.The authors note that the large-n assumption can be relaxed in real-world applications, where the comparison holds on average rather than at every selection step.
- Assumptions and scope: The analysis does not imply p_g ≈ p_η as n approaches infinity because it does not assume that m/n approaches zero.This explicitly limits the asymptotic interpretation of the probability comparison.
4. Methodology
The method measures structural redundancy separately in each convolutional layer using filter graphs, then prunes unimportant filters from the most redundant layers. It uses graph-based redundancy measures and a lightweight covering-number estimate to make pruning layer-adaptive.
- Notations and preliminaries: Channel pruning is formulated as finding parameters with fewer nonzero filters under an optimization objective.
- Graph establishment: Each convolutional layer is represented by an undirected graph whose vertices are filters and whose edges reflect distances between filter weights.Filters are flattened and normalized before graph construction.
- Filter selection: After identifying the most redundant layers, the approach ranks filters separately within each selected layer and prunes according to the remaining graph vertices.Different pruning amounts across layers make the procedure layer-adaptive.
- Graph redundancy: The method quantifies layer redundancy using quotient space size and ℓ-covering number, whose larger values indicate greater redundancy.The graph redundancy combines these quantities using probability weights w1 and w2.
- Covering-number estimation: A lightweight procedure estimates the ℓ-covering number because its exact calculation is NP-hard and time-consuming.Experiments report that the estimate is good enough and requires negligibly small computing time.
5. Experiments
Experiments evaluate single-shot pruning on ResNet models using CIFAR-10 and ImageNet, with comparisons against recent channel-pruning methods. The approach achieves strong FLOPs reductions with small or improved accuracy changes.
- Experiment settings: The evaluation uses single-shot pruning on ResNet20 and ResNet56 with CIFAR-10 and ResNet50 with ImageNet.Progressive-pruning results are provided in the Appendix.
- Experiment settings: The study compares the proposed approach with recent channel-pruning methods including MW, Taylor expansion, APoZ, SFP, DCP, NISP, SNN, and others.
- CIFAR-10: 45.8% FLOPs reduction on ResNet20 increases CIFAR-10 test accuracy by 0.21%.
- CIFAR-10: 53.8% FLOPs reduction on ResNet56 achieves 93.75% test accuracy, outperforming the baseline by 0.37%.
- ImageNet: 44.1% FLOPs reduction on ResNet50 loses 0.37% top-1 accuracy and 0.19% top-5 accuracy on ImageNet.
- Performance comparison: At 55.1% pruning, the proposed approach records 1.02% top-1 and 0.51% top-5 accuracy drops, while comparable prior methods usually lose more than 1% top-1 accuracy.Uniformly applying MW yields 71.24% top-1 accuracy, whereas redundancy-guided pruning improves performance by around 4%.
6. Analysis and ablation study
The ablation studies examine covering-number estimation, pruning sensitivity, distance-threshold effects, and alternative redundancy measurements. Results support efficient estimation, robustness to filter-selection criteria, and improved pruning when structural redundancy is measured more precisely.
- l-covering number estimate: n1 and n2 remain approximately equal as γ varies from 0.001 to 0.3, supporting the use of 2(n1 + n2) to estimate N_c^1.The same negligible deviation is reported across the experiments described.
- Computation time: The oracle method becomes computationally infeasible when N_c^1 > 4, whereas the proposed estimate is negligibly short and mainly depends on layer size and γ.The reported timing comparison motivates the lightweight estimator over exhaustive search.
- Filter selection criteria: Filter-selection criteria have little influence: minimum weight reaches 75.99% accuracy, while random pruning lowers accuracy by only 0.17%.This experiment prunes 30% FLOPs from AlexNet on CIFAR-10 and fine-tunes for 100 epochs.
- Distance threshold: Changing γ changes layer allocation: large γ concentrates pruning in the largest layers, small γ distributes removal similarly across layers, and γ = 0.034 identifies a distinct pattern.These observations come from pruning 40% of AlexNet filters.
- Distance threshold: When γ approaches infinity, the method favors the layer with the most filters as most redundant; when γ produces equal redundancy, pruning becomes uniform.The analysis interprets γ as controlling a dynamic architecture-search behavior.
- Structural redundancy measurements: 44% FLOPs pruning on ResNet50 shows progressively smaller top-1 and top-5 accuracy drops from NOF to PCA to graph-based redundancy measurement.The graph-based approach is reported to significantly improve pruning performance over the alternative redundancy measures.
7. Conclusion
The study frames CNN pruning as structural redundancy reduction, theoretically identifies redundant layers, and proposes graph-based layer-adaptive pruning. Experiments report improved state-of-the-art image-classification results, while more complicated tasks remain future work.
- Conclusion: The authors model pruning statistically and find that pruning the most structurally redundant layers is more important than pruning the least important filters across all layers.This conclusion is presented as the rationale for the proposed pruning strategy.
- Conclusion: The proposed approach builds a graph for each convolutional layer, measures redundancy with two graph-related quantities, and prunes selected layers using a simple filter criterion.The method is layer-adaptive because pruning targets are selected by measured layer redundancy.
- Conclusion: Experiments validate improved state-of-the-art performance on image-classification tasks, while object detection and image synthesis are left for future research.The conclusion explicitly limits the demonstrated scope to image classification.