Source-linked AI summary
EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning
Bailin Li, Bowen Wu, Jiang Su, Guangrun Wang, Liang Lin
TL;DR
DNN pruning must identify accurate sub-networks while reducing computational redundancy, but existing candidate-evaluation methods can be inaccurate or complicated. EagleEye uses adaptive batch normalization to estimate candidate potential quickly and reports better pruning performance than the studied methods, including up to 3.8% higher accuracy and 70.9% accuracy with 50% FLOPs pruned. Its broader integration potential remains to be explored in detail.
Problem
DNN pruning requires finding the highest-accuracy sub-network in a large constrained search space, while existing evaluation methods can be inaccurate or complicated.
Method
EagleEye applies correlation-based evaluation with adaptive batch normalization to estimate converged accuracy for pruned models using only a few inference iterations.
Results
EagleEye outperforms compared methods by up to 3.8% accuracy and achieves 70.9% ImageNet top-1 accuracy with 50% operations (FLOPs) pruned on compact MobileNet V1.
Takeaways & Limitations
The adaptive-BN evaluation module can quickly identify promising pruning candidates without fine-tuning them and can plug into existing pruning methods.
Takeaways & Limitations
The generality of the adaptive-BN module could not be analyzed in more detail because of the paper's limited length.
Abstract
from arXiv · showhide
Finding out the computational redundant part of a trained Deep Neural Network (DNN) is the key question that pruning algorithms target on. Many algorithms try to predict model performance of the pruned sub-nets by introducing various evaluation methods. But they are either inaccurate or very complicated for general application. In this work, we present a pruning method called EagleEye, in which a simple yet efficient evaluation component based on adaptive batch normalization is applied to unveil a strong correlation between different pruned DNN structures and their final settled accuracy. This strong correlation allows us to fast spot the pruned candidates with highest potential accuracy without actually fine-tuning them. This module is also general to plug-in and improve some existing pruning algorithms. EagleEye achieves better pruning performance than all of the studied pruning algorithms in our experiments. Concretely, to prune MobileNet V1 and ResNet-50, EagleEye outperforms all compared methods by up to 3.8%. Even in the more challenging experiments of pruning the compact model of MobileNet V1, EagleEye achieves the highest accuracy of 70.9% with an overall 50% operations (FLOPs) pruned. All accuracy results are Top-1 ImageNet classification accuracy. Source code and models are accessible to open-source community https://github.com/anonymous47823493/EagleEye .
1 Introduction
DNN pruning searches for accurate, computationally smaller sub-networks, but selecting promising candidates efficiently is difficult because existing evaluation methods can be inaccurate or complicated. EagleEye addresses this evaluation step with correlation analysis and adaptive batch normalization, enabling fast candidate selection and strong pruning results.
- Pruning challenge: DNN pruning searches a space of legitimate sub-networks for the highest-accuracy model under computational constraints.The search space consists of layer-wise pruning strategies, and exhaustive fine-tuning to convergence can be impractical.
- Pruning challenge: Evaluation modules estimate sub-network potential so pruning candidates can be selected without training every candidate to convergence.Their advantage is faster decision-making in a large search space.
- Existing evaluation methods: Existing evaluation methods are either inaccurate or complicated, limiting reliable and practical candidate selection.Inaccuracy means the evaluated winner may not achieve high accuracy after convergence; complexity may require auxiliary training or difficult hyper-parameter tuning.
- EagleEye: EagleEye introduces correlation-based analysis and adaptive batch normalization to improve the vanilla evaluation step.The method estimates converged accuracy for pruned models using only a few inference iterations and can plug into existing pruning methods.
- Results: 3.8% higher accuracy is achieved by EagleEye than compared algorithms in ResNet-50 experiments.The reported range is 1.3% to 3.8% higher accuracy, measured using ImageNet top-1 classification accuracy.
- Results: 70.9% ImageNet top-1 accuracy is achieved on compact MobileNet V1 with an overall 50% of operations (FLOPs) pruned.This was reported as the highest accuracy in the challenging compact-model pruning experiments.
2 Related work
Prior pruning approaches include hand-crafted candidate selection, sparsity-inducing training, and automated methods using auxiliary mechanisms. These approaches can require extensive convergence training, extra modeling, or hyper-parameter tuning, while BN adjustment had not been specifically applied to pruning.
- Existing approaches: Early pruning used hand-crafted heuristics and evaluated candidates by training them to converged accuracy, which could be time-consuming.The large number of plausible sub-networks made exhaustive candidate evaluation costly.
- EagleEye: EagleEye searches layer-wise pruning ratios jointly in one step, avoiding an issue associated with sequentially searching pruning ratios.The method's joint search is contrasted with strategies that update layers across separate steps.
- Existing approaches: Some pruning methods induce weight sparsity during training using group-LASSO, batch-normalization regularization, or Taylor-based filter ranking.These selection standards are described as orthogonal to EagleEye's algorithm.
- Existing approaches: Automated pruning methods use reinforcement learning, generative adversarial learning, and related mechanisms, but auxiliary models and hyper-parameters can make experiments difficult to reproduce and time-consuming.The cited approaches may require trial-and-error to make auxiliary models work well.
- BN adjustment: BN-statistics adjustment had been used for domain adaptation, but not specifically for model pruning purposes.Both settings involve adapting BN when the model or domain changes.
3 Methodology
EagleEye evaluates structured-pruning candidates by adapting Batch Normalization statistics during inference, targeting the inaccurate and costly candidate-selection stage. This produces stronger agreement with fine-tuned accuracy while avoiding candidate fine-tuning.
- Pruning formulation: Structured pruning searches layer-wise pruning-rate combinations under constraints such as parameters, FLOPs, or latency.A trained full-size network is pruned and then fine-tuned to recover accuracy.
- Vanilla evaluation: Existing vanilla evaluation can produce low inference accuracy and weak correlation with candidates’ eventual fine-tuned accuracy.The problem is attributed to applying full-model BN statistics to dimensionally changed pruned networks.
- Adaptive Batch Normalization: Adaptive BN freezes network parameters and resamples moving statistics through a few forward iterations on part of the training set.The method resets moving averages and updates them without backpropagation, producing statistics adapted to pruned connections.
- Adaptive Batch Normalization: Adaptive-BN evaluation shows stronger correlation with fine-tuned accuracy than vanilla evaluation across pruning rates and Pearson, Spearman, and Kendall metrics.Figure 4 reports the correlation comparison, while Figure 5 visualizes adaptive BN statistics as closer to validation-derived true statistics.
- EagleEye pruning algorithm: EagleEye combines pruning-strategy generation, filter pruning, and adaptive-BN-based evaluation in one pipeline.Its strategy generator can use uniformly sampled layer-wise pruning rates under predefined constraints, while other generators are also compatible.
4 Experiments
Experiments show that adaptive-BN evaluation correlates more strongly with fine-tuned candidate accuracy than vanilla evaluation, while supporting fast candidate selection and strong pruning results across models and datasets.
- 4.1 Quantitative analysis of correlation: Adaptive-BN evaluation shows stronger correlation than vanilla evaluation across Pearson, Spearman, and Kendall measures at different pruning ratios.On average, ρ is 0.67 higher, φ is 0.79 higher, and τ is 0.46 higher.
- 4.1 Quantitative analysis of correlation: Higher Spearman and Kendall correlations make adaptively evaluated winners more likely to rank highly after fine-tuning.Spearman emphasizes monotonic correlation, which is relevant to selecting high-accuracy candidates.
- 4.2 Generality of the adaptive-BN-based evaluation method: The adaptive-BN evaluation module is general enough to plug into existing pruning methods, including AMC and short-term fine-tuning-based selection.The paper also discusses upgrading the pipeline with evolutionary search methods.
- 4 Experiments: EagleEye outperforms all compared methods on the reported ResNet-50 and MobileNetV1 pruning experiments across CIFAR-10 and ImageNet.The comparisons include different FLOPs constraints and the challenging compact MobileNetV1 setting.
- 4 Experiments: EagleEye evaluates 1000 potential pruning strategies by adaptive-BN-based evaluation and fine-tunes only the top-2 candidates in the ResNet-50 ImageNet experiments.The best fine-tuned candidate is returned as the delivered pruned model.
5 Discussion and Conclusions
The paper concludes that EagleEye combines adaptive batch normalization with fast, accurate candidate evaluation and reports higher accuracy than the studied pruning methods on ImageNet.
- 5 Discussion and Conclusions: EagleEye uses adaptive batch normalization to provide fast and accurate pruning-candidate evaluation.The authors identify broader integration with existing methods and more advanced pruning-strategy generation as future work.
- 5 Discussion and Conclusions: EagleEye delivers higher accuracy than the studied methods in the reported ImageNet pruning experiments.