Source-linked AI summary
AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
Ning Liu, Xiaolong Ma, Zhiyuan Xu, Yanzhi Wang, Jian Tang, Jieping Ye
TL;DR
Large DNNs require substantial storage and computation, motivating automatic hyperparameter determination for structured pruning. AutoCompress combines structured pruning schemes with ADMM-based pruning, purification, and heuristic search, achieving ultra-high pruning rates and measured smartphone inference speedups. Its reported comparisons include up to 33× higher pruning rate and 120× actual parameter-count reduction under the same accuracy.
Problem
Large DNNs have high storage and computational requirements, while structured pruning exposes many flexible hyperparameters requiring automatic determination.
Method
AutoCompress combines structured pruning schemes, ADMM-based structured weight pruning with purification, and experience-enhanced heuristic search instead of deep reinforcement learning.
Results
Up to 33× improvement in pruning rate and 120× reduction in actual parameter count are reported under the same accuracy, with significant smartphone inference speedup observed.
Takeaways & Limitations
AutoCompress achieves ultra-high weight and FLOPs pruning rates on CIFAR-10 and ImageNet and supports measured mobile inference acceleration.
Abstract
from arXiv · showhide
Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference. An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters. This work proposes AutoCompress, an automatic structured pruning framework with the following key performance improvements: (i) effectively incorporate the combination of structured pruning schemes in the automatic process; (ii) adopt the state-of-art ADMM-based structured weight pruning as the core algorithm, and propose an innovative additional purification step for further weight reduction without accuracy loss; and (iii) develop effective heuristic search method enhanced by experience-based guided search, replacing the prior deep reinforcement learning technique which has underlying incompatibility with the target pruning problem. Extensive experiments on CIFAR-10 and ImageNet datasets demonstrate that AutoCompress is the key to achieve ultra-high pruning rates on the number of weights and FLOPs that cannot be achieved before. As an example, AutoCompress outperforms the prior work on automatic model compression by up to 33x in pruning rate (120x reduction in the actual parameter count) under the same accuracy. Significant inference speedup has been observed from the AutoCompress framework on actual measurements on smartphone. We release all models of this work at anonymous link: http://bit.ly/2VZ63dS.
1 Introduction
AutoCompress addresses the costly and difficult design of structured pruning by automating hyperparameter selection and combining ADMM-based pruning, purification, and guided heuristic search. Experiments report substantially higher pruning rates and practical smartphone inference speedups.
- Large DNNs have high storage and computational requirements, motivating compression techniques that reduce inference costs for mobile applications.
- Structured pruning exposes many choices, including pruning schemes, scheme combinations, and per-layer rates, making manual trial-and-error time-consuming and often sub-optimal.
- Prior automatic pruning using deep reinforcement learning considered only filter pruning and reported a maximum 5× pruning rate in the cited work.
- AutoCompress combines structured pruning schemes, ADMM-based pruning with purification, and experience-guided heuristic search instead of deep reinforcement learning.
- 33× higher pruning rate and 120× reduction in actual parameter count were achieved over prior automatic model compression under the same accuracy.
2 Related Work
Related work distinguishes structured pruning, which preserves regular matrix structure for hardware-friendly acceleration, from non-structured pruning and reviews ADMM and automated hyperparameter search as foundations for AutoCompress.
- DNN Weight Pruning and Structured Pruning: Non-structured pruning can remove arbitrary weights and achieve high reduction, but sparse storage and indices limit acceleration on actual hardware.
- DNN Weight Pruning and Structured Pruning: Filter, channel, and filter-shape pruning remove regular groups of weights, while structured pruning preserves full GEMM matrices with reduced dimensions.
- DNN Weight Pruning and Structured Pruning: Filter and channel pruning are correlated across adjacent layers, and bypass links make these relationships more complicated in ResNet and MobileNet.
- ADMM: ADMM decomposes optimization into iteratively solved subproblems and can handle the combinatorial constraints arising in DNN weight pruning.
- AutoML: AutoML applies machine learning to DNN hyperparameter determination, with AMC using deep reinforcement learning to generate layer-wise pruning rates.
3 The Proposed AutoCompress Framework for DNN Structured Pruning
AutoCompress automates structured DNN pruning by selecting per-layer pruning rates and schemes, using ADMM-based pruning, purification, and guided heuristic search across progressive rounds.
- Automatic process: The process selects per-layer pruning rates and structured pruning schemes to maximize weight or FLOPs reduction while minimizing accuracy loss.
- Automatic process: Each automatic round samples hyperparameters, evaluates actions with a fast heuristic, makes decisions, and generates the actual pruning result.Fast evaluation avoids time-consuming retraining-based pruning during search.
- Sources of improvement: AutoCompress combines filter and column pruning, using column pruning’s flexibility while retaining a hardware-friendly full matrix format in GEMM.
- Guided heuristic search: The framework replaces DRL-based search with heuristic search because the authors identify DRL as inherently incompatible with achieving high pruning rates in this process.The authors attribute the difficulty partly to randomized action samples, rough evaluations, and neural-network decisions based on them.
- Core algorithm: ADMM regularization dynamically adjusts its targets during iterations and decomposes structured pruning into two subproblems solved iteratively until convergence.The constrained problem requires remaining weights in each layer to satisfy predefined structures.
- Purification: Purification removes unused filters, columns, and small-weight structures after ADMM-based pruning for further weight reduction without accuracy loss.Layer-specific column-wise and filter-wise thresholds determine which structures are removed.
- Overall framework: AutoCompress uses two progressive phases—ADMM-based structured pruning followed by purification—with flexible rounds and hard weight or FLOPs reduction constraints.Each round targets approximately a 2× reduction relative to the previous round, with small variance.
4 Evaluation, Experimental Results, and Discussions
AutoCompress is evaluated across CIFAR-10 and ImageNet on VGG and ResNet models, comparing structured-pruning schemes, search strategies, and baselines. The results show that combining pruning schemes with ADMM-based pruning and enhanced search produces substantially higher compression, while retaining accuracy and improving measured mobile inference speed.
- CIFAR-10 results: 44.3× and 43.3× pruning rates are achieved with combined structured pruning on VGG-16 and ResNet-18, versus 9.3× and 5.2× with filter-only pruning under manual tuning.Filter-only pruning at similarly high rates can incur notable accuracy loss, including 88.78% accuracy at 47.4× on VGG-16.
- CIFAR-10 results: Enhanced SA-based hyperparameter determination significantly outperforms the DRL-based framework and improves over manual hyperparameter optimization.The comparison holds other sources of improvement constant when contrasting search strategies, supporting the reported incompatibility of DRL with ultra-high pruning rates.
- CIFAR-10 results: 33× improvement over NISP and AMC and 15.3× over 2PFPCE are reported for weight reduction under the same or higher accuracy.Accounting for differing parameter counts, the improvement over NISP and AMC is reported as 120× in actual parameter count.
- Additional discussion: Train-From-Scratch can recover similar accuracy after filter-only pruning, suggesting the discovered pruned structure can function as a smaller DNN model.The framework’s role in these cases is to discover the structure and layer-wise compression rates.
- ImageNet results: 9.2× non-structured pruning on ResNet-50 CONV layers is achieved without accuracy loss, versus 8× for manual ADMM pruning and 4.8× for AMC.The reported accuracy is 92.7% Top-5 on ImageNet.
5 Conclusion
AutoCompress combines structured pruning schemes, ADMM-based pruning with purification, and experience-guided heuristic search to automate hyperparameter determination. Experiments on CIFAR-10 and ImageNet show ultra-high reductions in weights and FLOPs.
- AutoCompress incorporates combinations of structured pruning schemes into its automatic hyperparameter determination process.
- ADMM-based structured weight pruning and an additional purification step enable further weight reduction without accuracy loss.
- Experience-based guided heuristic search replaces deep reinforcement learning because of its incompatibility with the target pruning problem.
- Experiments on CIFAR-10 and ImageNet demonstrate ultra-high pruning rates for both weight counts and FLOPs.
Supplementary Materials
The supplementary materials compare weight-count and FLOPs reduction objectives for VGG-16 and ResNet-18 on CIFAR-10, including per-layer pruning patterns for VGG-16.
- Figure 4 shows the per-layer portion of pruned VGG-16 weights under the Params# and FLOPs# objectives.
- Tables 6 and 7 report AutoCompress comparison results for VGG-16 and ResNet-18 on CIFAR-10.
- The Params# and FLOPs# objectives produce only slight differences in per-layer pruning portions for VGG-16.