Source-linked AI summary
To prune, or not to prune: exploring the efficacy of pruning for model compression
Michael Zhu, Suyog Gupta
TL;DR
Deploying accurate neural networks on resource-constrained devices motivates comparing pruning with reducing dense model width under similar memory footprints. The paper introduces a broadly applicable gradual pruning method and finds that large-sparse models consistently outperform comparable small-dense models across diverse architectures, despite sparse-storage overhead.
Problem
The paper asks whether large-sparse or small-dense models provide higher accuracy for a bounded memory footprint in on-device inference.
Method
The paper applies automated gradual magnitude pruning during training and compares pruned large models with dense smaller counterparts across vision and language architectures.
Results
Large-sparse models outperform comparably sized small-dense models across diverse neural network architectures, with up to 10x fewer nonzero parameters and minimal accuracy loss.
Takeaways & Limitations
The results support model pruning as a compression approach for resource-constrained deployment and motivate hardware support for sparse storage and computation.
Takeaways & Limitations
Sparse matrix indexing and auxiliary storage overhead diminish the compression ratio achievable through pruning.
Abstract
from arXiv · showhide
Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.
1 Introduction
The introduction frames model compression as necessary for deploying accurate neural networks on resource-constrained devices, especially for energy-efficient inference. It proposes comparing large-sparse and small-dense models under comparable size constraints while developing a broadly applicable gradual pruning method.
- Motivation: Deploying large, accurate neural networks on mobile phones and smart cameras creates challenges for resource-constrained on-device inference.The motivation includes privacy preservation and reduced user-perceived query times.
- Motivation: Model compression can reduce energy-intensive memory accesses and improve inference time in memory-bandwidth-bound workloads.
- Model compression: Pruning removes less salient connections to reduce nonzero parameters while aiming to preserve model quality.Sparse storage overhead must also be included when evaluating the net memory footprint.
- Study design: The study compares large-sparse and similarly sized small-dense models across image recognition, language modeling, and neural machine translation.The evaluated architectures include InceptionV3, MobileNets, stacked LSTMs, and seq2seq models.
2 Related work
The related work reviews second-order and magnitude-based pruning, structured sparsity, and quantization. The paper distinguishes its approach by emphasizing a simple, broadly applicable gradual pruning method and an extensive large-sparse versus small-dense comparison.
- Prior pruning methods: Early pruning methods estimated parameter saliency with second-order loss approximations, then removed low-saliency weights and retrained the network.Optimal Brain Damage used a diagonal Hessian approximation, whereas Optimal Brain Surgeon used the inverse Hessian.
- Magnitude-based pruning: Magnitude-based pruning is computationally efficient for large networks, and this work prunes smallest-magnitude weights toward a preset sparsity level.
- Proposed pruning method: Compared with prior gradual pruning, the proposed scheme avoids two phases, two manually chosen slopes, and layer-specific weight thresholds.The paper presents it as requiring little hyperparameter tuning across different models.
- Structured sparsity: Structured pruning targets coarse-grained patterns for dense-hardware speedups but may not extend directly to architectures such as LSTMs.The paper claims its method makes no assumptions about network or layer structure.
- Related compression methods: Quantization reduces the bits used per parameter and can be combined with pruning, but different schemes require different runtime software or hardware support.
3 Methods
The method prunes weights during training using binary masks and a gradual sparsity schedule, then evaluates sparse models through model-size and accuracy tradeoffs.
- Gradual pruning: Binary masks remove the smallest-magnitude weights in selected layers during training.Each mask matches its layer’s weight tensor and determines which weights participate in forward execution.
- Gradual pruning: Mask updates gradually increase network sparsity while allowing training to recover from pruning-induced accuracy loss.Updates occur every Δt steps and stop once the target sparsity is reached.
- InceptionV3 illustration: For sparse-InceptionV3, Figure 2 combines the gradual sparsity and exponentially decaying learning-rate schedules with accuracy evolution during training.The associated results show recovery after severe degradation and a gradual quality decline at higher sparsity.
- Sparsity schedule: Pruning rapidly removes redundant connections initially, then prunes fewer weights as the remaining network becomes sparser.The schedule is coordinated with the learning-rate schedule and can begin after several epochs or from a pretrained model.
- Evaluation: The evaluation compares sparse and dense models using model-size and accuracy tradeoffs, including MobileNet and Penn Treebank results.The supplied method-section captions identify these comparisons, while the reported InceptionV3 result provides the corresponding tradeoff example.
4 Comparing large-sparse and small-dense models
Across MobileNet, PTB, and NMT experiments, large-sparse models generally deliver better accuracy than small-dense models at comparable parameter counts or memory footprints. The comparison also shows that pruning schedules and storage overhead matter when evaluating compression trade-offs.
- MobileNets: Sparse MobileNets outperform dense MobileNets with comparable parameter counts, including 75% sparsity versus width multiplier 0.5 and 90% sparsity versus width multiplier 0.25.The 75% sparse model has 1.09 million parameters, while the 90% sparse model matches the dense 0.25 model at 0.46 million parameters.
- PTB language model: 3 million parameters: the 85% sparse medium PTB model achieves perplexity 85.17, outperforming the 95% sparse large model with 3.3 million parameters and perplexity 87.83.The results suggest that the best sparse model at a target size may come from pruning a dense model 5x-10x larger rather than pruning by 20x or more.
- Storage trade-offs: Sparse-model comparisons must include representation overhead: Table 6 calculates model size with sparse storage overhead and assumes 32-bit values per non-zero element.The paper notes that large-sparse models appear more accurate than small-dense models with comparable memory footprints, but storage format affects that comparison.
- NMT: 23M non-zero parameters: the 90% sparse 1024-unit NMT model is comparable to or outperforms the dense 512-unit model with 81M parameters.BLEU scores are 26.19 versus 26.05 for EN-DE and 28.81 versus 28.88 for DE-EN; the sparse model uses 3.5x fewer non-zero parameters.
5 Discussion
Sparse models retain an accuracy advantage over comparably sized dense models, despite storage overheads that depend on the sparse representation. This advantage is observed with 32-bit floating-point models, while reduced precision makes sparse-storage overhead more consequential.
- Memory overhead: Sparse storage overhead includes nonzero parameters and auxiliary indexing structures, reducing achievable compression ratios.Bit-mask storage uses one bit per matrix element, whereas CSR(C) associates each nonzero parameter with a zero-count index.
- Accuracy–size trade-off: Large-sparse models achieve higher accuracy than small-dense models with comparable memory footprints.For example, MobileNet width multiplier 1 at 50% sparsity has a similar footprint to width multiplier 0.75 but higher accuracy.
- Accuracy–size trade-off: The accuracy gap between large-sparse and small-dense models widens for larger PTB language models and NMT models.
- Precision boundary: Reduced-precision inference increases the relative memory cost of sparse-matrix storage.The paper notes that quantization and pruning require further study together.
6 Conclusion
The work finds that large-sparse models outperform comparably sized small-dense models across diverse architectures and presents an easily applied gradual pruning technique. The reported results and pruning library support compression-oriented deployment in resource-constrained environments.
- Conclusion: Large-sparse models outperform comparably sized small-dense models across a diverse set of neural network architectures.
- Conclusion: The paper presents a gradual pruning technique that can be applied across different architectures with ease.
- Release: The TensorFlow pruning library used to generate the reported results was open-sourced.