Source-linked AI summary
Dynamic Network Surgery for Efficient DNNs
Yiwen Guo, Anbang Yao, Yurong Chen
TL;DR
Deep neural networks are difficult to deploy on mobile platforms because their large parameter counts increase resource demands. The paper proposes dynamic network surgery, which combines on-the-fly pruning with connection splicing and continual importance updates. Experiments report 108× and 17.7× compression for LeNet-5 and AlexNet without decreased prediction accuracy, alongside improved learning efficiency.
Problem
Growing DNN parameter counts increase storage, computation, and battery demands, while pruning can suffer irreversible damage and inefficient retraining.
Method
Dynamic network surgery continually combines parameter pruning with splicing, allowing incorrectly pruned connections to recover as importance changes.
Results
108× and 17.7× parameter compression were achieved for LeNet-5 and AlexNet, respectively, without decreased prediction accuracy.
Takeaways & Limitations
The method deletes most DNN parameters while retaining prediction accuracy and requiring fewer training epochs than the recent pruning method.
Abstract
from arXiv · showhide
Deep learning has become a ubiquitous technology to improve machine intelligence. However, most of the existing deep models are structurally very complex, making them difficult to be deployed on the mobile platforms with limited computational power. In this paper, we propose a novel network compression method called dynamic network surgery, which can remarkably reduce the network complexity by making on-the-fly connection pruning. Unlike the previous methods which accomplish this task in a greedy way, we properly incorporate connection splicing into the whole process to avoid incorrect pruning and make it as a continual network maintenance. The effectiveness of our method is proved with experiments. Without any accuracy loss, our method can efficiently compress the number of parameters in LeNet-5 and AlexNet by a factor of $\bm{108}\times$ and $\bm{17.7}\times$ respectively, proving that it outperforms the recent pruning method by considerable margins. Code and some models are available at https://github.com/yiwenguo/Dynamic-Network-Surgery.
1 Introduction
DNNs achieve strong performance but their growing parameter counts hinder mobile deployment. Dynamic network surgery addresses pruning errors and inefficiency by combining pruning with recoverable connection splicing.
- AlexNet has 61 million parameters, increasing storage, FLOP, and battery demands on mobile platforms.
- Parameter redundancy makes substantial compression possible without significantly reducing prediction accuracy.
- Changing connection importance can make greedy pruning irreversibly damage accuracy and force conservative compression.
- Dynamic network surgery combines pruning with splicing so incorrectly pruned connections can be recovered when they become important.
2 Related Works
Prior DNN compression methods include hardware optimization, decomposition, quantization, and pruning, each with distinct accuracy, complexity, or compression trade-offs. This paper builds on pruning because it is effective and structure-independent.
- Hardware-oriented methods improve execution through data layout, batching, fixed-point instructions, or frequency-domain convolution.
- Low-rank decomposition can accelerate convolution but may incur severe accuracy loss under high compression requirements.
- BinaryConnect and Binarized Neural Networks compress DNNs by 32×, with noticeable accuracy loss considered difficult to avoid.
- Classical Hessian-based pruning methods reduce parameters but suffer high computational complexity on large networks.
- The authors build on Han et al.’s pruning because it compresses effectively without assuming a particular network structure and can combine with other methods.
3 Dynamic Network Surgery
Dynamic network surgery continually maintains a sparse DNN by jointly updating weights and connection masks. It uses importance-based pruning, recovery through splicing, stochastic mask updates, and separate treatment of convolutional and fully connected layers.
- 3.1 Notations: The method represents each layer with weights Wk and a binary mask Tk indicating whether connections are pruned.
- 3.2 Pruning and Splicing: The optimization minimizes network loss over masked weights while a discriminative function determines which connections remain active.
- 3.2 Pruning and Splicing: Weights corresponding to pruned connections continue updating, enabling incorrectly removed connections to be spliced back.
- 3.2 Pruning and Splicing: Pruning and splicing form a circular process that continually updates connection weights and mask entries.
- 3.3 Parameter Importance: Absolute weight magnitude determines importance: small-magnitude parameters are pruned, while large-magnitude parameters are kept or spliced.
- 3.4 Convergence Acceleration: Mask updates occur stochastically with a non-increasing probability, which may eventually reach zero to improve convergence.
- 3.4 Convergence Acceleration: Separately pruning convolutional and fully connected layers addresses slow convergence associated with vanishing gradients in thin networks.
4 Experimental Results
Experiments on XOR, MNIST networks, and AlexNet show that dynamic network surgery substantially reduces model complexity while preserving prediction accuracy and improving compression over prior pruning.
- 4.1 The Exclusive-OR Problem: 0.30% prediction error remained after compression, despite setting 40% of parameters to zero.The compressed model slightly outperformed its reference model.
- 4.2 The MNIST database: 108× and 56× parameter reductions were obtained for LeNet-5 and LeNet-300-100, respectively, while prediction accuracies were as good or slightly better.Less than 1% of LeNet-5 connections and 2% of LeNet-300-100 connections were retained.
- 4.2 The MNIST database: At least 4× greater compression than Han et al.’s method was achieved on the compared LeNet models.The comparison was performed layer by layer using the percentage of remaining parameters.
- 4.3 ImageNet and AlexNet: On AlexNet, dynamic network surgery achieved the best compression rate among compared lossless methods, with comparable or better prediction error after acceptable training epochs.Compared models used Han et al.’s pruning method and adaptive fastfood transforms.
- 4.3 ImageNet and AlexNet: At least 6.8× fewer epochs were needed than with Han et al.’s method, while reducing parameters on almost every AlexNet layer.The reductions also improved storage and FLOP requirements relative to that method.
5 Conclusions
Dynamic network surgery compresses DNNs by combining pruning with connection splicing in a dynamic process. It deletes most parameters without reducing prediction accuracy, achieving large compression factors and improved learning efficiency.
- Dynamic network surgery incorporates connection splicing into pruning and retraining, implementing the process dynamically.
- Most DNN parameters can be deleted without decreasing prediction accuracy.
- 108× compression is achieved for LeNet-5 and 17.7× for AlexNet.
- The method outperforms a recent pruning method by considerable margins and requires fewer training epochs.