Source-linked AI summary
Learning both Weights and Connections for Efficient Neural Networks
Song Han, Jeff Pool, John Tran, William J. Dally
TL;DR
Large neural networks demand prohibitive storage, memory bandwidth, and computation for embedded deployment. This paper learns important connections, prunes the rest, and retrains the sparse network, reducing connections 9× to 13× without accuracy loss.
Problem
Large neural networks require substantial storage, memory bandwidth, and computation, making embedded and mobile deployment difficult.
Method
The method learns important connections through training, prunes low-weight connections, and retrains the resulting sparse network, iterating as needed.
Results
The method reduces AlexNet and VGGNet connections by 9× to 13× on ImageNet without loss of accuracy.
Takeaways & Limitations
Pruning reduces memory capacity and bandwidth requirements, making real-time image processing easier to deploy on mobile systems.
Abstract
from arXiv · showhide
Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9x, from 61 million to 6.7 million, without incurring accuracy loss. Similar experiments with VGG-16 found that the number of parameters can be reduced by 13x, from 138 million to 10.3 million, again with no loss of accuracy.
1 Introduction
Large neural networks demand substantial storage, memory bandwidth, and computation, making embedded deployment difficult. The paper addresses this by pruning low-weight connections and retraining the resulting sparse network while preserving accuracy.
- Motivation: 60M parameters marked the scale of the 2012 ImageNet-winning CNN, compared with LeNet-5’s less than 1M parameters in 1998.The passage also notes that Deepface used 120M parameters, illustrating continued network growth.
- Motivation: Neural networks’ increasing size consumes substantial storage, memory bandwidth, and computational resources, making their demands prohibitive for embedded mobile applications.The passage attributes connection energy costs primarily to memory access, ranging from 5pJ to 640pJ for 32-bit coefficients.
- Method: The method prunes connections with weights below a threshold after initial training, thereby learning and retaining important topology.Pruning converts dense, fully connected layers into sparse layers by removing unimportant connections.
- Method: Retraining the sparse network fine-tunes the remaining connections while preserving the original accuracy.The method uses an initial training phase, pruning, and subsequent retraining.
2 Related Work
Prior work reduces neural-network redundancy through quantization, low-rank approximations, architectural replacements, pruning, and hashed parameter sharing. These approaches offer efficiency gains but involve trade-offs such as transfer-learning difficulty or additional computation.
- Redundancy reduction: Neural networks are often over-parameterized, wasting computation and memory; prior work explored 8-bit activations and low-rank parameter approximations.Vanhoucke et al. used 8-bit integer rather than 32-bit floating-point activations, while Denton et al. exploited linear structure through low-rank approximation.
- Architectural alternatives: Global average pooling can replace fully connected layers and achieve state-of-the-art benchmark results, but it makes ImageNet transfer learning more difficult.The difficulty arises when reusing ImageNet features for new tasks by fine-tuning only fully connected layers.
- Network pruning: Network pruning reduces complexity and over-fitting by removing connections, while Hessian-based methods may improve accuracy over magnitude-based pruning at the cost of additional computation.Biased weight decay, Optimal Brain Damage, and Optimal Brain Surgeon represent earlier pruning approaches.
- Parameter sharing: HashedNets reduce model size by sharing one parameter among connections assigned to the same hash bucket, and pruning may further improve parameter savings by reducing hash collisions.The passage identifies sparsity as a way to make feature hashing more effective.
3 Learning Connections in Addition to Weights
The method learns important connections alongside weights, prunes low-weight connections, and retrains the resulting sparse network to preserve accuracy. Iterative pruning and retraining progressively identify connections for removal, while retained weights and layer-wise retraining improve recovery after pruning.
- 3 Learning Connections in Addition to Weights: The three-step method learns important connectivity, prunes connections below a threshold, and retrains the remaining sparse network to learn final weights.Retraining is critical because using the pruned network without it significantly impacts accuracy.
- 3 Learning Connections in Addition to Weights: L2 regularization gives the best overall pruning results, although L1 regularization produces better accuracy before retraining.L1 creates more parameters near zero, whereas L2 yields better remaining connections after pruning and retraining.
- 3 Learning Connections in Addition to Weights: Retaining surviving weights from initial training is better than re-initializing pruned layers during retraining because CNNs contain fragile co-adapted features.Gradient descent finds a good solution during initial training but not necessarily after layers are re-initialized.
- 3 Learning Connections in Addition to Weights: Fixing CONV parameters while retraining FC layers, and vice versa, reduces computation and helps address vanishing gradients in deeper networks.Starting with retained weights avoids backpropagating through the entire network, while layer-wise retraining makes pruning errors easier to manage.
- 3 Learning Connections in Addition to Weights: 5× to 9× pruning on AlexNet can be achieved without accuracy loss through iterative pruning and retraining, compared with single-step aggressive pruning.Each iteration performs a greedy search for the best connections, and repeated iterations can find the minimum number of connections.
4 Experiments
Experiments on MNIST and ImageNet show that pruning substantially reduces neural-network parameters and computation while revealing task-relevant sparsity patterns. On VGG-16, pruning reduced the network to 7.5% of its original size, or 13× smaller.
- MNIST: 12× lower computation was achieved for LeNet-300-100, while LeNet-5 achieved 6× lower computation.The reported reductions apply to weights and computation for the two MNIST networks.
- MNIST: 12× fewer parameters were achieved on LeNet-300-100 and LeNet-5 through pruning on MNIST.LeNet-300-100 and LeNet-5 were retrained after pruning at 1/10 of the original learning rate.
- MNIST: 28 sparsity bands in LeNet-300-100’s first fully connected layer corresponded to centered digit pixels and identified visual attention regions.The layer matrix was 784 ∗300, with each band 28 pixels wide.
- ImageNet: 61 million parameters and 57.2% top-1 accuracy characterized the AlexNet reference model evaluated on ILSVRC-2012.The reference model also achieved 80.3% top-5 accuracy and was trained for 75 hours on an NVIDIA Titan X GPU.
- ImageNet: 13× smaller was the resulting VGG-16 network, reduced to 7.5% of its original size after five pruning-and-retraining iterations.Each of the two largest fully connected layers was pruned to less than 4% of its original size.
5 Discussion
Discussion shows that iterative pruning with retraining preserves accuracy deepest into parameter reduction, while pruning sensitivity varies across layers. The resulting sparse models reduce storage demands enough for on-chip deployment.
- Pruning and retraining: Retraining delays accuracy loss until 1/10 of the original connections, compared with 1/3 without retraining.The accuracy–parameter trade-off worsens as more parameters are pruned.
- Regularization: L1 regularization performs better immediately after pruning, whereas L2 performs better after retraining.Using L1 for pruning and L2 for retraining did not outperform L2 in both phases.
- Iterative pruning: 9× pruning causes no accuracy loss, while accuracy begins dropping sharply at 10× under iterative pruning.Iterative pruning repeatedly prunes and retrains the network and provides the largest gain.
- Accuracy effects: Two pruned models achieve slightly better accuracy than the original, possibly because pruning selects a capacity that reduces overfitting.The paper presents this explanation as a belief rather than a demonstrated mechanism.
- Layer sensitivity: CONV layers are more sensitive to pruning than fully connected layers, with the first convolutional layer most sensitive.Sensitivity is measured by accuracy decline as parameters are pruned layer by layer.
- Deployment: Sparse storage adds only 15.6% overhead, and pruned AlexNet and VGGNet can store all weights on chip instead of off-chip DRAM.Relative indices use 5 bits for fully connected layers and 8 bits for convolutional layers.
6 Conclusion
The paper presents a connection-learning method that improves neural-network energy efficiency and storage without affecting accuracy. It learns important connections, prunes unimportant ones, and retrains the resulting sparse network, with experiments on AlexNet and VGGNet using ImageNet.
- 6 Conclusion: The method improves neural-network energy efficiency and storage without affecting accuracy by finding the right connections.The approach is motivated in part by how learning works in the mammalian brain.
- 6 Conclusion: The three-step method learns important connections, prunes unimportant connections, and retrains the remaining sparse network.
- 6 Conclusion: Experiments on AlexNet and VGGNet used ImageNet to evaluate the method.The experiments addressed both fully connected and convolutional layers.