Source-linked AI summary
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang
TL;DR
The paper addresses the difficulty and cost of designing efficient deep networks with appropriate layer widths. It introduces Network Trimming, which uses activation statistics to prune weak neurons and alternates pruning with retraining. Experiments on LeNet and VGG-16 report substantial parameter reductions with preserved or improved accuracy.
Problem
Designing efficient deep architectures requires labor-intensive experiments and fine-tuning, while the appropriate number of neurons for each layer is difficult to determine.
Method
Network Trimming identifies weak neurons using activation statistics on a large validation dataset, prunes them, and retrains the remaining network iteratively.
Results
LeNet and VGG-16 achieve 2∼3× fewer parameters without accuracy loss, while trimmed VGG-16 models can surpass the original model.
Takeaways & Limitations
A network architecture can be tailored to different tasks and datasets by determining how many neurons to use in each layer without intensive computational power or human labor.
Takeaways & Limitations
Using validation data to obtain pruning statistics may overfit the validation set, especially given the trimmed model’s higher validation accuracy.
Abstract
from arXiv · showhide
State-of-the-art neural networks are getting deeper and wider. While their performance increases with the increasing number of layers and neurons, it is crucial to design an efficient deep architecture in order to reduce computational and memory costs. Designing an efficient neural network, however, is labor intensive requiring many experiments, and fine-tunings. In this paper, we introduce network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset. Our algorithm is inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero, regardless of what inputs the network received. These zero activation neurons are redundant, and can be removed without affecting the overall accuracy of the network. After pruning the zero activation neurons, we retrain the network using the weights before pruning as initialization. We alternate the pruning and retraining to further reduce zero activations in a network. Our experiments on the LeNet and VGG-16 show that we can achieve high compression ratio of parameters without losing or even achieving higher accuracy than the original network.
1 Introduction
The paper targets efficient neural architectures by determining layer-wise neuron counts from activation-based redundancy, rather than relying solely on labor-intensive empirical design. Network Trimming alternates pruning weak neurons with retraining to reduce model size while preserving performance.
- Neural networks improve performance through increasing depth, width, and parameter count, but this expansion raises efficiency concerns.The paper contrasts LeNet’s fewer than 1M parameters with Deepface’s more than 120M parameters.
- Choosing the number of neurons per layer is difficult because optimal layer widths cannot be determined directly and are usually selected empirically.Common empirically chosen widths include 128, 512, and 4096.
- The method is motivated by sparse activations and seeks to remove redundant neurons while maintaining or improving performance with fewer neurons.The paper links sparse top-layer outputs to potentially harmful overfitting and optimization difficulty.
- Network Trimming iteratively eliminates weak neurons through a pruning-retraining loop guided by activation analysis on a large validation dataset.The retained neurons initialize the pruned model, which is retrained or fine-tuned according to the performance drop.
2 Related Work
Related work reduces neural-network cost either by replacing architectural components or by pruning connections and weights. Network Trimming addresses the intermediate problem of determining the appropriate number of neurons in each layer for a fixed architecture and task.
- Prior architecture-level methods replace dense or computationally expensive components with alternative layers or modules.Examples include global average pooling, inception modules, and SqueezeNet’s Fire module.
- Connection- and weight-level methods prune parameters using numerical properties of weights or loss functions, sometimes followed by quantization and encoding.These approaches reduce parameters directly without external data in the cited earlier methods.
- Network Trimming targets the intermediate problem of determining the optimal neuron count in each layer for a given architecture and task.This focus differs from methods that redesign modules or directly remove connections and weights.
- The approach can save parameters without designing a new architecture while evaluating redundancy layer by layer to guide architecture optimization.The paper presents layer-wise redundancy evaluation as guidance for large neural networks.
3 Zero Activations and Network Trimming
The paper measures neuron redundancy through post-ReLU zero activations and trims neurons iteratively using APoZ-guided pruning and retraining. The procedure targets high-APoZ layers and progressively reduces neurons while preserving performance.
- Zero Activations: APoZ measures each neuron's percentage of zero activations after ReLU across validation examples and feature-map positions.Larger validation sets make APoZ measurements more accurate; the ImageNet validation set contains N = 50,000 examples.
- Zero Activations: 631 VGG-16 neurons have APoZ larger than 90%, demonstrating substantial zero-activation redundancy.Layer-wise mean APoZ indicates greater redundancy in higher convolutional and fully connected layers.
- Network Trimming: Network trimming trains a baseline, measures neuron APoZ on a large validation dataset, prunes high-APoZ neurons and connections, then retrains the trimmed network.The trimmed model is initialized with weights from before pruning, and retraining strengthens the remaining neurons after the performance drop caused by pruning.
- Network Trimming: Weight initialization preserves performance better than training a trimmed network from scratch, which produces a larger percentage of zero-activation neurons.The paper therefore uses weights from before trimming to initialize the reduced network.
- Network Trimming: Iterative, layer-wise trimming is preferred because pruning too many neurons or layers at once can cause unrecoverable performance loss.The authors begin with layers having high mean APoZ and progressively trim neighboring layers using an empirical threshold.
4 Experiments
Experiments on LeNet and VGG-16 show that iterative neuron trimming can substantially reduce parameters while preserving or improving accuracy. The results also indicate that weight initialization and gradual, targeted pruning are important to effective trimming.
- 4.1 Experiments: LeNet trimming of CONV2 and FC1 achieved more than 2∼3× parameter compression without loss in accuracy after four iterations.The first pruning reached 1.41× compression, and retraining improved accuracy from 98.75% to 99.29%.
- 4.1 Experiments: Proper weight initialization let trimmed LeNet retain or exceed the ancestor’s accuracy, whereas training without initialization deteriorated classification accuracy.Initialized models also showed lower mean APoZ, while retraining from scratch increased mean APoZ.
- 4.2 Experiments: VGG-16 trimming focused on CONV5-3 and FC6 because these layers exhibited higher mean APoZ and contained 100M of the network’s 138M parameters.The experiments analyzed APoZ on the ImageNet validation set before selecting these parameter-bottleneck layers.
- 4.2 Experiments: After trimming, the FC6 APoZ distribution shifted left and its right tail diminished, indicating reduced redundancy and disappearance of weak neurons.The authors associate the diminishing tail with the benefit of weight initialization.
- 4.2 Experiments: After six trimming iterations, VGG-16 reduced more than half of its parameters, reached 2.59× compression, and achieved 2∼3% higher Top-1/Top-5 accuracy than the original model.The trimmed model recovered the original accuracy in fewer than 5K fine-tuning iterations, while its reduced FC6 size also reduced overfitting risk.
- 4.2 Experiments: Progressive trimming of neighboring layers was more effective than trimming many layers at once, with CONV5-3, FC6, and FC7 reaching 2.7× compression and improved accuracy.The authors report that these layers contain around 86% of all parameters and are sufficient for the parameter-reduction goal.
5 Discussion
Network Trimming removes redundant neurons using activation statistics and achieves lower computational cost while preserving or improving accuracy. Additional experiments address concerns that using validation data for pruning could overfit.
- Network Trimming removes unneeded neurons entirely, avoiding their memory and computation costs; trimmed VGG-16 uses more than 2× fewer FLOPs in its first fully connected layer.
- The method targets redundancy where it is most prevalent: fully connected layers and connections from the last convolutional layer to the first fully connected layer.
- Using validation data to measure APoZs raised overfitting concerns, especially because trimmed VGG-16 achieved 2% higher top-5 validation accuracy than the original.
- Weak neurons selected from a 50K-image training subset overlapped by more than 95% with those selected using the validation set.
- On the non-overlapping ILSVRC2012 test set, trimmed VGG-16 achieved 10.02% error versus 13.02% for the original model, with a 2.00 compression rate.
- Two additional experiments dismissed the overfitting concern and supported using the validation set for APoZ analysis.
6 Conclusion
The conclusion presents Network Trimming as activation-based neuron pruning that reduces network complexity while preserving performance. It applies across mature architectures and can complement weight pruning.
- Network Trimming prunes redundant neurons using statistics of their activations and can iteratively remove low-activation neurons without damaging model performance.
- LeNet and VGG-16 achieved the same accuracy with 2∼3× fewer parameters after trimming.
- Trimmed VGG-16 models can surpass the original, which the authors suggest could result from reduced optimization difficulty.
- Neuron pruning occupies a middle ground between high-level network redesign and low-level weight pruning and can be combined with weight pruning.