Source-linked AI summary
NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
Xiaoliang Dai, Hongxu Yin, Niraj K. Jha
TL;DR
Finding suitable DNN architectures is difficult because fixed-architecture training and trial-and-error searches can produce inefficient, over-parameterized models. NeST jointly grows and prunes networks during training, producing compact, accurate DNNs with substantial parameter and FLOPs reductions across MNIST and ImageNet models.
Problem
Finding suitable DNN architectures remains difficult because back-propagation trains weights for fixed architectures, while trial-and-error searches are inefficient and can yield over-parameterized models.
Method
NeST starts from a sparse seed architecture and jointly adapts it through gradient-based growth and magnitude-based pruning of neurons and connections.
Results
NeST produces accurate compact DNNs across MNIST and ImageNet, reducing LeNet-300-100 parameters by 70.2× and FLOPs by 79.4× without accuracy loss.
Takeaways & Limitations
Grow-and-prune synthesis achieves substantial parameter and computation reductions, and outperforms pruning-only methods on the reported ImageNet comparisons.
Takeaways & Limitations
Mask-based growth and pruning incurs temporal training-memory overhead that may be uneconomical for deeper models such as ResNet or DenseNet.
Abstract
from arXiv · showhide
Deep neural networks (DNNs) have begun to have a pervasive impact on various applications of machine learning. However, the problem of finding an optimal DNN architecture for large applications is challenging. Common approaches go for deeper and larger DNN architectures but may incur substantial redundancy. To address these problems, we introduce a network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training. We propose a DNN synthesis tool (NeST) that combines both methods to automate the generation of compact and accurate DNNs. NeST starts with a randomly initialized sparse network called the seed architecture. It iteratively tunes the architecture with gradient-based growth and magnitude-based pruning of neurons and connections. Our experimental results show that NeST yields accurate, yet very compact DNNs, with a wide range of seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we reduce network parameters by 70.2x (74.3x) and floating-point operations (FLOPs) by 79.4x (43.7x). For the AlexNet and VGG-16 architectures, we reduce network parameters (FLOPs) by 15.7x (4.6x) and 30.2x (8.6x), respectively. NeST's grow-and-prune paradigm delivers significant additional parameter and FLOPs reduction relative to pruning-only methods.
1 Introduction
NeST addresses the difficulty of selecting efficient DNN architectures by jointly training weights and architectures, using growth followed by pruning to produce compact accurate models.
- DNN architecture selection remains difficult because fixed-architecture back-propagation trains weights but cannot improve the architecture.
- NeST starts from a seed architecture and trains both DNN weights and architectures.The seed architecture is the birth point for synthesis.
- Gradient-based growth adds connections and neurons, while magnitude-based pruning removes insignificant ones to avoid redundancy.
- NeST generates accurate and compact DNNs for MNIST and ImageNet, with drastic parameter and FLOPs reductions relative to baselines and no accuracy loss.
2 Related Work
Related work approaches DNN architecture selection through evolutionary search, reinforcement learning, or structure adaptation that constructs or removes network components during training.
- Evolutionary algorithms search architectures through repeated mutation, recombination, evaluation, and selection.
- Reinforcement-learning methods generate candidate networks or architectural building blocks and use performance as a reward or search signal.
- Structure adaptation incorporates architecture selection into training through constructive growth or destructive pruning of connections and neurons.
3 Synthesis Methodology
NeST synthesizes DNN architectures through sequential gradient-based growth and magnitude-based pruning, with specialized procedures for connections, neurons, feature maps, and convolutional areas.
- Synthesis Flow: NeST begins with a sparse, partially connected seed architecture and uses sequential growth and pruning phases to produce a lightweight DNN without accuracy degradation.
- Connection Growth: Connection growth activates dormant connections whose gradients indicate they can reduce the loss most efficiently.
- Connection Growth: Grown input-to-hidden connections in LeNet-300-100 are denser near the image center, consistent with centered MNIST digits.
- Neuron Growth: Neuron growth adds a shared intermediate neuron between highly correlated existing neuron pairs and initializes its weights from batch gradients.
- Neuron Growth: The neuron-growth algorithm evaluates bridging gradients, connects the top β × 100% pairs, and scales new weights using a birth-strength factor.
- Neuron Growth: Neuron weight initialization is designed to imitate the effect of a back-propagation update and applies across tanh, ReLU, and Leaky ReLU activations.
- Feature Map Growth: Feature-map growth randomly generates kernel sets and selects the set that reduces the loss most, yielding approximately twice the loss reduction of naive random initialization.
3.3 Magnitude-based Pruning
NeST prunes insignificant weights, neurons, and convolutional connections using magnitude-based criteria, including batch-normalization-adjusted weights and partial-area convolution. These iterative procedures remove redundancy while preserving performance and can substantially reduce FLOPs.
- NeST removes connections and neurons when their weight or output magnitude falls below a predefined threshold.
- Batch normalization is incorporated into pruning by defining effective weights and treating small effective weights as insignificant.
- Partial-area convolution prunes each kernel’s connections to image regions outside its area of interest rather than discarding the entire image.
- 1% of the least-significant feature-map elements are typically thresholded during each partial-area-convolution pruning iteration, followed by whole-network retraining.The threshold is set from the absolute feature-map values, and a mask disregards pruned convolution areas.
- 2.09× FLOPs reduction was achieved for LeNet-5 on MNIST with partial-area convolution without performance degradation.
- LeNet-5 kernels self-explore preferred area-of-interest shapes, with substantial overlap in the central image region.
4 Experimental Results
NeST synthesizes compact LeNet, AlexNet, and VGG-16 models by exploring varied sparse seeds, growing architectures, and pruning redundant structures. Smaller seeds generally produce smaller final networks but can require longer growth, while growth-and-prune outperforms pruning-only methods.
- LeNets on MNIST: Nine and four sparse seed architectures were derived for LeNet-300-100 and LeNet-5, respectively, using fewer neurons and connections than the originals.Only 10% of possible seed connections were randomly initialized, while all neurons remained connected.
- Growth Phase: Smaller seeds often yield smaller post-growth networks, but require more growth time to reach the target accuracy.The targets were 1.3% error for LeNet-300-100 and 0.8% for LeNet-5.
- Growth Phase: Once post-growth size saturates at a target accuracy, reducing the seed further no longer provides a size benefit.This saturation appears as flat left ends of the dashed curves in Figure 6.
- Pruning Phase: Larger pre-pruning networks have higher compression ratios but also larger post-pruning sizes, favoring smaller seeds within an appropriate range.The higher compression is attributed to greater redundancy in larger pre-pruning networks.
- Inference model comparison: 70.2× and 74.3× reductions in network connections were achieved for LeNet-300-100 and LeNet-5, with FLOPs reductions of 79.4× and 43.7×, respectively.These reductions were reported without accuracy loss relative to the baseline Caffe model.
- AlexNet and VGG-16 on ImageNet: On ImageNet, NeST's grow-and-prune paradigm outperformed pruning-only methods for AlexNet- and VGG-16-based inference models.The authors suggest growth can reduce redundancy inherited from large original networks.
- Limitations: Mask-based growth and pruning currently incur temporal training-memory overhead, which may be uneconomical for deeper ResNet or DenseNet models.The authors identify addressing this overhead as future work.
5 Discussions
NeST is discussed as a brain-inspired grow-and-prune process whose connection dynamics, rewiring, and sparsity motivate compact DNN synthesis. The resulting sparsity targets redundant neuron responses and reduces storage and computation.
- Brain-inspired synthesis: NeST's connection count follows a brain-inspired pattern: growth early in learning followed by steady pruning.The paper compares this trajectory with synapse evolution across human brain development.
- Brain-inspired synthesis: NeST mimics synaptic rewiring by activating new connections during learning and later pruning others.The paper relates this to substantial daily synaptic growth and pruning in the brain.
- Brain-inspired synthesis: Magnitude-based pruning removes insignificant neuron responses, producing sparsity and compactness with large storage and computation reductions.The motivation is that only a small fraction of brain neurons are active at any given time, unlike fully connected DNNs.
6 Conclusions
NeST synthesizes accurate, compact DNNs by starting from sparse seeds, applying gradient-based growth, and then using magnitude-based pruning. The paper reports substantial parameter and FLOPs reductions across MNIST and ImageNet architectures.
- Conclusions: NeST starts from a sparse seed architecture, grows structures using gradients, and prunes redundant structures using magnitudes to produce compact accurate DNNs.The method synthesizes models for LeNet-300-100, LeNet-5, AlexNet, and VGG-16.
- Conclusions: 70.2× and 74.3× parameter reductions with 79.4× and 43.7× FLOPs reductions were achieved for LeNet-300-100 and LeNet-5, respectively.For AlexNet and VGG-16, parameter reductions were 15.7× and 30.2×, while FLOPs reductions were 4.6× and 8.6×.
A Experimental details of LeNets
The LeNet experiments evaluate the smallest synthesized models for LeNet-300-100 and LeNet-5, using metrics that include partial-area convolution coverage and non-zero activation rates.
- Smallest synthesized LeNets: Tables 4(a) and 4(b) report the smallest DNN models synthesized for LeNet-300-100 and LeNet-5.Table 4(a) identifies a LeNet-300-100 model with a 1.29% error rate.
- Smallest synthesized LeNets: Conv% measures area-of-interest coverage for partial-area convolution, while Act% measures the average percentage of non-zero activations per inference.Both metrics describe sparsity or selective computation in the synthesized LeNets.
B Experimental details of AlexNet
NeST produces a compact AlexNet-based inference model through successive growth and pruning, reducing the model from a sparse seed to a smaller final network.
- 3.9M parameters remain after pruning, down from 28.3M after growth and 8.4M in the AlexNet seed.The final model requires 325M FLOPs at a top-1 error rate of 42.76%.
- The synthesized AlexNet has a 42.76% error rate.
- The paper states that its models will be released soon.
C Experimental details of VGG-16
NeST synthesizes a compact VGG-16-based inference model with substantially fewer parameters than the original architecture.
- 4.6M parameters are retained in the final VGG-16-based model, making it 30.2× smaller than the original VGG-16.
- The synthesized VGG-16 has a 30.72% error rate.