Source-linked AI summary
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, Edward Choi
TL;DR
MorphNet addresses the challenge of manually designing deep-network structures and the limited resource specificity of prior sparsification methods. It iteratively shrinks networks with resource-weighted activation regularization and expands them uniformly, achieving improved performance under fixed resource constraints across datasets.
Problem
Prior architecture design relies heavily on manual trial-and-error, while existing sparsification methods generally do not target specific resources such as FLOPs per inference.
Method
MorphNet iteratively shrinks and expands a DNN, using a resource-weighted sparsifying regularizer on activations followed by uniform multiplicative expansion.
Results
2.1% improvement in evaluation MAP on JFT was achieved at the same FLOPs per inference, while ImageNet Inception-v2 test accuracy increased 1.1% at the same FLOPs per instance.
Takeaways & Limitations
MorphNet discovers improved, constraint-specific structures across datasets and provides a scalable, simple tool for automating resource-constrained architecture design.
Takeaways & Limitations
The optimal number of MorphNet iterations and whether the process converges remain to be investigated.
Abstract
from arXiv · showhide
We present MorphNet, an approach to automate the design of neural network structures. MorphNet iteratively shrinks and expands a network, shrinking via a resource-weighted sparsifying regularizer on activations and expanding via a uniform multiplicative factor on all layers. In contrast to previous approaches, our method is scalable to large networks, adaptable to specific resource constraints (e.g. the number of floating-point operations per inference), and capable of increasing the network's performance. When applied to standard network architectures on a wide variety of datasets, our approach discovers novel structures in each domain, obtaining higher performance while respecting the resource constraint.
1. Introduction
MorphNet addresses the largely manual design of deep neural networks with a scalable, resource-constrained architecture-optimization method. It finds constraint-specific structures and can improve performance while maintaining targeted resource usage.
- DNN structure design has traditionally relied on expert creativity, intuition, experience, and trial-and-error, motivating automated architecture design.
- Existing neuron-focused sparsification reduces parameters but generally does not target a particular resource such as FLOPs per inference.
- MorphNet is a simple, general, scalable technique that optimizes DNN structures for specific resources while allowing untargeted resources to grow as needed.
- 2.1% improvement in evaluation MAP on JFT was achieved at the same FLOPs per inference, with resources only slightly greater than training once.
- 1.1% test-accuracy improvement on ImageNet was obtained by modifying Inception-v2 while maintaining the same FLOPs per instance.
- MorphNet finds unique, improved structures for different constraints, providing a general automated alternative to context-specific methods and manual labor.
2. Related Work
Prior work sparsified networks or searched architectures through extensive trial and error, but often lacked resource specificity or scalability. MorphNet extends sparsification toward application-specific constraints and structure changes.
- Weight sparsification can reduce nonzero weights, but pruning individual connections is poorly suited to modern hardware accelerators and often does not yield practical speedups.
- MorphNet follows a multi-step process in which a sparsity-inducing stage is followed by retraining the newer structure from scratch.
- Neuron-level sparsification improves hardware-relevant structure, yet previous techniques generally reduce parameters without targeting a particular resource.
- Earlier sparsification methods traditionally focused on model size, whereas MorphNet formulates constraints generally and evaluates both model size and FLOPs.
- Reinforcement-learning and genetic-algorithm approaches search architectures through many trial-and-error attempts, requiring substantial computation.
3. Background
The paper considers feed-forward convolutional networks and formulates architecture design as choosing layer widths to optimize performance under a resource constraint. The seed network’s topology and other design choices remain fixed.
- The considered DNNs typically stack convolutions, biases, fully connected layers, and pooling layers to produce a vector of scores.
- Each parameterized layer has input and output widths; for convolutions, these correspond to input and output channel counts.
- A convolution maps a w_L × x_L × I_L input to a y_L × z_L × O_L output using weights with dimensions I_L × O_L × f_L × g_L.
- 3.1. Problem Setup: The optimization is restricted to output widths O_1:M, while filter dimensions, topology, and other seed-network choices are fixed.
- 3.1. Problem Setup: The design problem seeks a high-performing network under F(O_1:M) ≤ ζ, where F is either FLOPs per inference or model size in this paper.
4. Method
MorphNet alternates resource-aware neuron sparsification with uniform width scaling to reshape a network under constraints such as FLOPs or model size. Its regularizer accounts for surviving neurons and layerwise resource costs, while repeated cycles can improve performance.
- Width multiplier: The width-multiplier baseline uniformly shrinks or expands every layer, selecting the largest multiplier that satisfies the resource constraint.Values below one shrink the network, while values above one expand it.
- Sparsifying regularizer: Sparsifying regularization changes relative layer widths by zeroing neurons, but it does not guarantee the resulting structure satisfies the resource constraint.It may also disproportionately sacrifice performance.
- MorphNet algorithm: MorphNet alternates neuron sparsification with uniform expansion, shrinking targeted resource usage before expanding all layer sizes as much as the constraint permits.The method starts from a seed network and repeats these steps for additional architectural improvement.
- MorphNet algorithm: During shrinking, resource-weighted regularization identifies both over-parameterized and bottlenecked layers, with the affected resolution pattern depending on whether FLOPs or model size is targeted.For FLOPs, higher-resolution lower-layer neurons tend to be sacrificed more; the pattern reverses for model size.
- Iteration and cost: A single MorphNet iteration can noticeably outperform uniform scaling, while later iterations may provide additional gains; the optimal iteration count remains unresolved.Each iteration may require several training runs for different λ values, commonly five or fewer.
- Resource constraints: For convolutional layers, the constrained cost is modeled from input and output dimensions, with total network cost obtained by summing layer costs.Bias costs are included separately but omitted from the displayed simplified expressions.
- Regularization: MorphNet uses an L1 penalty on batch-normalization scale variables, where setting a scale to zero effectively removes its neuron and associated computation.The regularizer uses indicator functions for alive inputs and outputs, and its total value sums contributions across layers.
- Regularization: The regularizer produces a clear separation between zeroed and nonzero γ values in a ResNet101 bottleneck layer trained with a FLOP regularizer.This separation supports extracting structured neuron sparsity after training.
5. Empirical Evaluation
MorphNet was evaluated across datasets, architectures, and resource targets, consistently finding structures that improved performance under constrained FLOPs or model size. On ImageNet, iterative sparsification and expansion improved Inception V2 accuracy while preserving FLOPs, and broader experiments showed gains on MobileNet, JFT, and AudioSet.
- Experimental setups: Experiments covered ImageNet, JFT, and AudioSet, using Inception V2, MobileNet, ResNet101, and a residual audio architecture.
- Inception V2 on ImageNet: The FLOP regularizer adequately targeted the desired constraint during Inception V2 training, despite being only a proxy for true FLOPs.
- Inception V2 on ImageNet: Sparsifying with the FLOP regularizer was more effective than a naïve width multiplier for producing downsized Inception V2 networks.
- Inception V2 on ImageNet: 1.1% total accuracy improvement over the Inception V2 baseline was achieved after two sparsification-and-expansion iterations at the original FLOP cost.The first iteration improved accuracy by 0.6%, and the second added 0.5%; dropout was introduced to mitigate stronger overfitting.
- Inception V2 on ImageNet: 30% fewer FLOPs produced an Inception V2 network matching the original model’s accuracy.
- Improved performance at no cost: Performance improved across all datasets under a FLOP constraint, including a 1% improvement on MobileNet and over 2.1% relative improvement on JFT.On JFT, one MorphNet iteration added 5% to the cost of training a single model for a fixed regularization strength, although selecting a suitable strength may cost more.
- Improved performance at no cost: 2.18% relative MAP increase on AudioSet corresponded to a FLOP-per-inference reduction of over 50% for an equivalent decrease from the seed model.
- Resource targeting: Targeting FLOPs and model size produced different tradeoff curves and structures, with FLOP targeting removing lower-layer neurons and model-size targeting removing upper-layer neurons.The resulting structures achieved MAP values of 0.428 and 0.421, compared with 0.405 for a similar-cost baseline; FLOP regularization can also remove entire layers.
6. Conclusion
MorphNet learns DNN structures under resource constraints through a simple, scalable, and fast iterative approach. The experiments show resource-dependent accuracy–cost tradeoffs and improvements over human-designed networks with little additional training cost.
- MorphNet learns DNN structures under a constrained resource and can navigate tradeoffs for either FLOPs or model size.
- The method improves large-scale, human-designed DNN structures with little extra training cost compared with training the DNN once.
- MorphNet is simple to implement and fast to apply as a tool for automating neural network architecture design.
B.1. Training protocol
The protocol trains a full MobileNet, applies MorphNet with FLOP regularization, extracts active channels, and briefly fine-tunes the resulting pruned network. It compares these models with width-multiplier baselines on 128 × 128 images.
- Training setup: The models operate on 128 × 128 images and use a slight one-iteration variant of MorphNet.This variability is reported to improve results and help MorphNet overtake the 50% width-multiplier model.
- Training setup: The full width-multiplier-1.0 network is trained for 2 million steps before initializing MorphNet training.Smaller width-multiplier networks may require around 10 million steps to converge.
- MorphNet training: MorphNet training continues for 10 million steps or until active-channel FLOPs converge, using λ values from 3 × 10^-9 through 11 × 10^-9.The λ range is chosen to keep converged FLOPs near the width-multiplier baselines.
- Pruning and fine-tuning: The converged checkpoint is converted into a pruned network containing only active channels, including both the structure and weights.The extracted network is then fine-tuned with a small learning rate to restore batch-normalization moving-average statistics.
- Pruning and fine-tuning: 20k fine-tuning steps suffice to outperform width-multiplier models, although longer training continues to improve accuracy.All training steps use the same optimizer and trainer configuration.
B.3. Observations
Each attempted λ contributes to the observed FLOPs-versus-accuracy tradeoff, while the 50% width-multiplier model is dominated by MorphNet models under the paper’s narrow optimality definition.
- Tradeoff observations: Each λ value contributes to the “optimal” FLOPs-vs-accuracy tradeoff because it yields a model not dominated in both FLOPs and accuracy.Here, optimality means that no other model is better in both dimensions.
- Tradeoff observations: The 50% width-multiplier model is dominated by MorphNet models.The comparison is made in the joint FLOPs-and-accuracy sense defined in the passage.
- Training cost: Each attempted λ requires around 12 million training steps, less than twice the roughly 10 million steps used to train a regular network.Multiple λ values are used, and each contributes to the tradeoff curve.
C. ResNet101 on JFT
The supplied passages describe regularizer settings, learned ResNet101 width changes, and variability measurements for ImageNet Inception V2 runs.
- Regularizer settings: The JFT experiments use FLOP regularizers of 0.7, 1.0, 1.3, and 2 × 10^-9, and size regularizers of 0.7, 1, and 3 × 10^-7.Width-multiplier values range from 1.0 to 0.375.
- Learned ResNet101 structures: Increasing FLOP-regularizer strength removes progressively more neurons from ResNet101 blocks and can remove entire residual units.The figure contrasts FLOP- and size-regularized widths with a uniformly expanded model matched to the seed model’s FLOPs.
- Stability analysis: The stability analysis repeats FLOP-regularized Inception V2 training independently 10 times with λ = 1.3 × 10^-9.The resulting architectures are compared through the number of filters in each layer.
- Learned ResNet101 structures: Figure 7 illustrates structures learned when applying the listed regularizers to ResNet101.The supplied passage links the width-multiplier settings to the structural illustration.
D. Stability of MorphNet
MorphNet’s stability is assessed across 10 independent Inception V2 runs with the same FLOP-regularization configuration. Most layer widths vary little, while the resulting FLOPs and test accuracy show low relative standard deviations.
- Experimental setup: The stability experiment trains Inception V2 10 times independently with a constant 10^-3 learning rate and λ = 1.3 × 10^-9.The final architecture from each run is represented by the number of filters in each layer.
- Outcome stability: Relative standard deviation is calculated by dividing standard deviation by the mean.This definition applies to the reported variability measure.
- Outcome stability: 1.12% FLOPs relative standard deviation and 0.208% test-accuracy relative standard deviation are reported across 10 runs.Figure 9 plots FLOPs versus test accuracy for the resulting Inception V2 models.
- Architecture stability: Most layers show little change in filter count across runs using the same MorphNet configuration.A few layers have slightly larger relative standard deviations, but their small widths keep absolute changes small.
E. Extensions of the method
The paper evaluates MorphNet primarily by optimizing layer output widths, while its shrink-and-expand process can extend to other network-design dimensions. Filter dimensions and depth are possible extensions using structured sparsity and uniform expansion.
- Scope of evaluation: The evaluation is restricted to optimizing the output widths O_1:M of all layers.
- Scope of evaluation: The same iterative shrinking and uniform multiplicative expansion process can be applied to other aspects of network design.
- Possible extensions: Filter dimensions and network depth could be shrunk using Group LASSO and residual connections, then expanded uniformly with a width multiplier.