Source-linked AI summary
Simple And Efficient Architecture Search for Convolutional Neural Networks
Thomas Elsken, Jan-Hendrik Metzen, Frank Hutter
TL;DR
Neural architectures are usually designed manually, making architecture search costly and dependent on expert knowledge. The paper proposes NASH, a hill-climbing method that applies network morphisms and short cosine-annealing runs, achieving competitive results with resources on the order of training a single network.
Problem
Manual architecture design is exhausting and time-consuming, while discrete and conditional search spaces make many classic optimization algorithms inapplicable and existing methods costly or non-competitive.
Method
NASH iteratively applies alternative network morphisms to a current network, briefly trains the resulting child networks with cosine annealing, and selects the most promising child.
Results
NASH finds CIFAR-10 CNNs with error below 6% in roughly 12 hours on one GPU, reducing error to almost 5% after one day.
Takeaways & Limitations
The method yields competitive results with computational resources of the same order of magnitude as training a single network and is positioned as a basis for future extensions.
Abstract
from arXiv · showhide
Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cumbersome trial-and-error process. We propose a new method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing. Surprisingly, this simple method yields competitive results, despite only requiring resources in the same order of magnitude as training a single network. E.g., on CIFAR-10, our method designs and trains networks with an error rate below 6% in only 12 hours on a single GPU; training for one day reduces this error further, to almost 5%.
1 INTRODUCTION
Neural architecture search is motivated by the cost and expertise required to design networks manually, while existing automated methods can be expensive or non-competitive. The paper proposes simple network-morphism-based approaches intended to reduce computation while retaining competitive performance.
- Manual neural-network design is exhausting and time-consuming, and the large configuration space requires expert knowledge to restrict the search.
- Discrete, conditional architecture spaces limit the applicability of optimization methods that rely on differentiability or independent parameters.
- Existing automated architecture-search methods are often very costly, requiring hundreds or thousands of GPU days, or yield non-competitive performance.
- The paper aims to dramatically reduce computational costs while still achieving competitive performance.
- The baseline randomly constructs networks and trains them with SGDR, achieving 6%-7% test error on CIFAR-10.
- NASH applies alternative network morphisms iteratively, trains child networks with short cosine-annealing runs, and selects the most promising child.
- The method is described as easy to use and extend, with the paper positioning it as a basis for future work.
2 RELATED WORK
Related work spans automated hyperparameter optimization, reinforcement-learning and evolutionary architecture search, and network morphisms. Prior approaches vary in computational cost and search-space expressiveness, motivating the paper’s approach.
- Automated hyperparameter optimization includes random search, Bayesian optimization, bandit-based approaches, and evolutionary strategies.
- Architecture search has shifted toward specialized techniques beyond treating architectural choices as categorical hyperparameters.
- Evolutionary methods generate networks through mutations such as inserting layers, modifying layer parameters, or adding skip connections.
- Real et al. used 250 GPUs for 10 days, whereas Suganuma et al. handled relatively small networks because they maintained a population of networks.
- Network morphisms include function-preserving transformations that make networks deeper or wider, while later work added operations such as changing kernel size.
- Cai et al. limited their architecture search to simple networks without skip connections by using only a restricted set of morphism operations.
3 NETWORK MORPHISM
Network morphisms transform a network into a functionally equivalent architecture while enabling depth, width, normalization, nonlinearities, and skip connections. These transformations provide composable building blocks for architecture search.
- Definition: A network morphism maps one neural network to another while preserving its function on every input.The mapping can change both the network architecture and the number of parameters.
- Network morphism types: Type I inserts a linear or convolutional layer while preserving the original network function through identity initialization.This transformation is called Net2DeeperNet.
- Network morphism types: Type II can widen fully connected or convolutional layers and formulate concatenative skip connections as function-preserving transformations.Widening increases units or channels, while a suitable replacement function can realize a skip connection.
- Network morphism types: Type III inserts an additional idempotent function, including an unweighted ReLU, by initializing the new function with the existing parameters.The construction composes the original function with the inserted copy.
- Network morphism types: Type IV mixes the original function with an arbitrary function and can incorporate nonlinearities or additive skip connections.Function preservation holds when λ is initialized to 1.
- Composition: Network morphisms compose, allowing standard blocks such as Conv-BatchNorm-Relu to be inserted after a ReLU layer.The paper states that every combination of network morphisms is itself a morphism.
4 ARCHITECTURE SEARCH BY NETWORK MORPHISMS
NASH uses hill climbing over network architectures: it generates function-preserving child networks from the current best model, briefly trains them, and selects the strongest candidate. The method can also be viewed as a simple evolutionary algorithm without crossover.
- Search strategy: NASH starts from a small network and repeatedly generates larger child networks by applying network morphisms to the current best model.The search proceeds until validation performance saturates.
- Search strategy: Function-preserving initialization lets child networks begin with their parent’s performance, reducing the cost of evaluating architectures without training from scratch.Children are then trained briefly to exploit the additional capacity introduced by the morphisms.
- Selection: Each search step trains neighboring child models briefly and moves to the best resulting child, while retaining the current model when no child improves upon it.The current best model is included among the candidates.
- Optimization: Cosine annealing provides the short optimization runs required for child networks and is also used during final training.The learning rate is annealed from λ_start to λ_end over the neighbor-training epochs.
- Interpretation: NASH is also interpretable as an evolutionary algorithm with population size n_neigh, network morphisms as mutations, no crossover, and best-member parent selection.This interpretation motivates possible extensions to the method.
5 EXPERIMENTS
Experiments evaluate the search procedure on CIFAR-10 and CIFAR-100, testing selection, weight inheritance, runtime, and comparisons with handcrafted and automated architectures. The method produces competitive results with substantially less computational expenditure than several alternatives.
- Experimental setup: The experiments evaluate the method on CIFAR-10 and CIFAR-100 using fixed data augmentation, training/validation splits, and Nvidia Titan X GPUs.The CIFAR training set contains 50,000 samples, split into 40,000 training and 10,000 validation samples; final performance is measured on the test set.
- Baselines: The baseline comparison tests random construction against hill-climbing selection with nneigh = 1 versus nneigh = 8.Both settings use nsteps = 5, nNM = 5, epochneigh = 17, and epochfinal = 100.
- Retraining from scratch: Retraining generated models from scratch changes error rates by less than 0.5% in all but one model.The differences sometimes favor NASH and sometimes favor retraining from scratch, indicating little consistent performance gap.
- Retraining from scratch: Architecture search adds roughly a factor-3 runtime overhead relative to training the final model.The paper characterizes this overhead as being in the same order of magnitude as training a single network.
- Comparison to other architectures: After 12 hours, the proposed method generates competitive architectures; after another 12 hours, it outperforms most automated architecture-search methods.It does not reach two handcrafted architectures or the architectures found by Zoph & Le (2017) and Brock et al. (2017).
- CIFAR-100 experiments: On CIFAR-100, the method is on a par with Real et al. (2017) after one day using a single GPU.Its snapshot ensemble performs similarly to Brock et al. (2017), while an ensemble across five runs can compete with WRN 28-10; Shake-Shake performance is not reached.
6 CONCLUSION
NASH is a simple, fast hill-climbing method for automated architecture search that combines network morphisms with SGDR training. Experiments on CIFAR-10 and CIFAR-100 found competitive results with considerably fewer computational resources than most alternatives.
- NASH combines hill climbing, network morphisms, and SGDR to search for CNN architectures.
- The approach can be extended with additional morphisms, evolutionary model generation, cheaper performance evaluation, or improved resource handling.
A SOME MODELS
The appendix presents networks produced by the algorithm at increasing search depths, showing the initial network and generated networks for nsteps = 5 and nsteps = 8.
- The initial network used by the architecture-search algorithm is presented first.
- A generated network is shown after nsteps = 5.
- A generated network is shown after nsteps = 8.