Source-linked AI summary
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
Arber Zela, Aaron Klein, Stefan Falkner, Frank Hutter
TL;DR
The paper addresses inefficiencies from separating neural architecture search from hyperparameter tuning and from comparing configurations across poorly correlated short and long training budgets. It jointly searches architectures and hyperparameters with BOHB, which combines Bayesian optimization and Hyperband, and reports competitive CIFAR-10 performance within 3 hours.
Problem
NAS commonly fixes hyperparameters during architecture search and tunes them afterward, while short- and long-budget rankings correlate weakly.
Method
The paper jointly searches architectural and hyperparameter choices with BOHB, combining Bayesian optimization and Hyperband with incrementally increasing budgets.
Results
Competitive CIFAR-10 performance is achieved within a limited 3-hour training runtime.
Takeaways & Limitations
Joint architecture and hyperparameter optimization is desirable and feasible, while incremental budget increases can sidestep short-training effects on configuration choices.
Takeaways & Limitations
The architecture search space is restricted to a fixed 10-dimensional representation because of limited compute, and 3-hour BOHB evaluations are skewed toward configurations performing well on smaller budgets.
Abstract
from arXiv · showhide
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we demonstrate that the common practice of using very few epochs during the main NAS and much larger numbers of epochs during a post-processing step is inefficient due to little correlation in the relative rankings for these two training regimes. To combat both of these problems, we propose to use a recent combination of Bayesian optimization and Hyperband for efficient joint neural architecture and hyperparameter search.
1. Introduction
Existing NAS methods often search architectures with fixed hyperparameters and short training budgets, then tune hyperparameters afterward with much larger budgets. The paper argues this separation and budget jump are inefficient and proposes joint, anytime optimization with gradually increasing budgets.
- Many NAS methods first search architectures with fixed hyperparameters and about 20 epochs, then tune the result using about 600 epochs.
- The two-step process is not anytime, so it cannot provide predictions after an arbitrary time budget.
- The relative rankings of configurations correlate weakly between 20- and 600-epoch budgets, potentially making much of the short-budget optimization void.
- The proposed approach jointly searches architectures and hyperparameters while gradually increasing computation for the best fraction of networks.
- Bayesian optimization combined with Hyperband provides the framework for efficient joint neural architecture and hyperparameter search.
- Within a limited 3-hour training runtime, joint optimization achieves competitive CIFAR-10 performance.
2. Related Work
Related work shows that hyperparameter tuning can materially affect neural-network performance, while NAS methods commonly search architectures separately from hyperparameters. Bayesian optimization addresses expensive evaluations by exploiting lower-cost fidelity information.
- A well-tuned LSTM outperformed a recurrent cell found by NAS on Penn Treebank, underscoring the practical effect of hyperparameters.
- Bayesian optimization methods accelerate expensive neural-network evaluations by exploiting fidelities such as learning curves or dataset subsets.
- Many NAS methods search only neural architectures while keeping hyperparameters fixed and optimizing them in a post-hoc step.
3. Efficient Joint Hyperparameter Optimization and Architecture Search
The paper casts NAS as hyperparameter optimization over categorical and conditional choices, enabling joint search of architecture and training settings. It uses BOHB to model interactions and allocate increasing budgets to promising configurations within a multiple-branch ResNet search space.
- Joint search formulation: Most NAS search spaces can be represented as hyperparameter-optimization spaces using categorical and conditional hyperparameters.
- Joint search formulation: Joint search extends the architecture hyperparameter space with standard training hyperparameters, allowing both to be optimized together.
- BOHB: BOHB combines Bayesian optimization and Hyperband to search the joint space efficiently.
- BOHB: BOHB exploits cheap low-fidelity evaluations and repeatedly applies Successive Halving to allocate more resources to promising configurations.
- BOHB: Its multivariate KDEs model parameter interactions and propose configurations with the highest expected improvement.
- Search space: The search space uses a multiple-branch ResNet with 10 architectural choices covering filters, residual blocks, branches, and widening factors.
- Search space: The architecture space is restricted to 10 dimensions because of limited compute, leaving more generic variable-length network representations for future work.
- Search space: The optimization tunes learning rate, batch size, momentum, L2 regularization, CutOut length, MixUp α, and ShakeDrop death rate.
4. Experiments
Experiments evaluate BOHB for joint architecture and hyperparameter search under budgets from 400 seconds to 3 hours. Results show strong budget-dependent interactions, weak long-range rank correlation, and improved performance from gradually increasing resources.
- Experimental setup: BOHB searches a 10-dimensional architecture space under budgets of 400s, 1200s, 1h, and 3h, with a maximum of three hours per configuration.The study uses a training-time-constrained NAS setting and default budget multiplier η = 3.
- Results: 3.18% test error was achieved within the 3h budget, outperforming several standard architectures evaluated with the same pipeline and hyperparameters.The search used 256 equivalent full-budget evaluations, totaling 32 GPU days.
- Results: The medium 26 2x64d Shake-Shake architecture outperformed the larger 26 2x96d model within 3h, opposite to comparisons using 1800 epochs.A WRN outside the search space performed slightly better at 3h but worse at large budgets, linking regularization efficacy to available time.
- Budget analysis: Rank correlation is roughly 0.87 for threefold budget increases but nearly vanishes when comparing 400s with 3h, so short-run rankings cannot identify the best long-run configurations.The study reports that correlation degrades quickly as the budget difference grows.
- Budget analysis: Learning rate importance remains stable across budgets, whereas residual-block count and its interaction with CutOut length change substantially with training time.With longer training, residual-block count becomes less important and the useful CutOut range becomes more restricted.
- Budget analysis: Architectural and hyperparameter choices interact with runtime budget, making optimization only at the smallest budget wasteful when evaluation occurs at the largest.The 3h evaluations are skewed toward configurations that performed well on smaller budgets because BOHB progressively allocates resources to promising configurations.
5. Conclusions
The paper concludes that jointly optimizing neural architectures and hyperparameters is desirable and feasible, while incremental budget increases address poor short-versus-long training correlation.
- Joint optimization of neural architectures and hyperparameters is desirable and feasible.
Appendix A. Joint Archicture and Hyperparameter Search Space
The search space combines 10 architectural choices and 7 hyperparameters, with the best configuration reported after BOHB optimization on a 3-hour budget.
- The configuration space contains 10 architectural choices and 7 hyperparameters.Table 3 also reports specific ranges and the best-performing configuration on the 3-hour budget.
- Table 3 maps the three main architectural blocks to numbered Residual Block, Residual Branches, and Widen Factor choices, and identifies the BOHB-selected best configuration.
Appendix B. Training details
Training uses PreAct ResNet-18 and WideResNet-28-10 with Shake-Shake regularization on one GTX 1080Ti, while CIFAR-10 is split into training, validation, and test sets with standard augmentation.
- Experiments use PreAct ResNet-18 and WideResNet-28-10 with Shake-Shake regularization on one Nvidia GTX 1080Ti GPU.Models use SGD with an initial learning rate of 0.1 and Nesterov momentum of 0.9.
- CIFAR-10 is split into 45k training, 5k validation, and 10k test examples, with normalization, padding, random cropping, horizontal flipping, and CutOut augmentation.
Appendix C. Correlation across budgets
Correlation is high between adjacent budgets but degrades quickly as budget differences grow, leaving the smallest budget uninformative about the largest-budget configurations.
- Adjacent budgets have high correlation, but correlation quickly degrades for larger budget differences.
- The smallest and largest budgets show no correlation in error rates, although the analyzed configurations are biased toward the best largest-budget configurations.