Source-linked AI summary
Neural Architecture Search: Insights from 1000 Papers
Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, Frank Hutter
TL;DR
Deep learning's broad success makes specialized architectures important, while evidence and guidance for navigating rapidly expanding NAS methods remain dispersed. This survey organizes NAS search spaces, algorithms, speedup techniques, and resources, while highlighting limitations in robustness, reliability, and evaluation across settings.
Problem
NAS research has expanded rapidly across diverse search spaces and methods, creating a need for an organized guide to techniques, benchmarks, best practices, and resources.
Method
The survey provides a taxonomy of NAS search spaces, search algorithms, performance and speedup techniques, extensions, applications, and research resources.
Results
The survey finds that macro spaces offer high representation power but are slow to search, while cell-based spaces search faster but face low performance variance and design-choice criticisms.
Takeaways & Limitations
NAS practice should consider search-space trade-offs and use appropriate baselines, including random search, when evaluating new algorithms.
Takeaways & Limitations
One-shot methods lack large-scale fair comparisons and require substantial expertise to apply robustly across new datasets and tasks.
Abstract
from arXiv · showhide
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, and reinforcement learning. Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas. Neural architecture search (NAS), the process of automating the design of neural architectures for a given task, is an inevitable next step in automating machine learning and has already outpaced the best human-designed architectures on many tasks. In the past few years, research in NAS has been progressing rapidly, with over 1000 papers released since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized and comprehensive guide to neural architecture search. We give a taxonomy of search spaces, algorithms, and speedup techniques, and we discuss resources such as benchmarks, best practices, other surveys, and open-source libraries.
1. Introduction
Neural architecture search automates the design of task-specific neural networks within a defined search space and computation budget. This survey organizes NAS methods around search spaces, search strategies, performance estimation, and related resources.
- Deep learning breakthroughs across vision, language, speech, and reinforcement learning have increased the importance of specialized neural architectures.
- NAS developed within AutoML but differs from typical hyperparameter optimization because neural architectures have intricate, usually discrete, graph-structured spaces.NAS-specific techniques can exploit architectural structure, including sharing weights among similar architectures to avoid training each from scratch.
- NAS research expanded rapidly after 2017, with over 1000 papers released in the last two years reported by the survey.
- A NAS search space contains possible architectures, while search strategies optimize within it and performance estimation methods accelerate evaluation.Search spaces may range from a few thousand to over 10^20 architectures, and domain knowledge can simplify search while introducing human bias.
- NAS searches for a high-performing architecture by selecting candidates from a predefined space and estimating their performance under a computational budget.The formal objective seeks an architecture a ∈ A with the highest possible validation accuracy using dataset D and pipeline P within budget t.
- The survey covers search spaces, black-box optimization, one-shot methods, speedup techniques, extensions, applications, and NAS resources.Its taxonomy includes approaches such as reinforcement learning, evolutionary algorithms, Bayesian optimization, and weight-sharing techniques.
2. Search Spaces
NAS search spaces determine which architectures can be explored, balancing search efficiency against human bias and the possibility of discovering novel designs. The survey distinguishes macro, chain-structured, and cell-based spaces, while emphasizing architecture encodings as an associated design choice.
- Search-space design: Search-space design trades off efficient optimization against human bias and architectural novelty.Smaller, more hand-designed spaces are easier to search, whereas larger spaces with primitive building blocks take longer but may reveal novel architectures.
- Macro search spaces: Macro search spaces represent entire architectures or vary macro-level hyperparameters, offering high representation power but slow search.Whole-architecture spaces can vary graph topology and node operations, while other macro spaces vary depth, width, resolution, or downsampling with topology and operations fixed.
- Chain-structured search spaces: Chain-structured spaces search sequential operation layers, making them simple to design and relatively quick to optimize but less likely to produce novel topologies.Examples include convolutional and transformer spaces that search layer-level choices such as kernel size, expansion ratio, depth, width, and number of heads.
- Cell-based search spaces: Cell-based spaces search reusable directed acyclic graph cells and stack them into full networks, reducing complexity while often preserving high performance and transferability.Cells can be transferred from small datasets to larger ones by increasing the number of cells and filters in the resulting architecture.
- Cell-based search spaces: Cell-based spaces have limited performance variation, ad-hoc fixed choices, and reduced expressiveness, motivating searches over macro connections as well as cell structure.In DARTS, sophisticated search strategies may yield only marginal gains over randomly sampled architectures because architecture-performance variance is small.
3. Black-Box Optimization Techniques
Black-box NAS searches by repeatedly evaluating architectures and using their measured performance to guide subsequent choices. Its methods include random, local, reinforcement-learning, evolutionary, Bayesian, and tree-search strategies, generally trading computational cost against flexibility and performance.
- Overview: Black-box optimization repeatedly samples, trains, and evaluates architectures, using performance estimates to guide the search.These methods commonly train architectures independently rather than sharing weights.
- Baselines: Random search fully trains randomly selected architectures and returns the one with the best validation accuracy.It is recommended as a baseline and can be strengthened with weight sharing, zero-cost proxies, or learning-curve extrapolation.
- Baselines: Local search iteratively trains and evaluates neighbors of the best architecture found so far, differing typically by one operation or edge.Studies report strong performance on both small and large search spaces, with substantial opportunities for speedup.
- Reinforcement Learning: Reinforcement-learning NAS uses a controller to generate architecture actions, trains sampled architectures, and updates the controller using validation performance as reward.The controller is usually a recurrent neural network whose parameters are updated to maximize expected reward.
- Reinforcement Learning: Reinforcement learning has been used less prominently because evolutionary methods and Bayesian optimization outperformed it in head-to-head comparisons.This comparison is reported for recent NAS research rather than as a universal ranking across all settings.
- Evolutionary Algorithms: Evolutionary NAS maintains a population by sampling parent architectures, mutating them into children, training the children, and replacing older or lower-performing individuals.Regularized evolution drops the oldest population member and outperformed random search and reinforcement learning in a head-to-head comparison while achieving state-of-the-art ImageNet performance at release.
- Bayesian Optimization: Bayesian optimization builds a probabilistic surrogate and selects architectures with an acquisition function that balances exploration and exploitation.The selected architecture is trained and added to the observed population before the surrogate is updated.
4. One-Shot Techniques
One-shot NAS avoids independently training every candidate by training a shared hypernetwork or supernetwork whose weights support architecture evaluation. Its families combine this shared model with black-box or differentiable search, improving scalability while relying on ranking and optimization assumptions that remain debated.
- Overview: One-shot methods reduce computational burden by implicitly training all architectures through one training run of a hypernetwork or supernetwork.They were among the most popular NAS techniques as of 2022.
- Supernet-Based Methods: A supernetwork is an over-parameterized architecture containing all search-space architectures as subnetworks, while a hypernetwork generates weights for other neural networks.Each candidate architecture can be represented as a subgraph within the supernet.
- Supernet-Based Methods: After supernet training, candidate architectures inherit weights from corresponding subnetworks for rapid evaluation.Adding candidate operations increases training cost linearly while the number of contained subnetworks grows exponentially.
- Assumptions and Limitations: The one-shot ranking assumption holds that supernet-based architecture rankings are relatively consistent with rankings from independently trained architectures.Evidence both supports and challenges this assumption across settings, and its validity depends on search-space design.
- Supernet-Based Methods: Supernet methods can decouple training and search, run non-differentiable search during training, or optimize architecture choices with differentiable methods such as DARTS.The search strategy may therefore be black-box, reinforcement-learning-based, or gradient-based.
- Supernet-Based Methods: Once-for-all supernets progressively shrink from large to smaller subnetworks and support later selection under latency and memory constraints.The approach trains networks of different sizes in one shared model and can use random or evolutionary search afterward.
- Differentiable Supernet-Based Methods: DARTS continuously relaxes discrete operation choices, jointly optimizes architecture hyperparameters and model weights with gradient descent, then discretizes the result.Its continuous relaxation enables gradient-based search in DAG-based spaces, including absent edges via a zero operation.
5. Speedup Techniques
NAS speedup techniques estimate architecture performance without full training, reducing search cost across black-box and one-shot methods. The survey covers performance predictors, multi-fidelity methods, meta-learning, and weight inheritance, with predictors including learning-curve extrapolation, zero-cost proxies, and low-fidelity data training.
- Performance prediction: Performance predictors estimate architecture accuracy faster than full training and ideally preserve correlation or rank correlation with final performance.They can accelerate both black-box optimization and one-shot NAS.
- Performance prediction: Learning-curve extrapolation predicts final performance from validation accuracies observed during partial training.Methods fit parametric models or combine partial curves with architecture encodings in surrogate models.
- Performance prediction: Zero-cost proxies score architectures using very fast computations, often a single minibatch forward and backward pass, before training.The scores are intended to correlate with final accuracies; synflow instead computes an L1 path-norm from initialized weights.
- Performance prediction: Simple baselines such as “number of parameters” and “FLOPs” are surprisingly competitive with leading zero-cost techniques.Zero-cost proxies may be unreliable on larger search spaces and can prefer larger models or wide channels.
- Performance prediction: Zero-cost proxies can act as “weak learners” combined with other proxies or NAS methods, including Bayesian optimization and one-shot NAS.Initial work reports successful integration into both approaches.
- Multi-fidelity methods: Low-fidelity data methods estimate final accuracy by training on subsets or synthetic datasets, using selection strategies such as random sampling, entropy, clustering, facility location, and k-center.Adaptive subset selection updates the subset during training to maximize validation accuracy; generative teaching networks create synthetic data matched to real-data accuracy.
- Multi-fidelity methods: Multi-fidelity methods approximate an expensive objective using cheaper evaluations parameterized by fidelity, such as training epochs or data-subset size.SuccessiveHalving progressively promotes promising architectures to higher fidelities, while BOHB and DEHB combine Hyperband with model-based or evolutionary search.
6. Extensions
NAS extensions address joint architecture and hyperparameter optimization, constrained or multi-objective deployment requirements, and ensembles of diverse networks. These extensions increase search complexity but broaden NAS beyond optimizing a single standalone model for accuracy.
- Joint NAS + HPO: Joint NAS and HPO matters because hyperparameters can strongly affect performance and may vary across architectures within the same search space.The combined search is harder because it expands the heterogeneous search space and complicates modeling architecture–hyperparameter interactions.
- Joint NAS + HPO: Proposed NAS + HPO methods homogenize the search space by representing architecture and hyperparameter choices with categorical or continuous-valued coefficients.These methods use search strategies resembling standard NAS algorithms such as Bayesian optimization.
- Constrained and multi-objective NAS: Constrained NAS adds requirements such as latency, memory usage, or class-specific error rates alongside the original objective.Hardware constraints can be incorporated through penalty functions, although one-shot methods require differentiable penalties and may relax discrete metrics continuously.
- Constrained and multi-objective NAS: Multi-objective NAS treats the original objective and constraints as separate objectives and can search for a Pareto front of non-dominated architectures.A solution is non-dominated when improving one objective requires worsening at least one other objective.
- Neural ensemble search: Neural ensemble search finds sets of diverse architectures whose aggregated predictions can outperform standard NAS in accuracy, uncertainty calibration, and robustness to dataset shift.The ensemble search space has cardinality |A|^M rather than |A|, increasing the optimization burden.
- Neural ensemble search: Ensemble-search methods include random search, regularized evolution, differentiable multi-headed networks, and supernet-based Bayesian sampling.These approaches target the ensemble objective while addressing the computational cost of black-box optimization.
7. Applications
NAS has been applied beyond image classification to graph neural networks, GANs, dense prediction, and transformers. These applications generally reuse standard search strategies while adapting search spaces and evaluation procedures to domain-specific architecture and computational demands.
- Graph neural networks: GNN NAS primarily differs from standard NAS in search-space construction because graph operations and graph data create more complex and costly architecture searches.Published approaches use GNN-specific operations and standard strategies including reinforcement learning, one-shot methods, and evolutionary algorithms.
- Generative adversarial networks: GAN NAS requires special techniques because generators and discriminators train together under brittle dynamics.Researchers search only the generator, use a predefined growing discriminator, or search both networks simultaneously, typically in cell-based spaces with upsampling operations.
- Dense prediction: Dense-prediction NAS uses more complex architectures containing components such as decoders, multi-scale feature modules, and task-specific heads.Many methods retain a pretrained backbone and search task-specific components, while computational demands remain higher than for image classification.
- Transformers: Transformers have been searched with evolutionary and one-shot methods across a wider variety of search spaces than convolutional architectures.Applications include decoder-only, encoder-only, encoder-decoder, vision-transformer, and hybrid CNN–transformer spaces.
8. Benchmarks
Queryable NAS benchmarks address inconsistent search spaces, training pipelines, and high computational costs in early NAS evaluation. The survey catalogs tabular, surrogate, and multi-task benchmark suites while warning that reliance on too few benchmarks can cause overfitting.
- Motivation and benchmark types: Early NAS evaluations often used final CIFAR-10 and ImageNet test accuracy, producing inconsistent pipelines and unnecessary computation.Training final architectures for 600 epochs became common even though accuracy increased only fractionally after 200 epochs.
- Motivation and benchmark types: A NAS benchmark fixes a train-test split, search space, and evaluation pipeline; tabular benchmarks additionally precompute evaluations for every architecture.Surrogate benchmarks provide a model that predicts performance for architectures in the search space.
- Tabular benchmarks: NAS-Bench-101 contains 423 624 cell-based architectures with precomputed CIFAR-10 validation and test accuracies for three seeds.NAS-Bench-1Shot1 extends the space to simulate one-shot algorithms through fixed-node subsets.
- Surrogate benchmarks: Surrogate benchmarks model performance from sampled architecture evaluations, including the DARTS and FBNet search spaces.NAS-Bench-x11 extends this idea by predicting full learning curves.
- Benchmark suites: Benchmark suites broaden coverage across tasks and domains, including seven Taskonomy vision tasks, NLP, speech recognition, and ten diverse NAS-Bench-360 problems.NAS-Bench-360 includes prosthetics control, PDE solving, protein folding, and astronomy imaging.
- Best practices: Queryable benchmarks enable hundreds of trials with different random seeds and facilitate statistically significant comparisons.However, over-reliance on a few benchmarks can make the field overfit them, motivating diverse benchmark sets.
9. Best Practices
Sound NAS research requires reproducible implementations, fair benchmark comparisons, and transparent reporting of tuning and variability. The survey recommends complete releases, controlled evaluations, baselines, ablations, and repeated trials.
- Motivation: Reproducibility and statistically significant, fair comparisons remain important because shortcomings in these areas impede progress in NAS.
- Reproducibility: Reproducibility requires full code, specified random seeds, and release of all training and search components.Released components should include training pipelines, search spaces, hyperparameters, random seeds, and the NAS method.
- Hyperparameters: NAS researchers should report how their method-specific hyperparameters were tuned and generally prefer methods with fewer hyperparameters.Hyperparameters may not transfer well across datasets and search spaces.
- Fair comparisons: NAS methods should be compared using identical benchmarks, because datasets alone do not control search spaces or evaluation pipelines.Without fixed benchmarks, performance differences may reflect the training pipeline rather than the NAS algorithm.
- Baselines: Researchers should compare new NAS methods with simple baselines such as random sampling and random search.Many NAS methods are anytime algorithms and can be stopped at any point within a time budget.
- Analysis and reporting: Ablation studies should identify which moving parts improve performance and how they work together.The survey also recommends multiple experimental trials and reporting the random seeds because NAS results can have high variance.
10. Resources
NAS resources include libraries, surveys, workshops, conferences, and continuously updated literature lists. These resources support method development, practical deployment, taxonomy building, and access to the broader AutoML ecosystem.
- Libraries: NAS libraries provide toolboxes for developing methods and applying NAS to new problem domains.Examples include DeepArchitect, NASLib, and Archai, which provide abstractions separating search spaces and optimizers or unifying NAS algorithms.
- Libraries: Archai targets reproducible rapid prototyping and turnkey NAS use, while PyGlove constructs NAS methods through symbolic programming.
- Survey literature: Earlier surveys introduced core NAS concepts, unified existing methods, or organized the field around historical challenges and their solutions.
- Survey literature: Specialized surveys cover evolutionary NAS, hardware-aware NAS, graph AutoML, dense prediction, and one-shot NAS.
- Survey literature: Broader AutoML and AutoDL surveys also devote sections to NAS, alongside the first open-access book on automated machine learning.
- Community resources: NAS-related workshops have expanded into dedicated events and contributed to the community’s growth toward an AutoML conference.
- Community resources: Continuously updated searchable lists provide access to NAS papers published across machine-learning venues and related resources.
11. Future Directions
Future NAS research should improve the robustness of efficient methods, reduce human bias in search spaces, and jointly optimize architectures with hyperparameters and broader deep-learning pipelines. The survey identifies these directions as promising but technically challenging.
- 11.1 Robustness of Efficient Methods: One-shot methods offer orders-of-magnitude speedups but still face performance issues and lack large-scale, fair comparisons.Applying them to new tasks also requires significant expertise, motivating robust and reliable approaches across datasets and tasks.
- 11.1 Robustness of Efficient Methods: Zero-cost proxies promise orders-of-magnitude speedups, but their effectiveness relative to simple baselines is inconsistent.Future work should clarify when and why these proxies work, particularly alongside existing NAS techniques.
- 11.2 Going Beyond Hand-Crafted, Rigid Search Spaces: Hand-designed search spaces reduce search time but constrain automation for non-experts and limit NAS to domains with strong available spaces.
- 11.2 Going Beyond Hand-Crafted, Rigid Search Spaces: Hierarchical search spaces offer a flexibility–searchability trade-off and reduce human bias through greater diversity than cell-based spaces.They remain relatively under-explored compared with cell-based search spaces.
- 11.2 Going Beyond Hand-Crafted, Rigid Search Spaces: Starting with large, diverse search spaces and iteratively pruning low-performing regions could expose a wider variety of architectures.
- 11.3 Beyond NAS: On popular spaces such as DARTS, optimizing training hyperparameters has sometimes improved performance more than optimizing architecture.The survey presents joint optimization of hyperparameters and architecture as the preferable best-case direction.
- 11.3 Beyond NAS: Jointly varying architectures and hyperparameters can reduce human bias and discover previously unknown combinations that substantially outperform existing methods.This problem is more challenging than NAS or HPO alone but may offer larger improvements.
- 11.3 Beyond NAS: AutoDL extends optimization beyond NAS and HPO to problem formulation, data processing, augmentation, deployment, and continuous monitoring.The survey identifies fully automated deep learning as a broader direction for real-world performance improvements.
A. Additional Figures and Tables
The additional materials summarize NAS benchmarks and illustrate how neural architectures can be encoded. Figure 11 contrasts adjacency-matrix and path-based representations with one-hot or categorical encoding.
- Additional figures: Figure 11 shows a neural architecture alongside adjacency-matrix and path-based representations.Both representations can use one-hot or categorical encoding.
- Additional tables: Table 2 provides an overview of NAS benchmarks.