Neural Architecture Search: Insights from 1000 Papers

1. Introduction

• NAS automates architecture design, a crucial factor in deep learning’s breakthroughs, and has outpaced human-designed architectures across many tasks and constrained settings.Research has accelerated rapidly, with over 1000 new NAS papers released in the last two years.Source: §1 · p. 2

1.1 A Brief History of NAS and Relation to Other Fields

• NAS emerged within AutoML and differs from typical HPO by optimizing complex, usually discrete neural-network topologies represented as graphs or conditional hyperparameters.NAS-specific methods can exploit architectural structure, including weight sharing among similar architectures, rather than treating the problem as generic black-box optimization.Source: §1.1 · pp. 2–3
• NAS research now spans reinforcement learning, evolutionary algorithms, Bayesian optimization, one-shot methods, reproducible benchmarks, and applications beyond image classification.One-shot methods often speed search substantially, while NAS-Bench-101 (Ying et al., 2019) supported more reproducible research and later work expanded into detection, segmentation, speech, scientific computing, protein folding, weather prediction, and NLP.Source: §1.1 · p. 3

1.2 Background and Definitions

• NAS is framed through search spaces, search strategies, and performance estimation, although one-shot methods couple the latter two components.The basic objective is to find, within a computation budget, an architecture in search space A with the best validation performance after training, with variants allowing constraints or meta-learning.Source: §1.2 · p. 4
• The survey covers NAS techniques in Sections 2–5 and extensions, applications, resources, and future directions in Sections 6–11.Its technique coverage includes search spaces, black-box optimization, one-shot methods, and speedup techniques.Source: §1.2 · p. 5

2. Search Spaces

• Search spaces balance human bias against search efficiency: smaller hand-designed spaces search faster, whereas larger primitive-based spaces may discover more novel architectures (Real et al., 2020).Search-space design is often the first NAS setup decision and is commonly inspired by task-specific manual architectures.Source: §2 · p. 5

2.1 Terminology

• NAS terminology distinguishes primitives, layers, blocks or modules, cells, and motifs according to their roles and structural granularity.A primitive is typically a fixed activation, operation choice, and normalization; a cell is a directed acyclic graph of operations.Source: §2.1 · pp. 5–6

2.2 Macro Search Spaces

• Macro search spaces encode entire architectures or vary macro hyperparameters, offering high representation power and novelty potential but requiring slow searches.They may search DAG topology and operations, or vary depth, width, resolution, and downsampling while keeping topology and operations fixed.Source: §2.2 · pp. 6–7

2.3 Chain-Structured Search Spaces

• Chain-structured spaces use sequential operation layers, are simple to design and search, and often find strong architectures quickly, but limit architectural novelty.They search choices such as convolution kernels, dilations, expansion ratios, transformer dimensions, layer counts, and attention heads over established backbones.Source: §2.3 · p. 7

2.4 Cell-based Search Spaces

• Cell-based spaces search small reusable cells within a fixed outer skeleton, reducing complexity while often producing high-performing and transferable architectures.Their popularity is tempered by low performance variance, ad-hoc fixed choices, and reduced expressiveness for discovering highly novel architectures.Source: §2.4 · pp. 7, 9

2.5 Hierarchical Search Spaces

• Hierarchical spaces design motifs across multiple levels, from two-level macro-plus-cell or chain designs to recursive hierarchies with flexible depth and topology.Higher-level motifs are often DAGs of lower-level motifs, enabling progressively structured architectural construction.Source: §2.5 · pp. 9–10

2.6 Architecture Encodings

• Architecture encodings compactly represent architectures for mutation, similarity assessment, and performance prediction, but the best encoding depends on the NAS subroutine.DAG, adjacency-matrix, graph-, path-, and learned encodings are used; even small encoding changes can substantially affect NAS performance.Source: §2.6 · pp. 10–11

3. Black-Box Optimization Techniques

• Black-box optimization is widely used because it is robust, conceptually simple, parallelizable, and adaptable, but training architectures independently generally requires more computation than one-shot methods.Its advantages include avoiding catastrophic failure modes, supporting joint hyperparameter optimization, and simplifying optimization of non-differentiable objectives.Source: §3 · p. 11

3.1 Baselines

• Random search is a recommended NAS baseline that can perform surprisingly well in engineered spaces, while weight sharing, zero-cost proxies, and learning-curve extrapolation can improve competitiveness.With k evaluations, random search finds architectures in the top 100/k% of the search space in expectation; local search is also strong across small and large spaces.Source: §3.1 · pp. 11–12

3.2 Reinforcement Learning

• RL controllers generate architectures sequentially and update their policy from validation accuracy, but RL has been outperformed in head-to-head comparisons by evolution and Bayesian optimization.Controllers are commonly RNNs trained with REINFORCE or proximal policy optimization; ENAS accelerates reward estimation through weight sharing.Source: §3.2 · p. 12

3.3 Evolutionary and Genetic Algorithms

• Evolutionary NAS iteratively selects, mutates, trains, and replaces population members; regularized evolution outperformed random search and RL and achieved state-of-the-art ImageNet performance at release.Parent selection balances exploration and exploitation, while regularized evolution removes the oldest population member regardless of performance.Source: §3.3 · p. 13

3.4 Bayesian Optimization

• Bayesian optimization models NAS performance with a probabilistic surrogate and acquisition function, adapting representations through encodings, latent spaces, or graph-based models for irregular search spaces.Acquisition functions balance exploration and exploitation, but their optimization is difficult in non-continuous spaces and commonly uses mutations of promising architectures.Source: §3.4 · pp. 14–15

3.5 Monte Carlo Tree Search

• Monte Carlo tree search balances exploration and exploitation by sampling decisions, rolling out rewards, and backpropagating updates, while NAS adaptations support modular spaces, neural accuracy estimates, pruning, and multi-objective search.MCTS was adapted from decision-making applications such as AlphaGo and AlphaZero to architecture and hyperparameter search.Source: §3.5 · p. 15

4. One-Shot Techniques

• One-shot methods implicitly train all architectures through a single hypernetwork or supernetwork training, avoiding the immense cost of independently training thousands of candidates.Supernets contain all search-space architectures as subnetworks, while hypernetworks generate weights for other networks.Source: §4 · pp. 15–16

4.1 Non-Differentiable Supernet-Based Methods

• Supernets enable exponential candidate coverage at linear training cost by evaluating subnetworks with inherited weights, then applying black-box, reinforcement-learning, or gradient-based search.The taxonomy includes non-differentiable optimization, differentiable optimization such as DARTS, and hypernetworks.Source: §4 · p. 16, Figure 6 · p. 17

4.2.1 Rank Disorder

• One-shot ranking can suffer rank disorder because supernet and independently trained architectures may disagree; its severity depends on the search space, training technique, and dataset.Progressive depth growth, operation pruning, and certainty or stability-based selection are used to reduce the problem.Source: §4 · p. 16, §4.2.1 · p. 20

4.1 Non-Differentiable Supernet-Based Methods

• Non-differentiable methods either search after supernet training or search during training; random architecture sampling reduces memory, while ENAS uses a controller and REINFORCE to select subnetworks.The selected architecture is ultimately trained from scratch in the described black-box approaches.Source: §4.1 · pp. 17–18

4.2 Differentiable Supernet-Based Methods

• DARTS relaxes discrete operation choices continuously and jointly optimizes architecture hyperparameters and weights by alternating gradient descent, then discretizes and retrains the resulting architecture.This finds a high-performing local optimum faster than black-box optimization, although its gradient approximation is not guaranteed to converge.Source: §4.2 · pp. 18–19

4.2.2 Operation Biases

• Differentiable NAS often favors skip connections, prompting remedies including early stopping, separate skip weights, independent operation weights, removal-based selection, and less hand-biased search spaces.The reported cause is possible supernet over-compensation for vanishing gradients.Source: §4.2.2 · p. 20

4.2.3 Poor Test Generalization

• DARTS generalization problems arise from sharp local minima and discretization; proposed fixes include robust training, neighborhood objectives, single-level or reparameterized optimization, and more accurate architecture gradients.These approaches are described as improving stability or generalization.Source: §4.2.3 · p. 21

4.2.4 High Memory Consumption

• Memory-efficient differentiable methods mask or sample operations or channels during training, including ProxylessNAS, Gumbel-softmax methods, PC-DARTS, and DrNAS.These techniques reduce the memory burden that otherwise scales linearly with the number of candidate operations.Source: §4.2.4 · p. 21

5.1 Performance Prediction

• Performance prediction avoids fully training every candidate by estimating accuracy or relative accuracy with faster functions that ideally preserve performance rankings.These predictors can accelerate black-box optimization and one-shot NAS, and include learning-curve extrapolation, zero-cost proxies, and other methods.Source: §5.1 · p. 22

5.1.1 Learning Curve Extrapolation

• Learning-curve extrapolation predicts final architecture performance from partial training, using parametric fits or surrogate models that combine architecture encodings with observed learning curves.It supports black-box NAS and multi-fidelity methods such as Hyperband and BOHB.Source: §5.1.1 · pp. 22–23

5.1.2 Zero-Cost Proxies

• Zero-cost proxies score architectures from a single minibatch or initialized weights, often in under 5 seconds, but simple parameter-count and FLOPs baselines can be surprisingly competitive.These proxies may be unreliable on larger search spaces and biased toward larger models or wider channels, though they can serve as weak learners combined with other techniques.Source: §5.1.2 · pp. 23–24

5.1.3 Other Low-Fidelity Predictions

• Other low-fidelity predictors reduce cost by training on data subsets or synthetic datasets, using selection strategies, generative teaching networks, or synthetic petri dishes to approximate full-training rankings.Adaptive subset selection updates the subset during training to maximize validation accuracy, while synthetic petri dish explicitly optimizes correlation between approximate and full-training rankings.Source: §5.1.3 · p. 24

5.2 Multi-Fidelity Algorithms

• Multi-fidelity algorithms evaluate many architectures cheaply at lower budgets, progressively promoting promising candidates to the highest fidelity through methods such as SuccessiveHalving and Hyperband.BOHB, DEHB, ASHA, ABOHB, and EcoNAS combine or extend this strategy with Bayesian optimization, evolution, asynchronous computation, early stopping, or hierarchical allocation.Source: §5.2 · p. 25

5.3 Meta-Learning

• Meta-learning accelerates NAS by reusing information from related datasets or tasks instead of searching from scratch, including learned dataset and architecture encodings for surrogate performance prediction.Some approaches meta-learn architecture parameters with MAML, REPTILE, and DARTS, while others adapt probabilistic surrogates to test-time datapoints.Source: §5.3 · pp. 25–26

5.4 Weight Inheritance and Network Morphisms

• Weight inheritance reuses unaffected layers from trained parent architectures in offspring candidates, providing an intermediate strategy between training every architecture independently and sharing one weight set.It is particularly useful for black-box optimization with small sequential architecture changes.Source: §5.4 · p. 26
• Network morphisms modify architectures while preserving their represented function through appropriate initialization, enabling weight reuse across evolutionary algorithms, reinforcement learning, Bayesian optimization, and one-shot methods.For any input, the original and morphed architectures produce identical outputs.Source: §5.4 · p. 26

6.1 Joint NAS + HPO

• Joint NAS + HPO is motivated by hyperparameters’ substantial performance impact and architecture-dependent optima, but their interaction and budget-dependent effects make the combined search harder than either problem alone.Proposed methods homogenize NAS and HPO spaces and use Bayesian optimization, evolution, or REINFORCE with weight sharing.Source: §6.1 · pp. 26–27

6.2 Constrained and Multi-Objective NAS

• Constrained and multi-objective NAS extends accuracy optimization to requirements such as latency, memory, fairness, and hardware efficiency, using constrained or multi-objective formulations.Constraints can be incorporated through penalty functions, while multi-objective methods often search for non-dominated architectures on a Pareto front.Source: §6.2 · pp. 27–28
• Hardware-aware NAS often requires surrogate or modular cost estimates because parameter counts may poorly reflect metrics such as memory and latency.Differentiable one-shot methods may relax discrete hardware metrics, for example with Gumbel softmax functions.Source: §6.2 · pp. 27–28

6.3 Neural Ensemble Search

• Neural ensemble search seeks diverse architecture sets that can outperform standard NAS in accuracy, uncertainty calibration, and robustness to dataset shift.The ensemble search space has cardinality |A|^M, and methods include random search, regularized evolution, differentiable multi-headed models, Bayesian sampling, and supernet-based evolutionary search.Source: §6.3 · pp. 28–29

7. Applications

• NAS applications extend beyond image classification to new datasets and tasks, with documented successes in graph neural networks, generative adversarial networks, dense prediction, and transformers.The section surveys these application areas as settings where NAS designs task-specific architectures rather than focusing only on established image-classification benchmarks.Source: §7 · p. 29

7.1 Graph Neural Networks

• GNN NAS primarily differs from standard NAS through its more complex search spaces, while using reinforcement learning, one-shot methods, or evolutionary algorithms as search strategies.GNN-specific spaces incorporate operations and features such as meta-paths, edge features, and fast sampling, addressing the substantial computational overhead of both GNNs and NAS.Source: §7.1 · p. 29

7.2 Generative Adversarial Network

• GAN NAS accommodates brittle two-network training by searching only the generator, using a predefined discriminator, or jointly searching generator and discriminator architectures.Cell-based spaces commonly augment convolutional generator cells with upsampling operations, and candidate scoring uses Inception Score or Fréchet Inception Distance.Source: §7.2 · pp. 29–30

7.3 Dense Prediction Tasks

• Dense prediction NAS searches decoders, multi-scale modules, task-specific heads, or complete architectures, often using pretrained, cached, reduced, or staged backbones to control computational cost.Applications include semantic segmentation, object detection, optical flow, and disparity estimation; efficient one-shot methods have accelerated progress, while disparity, panoptic, 3D, and optical-flow search remain open directions.Source: §7.3 · p. 30

7.4 Transformers

• Transformers lack a single best NAS search space, with evolutionary and one-shot methods spanning decoder-only, encoder-only, encoder-decoder, and vision architectures.Transformer NAS explores diverse designs, including evolutionary decoder search, compressed BERT supernets, resource-efficient sequence-to-sequence search, and one-shot vision-transformer methods.Source: §7.4 · pp. 31–32

Benchmark motivation and definitions

• Queryable benchmarks reduce computational costs and enable fair, statistically significant NAS comparisons by standardizing evaluation and querying performance instead of training architectures from scratch.A benchmark fixes the train-test split, search space, and evaluation pipeline; tabular benchmarks precompute all architecture evaluations, while surrogate benchmarks predict them.Source: §8 · p. 32

Tabular benchmarks

• NAS-Bench-101, NAS-Bench-201, NATS-Bench, and HW-NAS-Bench-201 provide tabular evaluations spanning cell and macro search spaces, multiple datasets, and hardware costs.NAS-Bench-101 covers 423 624 architectures, while NAS-Bench-201 covers 6466 unique architectures across CIFAR-10, CIFAR-100, and ImageNet-16-120, each with three seeds.Source: §8 · p. 32

Surrogate benchmarks

• Surrogate benchmarks extend queryable NAS to DARTS and FBNet spaces, while NAS-Bench-x11 predicts full learning curves for multi-fidelity simulations.Surr-NAS-Bench-DARTS was trained from 60 000 DARTS architectures on CIFAR-10 before fitting a surrogate model.Source: §8 · p. 32

Cross-domain benchmark suites

• Benchmark coverage has expanded from computer vision to NLP, automatic speech recognition, diverse scientific and control problems, and unified suites spanning 28 tasks.NAS-Bench-360 covers ten diverse problems, while NAS-Bench-Suite combines the majority of existing queryable benchmarks and NAS-Bench-Suite-Zero adds precomputed zero-cost proxy values.Source: §8 · pp. 32–33

Limitations and best practices

• Although queryable benchmarks support hundreds of trials across random seeds and statistically significant comparisons, relying on only a few can cause over-fitting and hinder novel methods.Researchers should therefore use a large, diverse set of NAS benchmarks whenever possible.Source: §8 · p. 33

9. Best Practices

• NAS researchers should follow a reproducibility checklist covering sound research and fair, statistically significant comparisons (Li and Talwalkar, 2019; Lindauer and Hutter, 2020; Yang et al., 2020).The checklist is also recommended for inclusion in paper appendices.Source: §9 · p. 33

9.1 Releasing Code and Important Details

• Release full, easy-to-use NAS code with training pipelines, search space, hyperparameters, random seeds, and the NAS method, while documenting hyperparameter tuning.Search and final-evaluation pipelines may differ; for evaluations across datasets, tune hyperparameters on one dataset and fix them elsewhere because transfer can be poor (Mehta et al., 2022). DARTS code released by Liu et al. (2019c) enabled numerous follow-up works.Source: §9.1 · pp. 33–34

9.2 Comparing NAS Methods

• Compare NAS methods on identical benchmarks, include random sampling and random search baselines, evaluate anytime performance over time, and report ablations and multiple seeded trials.Fixed datasets, search spaces, train-test splits, and evaluation pipelines isolate algorithmic differences; repeated trials help verify statistically significant comparisons amid high variance.Source: §9.2 · p. 34

10.1 Libraries

• NAS libraries support method development, reproducibility, deployment, and application across problem domains through unified abstractions and specialized programming tools.DeepArchitect separates search spaces from optimizers; NASLib unifies one-shot and BBO algorithms and benchmark search spaces; Archai supports reproducible prototyping and turnkey NAS; PyGlove uses symbolic programming for mutable machine-learning programs.Source: §10.1 · pp. 34–35

10.2 Other NAS Survey Papers

• Earlier surveys introduced foundational NAS concepts, unified the research landscape, and organized historical challenges and their solutions.Elsken et al. (2019b) presented the three pillars of search space, search strategy, and performance evaluation; Wistuba et al. (2019) categorized NAS methods; Ren et al. (2020) focused on historical challenges and remedies.Source: §10.2 · p. 35
• Specialized and broader surveys cover evolutionary NAS, hardware-aware NAS, graph AutoML, dense prediction, one-shot NAS, AutoML, and AutoDL.The broader AutoML and AutoDL literature includes sections on NAS, and Hutter et al. (2019) released the first open-access book on automated machine learning.Source: §10.2 · p. 35

10.3 Additional Resources

• NAS-related workshops, the first AutoML conference, and updated online literature repositories provide venues and discovery resources for the growing community.NAS-focused workshops were held at ICLR (2020, 2021) and CVPR (2021), while automl.org and Awesome-AutoDL maintain continuously updated NAS paper and resource lists.Source: §10.3 · p. 35

11. Future Directions

• NAS has improved efficiency and achieved state-of-the-art architectures on many tasks, but substantial unsolved problems remain.Tools now enable comparison of NAS algorithms without GPUs, alongside diverse techniques and search spaces.Source: §11 · p. 35

11.1 Robustness of Efficient Methods

• One-shot methods need fair large-scale comparisons and approaches that work robustly and reliably across new datasets and tasks.They provide orders-of-magnitude speedups but still face performance issues and require substantial expertise when applied to new tasks.Source: §11.1 · p. 36
• Zero-cost proxies require better understanding of when and why they work, despite inconsistent results against simple baselines (Ning et al., 2021) and potential when combined with NAS techniques (Xiang et al., 2021).Other work argues that zero-cost proxies have untapped potential.Source: §11.1 · p. 36

11.2 Going Beyond Hand-Crafted, Rigid Search Spaces

• Human-designed search spaces limit automation and domain coverage, motivating broader use of hierarchical and automatically optimized spaces.Hierarchical spaces offer a flexibility–searchability trade-off and reduce human bias, while iterative pruning can consider more architectures (Ru et al., 2020b; Guo et al., 2020a; Radosavovic et al., 2020).Source: §11.2 · p. 36

11.3 Fully Automated Deep Learning

• NAS should jointly optimize architectures and hyperparameters because hyperparameter optimization can outperform architecture optimization on some search spaces (Yang et al., 2020; Zela et al., 2020b).NAS + HPO can reduce human bias and discover previously unknown combinations, although it is more challenging than either problem alone.Source: §11.3 · pp. 36–37
• Fully automated deep learning should optimize the entire pipeline, including problem formulation, data processing, augmentation, deployment, and continuous monitoring (Dong et al., 2021a).AutoDL could substantially improve performance on real-world problems as NAS matures.Source: §11.3 · p. 37

• Figure 9 visualizes search-space terminology, while Figure 10 presents chain-structured and macro search spaces.Source: A. Additional Figures and Tables · p. 38
• Figure 11 shows that neural architectures can use adjacency-matrix or path-based representations with one-hot or categorical encoding.Source: Figure 11 · p. 39
• Table 2 provides an overview of NAS benchmarks.Source: Table 2 · p. 39