PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Architecture Search: A Survey

Thomas Elsken, Jan Hendrik Metzen, Frank Hutter

arXiv:1808.05377v3stat.MLcs.LGcs.NE

TL;DR

Manual architecture engineering is time-consuming, motivating automated neural architecture search. The survey organizes existing work by search space, search strategy, and performance estimation, and reviews reported NAS outcomes and constraints.

  • Problem

    Neural architectures are largely designed manually, a time-consuming and error-prone process, despite deep learning’s automated feature engineering.

    Source: §1 · p. 1

  • Method

    The paper surveys NAS work by categorizing methods along search space, search strategy, and performance estimation strategy.

    Source: §1 · p. 1

  • Results

    NAS methods have outperformed manually designed architectures on some image classification, object detection, and semantic segmentation tasks.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors consider it important to extend NAS beyond image classification to less explored domains.

    Source: §5 · p. 12

    One-shot NAS is restricted to subgraphs of its predefined supergraph, and the biases introduced by jointly optimizing architecture sampling with the one-shot model are not well understood.

    Source: §4 · p. 11

Abstract

from arXiv · show

Deep Learning has enabled remarkable progress over the last years on a variety of tasks, such as image recognition, speech recognition, and machine translation. One crucial aspect for this progress are novel neural architectures. Currently employed architectures have mostly been developed manually by human experts, which is a time-consuming and error-prone process. Because of this, there is growing interest in automated neural architecture search methods. We provide an overview of existing work in this field of research and categorize them according to three dimensions: search space, search strategy, and performance estimation strategy.

1. Introduction

Neural Architecture Search (NAS) automates architecture engineering, which has increasingly accompanied deep learning’s automated feature engineering. The paper organizes NAS around the search space, search strategy, and performance estimation strategy.

1. Introduction

  • NAS automates architecture engineering, and the survey groups methods by search space, search strategy, and performance estimation strategy.The search strategy selects an architecture from a predefined space and receives its estimated performance from a performance estimation strategy.Source: §1 · p. 1, Figure 1 · p. 2

• Search Space.

  • The search space defines which architectures are representable; incorporating prior knowledge can simplify search but may bias it against novel building blocks.Source: • Search Space · p. 2

• Search Strategy.

  • Search strategies explore often very large or unbounded spaces while balancing quick discovery of strong architectures against avoiding premature convergence.Source: • Search Strategy · p. 2

• Performance Estimation Strategy.

  • Performance estimation targets predictive performance on unseen data, but training and validating each architecture is computationally expensive and limits how many can be explored.Source: • Performance Estimation Strategy · p. 2

2. Search Space

The section describes search spaces ranging from sequential layers to multi-branch networks and repeated cells. These design choices shape search complexity, transferability, and how much of the architecture remains fixed or manually chosen.

2. Search Space

  • A chain-structured architecture is a sequence of layers whose operation types and operation-specific hyperparameters define the space, with hyperparameters conditionally dependent on operation type.The section notes that fixing the maximum layer count can make an otherwise unbounded space finite, with conditional dimensions.Source: §2 · pp. 2, 3, 5
  • Multi-branch spaces combine previous layer outputs, allowing skip connections; chains, residual networks, and DenseNets are special cases using prior outputs differently.Source: §2 · p. 3
  • Cell-based search optimizes normal and reduction cells, then builds architectures by stacking them in a predefined manner.Repeated building blocks are also described as a useful design principle, as in LSTM and residual blocks.Source: §2 · p. 4
  • Cell-based spaces reduce search size and support transfer: Zoph et al. (2018) estimate a seven-times speed-up over their previous work while achieving better performance.Zoph et al. (2018) also transferred CIFAR-10-optimized cells to ImageNet and achieved state-of-the-art performance.Source: §2 · p. 4
  • Cell-based search leaves a macro-architecture choice—how many cells to use and how to connect them—and fixing it can require manual engineering after cell optimization.The section presents hierarchical search as a step toward optimizing macro-architecture and cell structure jointly.Source: §2 · p. 5
  • Even a single-cell space with fixed macro-architecture remains non-continuous and relatively high-dimensional, while cell-based spaces can be represented as fixed-length vectors.The section gives Zoph et al.’s (2018) cell space as a 40-dimensional categorical example.Source: §2 · p. 5

3. Search Strategy

The survey organizes NAS search strategies around ways to explore architecture spaces, including reinforcement learning, evolution, Bayesian optimization, and gradient-based search. Reported comparisons find RL and evolution perform similarly in final accuracy, while evolution has better anytime performance and finds smaller models.

Search Strategy

  • RL methods can sample architectures with an RNN policy trained by REINFORCE or PPO, while another approach encodes partially trained architectures and applies network-morphism actions.Cai et al. (2018a) train the policy end-to-end with REINFORCE and do not revisit the same architecture state.Source: §3 · p. 6
  • In NAS evolution, populations generate offspring through mutations, with methods varying in parent selection, population removal, and whether learned weights are inherited.Real et al. (2019) remove the oldest individual, whereas Real et al. (2017) remove the worst; Elsken et al. (2019) use Lamarckian inheritance.Source: §3 · p. 7
  • RL and evolution perform equally well in final test accuracy; both beat random search by a small margin, while evolution has better anytime performance and finds smaller models.On CIFAR-10, random search achieved approximately 4% test error versus approximately 3.5% for RL and evolution after model augmentation; without augmentation, the difference was approximately 2%.Source: §3 · p. 7
  • Evolutionary optimization of millions of weights is competitive when gradient estimates are high-variance, such as in reinforcement learning, but supervised learning commonly uses gradient-based weight optimization.Source: §3 · p. 7
  • Bayesian optimization methods use Gaussian-process kernels or tree-based models for architecture spaces, while Monte Carlo tree search and hill climbing offer other discrete-space strategies.Tree-based methods can handle high-dimensional conditional spaces; a full comparison against evolutionary algorithms is lacking.Source: §3 · p. 8
  • Gradient-based NAS can relax operation choices into a convex combination, optimize architecture parameters alongside weights, and select the highest-weight operation to obtain a discrete architecture.Liu et al. (2019b) alternate weight updates on training data with architecture-parameter updates on validation data.Source: §3 · p. 8

4. Performance Estimation Strategy

Performance estimation guides architecture search but training each candidate from scratch can require thousands of GPU days. The survey reviews lower-fidelity estimates, learning-curve and surrogate predictions, network morphisms, and one-shot weight sharing, alongside their biases and search-space constraints.

4. Performance Estimation Strategy

  • Training every architecture from scratch can require thousands of GPU days, motivating methods that reduce the cost of estimating performance.Source: §4 · p. 8
  • Lower-fidelity evaluations and learning-curve or surrogate predictions can speed estimation, but proxies may bias performance estimates and even change architecture rankings when their fidelity gap is large.Gradually increasing fidelity is proposed to address ranking instability; the broad challenge is making good predictions from relatively few evaluations.Source: §4 · p. 9
  • Network morphisms transfer weights while preserving a network’s function, enabling continued training after architecture changes and reducing estimation to a few GPU days.Strict morphisms can only enlarge architectures, potentially making them overly complex; approximate morphisms can also shrink them.Source: §4 · p. 10
  • One-shot search trains a supergraph whose subgraphs inherit shared weights, allowing candidate architectures to be evaluated without separate training in a few GPU days.This estimator can severely underestimate the best architectures’ actual performance, and whether its rankings correlate strongly with true performance remains unclear.Source: §4 · p. 10
  • One-shot methods differ in how they train or sample the supergraph: ENAS uses a controller, DARTS relaxes candidate operations, SNAS learns their distribution, and ProxylessNAS binarizes architectural weights.Bender et al. (2018) use stochastic path dropout while training the one-shot model once.Source: §4 · p. 10
  • One-shot search is constrained to subgraphs of its predefined supergraph, and methods requiring the whole supergraph in GPU memory are limited to relatively small spaces.The biases from jointly optimizing the architecture-sampling distribution and one-shot model are not well understood and may reinforce early search imbalance.Source: §4 · p. 11

5. Future Directions

The survey identifies future directions for NAS: broaden its application domains, make search spaces more flexible, improve fair comparison, and better understand why architectures work.

5. Future Directions

  • NAS has mostly targeted image classification, and the survey calls for work in less-explored domains including restoration, segmentation, translation, reinforcement learning, and language or music modeling.It also names generative adversarial networks and sensor fusion as promising areas.Source: §5 · p. 12
  • Multi-task and multi-objective NAS are promising directions, with resource efficiency considered alongside predictive performance in multi-objective search.The survey relates multi-objective NAS to network compression because both seek architectures that perform well while remaining efficient.Source: §5 · p. 12
  • Conditioning RL or bandit search policies on task properties and resource requirements could support task-specific architecture choices; NAS for adversarial robustness is another direction.The survey also cites one-shot NAS work generating different architectures depending on task or instance.Source: §5 · p. 12
  • Cell-based search spaces transfer across image-classification tasks but rely on human experience and do not generalize easily to domains where repeated chain-like cells do not fit.The survey points to more general hierarchical search spaces as a possible route to broader applicability.Source: §5 · pp. 12–13
  • NAS for reinforcement learning remains insufficiently studied despite work using a versatile policy-network search space with convolutional and recurrent building blocks.The survey describes this as a full study of NAS for RL still being missing.Source: §5 · p. 12
  • NAS comparisons and reproducibility are complicated because performance measurements also depend on search space, budget, augmentation, training, and regularization choices.The survey argues for common benchmarks and describes one precomputed benchmark with roughly 423,000 architectures and measurements across training budgets and runs.Source: §5 · p. 13
  • NAS offers little insight into why particular architectures work or whether independent runs find similar ones; the survey calls for identifying motifs and testing their importance and generality.Source: §5 · p. 13

Related papers

Loading 1808.05377v3…