Source-linked AI summary
A Survey on Neural Architecture Search
Martin Wistuba, Ambrish Rawat, Tejaswini Pedapati
TL;DR
Deep learning architecture selection is important but computationally intensive and dependent on substantial domain knowledge, motivating automated neural architecture search. The survey unifies and categorizes search spaces and optimization methods, then examines broader objectives and related automation directions. It concludes that search-space design and experimental comparability remain central to interpreting architecture-search results.
Problem
Selecting effective neural architectures is critical but time-consuming, computationally intensive, and demanding of substantial domain knowledge.
Method
The survey provides a formalism covering architecture search spaces, reinforcement learning, evolutionary algorithms, surrogate models, one-shot search, and related automation methods.
Results
The survey finds that architecture-search outcomes depend strongly on search-space design and that comparisons may be confounded when architectures use unequal training conditions.
Takeaways & Limitations
Neural architecture search now extends beyond architecture selection to constrained and multi-objective optimization, model compression, data augmentation, and other deep-learning components.
Takeaways & Limitations
The practical usefulness and credibility of neural architecture search remain open questions, while task-dependent initial-architecture choices lack clear guidelines.
Abstract
from arXiv · showhide
The growing interest in both the automation of machine learning and deep learning has inevitably led to the development of a wide variety of automated methods for neural architecture search. The choice of the network architecture has proven to be critical, and many advances in deep learning spring from its immediate improvements. However, deep learning techniques are computationally intensive and their application requires a high level of domain knowledge. Therefore, even partial automation of this process helps to make deep learning more accessible to both researchers and practitioners. With this survey, we provide a formalism which unifies and categorizes the landscape of existing methods along with a detailed analysis that compares and contrasts the different approaches. We achieve this via a comprehensive discussion of the commonly adopted architecture search spaces and architecture optimization algorithms based on principles of reinforcement learning and evolutionary algorithms along with approaches that incorporate surrogate and one-shot models. Additionally, we address the new research directions which include constrained and multi-objective architecture search as well as automated data augmentation, optimizer and activation function search.
1. Introduction
Neural architecture search aims to automate the time-consuming and error-prone process of selecting network components and architectures. This survey unifies existing methods through a common formalism covering search spaces, optimization procedures, constraints, and related automation tasks.
- Architecture modifications can significantly improve deep-learning performance, but finding suitable architectures is time-consuming, arduous, and error-prone.
- Neural architecture search formulates automation as optimization over decisions defining network components, feasible solutions, and an optimizer.
- Search-space design affects search speed, search duration, and solution quality, progressing from chain-structured to branched architectures.
- The field increasingly addresses multiple objectives, including limiting model parameters for efficient mobile deployment.
- The survey organizes neural architecture search around architecture spaces, formal problem definitions, and reinforcement-learning, evolutionary, surrogate-model, and one-shot optimization methods.
- The survey also examines constraints, multiple objectives, model compression, transfer learning, early termination, and applications to related deep-learning components.
2. Neural Architecture Search Space
Neural architecture search spaces define feasible network designs by restricting operations, connections, and structural templates. They range from flexible global graphs to repeated cell-based structures, with cell-based spaces offering transferability and scalable complexity but global spaces retaining potential when initialized appropriately.
- Search-space formalism: A neural architecture search space restricts the general set of neural architectures by limiting operations and imposing structural constraints.An architecture is completely specified by each node’s parent set and operation, represented as α.
- Global search spaces: Global search spaces describe entire architectures and can allow substantial freedom in arranging operations, while templates constrain segments, connections, or operation choices.Examples include chain structures, skip connections, and segmented templates.
- Search-space variants: Global and cell-based spaces encode different design freedoms, including hierarchical segments, skip connections, reduction cells, and arbitrary or structured cell graphs.Tan et al. use repeated parameterized segments; NASNet uses normal and reduction cells; other spaces vary cell connectivity and meta-operation arrangements.
- Cell-based search spaces: Cell-based search spaces construct networks by repeating smaller directed acyclic graphs called cells within a prespecified architecture template.Cell topology is typically maintained across the network, while hyperparameters such as filter counts may vary between cells.
- Global vs. cell-based search spaces: Cell-based designs support flexible scaling and transfer across datasets, whereas global architectures are generally harder to transfer to different input shapes or to deepen.The survey notes that cell-based complexity can be varied by changing the number of filters and cells.
- Global vs. cell-based search spaces: Appropriate initial architectures can yield global-space models comparable in performance to cell-based discoveries, but useful initialization guidelines remain unclear and may depend on the task.The survey also notes that no detailed comparative study of the different search spaces had yet been conducted.
3. Optimization Methods
The survey formalizes neural architecture search as validation-based optimization over architectures and organizes methods into reinforcement learning, evolutionary, surrogate-model, and one-shot approaches. It also highlights efficiency strategies, comparison limitations, and unresolved questions about the sources of performance gains.
- Formal definition: Neural architecture search finds an architecture α* that maximizes an objective on a validation partition after training the corresponding model on training data.The architecture can encode topology, optimization choices, regularization, and other training hyperparameters.
- Reinforcement learning: Reinforcement-learning methods treat architecture construction as sequential decisions whose rewards reflect long-term performance, often measured by accuracy.The search algorithm acts as an agent that modifies the architecture state to maximize return.
- Evolutionary algorithms: Evolutionary methods maintain populations of architectures, generate new candidates through mutation or recombination, evaluate fitness, and repeat selection until termination.Recombination is often omitted because high-fitness parents do not reliably produce high-fitness offspring; genetic algorithms commonly use fixed-length encodings.
- Efficiency and comparison: Function-preserving transformations and weight sharing can shorten search duration, although their benefits for recurrent neural networks remain questionable.The survey reports that evolutionary methods ignoring GPU budgets used at least seventeen GPU days, while efficiency-focused methods reported comparable results within a day.
- One-shot architecture search: One-shot search trains a single over-parameterized network and derives candidate architectures from it, reducing search cost through shared weights.Differentiable structural parameters can select the final architecture, but methods that represent the entire search space may require all parameters to remain in memory.
- Limitations and open questions: Comparisons across architecture optimizers are difficult because studies differ in search spaces, search duration, data augmentation, and baseline quality.The survey emphasizes random search as a strong baseline and notes that the sources of reported performance gains remain unclear.
4. Early Termination of Training Processes
Early termination reduces neural architecture search cost by predicting whether a partially trained candidate can outperform the current best architecture. Approaches range from learning-curve models based on predefined function families to models using architecture, hyperparameter, and curve features.
- Learning-curve prediction estimates whether a candidate’s premature training trajectory will exceed a target accuracy, enabling unlikely candidates to be terminated early.These methods use the probability that the current training job will produce a higher-accuracy network.
- Hyperband trains candidate models in parallel and progressively allocates more iterations while stopping many runs using premature validation performance.
- Domhan et al. model learning curves with weighted combinations of eleven increasing and saturating function families, but require relatively long premature curves.
- Klein et al. extend this approach across learning curves with a Bayesian neural network trained on mature learning curves.
- Chandrashekaran and Lane avoid predefined curve families by fitting premature curves to mature observed curves, addressing schedules that violate monotonic or saturation assumptions.
- Baker et al. augment curve information with derivatives, architecture properties, and hyperparameters, finding ν-Support Vector Regression strongest among tested alternatives.
5. Transfer Learning
Transfer learning reuses information across related datasets or tasks to make neural architecture search more efficient. Surveyed methods transfer architectures directly, use dataset representations and surrogate models, or decide when reuse is appropriate.
- Architectures can transfer across datasets with matching image dimensions and from CIFAR-10 to ImageNet with few changes.
- Searching with few cell repetitions and filters, then scaling them up, is a common way to reduce search cost before constructing a larger final model.
- Transfer-based methods use dataset meta-features to describe task-specific properties and improve the efficiency of training sampled architectures.
- Wong et al. add task-specific embeddings to a reinforcement-learning controller that generates architecture fine-tuning decisions.
- Istrate et al. use an inexpensive CNN landmarker to identify similar datasets and predict architecture performance on a new dataset.
- Kokiopoulou et al. predict validation accuracy from learned raw-data meta-features and architecture choices for text classification tasks.
- Xue et al. reuse prior solutions for similar incoming datasets and launch a new search when a dataset is fundamentally different.
6. Constraints and Multiple Objectives
Neural architecture search can incorporate deployment constraints and multiple objectives such as accuracy, parameter count, FLOPs, and latency. The survey covers penalty methods, Pareto-based evolutionary search, surrogate models, and model-compression approaches.
- 6.1 Constrained Optimization: Constraints such as inference time or memory requirements convert architecture search into a constrained optimization problem.
- 6.1 Constrained Optimization: Penalty methods transform constrained optimization into an unconstrained problem by punishing constraint violations.
- 6.1 Constrained Optimization: Reinforcement-learning approaches use tunable penalties or hard zero rewards to enforce constraints while optimizing architecture accuracy.
- 6.2 Multi-Objective Optimization: Multi-objective search seeks Pareto-optimal architectures because improving one objective may worsen another, leaving users to select from the Pareto front.
- 6.2 Multi-Objective Optimization: Decomposition methods aggregate multiple objectives with functions such as weighted sums, but fixed weights generally do not recover the entire Pareto front.
- 6.2 Multi-Objective Optimization: NSGA-II and evolutionary methods maintain diverse, non-dominated candidate sets for multi-objective architecture search.
- 6.2 Multi-Objective Optimization: Surrogate models can select feasible, non-dominated candidates and replace non-differentiable objectives such as latency with differentiable predictions.
- 6.3 Model Compression: Model-compression methods search for pruning policies that satisfy parameter budgets while preserving substantial accuracy.
7. Other Related Methods
Neural architecture search methods have been extended to automate activation functions, optimizers, and data augmentation, with experiments examining their effectiveness against established strategies.
- Optimizer search: Optimizer search represents parameter-update rules as graph structures composed of blocks with operands, unary functions, and binary functions.This extends architecture-search methodologies to automate a training component.
- Activation function search: Activation-function search adapts a controller-based architecture-search setup by using one operand and operations such as powers, sigmoids, and sine.The controller is trained with Proximal Policy Optimization.
- Activation function search: The discovered activation function was more effective than commonly used activation functions, although it closely resembles the previously proposed swish activation.The authors report that it has been used by many other authors since.
- Data augmentation search: AutoAugment searches policies composed of sub-policies containing augmentation operations, while each image receives a randomly selected sub-policy.The described operations include rotation, shearing, brightness, contrast, sharpness, horizontal flipping, random crops, and cutout.
- Data augmentation search: Cutout combined with mixup often outperformed either method alone and provided a useful baseline that AutoAugment could not outperform.The combination achieved 1.33% error on CIFAR-10 object recognition; adding mixup to AutoAugment usually did not hurt performance.
8. Outlook and Future Applications of Neural Architecture Search Methods
Neural architecture search is expanding beyond CNNs and object recognition toward broader automated deep learning, but joint workflow optimization and the field’s practical significance remain unresolved.
- Scope and expansion: Neural architecture search currently addresses only one component of the broader deep-learning pipeline.Many approaches focus on CNNs for object recognition and RNN cells for language modeling.
- Scope and expansion: Recent work extends search to constrained and multi-objective settings, model compression, transfer across datasets, data augmentation, optimizers, activation functions, and more complex vision tasks.Other searched network types include autoencoders, transformers, and graph CNNs.
- Open challenges: The remaining challenge is jointly optimizing all configuration parameters of the deep-learning workflow, including augmentation, search-space hyperparameters, optimizers, architectures, and compression.Existing approaches generally treat individual components independently, with some still manually configured.
- Open challenges: The survey questions whether search algorithms discover genuinely innovative architectures or mainly benefit from the design and properties of established search spaces.The authors also note that dedicated efforts toward understanding model spaces or deep-learning interpretability have been lacking.
- Open challenges: General automation of deep learning remains in its infancy, leaving many concerns about practical relevance and usefulness unanswered.The paper identifies this as an exciting domain for future work.