Source-linked AI summary
A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, Xin Wang
TL;DR
Neural architecture design depends on human expertise, motivating NAS as an automated alternative. This survey reframes NAS research around early challenges and corresponding solutions, then compares methods and discusses future directions. It finds that modular search, incomplete training, and continuous gradient-based strategies are widely used to reduce search complexity, duration, and cost, while performance gains over manually designed networks remain limited.
Problem
Neural architecture design relies heavily on researchers’ prior knowledge, while existing surveys make it difficult to follow NAS research challenges and connections.
Method
The survey analyzes NAS from the perspective of early algorithmic characteristics, their challenges, subsequent solutions, performance comparisons, and future research directions.
Results
Modular search, incomplete training, and continuous gradient-based strategies are widely used to reduce search complexity, search duration, and search cost, while NAS performance gains over manually designed networks are limited.
Takeaways & Limitations
Understanding which NAS design choices produce performance gains remains critical, and replacing manually designed architectures requires further research and stronger theoretical foundations.
Takeaways & Limitations
NAS comparisons are difficult because methods differ in baselines, preprocessing, hyperparameters, search spaces, and training tricks.
Abstract
from arXiv · showhide
Deep learning has made breakthroughs and substantial in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers' prior knowledge and experience. And due to the limitations of human' inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search (NAS) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. Besides, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.
1 INTRODUCTION
NAS automates neural architecture design to reduce reliance on human expertise, while this survey organizes the field around early challenges and subsequent solutions.
- Neural architecture design strongly affects feature representation and final performance but depends heavily on researchers’ prior knowledge and experience.
- NAS aims to design high-performing neural architectures with limited computing resources, minimal human intervention, and automated search.
- Early NAS studies demonstrated the feasibility of automated architecture design, achieving state-of-the-art image-classification accuracy with reinforcement learning and evolutionary search.
- This survey reviews NAS through challenges and corresponding solutions rather than only search space, search strategy, and evaluation components.
- The survey compares existing NAS performance and optimization strategies, discusses future directions, and identifies two issues requiring vigilance.
2 CHARACTERISTICS OF EARLY NAS
Early NAS automatically generated architectures through broad, discrete, from-scratch, fully trained searches, but these characteristics created collective challenges that later work addressed.
- Early NAS generally generated candidate architectures from predefined operation sets, trained them, ranked validation accuracy, and used ranking feedback to guide search.
- NAS-RL used an RNN controller to generate variable-length architecture strings, while MetaQNN modeled layer selection as a Markov decision process optimized with Q-learning.
- Global search spaces cover all necessary architecture components, making the search cost increase with the size of the space.
- Early methods used discrete modifications, preventing gradient-based adjustment of architectures, and built networks from scratch without reusing existing architecture experience.
- Fully training every candidate from scratch increased cost even when successive architectures had similar network structures.
- The survey treats early NAS characteristics as collective challenges and organizes later research around their solutions.
3 OPTIMIZATION STRATEGY
NAS optimization strategies address the high cost and limited flexibility of early search by restructuring the search space and relaxing or improving search procedures. The survey organizes these solutions around modular spaces, continuous strategies, architecture recycling, and incomplete training, while noting trade-offs between search efficiency, generalization, and design freedom.
- Modular search space: Modular search spaces simplify NAS by searching a small cell or block and repeatedly stacking it into the final neural architecture.Cell-based spaces reduce complexity because they search only a few small structures rather than every component of the full architecture.
- Modular search space: Modular spaces can replace reduction cells with unit operations or use dense connections to perform downsampling while repeatedly applying searched cells.Dpp-net uses unit operations for downsampling in a densely connected architecture, while related approaches search only a normal cell.
- Modular search space: Cell-based architectures are easier to search and train, but their generalization effect is poor.This trade-off provides guidance for understanding the strengths and limitations of cell-based NAS.
- Modular search space: Diversified blocks can improve neural architecture performance while maintaining search costs similar to leading NAS methods.FPNAS treats the search as a bi-level optimization problem and reports reduced search cost compared with broader alternatives.
- Modular search space: Global search retains higher design freedom, whereas modular search more effectively reduces the search space and improves accessibility.The survey presents modular search as useful but insufficient for all task requirements, leaving global search with distinct research value.
- Continuous search strategy: Continuous strategies relax discrete operation selection into differentiable variables, jointly optimize architecture parameters and network weights, then discretize the result.DARTS uses softmax mixtures of candidate operations and bilevel optimization, but gradient-based methods still face local optimization problems.
4 PERFORMANCE COMPARISON
The survey compares NAS methods and optimization strategies across CIFAR-10 and ImageNet, while emphasizing that performance comparisons are difficult because methods differ in settings and techniques. It finds modular search, incomplete training, continuous gradient optimization, and architecture recycling to be prominent or effective strategies, but reports limited gains over manually designed networks.
- Comparison framework: NAS performance comparisons span reinforcement learning, evolutionary algorithms, gradient optimization, random search, and sequential model-based optimization on CIFAR-10 and ImageNet.The survey reports these comparisons in Tables 2 and 3 and classifies methods by mainstream search strategy.
- Search strategies: Modular search is widely used because it greatly reduces search-space complexity, although this does not establish superiority over global search.Global search retains value because it offers neural architecture design a higher degree of freedom.
- Optimization strategies: Incomplete training is widely used to accelerate candidate ranking and reduce search duration, while continuous gradient optimization can reduce search cost substantially.The survey also identifies random search as competitive and architecture recycling as promising but relatively underexplored.
- Evaluation setting: Transfer learning commonly searches on CIFAR-10 before transferring the architecture to ImageNet, so the reported search-time costs for the two datasets can be identical.The smaller-dataset search tasks are described as proxy or agent tasks.
- Overall comparison: NAS performance gains are limited compared with those achieved by manually designed networks.The survey notes that preprocessing, hyperparameters, search spaces, and training tricks complicate direct comparisons between NAS methods and artificial networks.
5 FUTURE DIRECTIONS
Future NAS research should expand beyond current image-classification and search-cost priorities while strengthening theoretical guidance, baselines, and experimental analysis. Promising directions include adaptive search spaces, joint hyperparameter optimization, and applications requiring more complex architectures.
- NAS remains concentrated on image-classification accuracy and search-cost reduction, despite broader potential applications such as detection, segmentation, and multi-object architecture search.
- Large-scale Evolution reduced manual restrictions but struggled with evolutionary efficiency and a huge search space, motivating later efforts to constrain search spaces without sacrificing performance.
- Random search is a strong baseline, yet insufficient baselines, sharable protocols, and ablations make it difficult to identify which NAS components produce performance gains.
- Parameter sharing improves search efficiency but may produce inaccurate candidate rankings, potentially preventing NAS from finding the optimal architecture.
- Adaptive search spaces could emphasize structural features common to promising architectures and reduce features associated with poorly performing ones without imposing fixed constraints manually.
- Jointly searching architectures with optimizer and training hyperparameters is promising, although earlier work mainly targeted small datasets and search spaces.
- NAS still requires additional theoretical guidance and experimental analysis before it can replace manually designed neural architectures.
6 REVIEW THREATS
The survey’s conclusions rely partly on open literature and on reported results that the authors did not independently reproduce. This limits the evidentiary basis for some assessments.
- The survey applies results from cited references without reproducing them, and notes that some views from unpublished papers require further confirmation.
7 SUMMARY AND CONCLUSIONS
The survey reframes NAS research around the challenges faced by early methods and the solutions developed afterward. It compares existing work and identifies future directions and issues requiring vigilance.
- The survey summarizes four shared characteristics of early NAS and organizes subsequent research around corresponding challenges and solutions.
- It provides comprehensive analysis and comparison of existing NAS work and discusses possible future research directions.
- The survey raises two issues requiring vigilance for future NAS research.