PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Evolutionary Neural Architecture Search

Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G. Yen, Kay Chen Tan

arXiv:2008.10937v4cs.NE

TL;DR

Manual neural architecture design is laborious and expertise-dependent, while existing literature lacked a comprehensive synthesis of evolutionary NAS methods. This paper surveys over 200 ENAS papers across core algorithmic components, applications, and challenges. It reports that ENAS can match or exceed manually designed architectures on image-classification benchmarks, while computational cost, fair comparison, domain-knowledge dependence, and interpretability remain important issues.

  • Problem

    Manual DNN architecture design is labor-intensive and expertise-dependent, while no comprehensive survey specifically covered ENAS algorithms.

  • Method

    The paper comprehensively surveys ENAS through population representation, encoding space, population updating, fitness evaluation, applications, performance tables, and challenges.

  • Results

    On CIFAR-10, many ENAS methods achieved lower error rates than the manually designed ResNet-110 baseline of 6.43%, with NSGANet reported at 2.02%.

  • Takeaways & Limitations

    ENAS shows promising ability and wide applicability, including architectures that can reach or exceed expert-designed performance levels.

Abstract

from arXiv · show

Deep Neural Networks (DNNs) have achieved great success in many applications. The architectures of DNNs play a crucial role in their performance, which is usually manually designed with rich expertise. However, such a design process is labour intensive because of the trial-and-error process, and also not easy to realize due to the rare expertise in practice. Neural Architecture Search (NAS) is a type of technology that can design the architectures automatically. Among different methods to realize NAS, Evolutionary Computation (EC) methods have recently gained much attention and success. Unfortunately, there has not yet been a comprehensive summary of the EC-based NAS algorithms. This paper reviews over 200 papers of most recent EC-based NAS methods in light of the core components, to systematically discuss their design principles as well as justifications on the design. Furthermore, current challenges and issues are also discussed to identify future research in this emerging field.

I. INTRODUCTION

Neural architectures strongly affect DNN performance, but manual design is expertise-intensive and problem-dependent. NAS addresses this challenge automatically, while ENAS applies evolutionary computation; this survey responds to the lack of a comprehensive ENAS review.

  • DNN performance depends on both neural architectures and their associated weights.
  • Manual DNN architecture design requires specialized expertise and may need redesign when the data distribution changes.
  • NAS formulates architecture selection as a non-convex, non-differentiable optimization problem over potential architectures and training and fitness datasets.
  • NAS methods are broadly categorized as reinforcement-learning, gradient-based, or evolutionary-computation approaches.
  • ENAS mainly searches architectures with evolutionary computation, while gradient-based algorithms subsequently obtain their optimal weights.
  • The literature lacks a comprehensive survey focused specifically on ENAS algorithms, motivating this paper’s review.

II. BACKGROUND

ENAS evolves encoded neural architectures through population initialization, fitness evaluation, and population updating. The survey organizes ENAS methods by evolutionary search strategy and target neural-network type.

  • Common ENAS flow: A common ENAS process initializes an encoded architecture population, evaluates fitness, and updates the population through evolutionary search.
  • Evolutionary search strategies: ENAS search strategies are grouped into evolutionary algorithms, swarm intelligence, and other evolutionary-computation methods.
  • Evolutionary search strategies: Evolutionary-algorithm ENAS methods dominate the field, with genetic algorithms forming the largest share because architecture representation is convenient.
  • Neural-network perspective: The survey classifies ENAS outputs by neural-network perspective, including CNNs, DBNs, SAEs, RNNs, and other networks.

C. Common Neural Networks in ENAS

ENAS is applied across several neural-network families, with CNNs receiving the greatest attention. The optimized parameters range from layer and block configurations to connectivity, recurrent unfolding, and selected weights or initialization settings.

  • ENAS commonly targets CNNs, DBNs, stacked auto-encoders, RNNs, and other neural-network types.
  • CNN: CNN architecture searches optimize layer counts, layer hyperparameters, and connectivity such as dense and skip connections.
  • DBN: DBN searches commonly optimize the number of RBM building blocks, neurons per layer, and network depth.
  • SAE: SAE searches may evolve only the encoder, because the decoder can be derived symmetrically, or optimize encoder and decoder hyperparameters separately.
  • RNN: RNN-oriented ENAS methods can optimize how many times the recurrent network is unfolded, in addition to architecture-related parameters.
  • Some ENAS methods also optimize weight initialization values or initialization methods, alongside architecture parameters.

III. ENCODING SPACE

The encoding space defines valid encoded individuals and is organized by basic units or topology, while constraints trade search efficiency against architectural novelty. Layer-, block-, cell-, and topology-based spaces differ in representational detail and search-space size.

  • Encoding space contains all valid individuals and includes layer-, block-, cell-, and topology-based categories.
  • Layer-based encoding space: Layer-based encoding can create a huge search space because it represents many primitive-layer choices, increasing search time.
  • Block-based encoding space: Block-based encoding combines layers into structured units such as ResBlocks and DenseBlocks, often requiring fewer parameters.
  • Cell-based encoding space: Cell-based encoding stacks repeated motifs and reduces encoding-space size, but no theoretical basis guarantees that it finds good architectures.
  • Topology-based encoding space: Topology-based encoding focuses on connections between units, including subgraphs selected from a supergraph.
  • Constraints: Fixed depth, rich initialization, and partially fixed structure constrain encoding spaces; such constraints can ease search but restrict novel architectures.
  • The encoding space generally underlies the search and initial spaces, with the initial space often a constrained subspace of the search space.

B. Initial Space

Initial spaces determine how ENAS populations begin, ranging from minimal or random architectures to manually designed ones. Smaller initial spaces can expand the subsequent search space, while encoding strategies differ in flexibility and computational consequences.

  • ENAS initialization uses trivial, random, or well-designed spaces, corresponding to minimal, encoding-space-randomized, or expertise-based starting populations.
  • Trivial space: A trivial initial space can start with single-layer models and has been experimentally shown to evolve into a competitive architecture.
  • Random initialization makes the initial and search spaces equivalent, whereas trivial or well-designed initialization usually starts from a smaller space.
  • Encoding strategies: Fixed-length encoding keeps individual lengths constant, while variable-length encoding allows different lengths during evolution.
  • Variable-length encoding: Variable-length encoding avoids requiring optimal depth in advance and represents more architectural detail, but requires redesigned operators and can produce over-deep architectures.
  • Architecture encoding represents both basic-unit configurations and the connections between units or within units.

A. Encoding for Configurations of Basic Units

Architecture encoding captures both the configurations of basic units and their connections. Linear structures offer simplicity, whereas nonlinear structures use skip or loop connections represented through graph-oriented encodings.

  • DNN architectures are represented as directed graphs whose encoding covers basic-unit configurations and connections.
  • Linear architecture: Linear architectures stack sequential basic units and are widely used because their skeleton is simple to construct.
  • Nonlinear architecture: Nonlinear architectures contain skip or loop connections at either the macro level between units or the micro level within units.
  • Linear architecture: Layer-based linear architectures can be built from unit parameters alone because their connections are implicit.
  • Nonlinear architecture: Adjacent matrices are a popular representation for nonlinear connections, with binary strings encoding whether node pairs are connected.
  • Nonlinear architecture: AmoebaNet-A illustrates nonlinear macro structure by stacking cells that receive inputs from the previous two cells.

V. POPULATION UPDATING

Population updating in ENAS varies with the evolutionary method, but EA-based approaches dominate and commonly use selection, mutation, and crossover. Methods also address exploration, computational cost, and multiple objectives such as accuracy and model size.

  • EA-based ENAS algorithms dominate the surveyed methods, with GA-based ENAS especially popular because architecture representations are convenient.
  • Selection: EA population updating begins with selection, which may choose parents or survivors for the next population.
  • Selection: Elitism preserves high-fitness individuals but can reduce diversity and cause premature convergence toward local optima.
  • Selection: Aging evolution discards older individuals to explore more of the search space instead of focusing too early on good models.
  • Evolutionary operations: Mutation changes one individual, whereas crossover combines two individuals to generate offspring.
  • Evolutionary operations: Guided mutation uses Gaussian regression or an RNN to direct offspring generation toward regions predicted to have high fitness.
  • Computational efficiency: Network morphisms change architectures while preserving their represented function, allowing acquired weights or experience to be retained.
  • Computational efficiency: Network morphisms generally increase network capacity, so they can produce larger and deeper architectures unsuitable for resource-limited devices.

B. SI for ENAS

Swarm-intelligence methods apply collective search mechanisms to ENAS, with ants constructing architectures through pheromone-guided paths and particles updating architecture-related representations. Multi-objective swarm-intelligence ENAS remains relatively scarce and uses diversity-aware selection or adaptive penalties.

  • Particle Swarm Optimization: Particle swarm optimization can update particles at the layer level, while gradient-priority variants address low convergence efficiency with many hyper-parameters.The cited implementations update particles based on layers rather than layer parameters.
  • Ant Colony Optimization: Ant colony optimization builds architectures by having ants follow pheromone instructions, reinforcing successful paths while evaporation promotes exploration.Pheromone is updated each generation.
  • Multi-objective ENAS: Multi-objective swarm-intelligence ENAS research is scarce, with methods using bias-variance modeling, crowding distance, crowding factors, or adaptive penalty-based boundary intersection.These mechanisms support more stable architectures or balance convergence and diversity.
  • Ant Colony Optimization: In ACO, the population is also termed a colony.Each ant belongs to the colony that constructs candidate architectures.

C. Other EC Techniques for ENAS

Other evolutionary and hybrid techniques broaden ENAS search and evaluation strategies, while the survey emphasizes that evaluation is often the most time-consuming stage. Methods reduce cost through inheritance, early stopping, smaller datasets or populations, one-shot models, hardware parallelism, proxies, and performance prediction, each with associated trade-offs.

  • Differential Evolution: Differential evolution generates offspring using information from three individuals, but differences among DE-based ENAS algorithms are reported as small.
  • Hybrid EC Techniques: A hybrid PSO-GA method uses PSO for decimal-encoded block parameters and GA for binary-encoded shortcut connections.The design matches PSO with continuous optimization and GA with binary optimization.
  • Other EC Techniques: Memetic algorithms combine evolutionary algorithms with local search, while other approaches include simple mutation-only evolution and virus-spread simulation.Gradient descent can serve as a fine-tuning operation in GP.
  • Efficient Evaluation: Evaluation is often the most time-consuming stage of ENAS, and training increasingly complex architectures to convergence requires substantial time.
  • Efficient Evaluation: Weight inheritance reuses weights from unchanged architectural parts, avoiding complete retraining from scratch; network morphisms can change architecture without losing acquired knowledge.
  • Efficient Evaluation: Early stopping, reduced training sets, and proxy models shorten evaluation, but proxies trade prediction accuracy for speed and require validation of their reliability.
  • Efficient Evaluation: Reducing populations can accelerate evolution but may weaken search-space exploration, motivating dynamic reduction that uses larger early populations and smaller later ones.
  • Efficient Evaluation: One-shot models train a single SuperNet and sample SubNets with shared parameters, but weight sharing makes architecture evaluation less accurate.

VII. APPLICATIONS

ENAS has been applied across the broad range of fields where DNNs are used. The survey presents applications spanning multiple categories rather than a single task domain.

  • ENAS applications cover the fields in which DNNs are applied.

A. Overview

Existing ENAS work spans image and signal processing, biomedical tasks, forecasting, engineering, and other applications, with image classification especially prominent. CIFAR-10, CIFAR-100, and ImageNet are used to report representative image-classification results, but comparisons are not fully fair because search spaces and training tricks differ.

  • Application Scope: Table VI groups ENAS applications into five categories: image and signal processing, biological and biomedical tasks, predictions and forecasting, engineering, and others.
  • Image and Signal Processing: Image and signal-processing applications include image classification, image restoration, denoising, super-resolution, inpainting, emotion recognition, speech recognition, language modeling, and face de-identification.
  • Biomedical Tasks: Biomedical applications include medical image segmentation, malignant melanoma detection, sleep-heart studies, and human-sperm assessment.
  • Predictions and Forecasting: Forecasting applications cover wind speed, car-park occupancy, time series, finance, usable life, electricity demand, traffic flow, electricity price, and municipal waste.
  • Engineering: Engineering applications include engine-vibration prediction, UAV tasks, bearing-fault diagnosis, and general-aviation flight-data prediction.
  • Other Applications: Other applications include concrete-crack detection, gamma-ray detection, multitask learning, galaxy identification, video understanding, and comics understanding.
  • Benchmark Comparisons: CIFAR-10, CIFAR-100, and ImageNet are used as image-classification benchmarks, with Table VII ranking methods by best CIFAR-10 accuracy and reporting error rates for the datasets.
  • Benchmark Comparisons: Table VII selects representative architectures when multiple results exist and may report classification errors from the literature.

VIII. CHALLENGES AND ISSUES

The survey identifies unresolved questions about ENAS effectiveness, operator contributions, and the role of domain knowledge. It also highlights the need for stronger experiments and broader evidence to distinguish evolutionary search from simpler baselines.

  • A. The Effectiveness: Random search can match or outperform some ENAS methods, but other studies report evolutionary search finding better architectures.The evidence is mixed across studies, motivating more controlled evaluation.
  • A. The Effectiveness: Crossover can efficiently generate architectures substantially different from their parents, while mutation explores neighboring regions incrementally.The survey characterizes crossover as more exploitative and mutation as more exploratory.
  • A. The Effectiveness: The contribution of crossover remains insufficiently explained, especially in methods that omit it.Additional experiments are suggested to clarify how evolutionary operators affect ENAS performance.
  • A. The Effectiveness: ENAS effectiveness may depend on embedding domain knowledge about neural architectures, despite its goal of reducing manual design expertise.The survey presents this tension as an important direction for future development.

B. Scalability

Scalability challenges arise from dataset size, expensive fitness evaluation, limited interpretability, and the difficulty of evolving deep architectures. The survey describes proxies and acceleration methods, but emphasizes unresolved resource and evaluation constraints.

  • B. Scalability: ImageNet contains more than 14 million manually annotated high-resolution images, making large-scale evaluation more convincing but more demanding than CIFAR-10.CIFAR-10 may be easy to over-fit, so proxy transfer to ImageNet or down-scaled ImageNet64 × 64 are alternatives.
  • B. Scalability: Fully training each architecture is prohibitively time-consuming, requiring compromises between evaluation accuracy and substantial reductions in evaluation time.The survey frames efficient evaluation as balancing these competing objectives under limited computing resources.
  • B. Scalability: Search acceleration remains immature: LargeEvo reduced cost through weight inheritance but still required several days and substantial computational resources.The field also lacks baseline methods and common assessment criteria for acceleration.
  • B. Scalability: GP-based ENAS methods have demonstrated interpretability through visualized evolved features and functions used for classification.Examples include edge detection, minimum-value aggregation for hand openness, and discriminative face features.
  • B. Scalability: Existing GP-based ENAS methods target shallow neural networks with relatively few generated features, whereas successful neural networks generally use deep architectures.The survey identifies deep-GP as a future direction for evolving and analyzing deeper models.

E. Future Applications

ENAS has been explored across multiple neural-network application areas and is presented as a way to automate architecture design. However, expert-designed encoding spaces, unfair comparisons, limited benchmarks, and unresolved search-strategy questions constrain the field’s current scope.

  • Applications: ENAS has been explored in various applications and is described as applicable wherever neural networks can be applied.The paper also summarizes image-classification successes as evidence of its applicability.
  • Applications: ENAS is not completely automated because experts still design the encoding space for different applications.Examples use CNN-oriented components for image processing and recurrent cells for time-series processing.
  • Fair comparisons: Comparisons among ENAS methods can be unfair because methods may use different training tricks and encoding spaces.The paper identifies cutout, ScheduledDropPath, and differing search spaces as major sources of unfairness.
  • Fair comparisons: NAS-Bench-101 and NAS-Bench-201 enable less expensive comparisons, but their cell-based spaces do not cover all existing ENAS search spaces.Layer- and block-based units can require more hyperparameters and produce larger encoding spaces.
  • Future directions: The survey organizes ENAS around population representation, encoding space, population updating, and fitness evaluation while reviewing applications, efficiency methods, and open challenges.It highlights computational cost, the need for fair standards, and uncertainty about whether evolutionary search outperforms simpler baselines.
Loading 2008.10937v4…