PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey on Evolutionary Deep Learning: Principles, Algorithms, Applications and Open Issues

Nan Li, Lianbo Ma, Guo Yu, Bing Xue, Mengjie Zhang, Yaochu Jin

arXiv:2208.10658v1cs.NE

TL;DR

Because DL model design and configuration require manual expertise and trial-and-error, the survey frames EDL as an optimization problem and maps EC methods across the DL pipeline. It synthesizes scale-dependent comparisons between pure and hybrid EC approaches.

  • Problem

    Practitioners struggle to manually design deep models and find configurations by trial and error, amid difficulty acquiring expert knowledge.

    Source: §1 · p. 2

  • Method

    The survey frames EDL as a complex optimization problem and organizes EC methods across feature engineering, model generation, and deployment, focusing on solution representations and search paradigms.

    Source: §7 · p. 24

  • Results

    Pure EC outperforms gradient-based methods on some small-scale problems, while hybrid EC and gradient-based methods are a better approach for large-scale models.

    Source: §4.1.4 · p. 12

  • Takeaways & Limitations

    The surveyed applications show EDL's potential ability to handle real-world problems.

    Source: §7 · p. 24

    Evidence remains mixed on whether EC outperforms random search, motivating unified comparisons under consistent search spaces and hyperparameter configurations.

    Source: §6.2 · p. 22

Abstract

from arXiv · show

Over recent years, there has been a rapid development of deep learning (DL) in both industry and academia fields. However, finding the optimal hyperparameters of a DL model often needs high computational cost and human expertise. To mitigate the above issue, evolutionary computation (EC) as a powerful heuristic search approach has shown significant merits in the automated design of DL models, so-called evolutionary deep learning (EDL). This paper aims to analyze EDL from the perspective of automated machine learning (AutoML). Specifically, we firstly illuminate EDL from machine learning and EC and regard EDL as an optimization problem. According to the DL pipeline, we systematically introduce EDL methods ranging from feature engineering, model generation, to model deployment with a new taxonomy (i.e., what and how to evolve/optimize), and focus on the discussions of solution representation and search paradigm in handling the optimization problem by EC. Finally, key applications, open issues and potentially promising lines of future research are suggested. This survey has reviewed recent developments of EDL and offers insightful guidelines for the development of EDL.

1 INTRODUCTION

Manual DL design requires trial and error and difficult-to-acquire expertise, while prior surveys rarely cover the full range of EDL. This survey reviews EC-driven DL across the pipeline and discusses its applications, open issues, and future trends.

1 INTRODUCTION

  • DL practitioners must manually design models and configurations by trial and error, while acquiring the required expert knowledge is difficult.The introduction identifies feature engineering, model generation, and deployment as stages where domain knowledge is used.Source: §1 · p. 2
  • AutoML automates feature engineering, parameter and hyperparameter optimization, neural architecture search, and model compression.The paper notes that automated design without manual intervention has attracted significant attention.Source: §1 · p. 2
  • EC has been widely applied to automatic DL, termed EDL, for feature engineering, model generation, and deployments; Web of Science publications and citations about EC and DL increased gradually until around 2012, then sharply.Source: §1 · p. 2
  • Prior surveys rarely cover EDL systematically across feature engineering, parameter and hyperparameter optimization, neural architecture search, and model compression.This survey frames its contribution as a comprehensive review, emphasizing solution representations and search paradigms and discussing applications, open issues, and trends.Source: §1 · pp. 2–4

2 AN OVERVIEW OF EVOLUTIONARY DEEP LEARNING

The overview frames deep learning as a pipeline of feature engineering, model generation, and deployment, and evolutionary computation as a population-based optimization approach. It defines EDL through that optimization lens and organizes methods by what they optimize and how they search.

2.1 Deep Learning

  • Deep learning comprises feature engineering, model generation, and model deployment, which respectively improve data quality, optimize models for tasks, and meet deployment constraints.Deployment concerns include latency, storage, and energy consumption under limited computational budgets.Source: §2.1 · p. 4

2.2 Evolutionary Computation

  • Evolutionary computation uses stochastic populations and environmental selection for optimization without gradient information; its framework includes initialization, evaluation, and solution updating.EDL evaluation can be computationally expensive; one cited architecture search used 3000 GPU days.Source: §2.2 · pp. 4–5

2.3.1 EDL from two perspectives.

  • EDL is both an automated way to find deep models for datasets or tasks and an evolutionary optimization process over model configurations and performance objectives.The survey presents these as the DL and EC perspectives, respectively.Source: §2.3.1 · p. 5

2.3.2 Definition and Framework of EDL.

  • The EDL framework maximizes model performance on tasks and datasets under constraints of no human assistance and limited computational resources.It initializes encoded individuals, evaluates objectives or constraints, updates solutions, checks termination, and outputs the best-performing solution.Source: §2.3.2 · pp. 5–6

2.3.3 Taxonomy of EDL Approaches.

  • The taxonomy separates what is optimized—feature, model-generation, and deployment problems—from how it is optimized through representations, search paradigms, and NAS acceleration.The covered problems include feature selection, construction, and extraction; parameter, architecture, and joint optimization; and pruning or other compression.Source: §2.3.3 · p. 6

3 FEATURE ENGINEERING

Feature engineering preprocesses raw data by filtering irrelevant features or creating new ones. The survey groups EC-based techniques into feature selection, construction, and extraction.

3 FEATURE ENGINEERING

  • Feature engineering preprocesses raw data by filtering irrelevant features or creating new features from the originals.Source: §3 · p. 6
  • EC-based feature engineering techniques can reduce data dimensionality, speed learning, or improve model performance.Source: §3 · p. 7
  • The common EC-based feature engineering categories are feature selection, feature construction, and feature extraction.Source: §3 · p. 7

3.1 Feature Selection

Feature selection searches for a representative subset without irrelevant or redundant features, but its search space grows exponentially with feature count and feature interactions affect performance. The survey reviews EC representations and search paradigms, including their comparative strengths and challenges.

3.1.1 Problem Formulation.

  • Feature selection has 2^n possible subsets for n features, and interactions among features can affect performance.Source: §3.1.1 · p. 7

3.1.2 Solution Representations.

  • Representations include linear vectors or strings, GP trees whose leaves are selected features, and graphs whose paths or subgraphs encode feature subsets.Source: §3.1.2 · p. 7

3.1.3 Search Paradigms.

  • Search paradigms include basic EC methods, co-evolution of classifier design and feature selection, and multi-objective optimization of accuracy and subset size.The multi-objective approach can return Pareto non-dominated candidate subsets.Source: §3.1.3 · pp. 7–8

3.1.4 Summary.

  • GP is commonly used for large or high-dimensional feature selection and outperforms GA on some small, high-dimensional datasets; PSO converges faster, while ACO graph flexibility brings encoding challenges.GA was initially used for datasets with at most 1000 dimensions; Brain Tumor-2 has 10367 features and 50 samples.Source: §3.1.4 · p. 8

3.2 Feature Construction

Feature construction creates higher-level features from original ones using function operators, but its combinatorial search space grows with the number of original features and operators. The survey reviews EC encodings and search paradigms, noting efficiency challenges in large-scale settings.

3.2.1 Problem Formulation.

  • Feature construction creates higher-level features from original features using operators, with a search space that grows exponentially as features and operators increase.Source: §3.2.1 · p. 8

3.2.2 Solution Representations.

  • EC representations include binary vectors with local operator search, trees with features as leaves and operators as internal nodes, and graphs linking features and operators.Graph encoding can become inefficient on high-dimensional feature sets because graph traversal complexity makes construction more difficult.Source: §3.2.2 · p. 8

3.2.3 Search Paradigms.

  • Search paradigms include basic EC, co-evolution of feature construction and classifier or object-detection tasks, construction of multiple features, and multi-objective optimization.Multi-objective formulations may consider feature count, mutual information, and classification accuracy.Source: §3.2.3 · pp. 8–9

3.2.4 Summary.

  • Feature construction can require more computational overhead than feature selection, and efficiency remains a challenge in high-dimensional or large-scale scenarios.Its performance may depend on the quality of features selected beforehand.Source: §3.2.4 · p. 9

3.3 Feature Extraction

Feature extraction reduces feature dimensions through transformations, but conventional extractors may lose important information and require tedious hyperparameter tuning. The survey describes EC representations and search paradigms, alongside limitations of existing encodings and operation sets.

3.3.1 Problem Formulation.

  • Feature extraction reduces dimensions by transforming original features or data; conventional PCA and LDA may lose important information and require tedious hyperparameter tuning.Source: §3.3.1 · p. 9

3.3.2 Solution Representations.

  • EC encodes map functions or their parameters linearly, or represents extractors as trees with features or constants at leaves and operators at internal nodes.An individual can represent a feature extractor or map function.Source: §3.3.2 · pp. 9–10

3.3.3 Search Paradigms.

  • Search paradigms include basic EC, co-evolution of classifier construction and feature extraction, and multi-objective optimization of transformation complexity and recognition performance.Other objectives may include model accuracy, computational time, complexity, and robustness.Source: §3.3.3 · p. 10

3.3.4 Summary.

  • Existing encodings and predefined operation sets may limit extractor performance, motivating efficient algorithms, operation control, and representations for high-dimensional extraction.Source: §3.3.4 · p. 10

4 MODEL GENERATION

Model generation searches for models with desirable learning capability on given tasks. This section introduces evolutionary parameter, architecture, and joint optimization, while directing readers to reviews for RL- and gradient-based approaches.

4 MODEL GENERATION

  • Model generation searches for models with desirable learning capability on given tasks.Source: §4 · p. 10
  • The section introduces evolutionary parameter optimization, architecture optimization, and joint optimization, from solution representation to search paradigms.Source: §4 · p. 10
  • Readers interested in RL-based or gradient-based model-generation approaches are referred to other reviews.Source: §4 · p. 10

4.1 Model Parameter Optimization

EC-based model-parameter optimization searches weights for a predefined architecture using direct or indirect representations and pure or gradient-hybrid search. Its suitability depends on scale and access to gradient information.

4.1.1 Problem Formulation.

  • Model parameter optimization seeks weights W* minimizing loss L for a predefined architecture A, where W may comprise millions of parameters and the problem is highly non-convex.Source: §4.1.1 · pp. 10–11

4.1.2 Solution Representations.

  • Direct encoding stores parameters explicitly but can impose huge computational overhead, whereas indirect encoding reduces representation dimensionality at the risk of information loss and degraded performance.The cited indirect approach represents weights as Fourier coefficients and omits high-frequency coefficients.Source: §4.1.2 · p. 11

4.1.3 Search Paradigms.

  • Pure EC searches parameters through basic or co-evolution paradigms, while gradient-based EC combines evolutionary and gradient methods in three execution orders.The hybrids apply EC before gradients, use gradient-derived population initialization, or alternate the two methods iteratively.Source: §4.1.3 · pp. 11–12, Figure 5 · p. 11

4.1.4 Summary.

  • Pure EC can outperform gradient-based methods on some small-scale problems, while EC–gradient hybrids may be more effective for large-scale models; EC also applies when gradients are difficult to obtain.For sparse or deceptive DRL rewards, gradient information may be unattainable.Source: §4.1.4 · p. 12

4.2 Model Architecture Optimization

The survey treats neural architecture search as a performance-oriented search over architectures, whose encodings and evolutionary paradigms vary in structure. It also reviews acceleration methods and their efficiency–quality trade-offs.

4.2.1 Problem Formulation.

  • Architecture optimization searches a space 𝒜 for the best-performing architecture under weights W, forming a bi-level problem because architecture search depends on parameter optimization.The survey omits detailed search-space design because of page limits.Source: §4.2.1 · pp. 12–13

4.2.2 Solution Representations.

  • Fixed-length encodings ease evolutionary operations but require a predefined maximum length, whereas variable-length encodings avoid prior depth knowledge but complicate standard genetic operators.Architecture representations also encode unit hyperparameters and connectivity; adjacency matrices require a fixed number of units, while ordered pairs can represent directed acyclic graphs.Source: §4.2.2 · pp. 13–14

4.2.3 Search Paradigms.

  • NAS paradigms include incremental construction, co-evolution of blueprints and components, and multi-objective search for Pareto-optimal architectures.Multi-objective criteria can include accuracy, inference time, model size, and energy consumption.Source: §4.2.3 · p. 14

4.2.4 Acceleration Strategies.

  • Acceleration strategies span initialization, evaluation, inheritance, surrogate prediction, population memory, and hardware; multiple strategies can be combined.Weight inheritance reuses supernet or parental weights, while population memory avoids reevaluating repeated architectures.Source: §4.2.4 · pp. 15–16
  • Many acceleration strategies trade search efficiency for possible suboptimality, and EC-based NAS still incurs high computational overhead despite such methods.Small populations may poorly cover a multi-objective front, parameter sharing may bias search, and accurate surrogates need time-consuming training data.Source: §4.2.4 · p. 16, §4.2.5 · p. 17

4.2.5 Summary.

  • Compared with EC, RL-based NAS can require immense resources and cannot find Pareto sets after converting multiple objectives to one; Bayesian and gradient methods have their own scaling or GPU-cost limits.Bayesian methods are more efficient on low-dimensional problems with a suitable distance function, while Gaussian-process cost grows with dimensionality.Source: §4.2.5 · p. 17

4.3 Joint Optimization

Joint optimization searches model architecture and parameters together because optimizing either independently may not yield the best task-specific model. The survey reviews encodings and EC search approaches while noting their computational cost.

4.3.1 Problem Formulation.

  • Joint optimization searches for an architecture A* and associated weights W* together because independent optimization of either is difficult to make optimal for a given task.Source: §4.3.1 · p. 17

4.3.2 Solution Representations.

  • Joint representations include linear encodings of weights and structure, tree encodings of topology and weights, and graph encodings whose edges record weights.The linear example uses a variable-length binary vector; the graph example uses real values for weights and biases and zero for absent connections.Source: §4.3.2 · pp. 17–18

4.3.3 Search Paradigms.

  • Basic EC can optimize architecture and weights simultaneously, while multi-objective search can jointly optimize network error and connection count.NEAT evolves small-network architecture incrementally while optimizing weights, keeping the search-space dimensionality lowest over generations.Source: §4.3.3 · p. 18

4.3.4 Summary.

  • Direct encoding is common for small networks, but larger networks motivate indirect encoding; joint EC methods can incur prohibitive computational costs, warranting more efficient approaches.The survey notes this cost can exceed that of architecture optimization alone.Source: §4.3.4 · p. 18

5 MODEL DEPLOYMENT

Model deployment uses evolutionary computation to automate compression of deep networks for resource-limited devices, with most reviewed approaches focused on pruning. The survey covers pruning formulation, representations, and search paradigms, then reviews evolutionary approaches to distillation, low-rank factorization, and jointly optimizing compression techniques.

5 MODEL DEPLOYMENT

  • Most reviewed evolutionary model-deployment methods focus on pruning to reduce model size and inference time for resource-limited devices.Traditional compression methods require substantial expert knowledge and manual effort.Source: §5 · p. 18

5.1.1 Problem Formulation.

  • Pruning seeks a lightweight model while keeping its loss close to the original by removing redundant, non-informative units.The survey focuses on EC-based pruning methods and refers readers to other surveys for traditional pruning categories.Source: §5.1.1 · p. 19

5.1.2 Solution Representations.

  • Binary encodings mark network components for removal, while indirect encodings can shorten representations or encode layer channel counts.Binary representations grow with model complexity; indirect encoding can improve search efficiency, and meta-networks can generate weights for encoded structures.Source: §5.1.2 · p. 19

5.1.3 Search Paradigms.

  • Pruning search uses single-objective and multi-objective EC paradigms, with the latter producing Pareto sets of lightweight models.Examples include sensitivity-guided weight pruning with recovery fine-tuning and NSGA-II optimizing filter count and performance.Source: §5.1.3 · p. 19, Table 3 · p. 20

5.2.1 Knowledge Distillation.

  • Evolutionary knowledge-distillation methods address teacher–student capability gaps through embedding learning or an online evolutionary teacher.The cited embedding-learning results report narrowing the teacher–student performance gap on given tasks.Source: §5.2.1 · p. 20

5.2.2 Low-rank Factorization.

  • Evolutionary low-rank methods automate rank and sparsity selection for weight matrices using classification error and compression rate as objectives.This addresses reliance on domain expertise for choosing compression hyperparameters and can generate models at different compression rates.Source: §5.2.2 · p. 20

5.2.3 EC for Jointly Optimization.

  • Evolutionary methods jointly optimize compression techniques, including quantization with compact architectures or distillation, and block-level shallowing.The cited methods use accuracy thresholds or objectives such as latency, model error, block count, and accuracy drop.Source: §5.2.3 · p. 21

5.2.4 Summary.

  • Evolutionary deployment still faces high computational overhead, coupling to model generation, and black-box concerns in security-critical domains.The survey identifies joint optimization of compression, NAS, and interpretability within acceptable time as a promising challenge.Source: §5.2.4 · p. 21

6 APPLICATIONS, OPEN ISSUES, AND TRENDS

EDL applications span computer vision, natural language processing, and other practical tasks, while the survey identifies unresolved challenges in efficiency, evaluation, scale, and end-to-end optimization. Future work also needs fair comparisons, greater interpretability, and stronger performance on real-world tasks.

6.1.1 Computer Vision.

  • NAT-M4 achieves 80.5% Top-1 accuracy on ImageNet with a small model size, and EDL structures show better performance than manually designed models, especially on image-classification benchmarks.The cited passage highlights CIFAR-10, CIFAR-100, and ImageNet.Source: §6.1.1 · p. 21

6.1.2 Natural Language Processing.

  • EC-inspired NLP methods address language modeling, entity recognition, text classification, and keyword spotting; cited work also optimizes translation inference time and accuracy.DE has also been used for feature selection in named entity recognition.Source: §6.1.2 · p. 22

6.1.3 Other Applications.

  • EDL applications include medical analysis, financial prediction, signal processing, industrial prediction, and remote sensing, where architecture complexity and prediction error are conflicting objectives.Examples include gene selection, bankruptcy analysis, and electromyography classification.Source: §6.1.3 · p. 22

6.2 Open Issues

  • Acceleration strategies such as surrogate models, supernets, and early stopping can trade some accuracy for efficiency, while EDL's advantage over random search remains debated.Supernet inheritance may not reliably evaluate every subnet because of catastrophic forgetting and weight coupling; consistent search spaces and hyperparameters are needed for comparisons.Source: §6.2 · p. 22
  • Large-scale EDL evaluation is costly, and complete pipeline optimization remains challenging because stages are highly coupled and the search space is large.The survey calls for assessing sensitivity to dataset scale and finding economical ways to verify methods on large datasets.Source: §6.2 · pp. 22–23

6.3 Challenges and Future Trends

  • Fair comparisons are impeded by nonuniform feature-engineering benchmarks, differing NAS and compression criteria, and platforms that cover only specific tasks.The survey points to task-specific platforms including BenchENAS, NAS-Bench-101, NAS-Bench-201, NAS-Bench-301, and HW-NAS-Bench.Source: §6.3 · p. 23
  • EDL remains a black-box optimization with limited theoretical explanation, while post-hoc accounts may lack trustworthy mathematical deduction.The survey links this interpretability challenge to slow EDL development in sensitive domains such as finance and medicine.Source: §6.3 · p. 23
  • EDL performs well on image benchmarks but loses ground to human-designed NLP models, and noisy or small real-world datasets can create additional challenges.The survey suggests unsupervised and self-supervised learning as possible ways to address real-world data issues.Source: §6.3 · p. 23

7 CONCLUSIONS

The conclusion frames EDL as an EC-based approach for automatically optimizing deep-model parameters or architectures, with competitive performance relative to manually designed approaches. It summarizes a pipeline-wide survey and highlights unresolved issues for future work.

7 CONCLUSIONS

  • EDL approaches automatically optimize deep-model parameters or architectures and show competitive robustness and search capability compared with manually designed approaches.The conclusion describes EDL as a growing research topic across machine learning and evolutionary computation.Source: §7 · p. 24
  • The survey formulates EDL as a complex optimization problem and organizes EC techniques across feature engineering, model generation, and model deployment.It discusses solution representations and search paradigms at different pipeline stages and compares EC-based with non-EC approaches.Source: §7 · p. 24
  • The conclusion identifies expensive optimization, large-scale datasets, fair comparisons, interpretability, real-world performance, and end-to-end development as continuing challenges.It calls for theoretical analysis of EDL search ability and further work on both large- and small-scale benchmarks.Source: §7 · p. 24
Loading 2208.10658v1…