PaperlayerView exact revision on arXiv

Source-linked AI summary

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

Edgar Galván, Peter Mooney

arXiv:2006.05415v1cs.NEcs.CVcs.LG

TL;DR

DNN architecture and training are difficult to configure, while existing neuroevolution research lacked a comprehensive survey focused on its strengths and limitations. The paper systematically reviews EA-based architecture search and training methods, finding broad evidence of successful applications but persistent training-time costs and open benchmark challenges. It identifies current issues and future directions for researchers in evolutionary computation and deep learning.

  • Problem

    Existing literature lacked a comprehensive survey focused specifically on the strengths and limitations of neuroevolution for DNN architecture configuration and training.

  • Method

    The paper conducts an extensive literature review and critically evaluates relevant state-of-the-art EA studies on DNN architecture configuration and training.

  • Results

    The survey finds successful EA-based DNN architecture methods, while identifying training time, benchmark diversity, and comparison issues as continuing challenges.

  • Takeaways & Limitations

    Neuroevolution is presented as a competitive approach for DNN design whose strengths, limitations, and future research directions warrant attention from deep learning and evolutionary computation communities.

Abstract

from arXiv · show

A variety of methods have been applied to the architectural configuration and learning or training of artificial deep neural networks (DNN). These methods play a crucial role in the success or failure of the DNN for most problems and applications. Evolutionary Algorithms (EAs) are gaining momentum as a computationally feasible method for the automated optimisation and training of DNNs. Neuroevolution is a term which describes these processes of automated configuration and training of DNNs using EAs. While many works exist in the literature, no comprehensive surveys currently exist focusing exclusively on the strengths and limitations of using neuroevolution approaches in DNNs. Prolonged absence of such surveys can lead to a disjointed and fragmented field preventing DNNs researchers potentially adopting neuroevolutionary methods in their own research, resulting in lost opportunities for improving performance and wider application within real-world deep learning problems. This paper presents a comprehensive survey, discussion and evaluation of the state-of-the-art works on using EAs for architectural configuration and training of DNNs. Based on this survey, the paper highlights the most pertinent current issues and challenges in neuroevolution and identifies multiple promising future research directions.

I. INTRODUCTION

Neuroevolution applies evolutionary algorithms to automate DNN architecture configuration and training, addressing a difficult and error-prone design process. This survey reviews the field’s methods, evidence, and open challenges across deep learning and evolutionary computation.

  • Architecture configuration and training strongly influence whether DNNs succeed or fail on applications.
  • Architecture search includes evolution-based and reinforcement-learning methods, alongside random search, Bayesian optimisation, hill-climbing, grid search, and Monte Carlo approaches.
  • Neuroevolution encompasses EA-based methods for automatically searching DNN architectures and training networks.
  • The survey considers landmark and newer EAs across autoencoders, CNNs, DBNs, RBMs, RNNs, and LSTMs.
  • Evolutionary Programming is identified as a landmark EA without known neuroevolution applications in the surveyed literature.
  • The paper aims to provide a timely, comprehensive review of neuroevolution in DNNs and identify strengths, challenges, and future research directions.

II. BACKGROUND

Deep learning uses layered neural representations to transform raw inputs into progressively more abstract features. The background introduces CNNs, autoencoders, DBNs, and related architectures used in neuroevolution.

  • Deep Neural Networks: Deep learning transforms raw data into progressively more abstract and composite representations through multiple layers.
  • Convolutional Neural Networks: CNNs process grid-like data using local receptive fields, convolutional filters, and feature maps.
  • Convolutional Neural Networks: CNN efficiency relies on sparse interaction, parameter sharing, and equivariant representations.
  • Convolutional Neural Networks: CNN architectures have expanded from early LeNet designs to deeper models such as AlexNet, with rectified linear units becoming standard in neuroevolutionary CNN design.
  • Autoencoders: Autoencoders combine encoder and decoder functions to reconstruct inputs while preserving information in an alternative representation.
  • Deep Belief Networks: DBNs support feature extraction and can be formed by stacking RBMs, but their many parameters require substantial training time.

4) Deep Learning Architecture: Other network types:

The section surveys recurrent and energy-based network types and introduces the evolutionary principles used to optimise neural systems. These architectures differ in their handling of sequences, representations, and search populations.

  • Recurrent Neural Networks: RNNs process sequences through recurrent connections and internal state, supporting temporal behaviour and sequential prediction.
  • Recurrent Neural Networks: RNN training is difficult because of exploding and vanishing gradients, and practical look-back may cover only a few steps.
  • Restricted Boltzmann Machines: RBMs are energy-based networks whose hidden-unit activities can provide training data for higher-level RBMs.
  • Restricted Boltzmann Machines: Stacking RBMs lets networks learn features from features toward higher-level representations.
  • Long Short Term Memory: LSTMs are recurrent networks designed to learn long-term dependencies using memory blocks, cells, and adaptive gating units.
  • Evolutionary Algorithms: EAs evolve populations of encoded candidate solutions across generations using fitness evaluation, selection, crossover, and mutation.

1) Evolutionary Algorithm: Genetic Algorithms (GAs):

Genetic algorithms and genetic programming provide evolutionary representations for searching neural designs. A common EA process repeatedly initializes, selects, recombines or mutates, evaluates, and outputs candidate networks.

  • Genetic Algorithms: Genetic algorithms are search methods that seek near-optimal solutions and have influenced machine learning and neural-network research.
  • Genetic Programming: Genetic programming is automated programming that commonly represents individuals as tree-like structures generated from functional and terminal sets.
  • Common EA Process: A common EA network-design process takes a dataset, generation count, population size, mutation probability, and crossover probability as inputs.
  • Common EA Process: Each generation applies selection, crossover, and mutation before evaluating the fitness of newly produced individuals.
  • Common EA Process: The final generation is returned as a set of evolved individuals with their fitness values.

3) Evolutionary Algorithm: Evolution Strategies (ES):

Evolution Strategies (ES) are evolutionary algorithms generally suited to real-valued optimisation, with mutation as the primary operator and crossover used secondarily or optionally.

  • ES were introduced in the 1960s by Rechenberg and Schwefel for real-valued optimisation problems.
  • Mutation is the main ES operator, while crossover is secondary and optional.
  • The two historical ES forms are (µ, λ)-ES and (µ + λ)-ES, distinguished by how parents and offspring participate in selection.
  • In ES notation, µ denotes the parent-population size and λ denotes the number of offspring produced before selection.
  • Evolutionary Programming differs from ES mainly because it does not use crossover and normally produces M offspring from M parents.

4) Evolutionary Algorithm:

The survey introduces several evolutionary algorithms relevant to neuroevolution, including DE, GE, and NEAT, while positioning EAs as a growing alternative for neural architecture search despite evaluation costs.

  • Differential Evolution (DE) is reported as efficient in continuous search spaces, robust, and fast-converging relative to other optimisation methods.
  • DE variants perturb population members using scaled differences between randomly selected, distinct population members.
  • Grammatical Evolution (GE) uses binary strings to select production rules in a Backus-Naur form grammar through genotype–phenotype mapping.
  • GE is presented as a special form of Genetic Programming in which evolution operates on representations rather than directly on programs.
  • NEAT evolves neural networks using historical marking, speciation, and incremental growth from a minimal starting structure.
  • EAs are gaining momentum in architecture search because they are gradient-free, population-based, and parallelisable, but their fitness evaluations can be computationally expensive.
  • With 20 individuals and 50,000 training samples, one EA generation requires one million fitness-function evaluations.

C. Deep Learning Architecture: Convolutional Neural Networks

Neuroevolutionary studies configure CNN architectures through varied encodings, search spaces, genetic operators, and hybrid training procedures, with reported competitive or encouraging results across image tasks.

  • CNN architectures have been evolved using tree-based GP, genetic algorithms, grammatical evolution, Cartesian GP, graph encodings, and constrained binary representations.
  • NASNet uses repeated Normal Cells and Reduction Cells, where Reduction Cells halve feature-map height and width while same-type cells share an architecture.
  • NASNet-inspired mutations select a random cell and pairwise combination before applying hidden-state or operation mutations.
  • Real et al. encoded architectures as graphs and found that three crossover forms did not improve mutation-based results.
  • Cartesian GP encodes CNN nodes and connections in a directed acyclic grid, with Suganuma et al. using a (1+λ) ES because individual evaluation was expensive.
  • Hybrid approaches combined evolutionary architecture or representation search with backpropagation for weight optimisation.
  • Reported experiments achieved competitive or encouraging results against state-of-the-art, reinforcement-learning, and human-designed configurations on datasets including CIFAR-10 and CIFAR-100.

D. Deep Learning Architecture: AutoEncoders

The surveyed autoencoder studies apply evolutionary methods to architecture, weights, training efficiency, and sequence-to-sequence models, spanning image restoration, language, and data-stream or fault-diagnosis applications.

  • Cartesian GP with a (1+λ) ES optimised symmetric convolutional autoencoders for image restoration and achieved competitive results without adversarial training.
  • So et al. evolved transformer networks for sequence-to-sequence language tasks using a NASNet-inspired search space seeded with a known transformer model.
  • EvoAE searches autoencoder network weights and structure simultaneously while decomposing large datasets into smaller batches to improve performance.
  • The discriminant autoencoder is a semi-supervised autoencoder proposed for fault diagnosis.
  • DEVDAN is an open-structure deep evolving denoising autoencoder designed for data-stream analytics and noise-perturbed input reconstruction.

E. Deep Learning Architecture: Deep Belief Networks

The surveyed studies apply evolutionary methods to deep belief networks and related architectures for feature extraction, prognostics, structure optimization, and recurrent-network hyperparameter selection.

  • Deep Belief Networks: Evolutionary methods were applied to deep belief networks for image feature classification and aircraft prognostic health management.EFACV uses DBN-extracted image features, while MODBNE evolves multiple DBNs simultaneously through a decomposition-based multiobjective evolutionary algorithm.
  • Deep Belief Networks: A directed-acyclic-graph representation was used with evolutionary and genetic algorithms to optimize DNN structure and parameters for phoneme recognition and spoken-digit detection.The experiments used 62 GPGPUs, with restricted Boltzmann machines in training.
  • Deep Belief Networks: Differential evolution was used to identify LSTM hyperparameters for forecasting over long time spans and nonlinear continuous functions.The study describes this as the first use of differential evolution for LSTM hyperparameter selection in forecasting applications.
  • Deep Belief Networks: The Semantic Learning Machine uses geometric semantic mutation in a 1 + λ hill-climbing strategy to generate child neural networks from a current best network.This representation focuses search on the current best network without imposing penalties for doing so.

G. Final Comments

Neuroevolution has become a practical approach for designing DNNs, using diverse evolutionary representations across several network types, while evolutionary training remains approximate and lacks convergence guarantees.

  • Final Comments: Evolution-based methods have produced state-of-the-art DNN designs using genetic algorithms, genetic programming, evolution strategies, hybrids, and diverse representations.Reported studies range from experiments using hundreds of computers to experiments using only a few GPUs.
  • Final Comments: Most neuroevolution studies focus on deep CNN design, while autoencoders, restricted Boltzmann machines, recurrent networks, LSTMs, and deep belief machines receive less attention.The survey describes only a few works involving these other network types.
  • Final Comments: The survey tables summarize evolutionary representations, individual encodings, genetic operators, parameters, datasets, and computational effort for selected neuroevolution studies.The selected papers were chosen ad hoc to demonstrate neuroevolution in DNN architecture design and training.
  • Final Comments: Evolutionary solutions are generally considered near optimal because evolutionary algorithms provide no guarantees of convergence.Population-based evolutionary methods approximate gradients from population objectives, whereas SGD computes an exact gradient.

C. Deep Learning Architecture: Convolutional Neural Networks

The surveyed CNN-related work applies evolutionary algorithms to weights, architectures, training schedules, and reinforcement-learning settings, with methods addressing large networks, limited evaluations, and sparse rewards.

  • Convolutional Neural Networks: A fixed-length genetic algorithm evolved convolutional DNN weights with more than four million free parameters and achieved competitive results against several state-of-the-art methods.The comparison included DQN, policy-gradient methods, and evolution strategies.
  • Convolutional Neural Networks: Genetic algorithms were used to evolve autoencoder weights, with individuals evaluated by training-sample root mean squared error and partly updated through backpropagation.The experiments used 10 individuals, 50% elitism, and the CIFAR-10 dataset.
  • Convolutional Neural Networks: Differentiable pattern-producing networks combine gradient-based learning with evolutionary optimization and reported superior results on MNIST.The approach uses a generic evolutionary algorithm for optimization.
  • Convolutional Neural Networks: Evolutionary methods can tolerate sparse rewards and long time horizons, although they can struggle when optimization becomes difficult.This contrast is discussed in relation to the sensitivity and brittle convergence properties of deep reinforcement learning methods.
  • Convolutional Neural Networks: Neuroevolution was also used to train a modular memory unit that separates memory from central computation without costly memory-management strategies.The modular memory unit is a memory-augmented recurrent architecture.

F. Final Comments

The survey presents neuroevolution as a renewed alternative to gradient-based training and identifies computational cost as a central limitation, with surrogate models offering one documented reduction in training time.

  • Final Comments: The research community largely shifted toward stochastic gradient descent and backpropagation, but evolutionary computation remains a distinct search paradigm for DNN learning.Comparisons require caution because a neuroevolutionary generation is not directly comparable to a gradient-descent epoch.
  • Final Comments: The selected DNN-training studies consistently use mutation and selection, sometimes omit crossover, and cover more varied, domain-specific datasets and problems.The survey summarizes these studies in a table of representations, operators, parameters, datasets, and computational effort.
  • Final Comments: Evolutionary algorithms have successfully designed DNN architectures, but training those methods commonly requires days to weeks to obtain competitive results.This training-time limitation is identified across state-of-the-art architecture-design methods.
  • Final Comments: Surrogate-assisted evolutionary computation reduces fitness evaluations by estimating candidate fitness with efficient meta-models.The approach is intended to accelerate evolutionary search while maintaining accurate fitness estimates.
  • Final Comments: An ensemble-based surrogate reduced CNN training time from 33 GPU days to 10 GPU days while retaining competitive accuracy against state-of-the-art algorithms.The reported limitation is that the number of training runs needed for good prediction performance remains unknown.

B. Mutations and the neutral theory

The paper identifies underexplored questions about neutrality, multi-objective optimisation, and genetic-operator choice in neuroevolution for DNNs. It argues that these issues require problem-specific analysis rather than general assumptions.

  • B. Mutations and the neutral theory: Researchers have used selection and mutation, and often crossover, to evolve DNN architectures or train DNNs.The paper notes that crossover can be difficult depending on the encoding.
  • B. Mutations and the neutral theory: Neutrality cannot be judged as universally beneficial or harmful because its effect depends on the problem class, representation, and genetic operators.The paper reports no known neuroevolution studies focused specifically on neutrality in DNNs.
  • B. Mutations and the neutral theory: If neutrality is beneficial in a particular setting, it may reduce training time by avoiding evaluation of some candidate solutions.This proposed benefit is conditional on the problem, representation, and operators used.
  • C. Multi-objective Optimisation: Most reviewed neuroevolution studies optimise a single objective, although DNN configuration can require simultaneous optimisation of conflicting objectives.Multi-objective optimisation seeks trade-off solutions when no single global optimum exists.
  • D. Fitness Landscape Analysis of DNNs and Well-posed Genetic Operators: No reviewed neuroevolution study explains why a particular genetic operator is well suited to its specific DNN problem.The paper notes that standard and specialised crossover variants have both been used.

E. Standardised Scientific Neuroevolution Studies in DNNs

The paper argues that neuroevolution studies should diversify beyond common DNNs, datasets, and image-classification benchmarks. It presents standardisation and open data as important for meaningful comparisons across studies.

  • E. Standardised Scientific Neuroevolution Studies in DNNs: Researchers use multiple DNN architectures and architectural elements, while EAs can flexibly combine methods and variants.The passage notes that this flexibility also creates many possible configuration choices.
  • E. Standardised Scientific Neuroevolution Studies in DNNs: Large datasets and stronger computational resources have enabled DNNs to address difficult problems in domains including image classification and speech processing.
  • F. Diversifying the use of benchmark problems and DNNs: Image classification is a primary DNN benchmark, but expanding to other domains is increasingly important for evaluating neuroevolution more broadly.The paper describes benchmark datasets as tools for comparing experimental setups across research groups.
  • F. Diversifying the use of benchmark problems and DNNs: Open benchmark datasets are critical because differing data and classification settings make published results difficult to compare.Proprietary data can further prevent direct comparison between studies.
  • VI. CONCLUSIONS: The survey concludes that EAs are a competitive and successful means of automatically creating and configuring DNNs.It targets researchers and practitioners in both Evolutionary Computation and Deep Learning.
Loading 2006.05415v1…