Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities

1. Introduction

• EA’s low sampling efficiency becomes more consequential as the solution space grows, motivating efforts to improve search.The paper describes RL as a learning approach that can use data generated during EA iterations.Source: §1 · p. 2
• RL-EAs combine reinforcement learning with evolutionary algorithms and have been used to address a wide range of complex optimization problems.The paper notes RL’s decision-making in complex and stochastic environments as motivation for this integration.Source: §1 · p. 2
• The paper situates RL-EA research within growing interest in the area, with particular emphasis on the three years preceding August 4, 2023.The authors attribute the growth to researchers’ increased understanding of these algorithms.Source: §1 · p. 2
• The survey covers RL-EA frameworks, implementation strategies, applications, and future research directions, including how RL assists EA search.It also categorizes application domains and collects public datasets used in prior work.Source: §1 · p. 2

2.1. Evolutionary Algorithms

• EAs iteratively initialize populations, generate offspring, select individuals for the next population, and stop when a termination condition is met.They can find approximate optima through population-based exploration but do not guarantee optimality on complex problems.Source: §2.1 · p. 3
• The survey highlights GA operations and problem-specific EA configuration, alongside variants such as MA, NSGA-II, and NSGA-III.In GA, candidate solutions are chromosomes, and crossover and mutation are primary evolutionary operations.Source: §2.1 · p. 3

2.2. Reinforcement Learning

• In RL, an agent selects actions using the current state, receives rewards through environmental interaction, and updates its policy to maximize rewards.The framework is described as an MDP comprising state and action spaces, a state-transfer function, reward function, and discount factor.Source: §2.2 · pp. 3–4
• The survey focuses on model-free RL, including policy-based methods and value-based methods such as Q-learning and DQN.Q-learning uses a Q-table and suits limited state spaces, while DQN uses artificial neural networks instead of Q-tables.Source: §2.2 · p. 4

3. The Taxonomy of RL-EA

• The RL-EA taxonomy covers integration approaches, assistance strategies, RL attribute settings, and applications across optimization tasks.The survey uses these categories to structure its discussions of how RL and EA interact, how RL assists EA, and where RL-EA is applied.Source: §3 · p. 4

4. Integration Methods for RL and EA

• RL-EA integration distinguishes direct interaction with EA search from indirect decisions over ensemble strategies used during search.The survey treats the form of information interaction as a prerequisite for RL assistance.Source: §4 · p. 4, §4.1 · p. 4, §4.2 · p. 4

4.1. Direct Integration

• Direct integration applies RL actions to EA search and uses search-performance feedback to update the RL model.Its simple structure makes states, actions, and rewards easy to define, but it generally regulates rather than replaces the EA search strategy.Source: §4.1 · p. 4

4.2. Indirect Integration

• Indirect integration has RL select ensemble strategies for EA search, then update those strategies based on their performance.Ensembles can incorporate multiple algorithms, operators, and other operations.Source: §4.2 · pp. 4–5
• Indirect integration can improve robustness through diverse strategies, but complex structures may consume substantial resources or mismatch strategies with problem features.Adaptive strategy choice using previous and current search information is proposed to reduce these risks.Source: §4.2 · p. 6
• RL models may be trained independently or with data generated during iterative RL-EA search; actual problem-solving requires no further RL training.Training data or scenarios differ from those used in actual problem-solving and need additional generation methods.Source: §4.2 · p. 6

5. RL Assistance Strategies Used in RL-EA

• RL assistance strategies include solution generation, learnable objective functions, selection of algorithms, populations, or operators, parameter adaptation, and other approaches.The survey also discusses strategies assisted by deep reinforcement learning.Source: §5 · p. 6

5.1. Generating Solution

• RL generates complete or partial solutions for EA search, either for the target problem or for one of its subproblems.For TSP, a pointer network decodes population individuals into routes, a method reported to reduce iterative search time.Source: §5.1 · p. 6

5.2. Learnable Objective Function

• A learnable objective function incorporates explored information and problem characteristics to guide EA search, with inverse reinforcement learning commonly used to obtain its reward function.One reported IRL-based moth-flame optimization approach used four reward functions and addressed large-scale real-parameter optimization problems.Source: §5.2 · pp. 6–7

5.3. Algorithm Selection

• RL-assisted algorithm selection chooses low-level heuristics for exploration or exploitation based on problem characteristics, but adding heuristics may increase search time.Reported approaches include Q-learning, A3C, DDQN, and dueling double deep Q networks.Source: §5.3 · p. 7

5.4.1. Adaptive Crossover

• RL adaptively selects crossover operators, including through Q-learning before each generation and DQN-based selection within MOEA/D.The cited GA approach used seven crossover operators; the MOEA/D approach used four candidate operators per mating pool.Source: §5.4.1 · pp. 7–8

5.4.2. Adaptive Mutation

• For differential evolution, RL selects among two or three mutation operators, retaining low computational complexity and global search performance.The survey also describes Q-learning for mutation-operator choice and consideration of problem-related heuristic operators.Source: §5.4.2 · p. 8

5.4.3. Adaptive Local Search

• Among five hybrid algorithms and three Q-learning local-search strategies, PSO combined with Q-learning yielded the optimal optimization performance.The comparison used six local-search operators to update individuals’ neighborhood structures based on agent decisions.Source: §5.4.3 · p. 9
• RL selects local-search operators across EA frameworks, while perturbation operators may be chosen to avoid local optima.One perturbation-based approach updated the known optimal solution in 71% of benchmark instances; operator selection studies predominantly use value-based RL.Source: §5.4.3 · pp. 8–9

5.5. Sub-population Selection

• In cooperative multi-population EA, Q-learning adjusts the number of individuals in each of seven sub-populations by increasing, decreasing, or maintaining their sizes.The approach was validated through simulation experiments and real-life examples.Source: §5.5 · p. 9
• In RL-assisted genetic programming, an ensemble population strategy presents four competing sub-populations and selects one for each search round.The populations include common, single/double elite, and mixed common-and-elite compositions.Source: §5.5 · p. 10

5.6. Parameter Adaptation

• Traditional EA tuning can restrict applicability or require complex processes and many parameters, while RL can adapt controls to problems and search performance.The passage describes parameter control as important to balancing early global exploration with later local exploitation.Source: §5.6 · p. 10
• In DE, RL methods range from adjusting a scale factor or choosing preset configurations to directly outputting control-parameter values through a policy network.The cited examples include scale-factor adjustment, seven configuration sets [90], and direct parameter output [91].Source: §5.6 · p. 10
• For GA, Q-learning selects among 18 experimentally determined parameter sets, while a GRU estimates crossover and mutation probabilities from population evolution.The reported overall trend is decreasing crossover probability and increasing mutation probability; the rates affect population diversity and exploration–exploitation balance.Source: §5.6 · p. 10, Figure 9 · p. 10

5.7. Other Strategies

• These designs are developed for specific problem scenarios, with the survey describing their optimization performance as superior.The authors identify problem design, fitness evaluation, and individual processing as ideas worth studying.Source: §5.7 · p. 10
• Q-learning can select a problem-domain subspace for targeted DE search, while RL can reduce the solution space by matching problem rules or constraints.These approaches analyze or process the solution space to enhance search efficiency.Source: §5.7 · p. 10
• RL can also mine sparse variables to reduce problem dimensionality and select among auxiliary fitness-assessment methods for evaluating population performance.The fitness-selection mechanism uses multiple auxiliary fitness functions to enhance algorithm efficiency.Source: §5.7 · p. 10

5.8. Discussion of DRL-assisted Strategies

• DRL supports auxiliary EA tasks such as operator selection, solution generation, and parameter optimization, and its neural models can set parameters from predicted outputs rather than preset combinations.The passage contrasts simple RL’s predefined parameter combinations with DRL’s predicted values within reasonable intervals.Source: §5.8 · pp. 10–11
• Listed strategy examples include algorithm probability-distribution choice, matching problem rules or constraints, sparse-variable mining, and multi-exemplar selection.The supplied examples describe these as auxiliary strategy tasks or design choices.Source: §5.8 · p. 11
• DRL training requires substantial data, time, and computational resources, and even sufficient data may not yield satisfactory outcomes.The survey also describes deep models as black boxes whose optimal training effect is difficult to guarantee.Source: §5.8 · pp. 11–12
• The survey warns that deep-model predictions alone may lack environmental perception and reflection, limiting generalization to novel problems.It advises evaluating overall EA performance rather than pursuing optimal DRL performance alone.Source: §5.8 · p. 12
• Simple RL may suffice for low-complexity tasks, while complex tasks or extensive state spaces may challenge simple RL; DRL also trades training cost against model effectiveness.The survey says DRL does not inevitably outperform simple RL and recommends considering cost, utilization efficacy, and generalizability when choosing.Source: §5.8 · p. 12

6.1. Setting of State

• Chromosomes can represent states directly, but large Q-tables can hinder sampling; population-level states, discretization, and ANN inputs offer alternatives for different state spaces.Discretization lets Q-learning handle continuous states to some extent but is limited to fewer dimensions, whereas ANN models address decision challenges from expanding state dimensions.Source: §6.1 · p. 12

6.2. Setting of Reward Function

• Reward designs include fitness changes between generations, adjusted or ratio-based rewards, performance mappings, and rewards autonomously explored during IRL training.Population-level rewards can remain independent of population size, whereas individual-level evaluations increase proportionally with population size; choosing attributes can be time-consuming and problem-specific.Source: §6.2 · pp. 12–13

6.3. Principle of Action setting

• Action design should remain simple, fit the EA’s search framework, and improve performance; actions may be selected before each generation, with complexity weighed against gains.GA examples use RL for diverse search strategies, exploration–exploitation balance, and neighborhood search.Source: §6.3 · p. 13

7.2. Combinatorial Optimization Problems

• RL-EA applications span continuous SOP, MOP, and MMOP, as well as discrete combinatorial problems including scheduling and routing; many COPs are NP-hard.The survey focuses its detailed COP discussion on production scheduling and vehicle routing, with other COPs treated more briefly.Source: §7.1 · p. 13, §7.2 · p. 15

7.1. Continuous Optimization Problems

• MOEA/D-DQN shows superior overall performance across 34 MOP benchmark functions, balancing global and local search through adaptive operator selection.The comparison is reported in Table 5.Source: §7.1 · p. 15, Table 5

7.2.1. Production Scheduling Problem

• CMAQ achieves the highest average HV values across ssd100 and ssd125 sizes, except for size (100, 5), in two energy-efficient distributed no-wait flow-shop cases.The comparison is reported in Table 7.Source: §7.2.1 · p. 15, Table 7 · p. 18

7.2.2. Vehicle Routing Problem

• HHASA and its variants find better planning results in 7 of 10 large CEVRP examples, while basic HHASA beats comparison algorithms in 6.The comparison is reported in Table 9.Source: Table 9 · p. 19, §7.2.2 · p. 19
• For VRPs, RL can adaptively select heuristic rules based on their iterative-search performance, and established RL methods can be integrated into EAs to expedite solution generation.The survey notes that this combination can facilitate rapid algorithm design while maintaining solution effectiveness.Source: §7.2.2 · p. 19

7.2.3. Other Combinatorial Optimization Problems

• On the TSP benchmark, RHGA significantly outperforms the comparison algorithms and approaches the known optimal solution for finding the shortest route that visits all cities and returns to the starting city.Source: §7.2.3 · p. 19
• RL-EA is also reported to outperform state-of-the-art algorithms in EOSSP scheduling, despite many EOSSPs being NP-hard.AEOS has a longer visible time window than CEOS.Source: §7.2.3 · pp. 19–20

7.3. Open Datasets in Main Application Domains

• CEC benchmarks are widely used for continuous optimization, while open datasets support assessment of objective value, convergence rate, and solution speed.PlatEMO provides access to evolutionary multi-objective optimization benchmarks.Source: §7.3 · p. 21

Integration of advanced RL models

• Explore advanced RL models, while simplifying their complex networks and large parameter sets to fit the functions they serve within EA.The survey expects advanced models to support exploration, exploitation, and generalization across problem scenarios.Source: §8 · p. 21

Assistance strategy based on multi-agent RL

• Investigate multi-agent RL to coordinate agents assisting sub-populations or individuals and to combine diverse search strategies.The survey describes information synchronization and ensemble strategies as potential benefits.Source: §8 · pp. 21–22

More complex algorithmic mechanism design

• Design richer RL-assisted mechanisms, including subspace selection, MOP direction choice, offspring retention, and interactions that preserve stability and search diversity.Many existing mechanisms resemble ensemble or traditional adaptive strategies.Source: §8 · p. 22

Digging deeper into the problem characteristics

• Use problem characteristics to guide search, and assess whether information discovered by the algorithm is valuable enough to inform later searches.The survey notes inverse RL as a possible fit for optimization problems.Source: §8 · p. 22

RL-assisted local search and population search information interaction

• Connect RL-assisted local search with population search, since an incorrectly directed local search can fail to improve outputs and may worsen search performance.The proposed information exchange should help the algorithm proceed in a more optimized direction.Source: §8 · p. 22

Low-overhead RL training methods; Efficient RL hyper-parameter setting methods

• Reduce RL training costs and make hyperparameter setting more efficient, since existing training is costly and integration adds parameters such as learning rate and discount factor.Suggested options include evolutionary-strategy training, pre-training, and transfer learning.Source: §8 · pp. 23–24

Theoretical analysis of RL-EA

• Analyze RL-EA components and whole algorithms theoretically; current work predominantly proposes mechanisms and validates them through simulations on specific problems.The survey argues theoretical analysis could guide later algorithm design.Source: §8 · p. 23

Benchmark for RL-EA

• Develop RL-EA-specific benchmarks because traditional EA benchmarks are insufficient to test newly proposed RL-EAs.One proposed route is to modify existing EA benchmarks.Source: §8 · p. 24

• The survey distinguishes direct and indirect RL-EA integration, helping researchers establish algorithm structures before focusing on strategy design.Source: §9 · p. 24
• It reviews RL-assisted solution generation, fitness evaluation, operator selection, and parameter control, alongside other RL attribute settings.Source: §9 · p. 24
• Its application analysis finds progress across domains but substantial room for further development, and it summarizes challenges and potential future directions.Source: §9 · p. 24