PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

Amjad Yousef Majid, Serge Saaybi, Tomas van Rietbergen, Vincent Francois-Lavet, R Venkatesha Prasad, Chris Verhoeven

arXiv:2110.01411v1cs.LGcs.AI

TL;DR

The paper addresses how to understand the relative strengths and limitations of DRL and ESs when choosing approaches for sequential decision-making. It surveys their foundations, compares their capabilities and applications, and considers hybrid systems; it reports distinct limitations for each approach and argues that hybridization has high potential for real-world agents.

  • Problem

    DRL and ESs have different strengths and weaknesses, and a broad comparison is needed to inform algorithm choice across sequential decision-making settings.

    Source: §I · p. 1

  • Method

    The paper surveys DRL and ES foundations, compares learning aspects and applications, and discusses hybrid systems combining both approaches.

    Source: §I · pp. 1–2

  • Results

    DRL has temporal credit-assignment and hyperparameter-sensitivity challenges and may have brittle exploration, while ESs have low data efficiency and struggle with large optimization tasks.

    Source: A. Comparison · p. 17

  • Takeaways & Limitations

    The authors believe hybridizing DRL and ESs has high potential to drive the development of agents that operate reliably and efficiently in the real world.

    Source: §VIII · p. 20

    DRL sample efficiency remains a challenge because many environment interactions may be infeasible for computational or interaction-availability reasons.

    Source: §VII · p. 19

Abstract

from arXiv · show

Deep Reinforcement Learning (DRL) and Evolution Strategies (ESs) have surpassed human-level control in many sequential decision-making problems, yet many open challenges still exist. To get insights into the strengths and weaknesses of DRL versus ESs, an analysis of their respective capabilities and limitations is provided. After presenting their fundamental concepts and algorithms, a comparison is provided on key aspects such as scalability, exploration, adaptation to dynamic environments, and multi-agent learning. Then, the benefits of hybrid algorithms that combine concepts from DRL and ESs are highlighted. Finally, to have an indication about how they compare in real-world applications, a survey of the literature for the set of applications they support is provided.

I. INTRODUCTION

The survey compares DRL and ESs across their strengths, weaknesses, learning aspects, and applications, while considering hybrid systems and broader developmental directions. It also introduces the foundations of RL and DRL, including their objectives and algorithm families.

I. INTRODUCTION

  • The survey contrasts DRL and ESs across scalability, exploration, dynamic-environment adaptation, and applications, and discusses how hybrids can combine their advantages.Unlike surveys focused on individual algorithms or narrower topics, it takes a bird’s-eye view of developmental directions.Source: §I · pp. 1–2

A. Reinforcement Learning

  • RL formalizes an agent interacting with an environment to maximize reward, commonly modeled as an MDP with state, action, transition, and reward components.The return weights immediate and future rewards with a discount factor; value functions estimate expected return for states or actions.Source: A. Reinforcement Learning · p. 2
  • Policy-based methods directly optimize a policy and accommodate continuous, discrete, or mixed action spaces, but are generally high-variance and sample-inefficient.Value-based methods learn a value function and extract a policy; they are more sample-efficient than policy-based methods, though convergence is not guaranteed under ordinary circumstances.Source: A. Reinforcement Learning · p. 2
  • Actor-critic architectures combine a policy-learning actor with a value-based critic that evaluates the actor’s actions.Model-based algorithms use or learn transition dynamics to imagine action consequences and make better use of gathered data, though realistic models can be difficult to produce.Source: A. Reinforcement Learning · pp. 2–3
  • DRL combines deep learning with RL, using neural networks to approximate learnable functions and organizing algorithms into value-based, policy-based, and model-based families.A policy-based network maps environmental states to actions, with its parameters optimized by backpropagation.Source: A. Reinforcement Learning · p. 3

B. Evolution Strategies

Evolution Strategies are population-based, gradient-free black-box optimizers that bias candidate sampling toward better-performing solutions. Their iterative process generates, evaluates, and selects candidates until a satisfactory solution is found.

B. Evolution Strategies

  • ESs are population-based black-box optimizers for continuous search problems that require fitness evaluation, but not an MDP formulation or a differentiable objective.The objective function must assign a fitness value to each input.Source: B. Evolution Strategies · p. 3
  • ESs bias candidate sampling toward better individuals, often by adjusting a normal distribution’s mean and standard deviation as strategic parameters.These parameters can be modified online to make search more efficient.Source: B. Evolution Strategies · p. 3
  • Each generation initializes a population, selects parents, recombines and mutates offspring, evaluates fitness, and retains the best individuals for the next generation.Mutation can add noise sampled around the mean produced by recombination; the process repeats until a satisfactory solution is found.Source: B. Evolution Strategies · p. 3

C. Comparison

The comparison distinguishes RL’s reward-based interaction and policy learning from ESs’ fitness-based search over candidate solutions. It also introduces foundational RL algorithms and the mechanisms used in representative methods.

C. Comparison

  • An RL objective of maximizing discounted rewards can be converted to an ES objective by using a terminal-state reward equivalent to the fitness function.ESs maintain multiple candidate solutions, whereas RL generally improves one agent’s policy through environment interaction.Source: C. Comparison · p. 3
  • ESs most resemble policy-based DRL because both search parameter spaces for objective-optimizing functions, but ESs use neither gradients nor backpropagation.The compared objectives are expected return for DRL and fitness score for ESs.Source: C. Comparison · p. 3
  • Value-based RL usually uses discrete action spaces, actor-critic extends RL to continuous actions, and ESs can handle discrete or continuous actions by default.Source: C. Comparison · p. 3

A. Reinforcement Learning Algorithms

  • SARSA is on-policy and follows the policy it evaluates, while Q-learning is off-policy and learns an optimal Q-value function from data collected by any policy.Off-policy algorithms are more data-efficient than on-policy ones because they can reuse collected data.Source: A. Reinforcement Learning Algorithms · p. 4
  • DQN combines Q-learning with a CNN for high-dimensional inputs and uses experience replay and a periodically updated target network to address instability.It takes a state as input and produces Q-values for possible actions.Source: A. Reinforcement Learning Algorithms · p. 4

B. Evolutionary Strategies Algorithms

The section introduces evolutionary strategies through mutation and selection, then describes population-based variants that adapt search parameters or distributions. It also outlines Natural Evolution Strategies’ gradient-based search updates.

(1+1)-ES

(μ/ρ+, λ)-ES

  • The (μ/ρ+, λ)-ES generates λ offspring from μ parents using recombination and mutation, with comma and plus variants differing in parent selection.Each individual carries strategy parameters that evolve through self-adaptation, removing the need for external adjustment settings.Source: B. Evolutionary Strategies Algorithms · p. 5

CMA-ES

  • CMA-ES [40, 41, 42, 43] samples offspring from a multivariate normal distribution and updates its mean, stepsize, and covariance toward fitter candidates.Its covariance update combines rank-1 and rank-μ components, while cumulative stepsize adaptation adjusts mutation strength.Source: B. Evolutionary Strategies Algorithms · p. 5

NES

  • NES samples candidates from a search distribution, estimates a gradient from their fitness, and applies gradient ascent to update distribution parameters.Salimans et al. estimate gradients for policy parameters by Gaussian smoothing the expected-return objective.Source: B. Evolutionary Strategies Algorithms · p. 5

C. Comparison

DRL and ES methods face trade-offs in data use and computation, while parallel variants distribute experience, gradients, or parameter updates in different ways. The section surveys parallel DRL architectures and their design choices.

C. Comparison

  • On-policy RL and ES are data-inefficient, while value-based DRL can reuse off-policy data but may become unstable over long horizons and high discount factors [45].ES often needs less computation per iteration because it avoids backpropagation; policy-based RL and ES both directly search for policies.Source: C. Comparison · p. 5

A. Parallelism

  • Parallel execution accelerates computationally intensive DRL and ES, motivating parallel methods for sequential decision-making.The comparison covers computation parallelization, exploration, and learning in multi-agent and dynamic settings.Source: A. Parallelism · p. 5, §IV · p. 5

1) Parallelism in Deep Reinforcement Learning

  • Gorila distributes actors, learners, a parameter server, and replay memory, whereas A3C uses CPU-thread agents with separate networks updating a global network asynchronously.Replay can stabilize updates but costs memory and computation and is limited to off-policy algorithms; A3C reduces distributed communication costs.Source: 1) Parallelism in Deep Reinforcement… · p. 6, Figure 3 · p. 6
  • Batched A2C synchronizes updates through a master node, while Ape-X distributes actors around a prioritized replay buffer and R2D2 adds recurrent agents.Batched A2C supports scalable actors on one machine but incurs higher communication costs; R2D2 is reported to outperform Ape-X.Source: 1) Parallelism in Deep Reinforcement… · p. 6
  • IMPALA uses V-trace to address actor-learner lag and supports synchronized learners; SEED moves inference to the learner, with actors sending observations and actions.The separation of acting and learning with V-trace supports stable learning and high throughput in IMPALA.Source: 1) Parallelism in Deep Reinforcement… · p. 7

2) Parallelism in Evolution Strategies

Parallel ES reduces communication through shared random seeds, while the exploration discussion surveys methods for balancing novelty-seeking with reward pursuit. It also notes a benchmark result for Agent57.

2) Parallelism in Evolution Strategies

  • OpenAI-ES [4] shares random seeds so workers exchange scalar fitness values rather than full parameter or gradient vectors.Workers sample noise, evaluate fitness, reconstruct perturbations from seeds, and update policy parameters.Source: 2) Parallelism in Evolution Strategies · p. 7, 3) Comparison · p. 7
  • NS-ES and its QD variant improve ES performance on sparse-reward RL tasks, while TRES reuses samples and PEL progresses from easier to harder tasks.TRES also uses shared random seeds for extremely low bandwidth; PEL follows OpenAI-ES parallelization.Source: 2) Parallelism in Evolution Strategies · p. 7

B. Exploration

  • The exploration-exploitation dilemma is choosing when to try suboptimal actions to improve policy estimates versus using the current estimate to make progress.It arises when agents interact with partially known environments.Source: B. Exploration · p. 8

1) Exploration in (Deep) Reinforcement Learning

  • UCB favors high estimated value or uncertainty, while Thompson sampling [60] adapts a parameter distribution toward promising choices; both become less exploratory over time.UCB’s reward bonus decreases as a state is visited more often; these methods are described as more efficient than ϵ-greedy [1].Source: 1) Exploration in (Deep) Reinforcement… · p. 8
  • High-dimensional or continuous DRL state-action spaces make original count-based UCB and state-action-distribution Thompson sampling impractical, prompting approximation-based methods.Pseudo-count approaches use density models or hashed state codes to produce bonuses that encourage visits to less-visited states.Source: 1) Exploration in (Deep) Reinforcement… · p. 8
  • Bootstrapped DQN samples an ensemble Q-function per episode for temporally extended exploration, which is especially relevant under sparse rewards.Related approaches combine ensemble uncertainty with UCB [69] or propagate Q-value uncertainty using Thompson sampling.Source: 1) Exploration in (Deep) Reinforcement… · pp. 8–9
  • Information-gain methods reward novelty using model uncertainty, prediction error, or feature-prediction error.Examples include Bayesian transition models, forward models with inverse-model filtering, and fixed randomly initialized feature networks.Source: 1) Exploration in (Deep) Reinforcement… · p. 9
  • Agent57 outperforms the standard human benchmark on all 57 Atari games.Memory-based approaches include episodic novelty, Go-explore’s two phases, and NGU’s episodic and lifelong novelty rewards; Agent57 adaptively selects exploration policies.Source: 1) Exploration in (Deep) Reinforcement… · p. 9

2) Exploration in Evolution Strategies

ES exploration methods perturb and recombine candidate solutions, but can still become trapped in local optima. The survey describes techniques that direct or structure search, while noting that high-dimensional exploration can remain sample-inefficient.

2) Exploration in Evolution Strategies

  • OpenAI-ES smooths a nonsmooth objective by adding Gaussian noise to parameters and uses stochastic gradient ascent, with auxiliary techniques for exploration, variance reduction, and local-optimum avoidance.Its listed techniques are virtual batch normalization, antithetic sampling, and fitness shaping.Source: 2) Exploration in Evolution Strategies · p. 9
  • Structured exploration uses orthogonal or Quasi Monte Carlo directions, while compact policies reduce problem dimensionality through parameter sharing without losing accuracy.These approaches were proposed to improve derivative-free optimization methods such as OpenAI-ES.Source: 2) Exploration in Evolution Strategies · p. 9
  • Guided ES samples a low-dimensional subspace informed by surrogate gradients to reduce search-direction variance, but depends on known gradient bias and can fail to improve on the surrogate direction when bias is small.A momentum-inspired follow-up combines surrogate and random-search directions and was reported superior to a standard ES on assessed tasks.Source: 2) Exploration in Evolution Strategies · p. 9
  • ASEBO adapts subspace dimensionality using prior gradient estimates, while SGES samples probabilistically from a guiding subspace built using historical estimated gradients.SGES was reported to outperform OpenAI-ES, Guided ES, CMA-ES, and vanilla ES.Source: 2) Exploration in Evolution Strategies · pp. 9–10
  • DGS-ES uses one-dimensional non-local exploration along orthogonal directions and Gauss-Hermite quadrature to address high-dimensional exploration and convergence speed.The survey reports comparisons with methods including OpenAI-ES and ASEBO.Source: 2) Exploration in Evolution Strategies · p. 10
  • NS-ES, NSR-ES, and NSRA-ES combine ES with novelty and quality-diversity signals to encourage exploration under sparse or deceptive rewards.NSR-ES combines reward and novelty; NSRA-ES dynamically adapts their gradient weights.Source: 2) Exploration in Evolution Strategies · p. 10

3) Comparison

  • Recombination and mutation may help ES algorithms suffer less from local optima than DRL, but high-dimensional exploration can still produce high-variance gradient estimates and poor sample efficiency.The survey identifies sparse or deceptive rewards as a setting requiring more sophisticated exploration, while noting that real-world efficacy is mostly unknown because benchmarks are largely simulated or game-based.Source: 2) Exploration in Evolution Strategies · p. 10

C. Non-Markov settings

The section considers settings beyond standard MDPs, including partial observability and adaptation across task distributions. It describes recurrent and gradient-based Meta-RL approaches, alongside limitations involving memory, task complexity, and computation.

1) Partially Observable

  • When an environment is partially observable, agents may need histories of observations, actions, and rewards to estimate hidden states; recurrent neural networks are one DRL approach to this problem.These settings are commonly modeled as partially observable Markov decision processes.Source: 1) Partially Observable · p. 11

2) Meta Reinforcement Learning

  • Meta-RL learns policies that generalize across task or environment distributions, then adapts them to a particular task sampled from a similar, unseen distribution.The section distinguishes recurrent and gradient-based learners.Source: 2) Meta Reinforcement Learning · p. 11
  • Recurrent Meta-RL trains hidden states across tasks and uses them to adapt to new observations, while an outer loop tunes an inner learner across sampled environments.RNN-based methods can lose performance as task complexity and temporal dependencies grow, and may struggle with short-term memory or identifying specific prior experiences.Source: 2) Meta Reinforcement Learning · p. 11
  • SNAIL combines temporal convolutions, which aggregate past experience, with attention, which can pinpoint particular information.The survey notes that selecting appropriate temporal-convolution block architectures remains challenging.Source: 2) Meta Reinforcement Learning · pp. 11–12
  • MAML learns an initialization whose parameters can be adapted to a sampled task through a few gradient steps, with the goal of modest loss after the updates.Reptile is described as a first-order approximation that avoids second derivatives and is less computationally demanding.Source: 2) Meta Reinforcement Learning · p. 12

3) Meta Evolution Strategies

The section surveys ES-based approaches to Meta-RL, including methods that promote behavioral diversity and methods that replace or modify gradient-based adaptation. It also describes sim-to-real applications and continuing sample-efficiency challenges.

3) Meta Evolution Strategies

  • Evolvability ES combines evolvability search, ES, and MAML to seek parameter vectors whose perturbations produce behavioral diversity, and is reported to support adaptation and generalization.Quality Evolvability ES optimizes task performance and evolvability together because the original method is restricted to problems where the two align.Source: 3) Meta Evolution Strategies · p. 12
  • ES-MAML uses ES to solve MAML problems without estimating second derivatives, and was reported competitive with other Meta-RL algorithms.A Hill-Climbing adaptation variant was demonstrated with a physical legged robot.Source: 3) Meta Evolution Strategies · p. 12
  • IW-IES weights offspring by newly acquired knowledge using instance novelty or instance quality, and was reported competitive with ES-MAML on robot navigation tasks.The method is presented as an adaptable and scalable ES-based meta-learner.Source: 3) Meta Evolution Strategies · p. 12
  • Meta-RL supports sim-to-real adaptation by training a general policy from simulation experience and fine-tuning it through real-world interaction.The survey cites physical-robot examples involving a legged millirobot, hockey-puck hitting, and a legged robot.Source: 3) Meta Evolution Strategies · pp. 12–13

4) Comparison

  • Meta-RL methods face challenges including derivative estimation, variance, and computation, while ES-based meta-RL also faces sample-efficiency challenges.The survey characterizes gradient-based and recurrent models as the two main Meta-RL approaches.Source: 3) Meta Evolution Strategies · pp. 12–13

D. Learning in multiagent settings

Multi-agent DRL addresses sequential decisions by multiple agents sharing an environment, where joint actions affect states and rewards. The section surveys coordination and control choices, together with challenges from non-stationarity, credit assignment, and scaling.

1) multi-agent Reinforcement Learning

  • In MADRL, agents’ joint actions affect the shared environment and rewards, so the single-agent MDP model does not directly apply; Markov games generalize MDPs to capture this interaction.Cooperative agents seek a common reward, whereas competitive agents may receive differing rewards from joint outcomes.Source: 1) multi-agent Reinforcement Learning · p. 13
  • Cooperative multi-agent credit assignment asks how to divide a team reward among learners when complex interactions make individual contributions difficult to determine.Examples of cooperative settings include foraging, exploration, and warehouse robots.Source: 1) multi-agent Reinforcement Learning · p. 13
  • Centralized control uses joint information but can become computationally intractable as agent numbers grow and creates a single point of failure.Decentralized control relies on local information and can offer greater scalability and robustness, while increasing system complexity.Source: 1) multi-agent Reinforcement Learning · p. 14
  • Concurrent learning makes the environment non-stationary, so convergence guarantees for algorithms such as Q-learning may no longer hold and replay data can become outdated.Lenient-DQN uses decaying temperature values to adjust policy updates from replay memory.Source: 1) multi-agent Reinforcement Learning · p. 14
  • Centralized training can address non-stationarity, but joint state-action spaces grow exponentially with agent numbers, motivating centralized training with decentralized execution.Independent learning avoids the scalability problem but makes the environment non-stationary from each agent’s perspective and conflicts with experience replay.Source: 1) multi-agent Reinforcement Learning · p. 14
  • MADDPG uses local observations for actors and globally observable actions, observations, and target policies for the critic during training to address non-stationarity.The survey says global critic observability makes the environment stationary despite changes in other agents’ policies.Source: 1) multi-agent Reinforcement Learning · p. 14
  • Value decomposition combines centralized action-value learning with local agent values, seeking coordination without the difficulty of learning a large joint value function.VDN assumes additive decomposition, while QMIX relaxes some additivity constraints and enforces positive mixer weights.Source: 1) multi-agent Reinforcement Learning · p. 14
  • RIAL and DIAL learn discrete communication protocols, while CommNet uses a continuous channel; the survey reports improved performance over non-communicating agents.DIAL passes gradients through the communication channel during learning.Source: 1) multi-agent Reinforcement Learning · p. 15

2) Multi-agent Evolution Strategies

Evolution Strategies are used in multi-agent learning for swarm and adversarial tasks, while PES-VD combines parallel ES with value decomposition to address shared-reward credit assignment. The survey also identifies multi-agent ES as a growing area with potential for collective robotics and cloud robotics.

2) Multi-agent Evolution Strategies

  • ES controllers have been applied to robotic swarms, including LED-based congestion management and swarm foraging or aggregation tasks.The aggregation study assessed CMA-ES, PEPG, SES, GA, and OpenAI-ES; another combined DRL with CMA-ES for foraging.Source: 2) Multi-agent Evolution Strategies · pp. 15–16
  • Because ES does not require an MDP formulation, the survey says single-agent ES is relatively easy to extend to multi-agent settings without environmental non-stationarity.The authors hypothesize that this helps explain why ES multi-agent literature focuses on enabling applications.Source: 2) Multi-agent Evolution Strategies · pp. 15–16
  • Tang et al. [155] trained a quadruped protagonist against diverse escaping adversaries using an outer loop for adversary and protagonist training and inner loops for policy optimization.The policies are feedforward neural networks optimized with CMA-ES.Source: 2) Multi-agent Evolution Strategies · p. 15
  • Chen and Gao report that ES-trained predators collectively formed formations and avoided collisions while pursuing prey.The predators used ES methods including OpenAI-ES and CMA-ES; prey followed a fixed evasion strategy.Source: 2) Multi-agent Evolution Strategies · p. 15
  • PES-VD combines Parallelized ES with a Value Decomposition Network to compute fitness for each policy when agents share a team reward.Workers evaluate policies and compute network gradients; a master node updates the policies and network.Source: 2) Multi-agent Evolution Strategies · p. 15
  • The survey describes multi-agent ES as a growing field with potential applications in collective robotic learning and cloud robotics.It points to ES parallelism as one relevant advantage.Source: 2) Multi-agent Evolution Strategies · p. 16

V. HYBRID DEEP REINFORCEMENT LEARNING AND EVOLUTION STRATEGIES ALGORITHMS

The survey reviews hybrids that combine DRL and ES strengths, including population-guided learning and methods that use ES to tune policies or losses. It reports that some hybrids outperform either approach alone, while emphasizing that hybridization remains relatively new.

V. HYBRID DEEP REINFORCEMENT LEARNING AND EVOLUTION STRATEGIES ALGORITHMS

  • DRL can be sample-efficient, while ES offers robust convergence properties and exploration strategies; hybrids aim to combine these strengths.The survey contrasts DRL’s temporal credit assignment and hyperparameter sensitivity with ES’s low data efficiency and difficulty on large optimization tasks.Source: §V · p. 16, A. Comparison · p. 17
  • CEM-RL uses CEM-generated actors to diversify training data for a DDPG or TD3 agent, then inserts agent gradients into the population to guide search.The survey reports CEM-RL as superior to CEM, TD3, and ERL; in a separate sequential-game study, ES performed better on the obstacle-rich map.Source: §V · p. 16, Figure 10 · p. 16
  • EPG uses stochastic gradient descent for policy updates and an outer ES loop to tune the loss function; the survey reports faster learning than a standard RL agent.The loss is tuned according to the environment and agent history.Source: §V · pp. 16–17
  • A two-stage MORL hybrid first learns multiple policies collaboratively with multi-policy soft actor-critic, then applies MO-CMA-ES to policy-independent parameters for a uniform Pareto frontier.The approach addresses the stated contrast that DRL cannot guarantee uniform approximation, while ES has difficulty optimizing a DNN.Source: §V · p. 17
  • Another hybrid learns state representations and initial policies with DQN or DDPG, then uses CMA-ES to fine-tune final action-selection parameters.The survey says this can surpass the gradient-based counterpart using fewer trials than a pure gradient-free policy.Source: §V · p. 17
  • Other hybrids apply ES with Efficient NAS to search for RL policies or use ES to learn agent-agnostic synthetic environments.The survey notes that DRL–ES hybridization is still relatively new.Source: §V · p. 17, A. Comparison · p. 17

A. Comparison

  • Some hybrid methods reported in the literature seem to outperform DRL or ES used alone.The survey also says combining the approaches can help address some of their identified challenges.Source: A. Comparison · p. 17

VI. APPLICATIONS

The survey tracks DRL applications across domains and uses a Google Scholar title search to indicate the literature’s application record. It highlights achievements in games and robotics alongside work in finance, communications, energy, and transportation.

VI. APPLICATIONS

  • The applications overview uses a Google Scholar search and Table VIII to track DRL and ES application papers by title keywords.The table caption specifies that only paper titles are considered.Source: §VI · p. 17, Table VIII · p. 18

A. Deep Reinforcement Learning applications

  • DRL has reached superhuman performance in Atari games, defeated a world champion in Go, and attained elite results in Dota 2 and StarCraft II.AlphaGo Zero beat AlphaGo using RL without human data.Source: A. Deep Reinforcement Learning… · p. 17
  • DRL robotics applications include navigation, obstacle avoidance, manipulation, and locomotion, with reported results in both simulation and physical robots.Examples include navigating unknown static environments after simulation training and learning 3D manipulation skills on real robots.Source: A. Deep Reinforcement Learning… · p. 17
  • The survey also describes DRL applications in trading and investment, communications, energy systems, and transportation.Examples include adaptive bitrate selection, smart-grid control, traffic-signal control, and ride-sharing fleet utilization.Source: A. Deep Reinforcement Learning… · pp. 17–18

B. Evolution Strategy applications

The survey reviews ES applications across several domains and compares DRL and ESs, then identifies unresolved challenges and argues that hybridization has potential for reliable, efficient real-world agents.

B. Evolution Strategy applications

  • ES applications span gaming, robotics, finance, communications, energy, and transportation.The reviewed work includes Atari games, robotic control, portfolio optimization, network design, energy systems, and traffic or vehicle-routing problems.Source: B. Evolution Strategy applications · pp. 18–19

C. Comparison

  • Both DRL and ESs are used in domains including robotics, games, and finance, while DRL-based solutions seem to excel in scalable and adaptive settings.The survey reports substantially more scientific attention to DRL and suggests its richer structure and overlap with related families such as genetic algorithms as possible reasons.Source: C. Comparison · p. 19

VII. CHALLENGES AND FUTURE RESEARCH DIRECTIONS

  • DRL sample efficiency remains limited by the large number of environment interactions needed, while ESs are even less sample-efficient because they use full-length episodes and no memory.The survey says further innovation is needed for both approaches; accurate environment models are usually difficult to obtain, and ES improvements such as sample reuse and importance mixing still need more research.Source: §VII · p. 19
  • ES exploration through recombination and mutation can still become trapped in local optima, and learning with sparse rewards remains an open challenge despite proposed remedies.Directing ES exploration against sparse or deceptive rewards is identified as a key challenge for scaling to more complex environments and improving efficiency.Source: §VII · p. 19
  • Policies learned in simulation often do not work as expected in the real world, and the sim-to-real gap remains unresolved despite mitigation efforts.Simulator randomization requires manually specifying what to vary; domain adaptation using many simulated examples and a few real ones is another described approach.Source: §VII · p. 20

VIII. CONCLUSION

  • DRL and ESs share a sequential decision-making objective but use different learning mechanisms, and the survey sees hybridization as having high potential for reliable, efficient real-world agents.The comparison covers parallelism, exploration, meta-learning, and multi-agent learning rather than focusing on individual algorithms.Source: §VIII · p. 20

Related papers

Loading 2110.01411v1…