Source-linked AI summary
Evolution-Guided Policy Gradient in Reinforcement Learning
Shauharda Khadka, Kagan Tumer
TL;DR
DRL struggles with sparse-reward credit assignment, exploration, and brittle convergence, while EAs face high sample complexity and difficulty optimizing many parameters. ERL combines EA populations with off-policy DRL gradients, and experiments across challenging continuous-control benchmarks show that it significantly outperforms prior DRL and EA methods. The method also incurs approximately 3% more runtime than DDPG in the reported nonparallelized implementation.
Problem
DRL is limited by sparse-reward temporal credit assignment, ineffective exploration, and brittle convergence, whereas EAs have high sample complexity and struggle with many parameters.
Method
ERL uses an EA population to generate diverse experiences for an RL agent and periodically reinserts the RL agent to inject gradient information into the population.
Results
ERL significantly outperforms prior DRL and EA methods across challenging continuous-control benchmarks.
Takeaways & Limitations
ERL combines EA’s sparse-reward invariance, diverse exploration, and population stability with DRL’s lower sample complexity and gradient-based learning.
Takeaways & Limitations
The reported ERL implementation is not parallelized, and future work proposes more complex evolutionary mechanisms and extensions to multiagent settings.
Abstract
from arXiv · showhide
Deep Reinforcement Learning (DRL) algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically suffer from three core difficulties: temporal credit assignment with sparse rewards, lack of effective exploration, and brittle convergence properties that are extremely sensitive to hyperparameters. Collectively, these challenges severely limit the applicability of these approaches to real-world problems. Evolutionary Algorithms (EAs), a class of black box optimization techniques inspired by natural evolution, are well suited to address each of these three challenges. However, EAs typically suffer from high sample complexity and struggle to solve problems that require optimization of a large number of parameters. In this paper, we introduce Evolutionary Reinforcement Learning (ERL), a hybrid algorithm that leverages the population of an EA to provide diversified data to train an RL agent, and reinserts the RL agent into the EA population periodically to inject gradient information into the EA. ERL inherits EA's ability of temporal credit assignment with a fitness metric, effective exploration with a diverse set of policies, and stability of a population-based approach and complements it with off-policy DRL's ability to leverage gradients for higher sample efficiency and faster learning. Experiments in a range of challenging continuous control benchmarks demonstrate that ERL significantly outperforms prior DRL and EA methods.
1 Introduction
DRL is limited by sparse-reward credit assignment, ineffective exploration, and brittle convergence. ERL combines evolutionary population-based learning with gradient-based optimization to address these challenges while improving sample efficiency.
- Challenges: DRL faces three major challenges: temporal credit assignment with sparse rewards, ineffective exploration, and brittle convergence.These limitations restrict adoption in real-world problems.
- Evolutionary algorithms: EAs can address sparse rewards because episode-wide fitness consolidates returns across entire trajectories.This makes them robust to long time horizons and sparse reward distributions.
- Evolutionary Reinforcement Learning: ERL uses an EA population to generate diverse experiences for an RL agent and periodically reinserts the RL agent into the population.The reinsertion transfers gradient information into the evolutionary process.
- Evolutionary Reinforcement Learning: ERL combines population-based exploration and stability with DRL gradients for higher sample efficiency.The hybrid design is intended to retain complementary strengths of EAs and DRL.
- Results: Experiments across challenging continuous-control benchmarks show that ERL significantly outperforms prior DRL and EA methods.The same evolutionary experiences are reused by the reinforcement learner.
2 Background
The paper formalizes RL as interaction between an agent and environment in an MDP, then describes policy-gradient learning and evolutionary search as relevant foundations for ERL.
- Reinforcement learning: An RL agent interacts with an environment over discrete time steps, mapping states to actions and receiving rewards until an episode ends.The objective is to maximize expected return.
- Reinforcement learning: The return is the discounted total reward accumulated from a time step onward.The discount factor satisfies γ ∈ (0, 1].
- DDPG: DDPG is a model-free policy-gradient method for continuous high-dimensional action spaces using deterministic actor and critic networks.The actor represents the policy, while the critic approximates the action-value function.
- DDPG: DDPG explores with a noisy behavioral policy and stores transition tuples in a cyclic replay buffer.The buffer supports repeated learning from past experience.
- DDPG: The actor’s policy gradient is computed by backpropagation through the combined actor and critic network.The critic is trained using a sampled loss.
- Evolutionary algorithms: EAs generate, alter, and select candidate solutions within a population, favoring solutions with higher fitness.These operators continually produce novel candidates while retaining promising ones probabilistically.
3 Motivating Example
The Inverted Double Pendulum example contrasts gradient-based and evolutionary methods under ordinary and extremely sparse rewards. ERL combines DDPG’s gradient efficiency with EA’s robustness to delayed feedback.
- Standard task: In the standard Inverted Double Pendulum, ERL and DDPG solve the task under 3000 episodes, while EA requires approximately 22000 episodes.The task is a continuous-control benchmark with an 11-dimensional state space and one-dimensional action space.
- ERL: ERL inherits gradient-based learning from DDPG while retaining EA’s robustness to sparse, deceptive rewards.The comparison illustrates complementary strengths of the hybrid design.
- Hard task: The hard task gives zero reward until the final step of episodes lasting up to 1000 timesteps, creating a severe temporal credit assignment challenge.The controller must wait for cumulative episode feedback.
- Hard task: On the hard task, EA retains its standard-task performance and solves it after 22000 episodes, whereas DDPG fails entirely.For EA, both task versions are equivalent because fitness uses only cumulative episode reward.
- ERL: ERL successfully solves the hard task by using episode-wide fitness and selection pressure toward policies with higher long-term return.This biases replay-buffer states toward states with higher long-term payoff.
4 Evolutionary Reinforcement Learning
ERL combines an evolutionary population with an off-policy actor-critic learner: population experiences train the RL components, and the learned actor periodically returns to the population. This design reuses data, supports exploration and episode-level fitness selection, and transfers gradient information into evolution.
- Algorithm overview: ERL combines a standard EA with DDPG, while allowing any off-policy actor-critic learner to serve as its policy-gradient component.The evolutionary population generates diverse experiences, while gradient-based DRL learns from them.
- Algorithm overview: A population of actors is evaluated episodically for fitness, while a separate rlactor and critic learn alongside the population.The algorithm initializes population actors, an rlactor, a critic, and a replay buffer before iterative evaluation and learning.
- EA → RL: ERL stores population transitions in a replay buffer so the critic and rlactor can repeatedly learn from experiences generated across the entire evolutionary population.The critic updates from random minibatches, and the rlactor is trained with sampled policy gradients.
- Temporal Credit Assignment: Episode-wide fitness scores bias selection toward individuals with higher long-term returns, providing implicit prioritization of experiences from favorable state distributions.The selection process uses cumulative episode returns rather than only timestep-level feedback.
- Diverse Exploration: ERL combines parameter-space exploration by population actors with action-space Ornstein-Uhlenbeck noise for the rlactor.The two exploration processes collectively explore the policy space.
- RL → EA: Every synchronization period, ERL copies the rlactor’s weights into the evolving population, replacing a weakest actor so gradient information can enter evolution.A good rlactor can survive through selection, whereas a poor one can be discarded.
5 Experiments
Across six MuJoCo continuous-control benchmarks, ERL was evaluated against EA, DDPG, and PPO using champion-policy scores and repeated statistical runs. ERL outperformed DDPG across all benchmarks, exceeded PPO in four of six, and showed complementary EA–RL behavior, while selection was important and runtime was close to DDPG.
- Results: ERL significantly outperformed DDPG across all six benchmarks and learned the Ant task where DDPG normally failed to make progress.ERL also consistently outperformed EA except on Swimmer, where their performances were approximately equivalent.
- Results: ERL significantly outperformed PPO in 4 out of the 6 benchmark environments.In Hopper and Walker2D, ERL eventually matched and exceeded PPO but was less sample efficient.
- Results: ERL eventually escaped Hopper and Walker2D local minima through temporally correlated action-space exploration and policy-gradient updates from replay-buffer batches.EA converged to policies that survived by balancing while staying still, whereas ERL combined evolutionary and policy-gradient mechanisms.
- Ablation Experiments: Removing ERL’s selection operator caused approximately 80% degradation in learning performance across HalfCheetah and Swimmer.The ablation normalized each benchmark’s performance by the best score achieved using full ERL.
- Interaction between RL and EA: Synchronized RL actors showed task-dependent selection rates, and HalfCheetah benefited from evolutionary critical interventions that supported robustness.The reported selection-rate distribution suggested tight integration between the RL actor and evolutionary population.
- Runtime: ERL took approximately 3% more runtime than DDPG on average in the non-parallelized implementation.Most added computation came from mutation, and the authors anticipated faster execution with parallelization.
6 Related Work
Prior work has combined evolutionary algorithms with reinforcement learning in several forms, including evolving value-function approximators and generating exploratory goals. Evolutionary Strategies also achieved competitive results, with large parallel deployments improving clock time.
- 6 Related Work: Earlier studies combined evolutionary algorithms and reinforcement learning through Learning Classifier Systems, NEAT-based value-function evolution, and goal-exploration methods.These approaches applied evolutionary mechanisms to autonomous control, Q-learning function approximators, or exploration-goal generation.
- 6 Related Work: Evolutionary Strategies achieved results competitive with DRL in Atari and robotic-control tasks, while over a thousand parallel workers substantially improved clock-time scalability.The passage presents parallel execution as a notable scalability property of ES approaches.
7 Discussion
ERL combines evolutionary populations with gradient-based reinforcement learning to address sparse rewards, exploration, stability, and sample efficiency. The paper reports strong benchmark performance while identifying richer evolutionary mechanisms and multiagent extensions as future directions.
- ERL uses an evolutionary population to generate diverse experiences, then periodically reinserts the RL agent to inject gradient information into the population.
- ERL recycles population-generated data through the replay buffer, enabling repeated learning from each experience and improving sample efficiency.
- ERL outperforms state-of-the-art DRL algorithms including PPO and DDPG across challenging continuous-control benchmarks.
- From the RL perspective, ERL guides exploration toward higher-return states while preserving policy diversity and population redundancy for stability.
- From the evolutionary perspective, ERL provides a Lamarckian mechanism for incorporating gradient-based learning at the resolution of individual experiences.
- The study uses a standard EA, leaving more informative evolutionary operators, adaptive exploration, diversity maintenance, curriculum methods, information-theoretic techniques, and multiagent settings for future work.
A Experimental Details
The experimental details specify shared ERL settings and task-dependent hyperparameters. These settings control population exploration, fitness estimation, actor–critic optimization, replay, and information flow between the RL actor and evolutionary population.
- Shared hyperparameters: ERL uses a population size k = 10, with one RL actor exploring through action-space noise and ten actors exploring through parameter-space mutation each generation.
- Optimization settings: The actor and critic learning rates are 5e−5 and 5e−4, respectively, with discount rate 0.99 and target weight 1e−3.
- Replay settings: The replay buffer holds 1e6 experiences and gradients are computed with batch size 128.
- Network settings: The actor and critic use two hidden layers, with actor architecture [128, 128] and critic architecture [200 + 200, 300].
- Task-varied hyperparameters: The elite fraction ψ controls the exploration–exploitation balance by protecting elite actors from mutation, while higher values can reduce fitness variance in stochastic tasks.
- Task-varied hyperparameters: The number of trials ξ averages cumulative scores across full episodes to reduce fitness variance, and synchronization period ω controls information flow from the RL actor to the evolutionary population.