Source-linked AI summary
CEM-RL: Combining evolutionary and gradient-based methods for policy search
Aloïs Pourchot, Olivier Sigaud
TL;DR
Policy search faces a trade-off between the stability but low sample efficiency of evolutionary methods and the sample efficiency but instability of deep RL. This paper combines CEM with DDPG or TD3 in CEM-RL and evaluates it on standard deep-RL benchmarks, finding competitive performance and sample efficiency relative to the stated alternatives. The study also identifies unresolved questions about why some mechanisms and benchmark behaviors occur.
Problem
Evolutionary methods are stable but sample-inefficient, whereas sample-efficient off-policy deep RL methods are unstable and sensitive to hyper-parameter settings; combining them remains relatively unexplored.
Method
CEM-RL combines the cross-entropy method with DDPG or TD3, applying gradient-based policy improvement within an evolutionary search process.
Results
CEM-RL outperformed the stated evolution-strategy, off-policy deep-RL, and ERL competitors in most cases and remained competitive with the state of the art in sample efficiency.
Takeaways & Limitations
The results support combining evolutionary and deep RL methods and indicate substantial unexplored potential in designing new combinations.
Takeaways & Limitations
The study does not establish why CEM performs well on swimmer-v2 or why importance mixing and action noise lack clear benefits, leaving these mechanisms for further investigation.
Abstract
from arXiv · showhide
Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting. So far, these families of methods have mostly been compared as competing tools. However, an emerging approach consists in combining them so as to get the best of both worlds. Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm. In this paper, we propose a different combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (td3), another off-policy deep RL algorithm which improves over ddpg. We evaluate the resulting method, cem-rl, on a set of benchmarks classically used in deep RL. We show that cem-rl benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency.
1 Introduction
Policy search methods based on evolution and deep RL offer complementary strengths: evolutionary methods are stable and parallelizable but sample-inefficient, while off-policy deep RL is sample-efficient but unstable and sensitive to tuning. The paper proposes combining these families through CEM with DDPG or TD3 and evaluates the resulting CEM-RL method.
- Motivation: Evolutionary methods use complete episodes and are less sample-efficient, whereas deep RL uses elementary system steps and replay buffers to reuse samples.The passage attributes the sample-efficiency difference to how each family uses experience.
- Motivation: Off-policy deep RL methods such as DDPG are known to be unstable and highly sensitive to hyper-parameter settings.
- Motivation: The paper advocates combining evolutionary and deep RL methods rather than treating them as competing policy-search solutions.The authors motivate the combination as a way to seek complementary benefits from both families.
- Contribution: CEM-RL combines the cross-entropy method with DDPG or TD3, an off-policy deep RL algorithm that improves over DDPG.
- Contribution: Experiments investigate CEM-RL on deep-RL benchmarks and report advantages over its components and a competing approach.
2 Related work
Prior work mostly compared evolutionary and deep RL methods, with a small number of hybrid approaches combining exploration, evolutionary search, or gradient information. The paper positions CEM-RL as a hybrid that uses CEM and TD3 while addressing a limitation of the closest ERL approach.
- Existing combinations: Earlier research primarily compared evolutionary and deep RL methods, although CEM was shown to be a strong policy-search baseline.
- Existing combinations: GEP-PG sequentially fills a replay buffer with exploratory trajectories before applying DDPG, improving sample efficiency and final solutions on two benchmarks.
- Existing combinations: Surrogate-gradient ES modifies its sampling covariance using gradient information and outperformed gradient descent and pure ES on simple benchmarks, but lacked a practical RL demonstration.
- ERL: ERL combines a population-based evolutionary algorithm with DDPG, training one DDPG agent from population-generated samples before periodically reinserting it.
- ERL: The ERL combination does not exploit the search efficiency of evolution strategies, where importance mixing can provide further sample-efficiency improvement.
- CEM-RL: CEM-RL combines CEM and TD3 to use TD3 policy improvement, ES stability, and potentially importance mixing.
3 Background
The background introduces evolutionary algorithms, CEM, CMA-ES, and off-policy actor-critic methods used in the paper. CEM samples noisy individuals, updates its distribution from elite performers, and uses a diagonal covariance in practice, while DDPG and TD3 provide sample-efficient deep RL components.
- Evolutionary methods: Evolutionary algorithms maintain a population and generate new individuals near elite individuals; evolution strategies retain a distribution mean as the next-generation individual.
- CEM: CEM fixes the number of elite individuals, updates the sampling mean and variance from them, and adds variance to prevent premature convergence.
- CEM: CEM samples individuals from N(µ, Σ), evaluates their fitness, and uses the top K_e individuals to update the distribution parameters.
- CEM: CEM weights may be uniform or rank-based, with rank-based weights assigning greater importance to better individuals.
- CMA-ES: CMA-ES also fixes an elite count but constructs its mean and covariance more elaborately using performance-based ranking weights.
- CEM implementation: The paper uses a CEM variant with the current mean for covariance estimation and added noise with exponentially decaying magnitude to prevent premature convergence.
- CEM implementation: Because full covariance computation becomes intractable for many actor parameters, the implementation constrains Σ to be diagonal.
- Deep RL: DDPG and TD3 are off-policy, actor-critic, sample-efficient deep RL algorithms, while DDPG suffers from critic overestimation bias and tuning-sensitive instabilities.
4 Methods
CEM-RL combines CEM population search with DDPG or TD3 gradient updates, allowing evolutionary and deep-RL components to exchange information through actor selection and replay-buffer learning.
- Method overview: CEM-RL combines CEM with DDPG or TD3, producing the cem-ddpg and cem-td3 algorithms.The mean actor is initialized randomly, alongside a critic managed by DDPG or TD3.
- Population search and gradient updates: Actors are sampled around the current mean using Gaussian noise, then half are evaluated directly while the others receive critic-guided actor updates before evaluation.CEM computes the new mean and covariance from the top-performing half of the resulting population.
- Information exchange: Environment experience from actor evaluations enters a replay buffer, which trains the critic in proportion to the collected data.This couples evolutionary evaluation with off-policy deep-RL learning.
- Information exchange: Good gradient-improved actors can enter the evolutionary population, while good evolved actors provide experience for the replay-buffer learner.The two components therefore exchange information in both directions.
- Comparison with ERL: Unlike ERL, CEM-RL applies gradient steps at each iteration and does not inject a separate deep-RL actor into the population.The cited comparison identifies these as distinguishing properties of CEM-RL.
- Design choice: CMA-ES was not used because its evolutionary path mechanism introduces inertia in covariance updates that resists the beneficial effect of RL gradient steps.The paper therefore uses the simpler CEM within CEM-RL.
5 Experimental study
The experimental study evaluates cem-rl against evolutionary, gradient-based, multi-actor, and hybrid baselines across continuous-control benchmarks. Results generally favor cem-td3, while additional analyses identify stability, exploration, noise, architecture, and benchmark-specific effects.
- Experimental setup: The study evaluates cem-rl methods on five MuJoCo continuous-control benchmarks commonly used for policy search.The benchmarks are half-cheetah-v2, hopper-v2, walker2d-v2, swimmer-v2, and ant-v2.
- Comparison to cem, td3 and a multi-actor td3: Cem-td3 outperforms cem and td3 on half-cheetah-v2, hopper-v2, and walker2d-v2, with slightly lower variance than td3 on most benchmarks.On ant-v2, cem-td3 outperforms cem and performs on par with td3.
- Comparison to cem, td3 and a multi-actor td3: Cem-td3 is generally superior to cem, td3, and multi-actor td3 by accelerating updates when td3 provides useful gradients and reducing learning variance.The comparison summarizes final performance across the methods in Table 1.
- Comparison to erl: After 1 million steps, both cem-rl methods outperform erl on half-cheetah-v2, hopper-v2, and walker2d-v2.Cem-td3 also outperforms cem-ddpg on walker2d-v2; on ant-v2, cem-ddpg and erl are initially on par after 1 million steps.
- Comparison to erl: Cem-td3 outperforms cem-ddpg on most benchmarks in final performance, convergence speed, and learning stability, especially on walker2d-v2 and ant-v2.Swimmer-v2 is identified as the exception.
- Additional results: Additional analyses find limited impact from importance mixing, no conclusive benefit from action noise, and often stronger performance with tanh than relu actor nonlinearities.The evolutionary population in erl tends to collapse toward one individual, whereas cem maintains exploration through its sampling method.
6 Conclusion and future work
The paper argues for combining evolutionary and deep RL methods, proposing cem-rl as a strong and sample-efficient hybrid. It also identifies unresolved questions about why some components and benchmarks behave unexpectedly.
- cem-rl outperformed several evolution strategies, sample-efficient off-policy deep RL algorithms, and the erl combination in most cases.
- Despite being mainly evolutionary, cem-rl remained competitive with the state of the art when sample efficiency was considered.
- The study leaves unexplained why simple cem performs so well on swimmer-v2.
- Importance mixing and action noise showed no clear general benefit, motivating further investigation.
A Architecture of the networks
The networks follow architectures similar to those of Fujimoto et al. (2018), while changing the nonlinearities used in the actor and critic. The architecture is documented from input to output layers.
- The network layer sizes remain the same as in Fujimoto et al. (2018).
- The actor uses tanh operations between layers instead of relu.
- The critic uses leaky relu instead of simple relu.
- Table 3 presents the network architecture from the input layer to the output layer.
B Importance mixing
Importance mixing reuses samples across generations to reduce reevaluation costs, but its experiments found little sample-efficiency benefit and sometimes reduced performance. The study discusses possible reasons for this discrepancy and adapts the mechanism for cem-rl.
- Importance mixing: Importance mixing reuses samples from previous or archived generations to avoid reevaluating corresponding policies in the environment.
- Implementation in cem-rl: In cem-rl, importance mixing is applied only to the population half that does not receive RL gradient steps.
- Empirical results: Importance mixing introduced minor instability in cem without noticeably increasing sample efficiency.
- Empirical results: On half-cheetah-v2, swimmer-v2, and walker2d-v2, cem performance decreased with importance mixing.
- Empirical results: For cem-rl, sample reuse gains were almost null overall, although performance increased on swimmer-v2.
- Reported comparisons: Tables 4 and 5 report means and medians over 10 runs of 1 million steps for the corresponding benchmark comparisons.
- Interpretation: The authors attribute the disagreement with earlier results partly to larger search spaces and harder MuJoCo environments.
- Action space noise: The action-noise experiments found no conclusive performance improvement, with cem exploration appearing sufficient on its own.
D Parameter space exploration in cem-rl and erl
cem-rl and erl use gradient information differently, producing distinct parameter-space dynamics. erl populations tend to collapse around one actor, whereas cem-rl maintains diversity and uses gradient-updated actors as scouts.
- Parameter update dynamics: Figure 6 compares actor-parameter trajectories in erl and cem-rl using sampled-parameter dots and gradient-step lines.
- Population diversity: erl populations become less diverse, with redundant parameters and apparent convergence toward a single individual.
- Population diversity: cem-rl introduces completely new samples each generation, yielding better apparent parameter-space exploration.
- Population similarity: Around 55% of populations encountered during erl runs exhibited high population similarity.
- Gradient information: In erl, ddpg parameters are repeatedly introduced and quickly spread through the population, concentrating it around the ddpg actor.
- Population similarity: Figure 7 reports population-similarity distributions averaged over 10 erl runs with 68% confidence intervals for the mean.
- Gradient information: The ddpg actor was the population elite 80% of the time and introduced into the population 98% of the time.
- Gradient information: In cem-rl, gradient steps from half the actors act as scouts that identify promising directions for subsequent evolutionary sampling.
E The case of the swimmer-v2 benchmark
Swimmer-v2 produces an unusual ranking: CEM outperforms TD3 and the tested combinations, while ERL outperforms CEM-DDPG, which outperforms CEM-TD3. The likely explanation is that deceptive deep-RL gradients damage actor optimization, reducing CEM-RL’s effective population.
- CEM outperforms TD3, CEM-TD3, and multi-actor TD3 on swimmer-v2.
- ERL outperforms CEM-DDPG, which itself outperforms CEM-TD3 on swimmer-v2.
- Deceptive gradient information from deep RL may hinder convergence toward efficient actor parameters in swimmer-v2.
- CEM-RL behaves like CEM with half a population, while ERL can ignore an underperforming DDPG actor; DDPG was rejected 76% of the time.
F Using the relu or tanh non-linearity
Changing actor nonlinearities from tanh to ReLU can substantially reduce performance in CEM-TD3 and CEM, with especially large drops on ant-v2 and swimmer-v2.
- Figure 9 compares CEM-TD3 and CEM learning performance with ReLU or tanh actors across five benchmarks.
- ReLU instead of tanh causes substantial performance drops on some benchmarks for CEM-TD3 and CEM.
- 46%: average performance drops on ant-v2 when changing actor nonlinearities from tanh to ReLU.
- 60%: CEM performance drops on swimmer-v2 when using ReLU instead of tanh.
G Additional results on ant-v2
The ant-v2 results are presented through learning curves comparing CEM, TD3, multi-actor TD3, CEM-DDPG, and CEM-TD3.
- Figure 10 represents ant-v2 learning performance for CEM, TD3, multi-actor TD3, CEM-DDPG, and CEM-TD3.
- Figure 10 compares learning curves for CEM, TD3, and CEM-RL on ant-v2.