PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Evolutionary Reinforcement Learning

Shauharda Khadka, Somdeb Majumdar, Tarek Nassar, Zach Dwiel, Evren Tumer, Santiago Miret, Yinyin Liu, Kagan Tumer

arXiv:1905.00976v2cs.LGcs.AIstat.ML

TL;DR

Deep reinforcement learning struggles with effective exploration and hyperparameter-sensitive convergence. CERL combines learners with varied time horizons, shared experience, dynamic resource allocation, and neuroevolution. Across continuous-control experiments, its emergent learner can outperform its component learners, though it can be less sample-efficient on simple tasks when ideal hyperparameters are already known.

  • Problem

    Deep reinforcement learning faces challenges in effective exploration and brittle convergence that require careful hyperparameter tuning.

    Source: §1 · p. 1

  • Method

    CERL combines learners with different time horizons, a shared replay buffer, dynamic resource allocation, and neuroevolution in a gradient-free evolutionary framework.

    Source: §1 · p. 2, §5 · pp. 5–6, §7 · p. 8

  • Results

    Across continuous-control experiments, CERL’s emergent learner can outperform its composite learners; on Humanoid, participating learners that fail alone contribute to effective solutions in CERL.

    Source: §7 · p. 8

  • Takeaways & Limitations

    CERL is presented as a simpler alternative to complex hyperparameter tuning that appears to generalize across multiple tasks.

    Source: §7 · p. 8

    CERL can be less sample-efficient on simple tasks when ideal learner hyperparameters are known, though it eventually matches the learner using those parameters.

    Source: §7 · p. 8

Abstract

from arXiv · show

Deep reinforcement learning algorithms have been successfully applied to a range of challenging control tasks. However, these methods typically struggle with achieving effective exploration and are extremely sensitive to the choice of hyperparameters. One reason is that most approaches use a noisy version of their operating policy to explore - thereby limiting the range of exploration. In this paper, we introduce Collaborative Evolutionary Reinforcement Learning (CERL), a scalable framework that comprises a portfolio of policies that simultaneously explore and exploit diverse regions of the solution space. A collection of learners - typically proven algorithms like TD3 - optimize over varying time-horizons leading to this diverse portfolio. All learners contribute to and use a shared replay buffer to achieve greater sample efficiency. Computational resources are dynamically distributed to favor the best learners as a form of online algorithm selection. Neuroevolution binds this entire process to generate a single emergent learner that exceeds the capabilities of any individual learner. Experiments in a range of continuous control benchmarks demonstrate that the emergent learner significantly outperforms its composite learners while remaining overall more sample-efficient - notably solving the Mujoco Humanoid benchmark where all of its composite learners (TD3) fail entirely in isolation.

1. Introduction

Deep reinforcement learning faces challenges in effective exploration and brittle convergence that demands careful hyperparameter tuning. CERL combines learners with different time horizons, shared experience, and neuroevolution to explore and exploit diverse regions of the solution space.

1. Introduction

  • Effective exploration and brittle convergence requiring careful hyperparameter tuning limit deep reinforcement learning’s widespread adoption for real-world problems.Standard learners often use a noisy operating policy for exploration, placing both exploration and exploitation demands on the same hyperparameters.Source: §1 · p. 1
  • CERL uses a portfolio of learners with different time horizons and dynamically reallocates computational resources toward better-performing learners.The resource manager makes this an online algorithm-selection process.Source: §1 · p. 2
  • A shared replay buffer lets learners use diverse experiences generated across the portfolio to improve sample efficiency.Source: §1 · p. 2
  • Neuroevolution adds policy-parameter exploration, redundancies, and crossover, binding the process into an emergent learner that exceeds its component learners.The introduction reports that CERL inherits the strengths of its composite learners while remaining more sample-efficient across continuous-control benchmarks.Source: §1 · p. 2

2. Background

The background introduces the reinforcement-learning setting, describes TD3’s actor-critic learning and replay-buffer updates, and outlines evolutionary search over neural-network policies.

2. Background

  • An agent interacts with an environment over discrete time steps, receiving rewards and seeking to maximize its expected discounted return.The return is defined from a time step using discount factor γ.Source: §2 · p. 2

2.1. Twin Delayed Deep Deterministic Policy Gradients

  • TD3 extends DDPG to address overestimation, using a deterministic actor and two action-value critics.Separate target networks are updated using a weighting parameter and delayed policy-update frequency.Source: §2.1 · p. 2
  • TD3 explores with a noisy behavioral policy, stores transitions in a replay buffer, and trains its actor and critic from sampled mini-batches.For the Bellman update, clipped noisy actions smooth value estimates and are described as addressing overfitting of the deterministic policy.Source: §2.1 · p. 2
  • The actor’s sampled policy gradient is computed by backpropagating through the combined actor and critic network.Source: §2.1 · p. 3

2.2. Evolutionary Algorithms

  • Neuroevolution searches over neural-network policies by generating, altering, and selecting candidate solutions across a population.In this work, mutation randomly perturbs network weights, while selection favors candidates with higher fitness.Source: §2.2 · p. 3

3. Related Work

CERL combines ideas from population-based training, evolutionary-policy-gradient methods, and online algorithm selection, while differing in how it selects learners and shares experience.

3. Related Work

  • Unlike population-based training, CERL dynamically reallocates computation among learners and shares experiences for collective exploitation.PBT relies on its evolutionary process for learner selection, and its learners are isolated from one another’s experience.Source: §3 · p. 3
  • Prior work has combined evolutionary methods for generating diverse samples with policy-gradient fine-tuning or run evolutionary and gradient-based processes concurrently.The passage relates these approaches to incorporating learning with evolution.Source: §3 · p. 3
  • CERL’s resource manager builds on ESBAS’s algorithm-selection formulation but uses soft selection through resource allocation rather than hard binary selection.Source: §3 · p. 3

4. Motivating Example

In Hopper, CERL combines TD3 learners with extreme discount rates and neuroevolution. The combined system continues learning where its components fail or plateau, reaching a score of 2136 ± 512.

4. Motivating Example

  • The two TD3 variants represent immediate-reward and virtually infinite optimization horizons, illustrating the sensitivity of the task to discount-rate choice.TD3 can solve Hopper with a tuned discount rate of γ = 0.99.Source: §4 · p. 3
  • 2136 ± 512: CERL continued learning beyond its component learners, while neuroevolution failed to solve Hopper within 5 million tested steps.Figure 2 compares CERL with neuroevolution and TD3 at γ = 0.0 and γ = 1.0.Source: §4 · p. 4
  • TD3 at γ = 0.0 failed to learn, while TD3 at γ = 1.0 reached a reward ceiling of 1000 because of variance in its computed return.Source: §4 · p. 4
  • The authors attribute CERL’s collaborative success to using diverse learners as behavioral policies while reallocating resources toward better performers for exploitation.They describe this as an alternative to finding the exploration–exploitation trade-off by tuning the discount rate.Source: §4 · p. 4

5. Collaborative Evolutionary Reinforcement Learning

CERL combines learners with different time-horizons, adaptive resource allocation, shared experience replay, and evolution. Together, these mechanisms support diverse exploration and integrate policies from the portfolio and evolutionary population.

5. Collaborative Evolutionary Reinforcement Learning

  • The evolutionary population selects actors by rollout fitness, preserves elites from mutation, and creates new actors through selection, mutation, and crossover.The population is initialized with random actor weights and evaluated in environment rollouts.Source: §5 · p. 5
  • Learners use distinct discount rates to optimize over different time-horizons, balancing greedier but more learnable objectives against longer-term objectives with higher return variance.The portfolio can also vary other hyperparameters or learning algorithms.Source: §5 · p. 5
  • UCB scores are normalized into a probability distribution and sampled to allocate rollout workers among learners for the next generation.The allocation is intended to balance exploitation and exploration across learners.Source: §5 · p. 5
  • CERL pools each actor’s transitions in a collective replay buffer, where portfolio learners repeatedly sample experiences for gradient updates.Unlike a standard evolutionary algorithm, it retains rollout experiences; the authors say this enables greater information extraction and sample efficiency.Source: §5 · pp. 5–6
  • Learners exploit behavioral policies from other learners and the evolutionary population, while evolution explores policy parameters through mutation.The authors describe these action-space and parameter-space processes as complementary forms of policy-space exploration.Source: §5 · p. 6
  • Periodic Lamarckian transfer copies learner networks into the evolutionary population, while selection and reproduction amplify good policies and crossover combines policy sub-components.The authors describe this as a way for evolution to leverage gradient-based learning and exploit decomposability in policy space.Source: §5 · p. 6

6. Results

CERL solves Humanoid where its composite learners fail in isolation, and succeeds on Swimmer where all tested TD3 learners fail. Its results also show that resource allocation can favor different learners across tasks, while CERL may use more samples than a well-tuned learner.

Compared Baselines

  • Comparisons use an EA and four untuned TD3 learners, with isolated learners and single-learner ERL as baselines; cumulative steps and five-run averages are reported.TD3 learners differ only in discount rate; scores are reported with 95% confidence intervals.Source: §6 · p. 6

Humanoid

  • 4702.0±356.5: CERL solves Humanoid within 1 million environment steps, while isolated TD3 learners fail and neuroevolution alone required 62.5 million roll-outs.Figure 3 compares CERL with its composite learners; single-learner ERL also fails on Humanoid.Source: Figure 3 · p. 7, §6 · p. 7

Resource-manager’s Sensitivity to Exploration

  • CERL with c = 0.9 performs best on Humanoid, while c = 0.0 and c = 5.0 also learn the benchmark.The passage also notes that c = 0.0 does not eliminate exploration, whereas c = 10 causes very high exploration and lower sample efficiency.Source: Figure 4 · p. 7, §6 · p. 7

Additional Mujoco Experiments

  • CERL is more sample-efficient than the other TD3-based learners, but may use more samples than a known ideal hyperparameter; it identifies and exploits hyperparameters online.The paper describes this as a simpler alternative to complex hyperparameter tuning methodologies.Source: §6 · pp. 7–8
  • On Swimmer, CERL solves the task where all TD3 learners fail; on three of four simpler benchmarks, TD3-0.99 is more sample-efficient.CERL also solves the other three simpler benchmarks, though less sample-efficiently than TD3-0.99.Source: §6 · p. 7, Figure 5 · p. 8

Allocation

  • Across five Mujoco benchmarks, L2 is generally preferred, but Swimmer receives more resources for the higher-discount L3 and L4 learners.Table 1 reports average cumulative resource-allocation rates; the paper says this flexibility, with the evolutionary population, enables CERL to solve Swimmer.Source: Table 1 · p. 8, §6 · p. 8

7. Discussion

CERL combines gradient-based learners and evolution into an emergent learner that can outperform its components while remaining sample-efficient. The discussion highlights robustness and hardware parallelism, alongside a simple-task efficiency trade-off and a homogeneous-learner scope.

7. Discussion

  • CERL’s emergent learner can outperform its composite learners while remaining overall sample-efficient compared with traditional approaches.The discussion also describes CERL as generally insensitive to its own and individual learners’ hyperparameters.Source: §7 · p. 8
  • CERL can find effective Humanoid solutions using participating learners that fail completely on their own.The discussion names Humanoid and Swimmer as cases where state-of-the-art algorithms show high hyperparameter sensitivity while CERL required no tuning.Source: §7 · p. 8
  • CERL’s parallel gradient-based and gradient-free methods can use hardware suited to backpropagation and forward-propagation, respectively.The passage notes that forward-propagation is highly scalable and links the combined modes to parallel learning and adapting algorithms to available hardware.Source: §7 · p. 8
  • CERL can be less sample-efficient on simple tasks when ideal hyperparameters are known a priori, but eventually matches the learner using them.The discussion attributes this trade-off to learner-selection exploration and notes that deriving ideal parameters can itself consume significant resources.Source: §7 · p. 8
  • The experiments use homogeneous learners over varying task time-horizons; future work proposes different algorithms, including stochastic SAC with deterministic TD3 actors.The passage also proposes learning within the resource manager to augment its current UCB formulation.Source: §7 · p. 8

Appendix

The appendix lists CERL’s benchmark hyperparameters, covering learner networks and updates, evolutionary operations, and resource allocation. It also describes how population, rollout, mutation, and transfer settings configure exploration and information flow.

Appendix

  • Table 2 lists the hyperparameters used for CERL across benchmarks.The appendix presents settings for learner networks, evolution, and computational resources.Source: Table 2 · p. 11, Appendix · p. 11
  • Population size k=10 and rollout size b=10 jointly set the balance between parameter-space and action-space exploration; learner action noise is σ=0.1.Source: Appendix · pp. 11–12
  • The described parameters control the magnitude of soft updates between actor and critic networks and their targets, actor and critic learning rates, and replay-buffer size; when full, oldest experiences are deleted.Source: Appendix · p. 11
  • Batch size is 256; actor and critic entries list [400, 300] networks with ELU and layer normalization, and Adam updates both networks.Source: Appendix · p. 11
  • The evolutionary population uses e=k/5 elites and mutation probability 0.9, fraction 0.1, strength 0.1, super-mutation probability 0.05, and reset probability 0.05.The elite fraction is intended to help preserve good actors amid high fitness variance in tasks with more stochastic dynamics.Source: Appendix · pp. 11–12
  • The value learning rate is α=0.2, and the Lamarckian transfer period is ω=5 for information flow between learners and the evolutionary population.The passage says higher ω generally allows more expansive evolutionary exploration, while lower ω can allow a narrower search.Source: Appendix · p. 12

Related papers

Loading 1905.00976v2…