Source-linked AI summary
GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer
TL;DR
Continuous-action deep RL faces inefficient exploration in sparse or deceptive reward problems, while exploration-focused methods are less efficient at fine-tuning. GEP-PG sequentially combines exploratory GEP with DDPG-based policy improvement, outperforming DDPG across the studied benchmarks.
Problem
Continuous-action deep RL methods such as DDPG face inefficient exploration in sparse or deceptive reward problems, where useful policy-improvement gradients may be unavailable until reward is found.
Method
GEP-PG first uses Goal Exploration Processes to explore behavioral space and fill a replay buffer, then uses DDPG for sample-efficient policy improvement and fine-tuning.
Results
Across two benchmarks, GEP-PG was more sample efficient, achieved better absolute final performance, and had lower variance than DDPG trained from scratch; it led on Half-Cheetah, while GEP alone led on Continuous Mountain Car.
Takeaways & Limitations
Sequentially decoupling exploration from exploitation combines robust behavioral exploration with gradient-based fine-tuning, extending performance beyond DDPG on the larger Half-Cheetah benchmark.
Takeaways & Limitations
The study evaluates the algorithms on two benchmarks, and its final-performance metric can miss the best policies when learning progress is unstable.
Abstract
from arXiv · showhide
In continuous action domains, standard deep reinforcement learning algorithms like DDPG suffer from inefficient exploration when facing sparse or deceptive reward problems. Conversely, evolutionary and developmental methods focusing on exploration like Novelty Search, Quality-Diversity or Goal Exploration Processes explore more robustly but are less efficient at fine-tuning policies using gradient descent. In this paper, we present the GEP-PG approach, taking the best of both worlds by sequentially combining a Goal Exploration Process and two variants of DDPG. We study the learning performance of these components and their combination on a low dimensional deceptive reward problem and on the larger Half-Cheetah benchmark. We show that DDPG fails on the former and that GEP-PG improves over the best DDPG variant in both environments. Supplementary videos and discussion can be found at http://frama.link/gep_pg, the code at http://github.com/flowersteam/geppg.
1. Introduction
Deep RL methods are sample-efficient for fine-tuning but can explore poorly in sparse or deceptive continuous-action problems. GEP-PG sequentially combines GEP exploration with DDPG policy improvement and is evaluated on CMC and Half-Cheetah.
- Deep RL combines neural-network function approximation with reinforcement learning and has achieved strong results in discrete and continuous domains.
- DDPG explores by adding action noise, but exploration can be inefficient when rewards are sparse or gradients are flat or deceptive.
- Evolutionary and developmental methods emphasize exploration and are especially suited to rare or deceptive rewards in continuous action spaces.
- GEP-PG first uses Goal Exploration Processes to explore the continuous state-action space, then places the resulting samples in DDPG’s replay buffer for policy improvement.
- The study evaluates GEP, DDPG variants, and their sequential combination on Continuous Mountain Car and Half-Cheetah, examining performance, sample efficiency, and variability.
2. Related work
Exploration methods range from undirected noise to directed coverage of state-action or behavioral spaces. The paper positions its approach as sequentially combining evolutionary exploration with deep-RL fine-tuning.
- As deep RL addresses larger and harder environments, exploration becomes increasingly important, particularly with sparse rewards and continuous actions.
- Undirected exploration methods rely mainly on stochastic policies, action noise, or noise injected into neural-network parameters.
- Directed exploration seeks broad state-action coverage through count-based methods, intrinsic rewards, exploration bonuses, and option discovery.
- Novelty Search, Quality-Diversity, and curiosity-driven methods instead cover experimenter-defined behavioral feature spaces.
- The proposed approach combines evolutionary or developmental exploration with traditional deep-RL algorithms that fine-tune policy parameters.
3. Methods
The methods define DDPG and GEP, combine them sequentially in GEP-PG, evaluate them on CMC and Half-Cheetah, and use multi-seed comparisons with alternative performance metrics.
- DDPG: DDPG is a deterministic-policy-gradient algorithm using a replay buffer and target network, evaluated here with action or parameter perturbations.
- GEP: GEP searches policy parameters through an experimenter-designed outcome space of behavioral trajectory features.
- GEP: GEP bootstraps random policies, stores parameter-outcome pairs, and then samples outcome goals to expand its reaching capabilities.
- GEP-PG: GEP can observe task rewards during roll-outs and reuse them to initialize DDPG’s replay buffer and evaluate discovered policies.
- GEP-PG: GEP-PG generates diverse GEP samples first, loads them into DDPG’s replay buffer, and then performs policy learning with action- or parameter-perturbation variants.
- Benchmarks: The benchmarks are Continuous Mountain Car, with deceptive energy-penalized rewards, and Half-Cheetah, with 17-dimensional observations and 6-dimensional actions.
- Evaluation: The evaluation uses standardized implementations, 20 random seeds, paired t-tests, bootstrap confidence intervals, and means with standard errors across runs.
- Evaluation: The authors criticize averaging only the last 100 episodes and instead report performance for the best policy found throughout training.
4. Results
The experiments compare exploration strategies on CMC and Half-Cheetah, then evaluate GEP-PG against GEP and DDPG variants. GEP-PG improves performance, variability, and sample efficiency over DDPG on Half-Cheetah, while CMC exposes distinct exploration and stability effects.
- 4.1. Undirected exploration in DDPG: Parameter perturbation outperforms action perturbation on both CMC and Half-Cheetah across the reported performance metrics.All tests were significant at the 5% level.
- 4.1. Undirected exploration in DDPG: On CMC, noiseless DDPG performs poorly because it lacks the exploration needed to reach the reward, while decreasing noise improves over static noise.The authors conclude that stronger initial noise helps find the goal more often on CMC.
- 4.1. Undirected exploration in DDPG: On Half-Cheetah, no noise, static OU noise, and decreasing noise show no statistically different performance at 2M steps, although earlier differences may occur.The authors suggest that stronger early exploration can help CMC but may hurt Half-Cheetah.
- 4.2. Exploration efficiency of GEP and DDPG on CMC: GEP reaches the CMC goal faster than DDPG variants: its linear and complex policies average 3875 and 3773 steps, while DDPG succeeds before 5.104 steps in 42% and 22% of trials.The comparison uses parameter perturbation and action perturbation for the two DDPG variants.
- 4.2. Exploration efficiency of GEP and DDPG on CMC: GEP policy complexity does not appear decisive; performance depends on the ratio between the parameter space and the subspace of successful policy parameters.This conclusion concerns the CMC exploration experiments.
- 4.3. Combining GEP and DDPG into GEP-PG: On CMC, GEP remains stable at good performance, whereas DDPG with standard or parameter noise performs poorly and GEP-PG improves only slightly over DDPG without matching GEP.The plotted CMC learning performance reflects the ratio of trials finding the goal, with absolute metrics reported separately.
- 4.3. Combining GEP and DDPG into GEP-PG: On Half-Cheetah, GEP-PG significantly outperforms its DDPG counterparts on both metrics and reaches a new state-of-the-art performance.GEP alone with the linear policy performs statistically worse than the other methods.
- 4.3. Combining GEP and DDPG into GEP-PG: GEP-PG produces less variability than DDPG counterparts and remains stable across approximately 100 to 600 GEP episodes used to fill the replay buffer.Lower variability means efficient policies are found more consistently, which matters when performance must not fall below a specified level.
5. Discussion and conclusion
The paper decouples exploration from exploitation by combining GEP exploration with DDPG fine-tuning, improving performance across two continuous-action benchmarks. It also identifies scope boundaries and several directions for extending the approach.
- Decoupling exploration and exploitation: GEP-PG first explores behavioral diversity with GEP, then fine-tunes using DDPG initialized with GEP-generated replay-buffer samples.The two phases separate robust exploration from gradient-based exploitation.
- Results: Across two benchmarks, GEP-initialized DDPG is more sample efficient, achieves better absolute final performance, and has lower variance than DDPG trained from scratch.
- Results: GEP alone is preferred on simple deceptive CMC, whereas GEP-PG leads on the larger HC benchmark.
- Limits and future work: The evaluation remains limited to two benchmarks, with broader assessment identified as future work.
- Limits and future work: Future extensions include adaptive switching, alternative exploration methods, alternative exploitation algorithms, discrete-action settings, and goal-parameterized reinforcement learning.
- Limits and future work: Behavioral-feature spaces are commonly defined manually, leaving unsupervised feature learning and its effect on GEP-PG as open questions.
A. Study of DDPG variability
The study examines variability in DDPG by using many random seeds and repeated comparisons, addressing instability concerns in standard benchmark evaluations.
- Evaluation design: The authors use 20 random seeds rather than the commonly used five or fewer to reduce variance in evaluation.They also run 40 baseline trials and repeatedly resample sets of 20 runs.
- Evaluation design: The variability analysis compares sets of DDPG OU(0.3) runs separately on CMC and HC.
B. Correlation between evaluation metrics
The paper compares absolute and final performance metrics, showing that their relationship depends on benchmark stability and can be misleading when final performance is highly variable.
- Metric definitions: The absolute metric averages performance of the best controller found during training, whereas the final metric averages the last 100 test episodes.
- Metric comparison: Figure 4 reports correlations between the two metrics separately for CMC and HC, including best-fit lines, Pearson coefficients, and p-values.
- Metric comparison: On CMC, unstable learning makes the final metric highly variable, while the absolute metric usually indicates good performance.
C. Individual runs on CMC
Individual CMC runs reveal that standard DDPG with OU noise often fails to find the rewarding goal and is unstable even when it succeeds.
- Individual-run behavior: Most of 20 DDPG OU-noise runs on CMC never find the rewarding goal.
- Individual-run behavior: Runs that do find the goal remain unstable, producing a non-normal performance distribution.
D. Performance comparisons
The comparisons examine DDPG exploration variants, GEP, and GEP-PG using final and absolute performance, with statistical tests and confidence intervals. GEP-PG significantly outperforms DDPG under both action and parameter perturbation, while its absolute-performance distributions show smaller variance.
- Evaluation: Statistical comparisons use 2-sample t-tests and bootstrapped 95% confidence intervals for algorithm pairs on Half-Cheetah.Differences are treated as significant when p ≤ 0.05; positive bootstrap intervals exclude zero.
- DDPG variants: DDPG with parameter perturbation achieves significantly higher final and absolute performance than DDPG with action perturbation.Both metrics are significant in 2/2 tests.
- GEP comparison: DDPG with action perturbation achieves significantly higher final and absolute performance than GEP.Both metrics are significant in 2/2 tests.
- GEP-PG comparisons: GEP-PG achieves significantly higher final and absolute performance than DDPG with action perturbation.Both performance metrics are significant in 2/2 tests.
- GEP-PG comparisons: GEP-PG achieves significantly higher final and absolute performance than DDPG with parameter perturbation.Both performance metrics are significant in 2/2 tests.
- Variability: GEP-PG versions of DDPG show smaller variance in the Half-Cheetah absolute-performance histograms.The comparison is presented in Figure 6.
F. Influence of policy complexity in GEP
The section tests whether GEP policy complexity affects performance. Linear and two-hidden-layer policies have similar final performance, while the linear policy reaches higher Half-Cheetah performance sooner.
- Final performance: Linear and two-hidden-layer GEP policies show no significant difference in final performance on CMC or Half-Cheetah.No test over five shows significance.
- Sample efficiency: 2.105 steps marks a significant Half-Cheetah sample-efficiency difference favoring the linear GEP policy.The reported tests include p = 7.3 × 10−4, p = 2.3 × 10−4, and a bootstrapped difference of 738 with interval (395, 1080).
- Implication: A DDPG replay buffer of 2.105 samples filled by GEP would probably be more useful with a linear policy.The authors connect this to faster exploration of a smaller policy parameter space.
- Experimental comparison: Figure 7 compares GEP performance for a linear policy and a policy with two hidden layers of (64,64) neurons.The figure focuses on policy-architecture comparison.
G. Influence of the initial replay buffer content
The section studies how GEP-filled replay buffers affect GEP-PG. Performance depends on the quality and diversity of explored content more than buffer size within the tested range, and GEP-PG remains superior to undirected DDPG exploration strategies.
- Buffer content: GEP-PG performance correlates with the best and average performance of GEP policies.The reported correlations are p < 2 × 10−6 for the best policy and p < 4 × 10−8 for average policy performance.
- Alternative initialization: Replay buffers filled through parameter-space exploration do not impede DDPG performance and can reduce variance.The comparison includes DDPG parameter perturbations and random-policy samples against action-perturbation exploration.
- Overall comparison: GEP-PG outperforms all tested DDPG variants combined with undirected exploration strategies.Both final-metric tests are positive, while only the bootstrap test is positive for the absolute metric.
- Experimental comparison: Figure 10 compares replay-buffer initialization using DDPG parameter perturbations, random policy search, and GEP during an exploration phase before network updates.The exploration phase lasts fewer than 500 epochs.