Source-linked AI summary
Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
Riashat Islam, Peter Henderson, Maziar Gomrokchi, Doina Precup
TL;DR
The paper addresses the difficulty of reproducing DDPG and TRPO baseline results in continuous-control reinforcement learning because of hyper-parameter sensitivity and external variance. It systematically varies experimental settings and random seeds, finding substantial cross-environment variation and developing reporting recommendations for fairer comparisons.
Problem
Reproducing and fairly comparing DDPG and TRPO is difficult because hyper-parameter choices, environment stochasticity, and other randomness produce varying reported results.
Method
The authors vary individual hyper-parameters while holding others at defaults, average results across five runs, and conduct ten-trial random-seed experiments on Hopper and Half-Cheetah.
Results
Performance variation from hyper-parameter tuning is larger in Half-Cheetah than Hopper, while results across continuous-control domains are not always consistent.
Takeaways & Limitations
Baseline comparisons should report hyper-parameters, averages, standard deviations, and results averaged over more than five trials using different random seeds.
Takeaways & Limitations
Further investigation is needed to determine how many trials are necessary to ensure fair comparisons with similarly distributed returns.
Abstract
from arXiv · showhide
Policy gradient methods in reinforcement learning have become increasingly prevalent for state-of-the-art performance in continuous control tasks. Novel methods typically benchmark against a few key algorithms such as deep deterministic policy gradients and trust region policy optimization. As such, it is important to present and use consistent baselines experiments. However, this can be difficult due to general variance in the algorithms, hyper-parameter tuning, and environment stochasticity. We investigate and discuss: the significance of hyper-parameters in policy gradients for continuous control, general variance in the algorithms, and reproducibility of reported results. We provide guidelines on reporting novel results as comparisons against baseline methods such that future researchers can make informed decisions when investigating novel methods.
1 Introduction
The paper examines why reproducing DDPG and TRPO results in continuous-control reinforcement learning is difficult, focusing on algorithmic variance, hyper-parameter choices, and experimental consistency. It argues that careful reporting and consistent baselines are needed for meaningful comparisons.
- Motivation: DDPG and TRPO are widely used as baseline algorithms, yet few studies reproduce existing policy-gradient methods before comparison.The study uses implementations that appear in several prior works.
- Performance measures: The paper compares performance using Average Return because Maximum Average Return and Maximum Return are considered highly biased, while Average Return and Standard Deviation are more stable measures.Final results display all metrics, but Average Return is used as the primary comparison measure.
- Sources of variation: Hyper-parameter choices can produce large variations in reported DDPG and TRPO baseline results, making comparisons difficult without detailed and fair tuning.Different cited works use different experimental hyper-parameters, and suboptimal settings may yield inaccurate baseline comparisons.
- Research objective: The authors recommend consistent reporting and experimentation because DDPG and TRPO are susceptible to hyper-parameters and external sources of variance or randomness.These sources include environment stochasticity and implementation choices.
2 Experimental Analysis
The experiments evaluate DDPG and TRPO across MuJoCo Hopper and Half-Cheetah tasks, varying hyper-parameters while holding others constant and averaging runs. Results show that architecture, batch size, algorithm-specific settings, and random seeds can substantially affect performance and reproducibility.
- Experimental setup: The study evaluates DDPG and TRPO on Hopper and Half-Cheetah using MuJoCo and OpenAI Gym implementations.Experiments build on TensorFlow implementations of TRPO and DDPG.
- Experimental setup: Hyper-parameter variations cover batch size, network architecture, step size, regularization, GAE λ, reward scale, and actor-critic learning rates.Each investigated parameter is varied across commonly used values while the others remain at defaults; results are averaged over 5 runs.
- Common hyper-parameters: The (400,300) architecture significantly outperforms smaller architectures on Half-Cheetah for both TRPO and DDPG, while architecture matters less on Hopper.The architecture used as best in these experiments differs from architectures used in some reported baseline results.
- Common hyper-parameters: DDPG shows similar Hopper performance with mini-batches of 32 and 64, whereas batch size 128 significantly improves Half-Cheetah performance.The analysis compares mini-batch sizes 32, 64, and 128.
- Common hyper-parameters: For TRPO, batch size 25000 performs best, while smaller batch sizes plateau at a lower optimum on Half-Cheetah.The study evaluates batch sizes of 1000, 5000, and 25000 over 5000 iterations.
- Algorithm-specific hyper-parameters: TRPO’s default step size 0.01 generally performs best, lower GAE λ improves longer Half-Cheetah sequences, and reward rescaling does not improve DDPG.For Half-Cheetah, no reward scaling (RS=1) yields much higher returns than the reported value RS=0.1.
- General variance: Averaging different groups of 5 runs can produce substantially different learning curves, and fine-tuned DDPG remains unstable across both environments.The paper therefore emphasizes averaging many runs with different random seeds when reporting results.
3 Discussion and Conclusion
The paper finds that reported TRPO and DDPG baselines vary substantially across implementations, metrics, hyper-parameters, environments, and random trials, complicating fair reproduction and comparison. It therefore recommends more complete reporting and consistent experimental procedures.
- Results averaged over five runs vary across related works, with some reports similar to the paper’s findings and others differing substantially.The comparison covers TRPO and DDPG baseline results across Hopper and Half-Cheetah.
- The authors caution that maximum average return can mislead comparisons and recommend reporting averages and standard deviations instead.Maximum Return is also omitted because high-variance policies and environments can produce unusually large outlying values.
- Half-Cheetah is more sensitive to hyper-parameter tuning than Hopper, while Hopper exhibits larger external variance from quick failure modes.The paper associates these differing patterns with environmental stochasticity and failure behavior.
- The number of trials needed to ensure fair comparisons remains unresolved and requires further investigation.The paper asks for the N at which N-sample averages reliably produce similarly distributed returns.
- The paper recommends reporting all hyper-parameters and trial counts, using averages and standard deviations, and averaging many trials with different random seeds.These recommendations address variance from hyper-parameters, environment randomness, and random seeds.