Source-linked AI summary
Deep Reinforcement Learning that Matters
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger
TL;DR
Deep RL results are difficult to reproduce and interpret because algorithmic variance and benchmark non-determinism can produce widely varying outcomes. The paper experimentally investigates these sources of variation in policy-gradient continuous-control methods, evaluates significance-analysis techniques, and proposes reporting and experimental recommendations. It concludes that careful significance analysis, multiple trials, matched baselines, and complete implementation details are important for meaningful comparisons.
Problem
Deep RL lacks sufficiently reproducible experiments, standardized reporting, and significance measures for determining whether reported improvements are meaningful.
Method
The paper experimentally studies intrinsic and extrinsic sources of non-determinism in policy-gradient continuous-control methods and evaluates confidence bounds and bootstrap power analysis.
Results
The experiments find numerous sources of non-determinism and show that significance analysis, bootstrapping, and power analysis can provide insight into the reliability and required number of trials for performance comparisons.
Takeaways & Limitations
Meaningful deep RL comparisons require multiple random-seed trials, matched baseline implementations, tuned hyperparameters, significance analysis, and complete reporting of experimental details.
Takeaways & Limitations
The investigated significance metrics are an initial set and may not be the best fit for comparing all RL algorithms.
Abstract
from arXiv · showhide
In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.
Introduction
Deep RL's rapid growth makes reproducible experiments and reliable comparisons increasingly important, yet reported results vary because of methodological and environmental factors. The paper studies these challenges in policy-gradient methods and advocates more careful evaluation and reporting.
- Motivation: Reproducing deep RL results is difficult because reported baseline performance varies widely across studies.Both extrinsic factors, such as hyperparameters and codebases, and intrinsic factors, such as random seeds and environment properties, contribute to this variability.
- Scope: The study focuses on policy-gradient methods for continuous control, examining reproducibility across commonly used algorithms and implementations.The examined methods include TRPO, DDPG, PPO, and ACKTR.
- Evaluation: The paper demonstrates that significance testing can help assess whether reported performance gains and evaluation metrics are reliable.It applies techniques from machine learning and statistics to evaluate confidence in algorithm performance gains.
- Experimental focus: The experiments investigate how hyperparameters, random seeds, trial averaging, environment characteristics, stochasticity, and codebases affect algorithm comparisons.The analysis poses questions about these factors while holding other conditions as constant as possible where appropriate.
- Reporting: The authors argue that detailed experimental procedures and reporting are necessary for reproducing and accurately comparing new and baseline methods.Their recommendations include reporting implementation details, experimental setup, hyperparameters, and evaluation methods.
Experimental Analysis
The experiments examine how hyperparameters, reward scaling, random seeds, implementations, and benchmark environments affect deep RL reproducibility and reported performance. Results show that these choices can substantially alter outcomes and complicate fair baseline comparisons.
- Experimental Design: The study varies hyperparameters, random seeds, environment characteristics, stochasticity, and codebases to investigate factors affecting reproducibility.Five trials with different preset seeds are generally used, and results report means and standard errors across seeds.
- Network Architecture: Network architecture and activation choices significantly affect performance, but their effects are inconsistent across algorithms and environments.The authors report that ReLU or Leaky ReLU usually perform best, while larger networks may require retuning clipping or learning-rate hyperparameters.
- Reward Scale: Reward rescaling can have a large but inconsistent effect across environments and scaling values, with small settings below σ̂ = 0.01 causing failure to learn in one DDPG example.Layer normalization changes how the rescaling factor affects results.
- Random Seeds and Trials: Different random seeds can produce statistically different performance distributions even with identical hyperparameters, making small or selective trial averages potentially misleading.Ten trials were split into two groups of five; on HalfCheetah, the resulting learning curves could fall outside the same distribution.
- Environments: Benchmark environment dynamics change algorithm rankings: DDPG performs best in stable HalfCheetah-v1, while its gains diminish in unstable Hopper-v1, and TRPO outperforms others in Swimmer-v1.Swimmer returns can reflect a local optimum rather than the desired behavior, so the authors recommend showing policy demonstrations alongside returns.
- Implementation Differences: Implementation differences that publications often omit can dramatically change final performance after training on 2M samples.The authors argue that implementation details should be enumerated and codebases packaged with publications.
Reporting Evaluation Metrics
Evaluation in deep RL should distinguish learning views, use metrics suited to the evaluation setting, and quantify uncertainty and significance rather than relying on raw return summaries alone. The paper examines bootstrap confidence bounds, power analysis, and significance tests while noting that existing corrected tests may not fit RL data assumptions.
- Maximum returns are typically inadequate for fair comparison because many RL algorithms are unstable; average returns are also commonly reported.
- Online View vs. Policy Optimization: Policy optimization evaluates a target policy offline across the full task trajectory, whereas the online view evaluates returns across the learning process.
- Confidence Bounds: Bootstrap resampling estimates means and 95% confidence bounds, which can vary widely across algorithms and environments.TRPO and PPO show small confidence bounds in the reported environment experiments.
- Power Analysis: Bootstrap power analysis can indicate whether more trials are needed by measuring how often a uniformly lifted sample produces statistically significant values.A small percentage of significant simulations suggests increasing the sample size.
- Significance: The paper examines 2-sample t-tests, Kolmogorov-Smirnov tests, and bootstrap percent differences with 95% confidence intervals to assess reported performance gains.These measures are intended to indicate whether a novel algorithm is truly performing as the state-of-the-art.
- Significance: Corrected significance tests developed for supervised learning may rely on assumptions that do not necessarily apply in reinforcement learning.The paper identifies further work as necessary to investigate properly corrected significance tests for RL.
Discussion and Conclusion
The experiments identify intrinsic and extrinsic sources of non-determinism that hinder reproducibility in deep RL and motivate significance analysis. The authors recommend standardized reporting, careful baseline validation, and further investigation of hyperparameter-agnostic algorithms and evaluation methods.
- Intrinsic factors such as random seeds and environment properties, together with extrinsic factors such as hyperparameters and codebases, hinder reproducing baseline algorithms.The study focuses on policy-gradient methods for continuous control and examines these sources through experiments.
- Hyperparameter searches should recover the original reported baseline performance, and comparison implementations should match original codebase results when available.The authors note that hyperparameters can have significantly different effects across algorithms and environments.
- Significance testing is needed to determine whether higher average returns represent better performance, with bootstrapping and power analysis informing the number of trial runs.The paper highlights confidence intervals, bootstrapping, and power analysis as useful tools for evaluating performance gains.
- Reproducibility requires reporting hyperparameters, implementation details, experimental setup, and evaluation methods for both baselines and novel methods.The paper links incomplete implementation and reporting details to wasted effort reproducing state-of-the-art work.
- Future work includes hyperparameter-agnostic algorithms and improved statistical methods for determining the significance of RL performance gains.The authors specifically highlight reward scaling and the need to identify better-fitting significance metrics.
- Benchmark results should be accompanied by demonstrated real-world application settings because no clear winner exists across all benchmark environments.The authors argue that new methods should address in what settings they would be useful.
Supplemental Material
The supplemental material documents experiment configurations and reported metrics for policy-gradient methods in continuous-control MuJoCo tasks. It also contains the paper’s experimental results for several baseline algorithms.
- The supplement reviews experiment configurations from related policy-gradient work in continuous-control MuJoCo tasks from OpenAI Gym.It includes detailed hyperparameter lists and reported metrics used in the literature.
- The supplemental material includes experimental results for DDPG, TRPO, PPO, and ACKTR baseline algorithms.
Literature Reviews
The literature review finds inconsistent reporting and substantial variability in baseline results across related deep RL studies. It emphasizes that network choices, evaluation metrics, trial counts, and codebases complicate comparisons across Hopper and HalfCheetah experiments.
- Network architectures and activation functions are often reported inconsistently across related literature, and these hyperparameters can significantly affect algorithm performance.
- The review organizes reported evaluation hyperparameters, related Hopper and HalfCheetah comparisons, trial counts, and evaluation metrics.These materials are represented in Tables 4–8.
- Reported baseline results vary across related work on the Hopper-v1 and HalfCheetah-v1 environments.
- Using the same hyperparameter configuration with different codebase implementations can produce significantly different experimental results.
Experimental Setup
The experiments evaluate DDPG, TRPO, PPO, and ACKTR implementations in Hopper-v1 and HalfCheetah-v1 using configurations drawn from related work. They examine evaluation procedures, network architectures, activation functions, and their effects on performance.
- Experimental Setup: Experiments use policy-gradient algorithms DDPG, TRPO, PPO, and ACKTR in the OpenAI Gym Hopper-v1 and HalfCheetah-v1 environments.Available implementations from OpenAI rllab and OpenAI Baselines are used, with each experiment performed over five trials.
- Experimental Setup: The study uses reported configurations spanning policy and value-network architectures, activation functions, optimization settings, discounting, and evaluation parameters.Examples include ReLU, tanh, ELU, linear outputs, learning rates, batch sizes, GAE, and KL constraints.
- Experimental Setup: DDPG evaluation is modified to average performance over 10 full trajectories at the end of each epoch.This replaces evaluation based on a partial trajectory evaluated across several policies.
- Network Experiments: The experiments compare ReLU, tanh, and Leaky ReLU activations while examining the effects of network sizes and configurations.The configurations are taken from related work to assess the importance of careful fine tuning.
- Network Experiments: The ACKTR value-network structure can significantly affect performance, whereas TRPO performance is strongly affected by the policy architecture.
- Network Experiments: DDPG experiments separately vary critic and actor activation functions and network sizes while holding the other network’s configuration at its default setting.
Reward Scaling Parameter in DDPG
DDPG performance is sensitive to reward scaling, with the effect depending on the continuous-control environment and influencing both stability and performance.
- Reward Scaling Parameter in DDPG: Figures 20 and 21 compare DDPG reward rescaling with and without layer normalization on Hopper-v1 and HalfCheetah-v1.
- Reward Scaling Parameter in DDPG: Reward scaling can significantly affect DDPG stability and performance on HalfCheetah-v1 and Hopper-v1.The study examines several reward scaling parameters across both environments.
Batch Size in TRPO
TRPO performance varies with batch size, and experiments using two implementations identify 1024 as the best-performing batch size in HalfCheetah-v1 and Hopper-v1.
- Batch Size in TRPO: 1024 is the best-performing TRPO batch size in both HalfCheetah-v1 and Hopper-v1.Performance degrades consecutively as batch size increases beyond 1024.
- Batch Size in TRPO: The batch-size experiments compare the original TRPO code with the OpenAI baselines implementation.The corresponding experiments are reported in Figures 22 and 23.
Random Seeds
Random seeds can materially change observed policy-gradient results, while algorithm performance also varies substantially across environments.
- Random Seeds: Identical hyperparameters can produce statistically different result distributions solely from random-seed partitioning.Ten trials were split into two groups of five seeds for DDPG and TRPO experiments.
- Random Seeds: No single policy-gradient algorithm performs consistently best across Hopper, HalfCheetah, Swimmer, and Walker.TRPO performs strongly on Swimmer but poorly on HalfCheetah, with only a marginal advantage over PPO on Hopper.
- Random Seeds: Environment choice can bias reported algorithm-performance comparisons.The study links this variation to the environments used when reporting results.
Codebases
Algorithm codebases can change observed performance even when implementations share nominal hyperparameters, network comparisons, and experimental settings.
- Codebases: The study compares TRPO policy and value-network structures and activations across implementations.Figures 27–29 summarize the original TRPO and rllab network experiments.
- Codebases: DDPG network structures and activations are examined for rllab++ and rllab implementations.Figures 30–33 report the corresponding DDPG network experiments.
- Codebases: TRPO has multiple commonly used implementations, including OpenAI Baselines, OpenAI rllab, and the original TRPO codebase.The paper analyzes how codebase choice affects performance across these implementations.
- Codebases: TRPO and DDPG codebases are compared using default hyperparameters while holding hyperparameters constant across implementations.The TRPO comparison uses a larger 20k-sample batch for rllab and original TRPO code, as optimized in separate experiments.
Significance
The paper evaluates performance significance using multiple statistical tests, bootstrap confidence intervals, and power analysis across several benchmark environments. Results are organized by environment and algorithm in Tables 9–14.
- Testing procedure: Performance significance is assessed from five sample trials using 10k bootstrap resamples to obtain confidence intervals.The analysis also compares sorted average returns with a normal two-sample t-test and a Kolmogorov-Smirnov test.
- Environment comparisons: Tables 9–12 report sorted two-sample t-tests, Kolmogorov-Smirnov tests, and bootstrap A/B percentage differences with 95% confidence bounds for HalfCheetah, Hopper, Walker2d, and Swimmer.Each table presents significance values and metrics for different algorithms within one environment.
- Bootstrap estimates: Table 13 reports bootstrap means with 95% confidence bounds across environments.This complements the significance-test tables with interval estimates for the environment-level results.
- Power analysis: Table 14 analyzes the predicted significance of a 25% lift using the percentages of insignificant, positive-significant, and negative-significant simulations.The power analysis evaluates how often the predicted improvement is classified under these three outcomes.