Source-linked AI summary
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, Olivier Bachem
TL;DR
Deep RL performance depends on numerous implementation choices that are often underreported, complicating comparisons and attribution of progress. The paper implements more than 50 choices in a unified on-policy framework, trains over 250’000 agents across five continuous-control environments, and derives practical recommendations. A key finding is that policy initialization can significantly improve training speed.
Problem
Underreported implementation choices and differing code bases make it difficult to determine whether RL improvements come from algorithms or implementations.
Method
The paper implements more than 50 choices in a unified on-policy agent and evaluates them across five continuous-control environments using grouped large-scale experiments.
Results
Policy initialization significantly influences performance, and an observation-independent initial action distribution with zero mean and low standard deviation improves training speed.
Takeaways & Limitations
The study provides practical insights and recommendations for on-policy training of continuous-control RL agents.
Abstract
from arXiv · showhide
In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.
1 Introduction
Deep RL implementations contain many low- and high-level choices that can strongly affect agent performance, yet these choices are often underreported. This paper studies them systematically in a unified framework and finds that policy initialization can significantly influence training speed.
- Unreported implementation choices can be crucial for, or even drive, good deep RL performance.
- Comparing algorithms implemented in different code bases makes it difficult to attribute improvements to algorithms rather than implementations.
- The study implements >50 choices in a unified on-policy algorithm and trains more than 250’000 agents across a large-scale experimental study.
- Policy initialization significantly influences performance, although it is rarely mentioned in RL publications.
- Initializing action distributions with zero mean, rather low standard deviation, and observation independence significantly improves training speed.
2 Study design
The study evaluates configurable on-policy RL choices for continuous control through unified experiments across five environments. It groups potentially interacting choices, samples configurations at scale, and scores agents by learning speed and final performance.
- The study defines on-policy learning as iterating between generating experience with the current policy and using it to improve that policy.The benchmark includes Hopper-v1, Walker2d-v1, HalfCheetah-v1, Ant-v1, and Humanoid-v1.
- The unified agent exposes commonly used losses, architectures, heuristics, and hyperparameters as configurable choices.The implementation is based on the SEED RL code base and labels choices for reference in Appendix B.
- Randomly sampling many choices can produce sub-optimal settings that prevent learning, with failure probability increasing exponentially as the number of choices grows.
- Interacting choices must be tuned together because varying one while fixing related choices can yield misleading conclusions.The study groups choices thematically and includes Adam learning rate in every group to account for broad interactions.
- The conditional 95th percentile estimates performance when other choices in the group are tuned by random search with a limited budget.The study also examines value distributions among the top 5% of configurations to identify over-represented choices.
- Each sampled configuration trains three independently seeded models, and its performance score is the median score across seeds based on average undiscounted returns during training.Training lasts one or two million environment steps depending on the environment, with evaluations every 100,000 steps.
3 Experiments
The experiments compare policy losses, network architectures, normalization, advantage estimation, training procedures, timestep handling, optimizers, and regularization in on-policy continuous control. Across these choices, performance is often sensitive to implementation details, with practical recommendations varying by environment and training objective.
- Policy Losses: PPO outperforms other policy losses on 4 of 5 environments and remains among the top performers after tuning loss-specific hyperparameters.PPO also outperforms other losses on Humanoid and Ant when optimal loss hyperparameters are selected.
- Networks architecture: Separate value and policy networks generally perform better, while policy width requires tuning and wider value networks show no apparent downside.Separate networks improve performance on four of five environments, but overly narrow or wide policy networks can significantly reduce performance.
- Networks architecture: A carefully initialized policy with small final-layer weights and low initial action standard deviation substantially improves training performance.The initial action distribution should be nearly observation-independent, centered around zero, and have a small standard deviation; smaller final-layer weights alone boost Humanoid performance by 66%.
- Normalization and clipping: Observation normalization is broadly important, whereas value normalization is strongly environment-dependent and gradient clipping provides only a small boost.Value normalization helps on HalfCheetah, Humanoid, Hopper, and Ant but significantly hurts Walker2d.
- Advantage Estimation: GAE and V-trace outperform N-step returns, while Huber loss and PPO-style value-loss clipping perform worse than MSE or no value clipping.The study recommends GAE with λ = 0.9 and avoiding both Huber loss and PPO-style value-loss clipping.
- Training setup and timestep handling: Repeated experience passes and advantage recomputation improve training, while discounting is highly influential and optimizer differences are comparatively small.The recommended starting points include recomputing advantages once per data pass, tuning γ per environment from 0.99, and recognizing that many parallel environments may hurt sample complexity.
4 Related Work
Prior work identifies reproducibility problems in reinforcement learning, including code-base differences, hyperparameter sensitivity, and stochasticity from random seeds. It also shows that reported gains can sometimes originate from implementation details rather than the attributed algorithmic innovation.
- Prior studies report performance differences across RL code bases, sensitivity to hyperparameter tuning, and substantial stochasticity from random seeds.
- Tucker et al. found that gains attributed to a policy-gradient improvement were instead caused by implementation details.
- This paper is closely related to earlier work on reproducibility issues and implementation-dependent performance in reinforcement learning.
5 Conclusions
The study evaluates a broad set of high- and low-level choices in on-policy learning through more than 250’000 experiments, identifying implementation decisions that affect training and performance. It examines data handling, advantage estimation, losses, regularization, episode termination, and action parameterization, while providing practical recommendations.
- Study scope: More than 250’000 experiments across five continuous control environments evaluate the impact of high- and low-level on-policy learning choices.The unified framework exposes these choices as configuration options for systematic comparison.
- Data and advantage handling: Experience can be organized as fixed trajectories, shuffled trajectories, shuffled transitions, or shuffled transitions with advantages recomputed each epoch.Shuffling transitions increases minibatch diversity but can make advantage estimates somewhat stale; recomputing them avoids that staleness.
- Advantage estimation: N-step returns and GAE(λ) provide alternative advantage-estimation choices, with their parameters controlling the bias–variance trade-off.Larger N produces less bias and more variance, while λ controls the corresponding trade-off for GAE.
- Losses: The value network may use MSE or Huber loss, while policy optimization includes PPO, V-trace, V-MPO, and the newly introduced RPA loss.Huber loss is quadratic near zero and linear beyond a threshold; RPA repeats positive-advantage actions and is described as a limiting case of AWR and V-MPO.
- Episode handling: Handling abandoned episodes requires distinguishing fixed step-limit terminations from ordinary terminal transitions when constructing advantages and value targets.One option treats the final transition as terminal; another sets the final advantage to zero and bootstraps its value target from the current value function.
- Action and regularization choices: Action parameterization can clip Gaussian actions to a bounded range or transform them with tanh, which changes action density while leaving specified policy-loss and KL terms unaffected.The environments commonly require actions in [−1, 1], motivating explicit handling of the Gaussian distribution’s unbounded outputs.
D.2 Results
This experiment reports performance quantiles and training curves for sampled training, policy-loss, and architecture choices. It also analyzes how individual choices and their sub-choices relate to high-performing configurations.
- Aggregate results: Performance quantiles summarize outcomes across sampled choice configurations.The aggregate results are presented in Table 3, with training curves in Figure 3 and per-choice analyses in Figures 5–13.
- Policy losses: Policy-loss comparisons evaluate the 95th percentile of performance conditioned on policy-loss hyperparameters.Figure 5 compares different policy losses, while Figure 4 shows performance distributions conditioned on Policy loss (C14).
- Experimental design: The experiment samples 4000 configurations per environment with choices drawn independently and uniformly from specified ranges.Each sampled configuration is evaluated using three agents trained with different random seeds; unspecified choices use Appendix C defaults.
- Architecture choices: Separate policy and value networks perform better than shared networks in the rerun restricted to the separate-network variant.The authors report rerunning the experiment after observing better performance for separate policy and value function networks.
- Architecture choices: The architecture analysis covers network sharing, layer widths and depths, initialization, activation, scaling, standard-deviation parameterization, and action transformation.Figures 10–15 analyze these choices and their sub-choices, including separate versus shared policy and value networks.
E.2 Results
This experiment examines normalization, clipping, optimization, and network-design choices using aggregate performance quantiles, training curves, and per-choice analyses. It samples these choices across five environments and evaluates each configuration with multiple seeds.
- Aggregate results: Performance quantiles summarize outcomes across configurations, with training curves and per-choice analyses reported separately.Table 4 gives aggregate statistics, Figure 16 shows training curves, and Figures 17–30 provide per-choice analyses.
- Per-choice analysis: Per-choice figures report the 95th percentile of performance conditioned on each choice and the distribution of choices among the top 5% of configurations.This format is used throughout the choice analyses in Figures 17–30.
- Experimental design: The experiment samples 2000 configurations per environment with choices drawn independently and uniformly from specified ranges.Each sampled configuration is evaluated using three agents trained with different random seeds; unspecified choices use Appendix C defaults.
F.2 Results
This experiment evaluates advantage estimation, value losses, environment parallelism, normalization, clipping, and optimization choices. Results are organized around aggregate quantiles, training curves, and per-choice analyses across five environments.
- Aggregate results: Performance quantiles summarize the sampled configurations, complemented by training curves and per-choice analyses.Table 5 reports aggregate statistics, Figure 31 shows training curves, and Figures 32–38 analyze individual choices.
- Experimental design: The experiment samples 4000 configurations per environment and evaluates each configuration with three agents using different random seeds.The sampled choices include environment count, value-function loss, PPO-style value clipping, advantage estimator, and Adam learning rate.
- Value losses: The study also analyzes value-loss choices and their associated Huber delta and PPO-style value-clipping settings.Value-function loss compares Huber and MSE, while Huber delta and clipping thresholds are sampled as sub-choices.
- Advantage estimation: Advantage-estimator analysis compares GAE, N-step, and V-Trace, with additional sampling of estimator-specific hyperparameters.The sub-choices include GAE λ, N-step N, and V-Trace λ and c, ρ parameters.
G.2 Results
This experiment studies data-collection, batching, optimization, and advantage-estimation choices through aggregate statistics, training curves, and per-choice analyses. It compares performance distributions and high-performing choice frequencies across five environments.
- Aggregate results: Performance quantiles summarize the experiment, with training curves and per-choice analyses reported in Figures 40–50.Table 6 reports aggregate statistics and Figure 39 presents training curves.
- Advantage estimation: Advantage-estimator results compare GAE, N-step, and V-Trace using their sampled hyperparameters.The study reports that GAE and V-Trace appear to outperform N-step returns, with no significant difference found between GAE and V-Trace.
- Value losses: PPO-style value-loss clipping hurt performance regardless of the clipping threshold.The result is reported for the value-clipping choice analyzed in Figure 43.
- Advantage estimation: λ = 0.9 performed well for both GAE and V-Trace across all tasks, although per-environment tuning may provide modest gains.This finding concerns the GAE λ and V-Trace advantage λ sub-choices.
- Experimental design: The experiment samples 2000 configurations per environment across iteration size, batch mode, epochs, environments, learning rate, and batch size.Each configuration is evaluated using three agents trained with different random seeds, while unspecified choices use Appendix C defaults.
H.2 Results
This experiment reports aggregate performance statistics, training curves, and per-choice analyses for several rollout, batching, and optimization choices. It samples configurations across five environments and evaluates three random-seed agents per configuration.
- The study reports aggregate performance quantiles, training curves, and per-choice analyses for choices C1–C5 and C24.The per-choice analyses cover iteration size, batch mode, number of epochs, number of environments, batch size, and Adam learning rate.
- For each of five environments, 2000 configurations were sampled by independently and uniformly varying the investigated choices.The varied choices included frame skip, handling abandoned episodes, and Adam learning rate; other choices used Appendix C defaults.
- Each sampled configuration was evaluated by training 3 agents with different random seeds and computing the Section 2 performance metric.
I.2 Results
This experiment reports aggregate statistics, training curves, and per-choice analyses for learning-rate decay and optimizer-related choices. Configurations were sampled across five environments and evaluated across three random seeds.
- The study reports aggregate performance quantiles, training curves, and per-choice analyses for learning-rate decay, optimizer, and optimizer sub-choices.The analyses include Adam and RMSProp settings such as momentum, epsilon, and learning rate.
- For each of five environments, 2000 configurations were sampled by independently and uniformly varying learning-rate decay and optimizer choices.Other choices were fixed to Appendix C defaults.
- Adam configurations varied momentum, epsilon, and learning rate, while RMSProp configurations varied centeredness, momentum, epsilon, and learning rate.
- Each sampled configuration was evaluated by training 3 agents with different random seeds and computing the Section 2 performance metric.
J.2 Results
This experiment reports aggregate statistics, training curves, and per-choice analyses for regularization strategies and their coefficients or thresholds. It samples regularization configurations across five environments and evaluates three random-seed agents per configuration.
- For each of five environments, 4000 configurations were sampled by independently and uniformly varying regularization-related choices.The sweep included constraint, no regularization, and penalty strategies, with associated regularizers, coefficients, and thresholds.
- The experiment varied coefficients and thresholds for KL terms, entropy, and the mean and standard-deviation components of decoupled KL(µ||π).
- Penalty and constraint settings included KL(µ||π), KL(π||µ), KL(ref||π), decoupled KL(µ||π), and entropy alternatives.
- Each sampled configuration was evaluated by training 3 agents with different random seeds and computing the Section 2 performance metric.
K.2 Results
This experiment reports aggregate performance statistics, training curves, and per-choice analyses for regularization approaches. The figures examine conditioned performance and the distribution of choices among top-performing configurations.
- The study reports aggregate performance quantiles, training curves, and per-choice analyses for regularization type and regularizer choices.The analyses span Figures 76–92 and include penalty, constraint, entropy, and KL-related settings.
- Figure 76 compares the 95th percentile of performance across different regularization approaches conditioned on regularization type.
- The analyses cover coefficients and thresholds for KL(µ||π), KL(π||µ), KL(ref||π), decoupled KL(µ||π), and entropy.