Source-linked AI summary
Proximal Policy Optimization Algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov
TL;DR
Reinforcement learning needs policy-gradient methods that are more data-efficient and robust without TRPO's complexity. The paper introduces PPO, which alternates sampling with multiple minibatch optimization epochs using a clipped surrogate objective. Across continuous-control and Atari tasks, PPO performs strongly while offering a favorable balance of simplicity, sample complexity, and overall performance.
Problem
Reinforcement learning methods need improved scalability, data efficiency, robustness, and architectural generality, while existing policy-gradient and trust-region approaches have important limitations.
Method
PPO alternates policy sampling with several optimization epochs and uses a clipped probability-ratio surrogate objective as a first-order policy-optimization method.
Results
PPO outperforms the compared methods on almost all continuous-control environments, performs better than A2C in Atari sample complexity, and performs similarly to ACER while being simpler.
Takeaways & Limitations
PPO combines trust-region-like stability and reliability with simpler implementation, broader applicability, and better overall performance.
Takeaways & Limitations
Adaptive KL penalties can occasionally produce policy updates whose KL divergence differs significantly from the target, although these cases are rare and the coefficient quickly adjusts.
Abstract
from arXiv · showhide
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
1 Introduction
Reinforcement learning methods still face trade-offs among scalability, data efficiency, robustness, and implementation complexity. PPO addresses these goals with a clipped objective and performs favorably across continuous-control and Atari benchmarks.
- Motivation: Existing approaches trade off scalability, data efficiency, robustness, and implementation simplicity.Q-learning can fail on simple problems, vanilla policy gradients have poor data efficiency and robustness, and TRPO is complicated or architecture-limited.
- Proposed approach: PPO introduces clipped probability ratios to support multiple optimization epochs on sampled data while targeting TRPO-like data efficiency and reliable performance with first-order optimization.The method alternates between sampling from the policy and optimizing the sampled data for several epochs.
- Empirical findings: The clipped-ratio surrogate performs best among the tested surrogate-objective variants.
- Empirical findings: PPO outperforms the compared algorithms on continuous-control tasks and has better sample complexity than A2C on Atari, with similar performance to ACER.On Atari, PPO is reported to be much simpler than ACER.
2 Background: Policy Optimization
Policy optimization estimates policy gradients from sampled trajectories, but repeatedly optimizing the same loss can produce destructively large updates. Trust-region methods constrain updates, yet their penalties and optimization procedures are difficult to tune and implement broadly.
- Policy gradients: Policy gradient methods estimate a policy gradient from sampled data and use it in stochastic gradient ascent.The estimator is represented through an objective whose gradient equals the policy-gradient estimator.
- Policy gradients: The empirical average is computed over a finite batch in an algorithm alternating between sampling and optimization.
- Policy-gradient limitation: Multiple optimization steps on the same policy-gradient loss are not well-justified and often produce destructively large policy updates.
- Trust-region methods: TRPO maximizes a surrogate objective subject to a constraint on policy-update size and approximately solves the problem with conjugate gradients after local approximations.
- Trust-region methods: A fixed KL-penalty coefficient is difficult to choose across problems or throughout learning, so first-order emulation of TRPO requires additional modifications.
3 Clipped Surrogate Objective
PPO clips probability ratios in a surrogate objective to limit incentives for excessively large policy updates while preserving a pessimistic performance bound. The clipped objective matches the conservative objective near the starting policy and becomes more restrictive as the policy moves away.
- The probability ratio r_t(θ) compares the updated policy’s action probability with the old policy’s, and equals 1 at θ_old.
- The clipped surrogate objective uses a minimum between the unclipped and clipped terms, producing a pessimistic lower bound on the unclipped objective.The clipping removes incentives to move the probability ratio outside [1 −ϵ, 1 +ϵ] when that movement would improve the objective.
- L_CLIP and L_CPI agree to first order around θ_old, but diverge as the policy moves farther from the starting parameters.
- On Hopper-v1, the updated policy after one PPO iteration has KL divergence about 0.02, where L_CLIP is maximal along the interpolated update direction.
4 Adaptive KL Penalty Coefficient
PPO also considers adapting a KL-divergence penalty coefficient toward a target divergence during policy updates. In experiments, this penalty performed worse than the clipped surrogate objective, while its coefficient generally adjusted quickly despite occasional deviations.
- The KL-penalty alternative adapts its coefficient to target a specified KL divergence for each policy update.
- The KL penalty performed worse than the clipped surrogate objective in the experiments.
- The adaptive procedure performs several epochs of minibatch SGD on the KL-penalized objective during each policy update.
- KL divergence sometimes differs significantly from the target, but such updates are rare and the penalty coefficient quickly adjusts.
5 Algorithm
PPO alternates collecting trajectory data and optimizing a surrogate loss with minibatch stochastic optimization. The algorithm can combine the policy surrogate with value-function and entropy terms, and it supports fixed-length segments collected by parallel actors.
- Implementations replace the usual policy-gradient loss with L_CLIP or L_KLPEN and perform multiple stochastic-gradient-ascent steps.
- The actor-critic objective combines the policy surrogate with a value-function error term and may include an entropy bonus for exploration.
- Advantage estimates can use finite-horizon returns or truncated generalized advantage estimation, with λ = 1 recovering the finite-horizon form.
- Each PPO iteration collects T timesteps from N parallel actors, constructs a surrogate loss over NT samples, and optimizes it for K epochs.
6 Experiments
The experiments evaluate PPO surrogate objectives and compare PPO with prior algorithms across continuous-control, humanoid, and Atari benchmarks. PPO generally performs strongly, with clipped ratios selected as the best surrogate variant and PPO outperforming previous methods on almost all continuous-control environments.
- 6.1 Comparison of Surrogate Objectives: The clipped surrogate objective was compared with natural variations and ablated versions under different hyperparameters.
- 6.1 Comparison of Surrogate Objectives: The continuous-control benchmark used 7 MuJoCo robotics tasks trained for one million timesteps, with scores averaged over 21 runs.
- 6.2 Comparison to Other Algorithms in the Continuous Domain: PPO outperforms the previous methods on almost all the continuous control environments.
- 6.3 Showcase in the Continuous Domain: Humanoid Running and Steering: The humanoid experiments test forward locomotion, target-directed running with changing targets, and running while recovering from cube impacts.
- 6.4 Comparison to Other Algorithms on the Atari Domain: The Atari evaluation compares PPO with well-tuned A2C and ACER implementations using the same policy-network architecture.
7 Conclusion
PPO uses multiple epochs of stochastic gradient ascent for each policy update. The methods combine trust-region-like stability and reliability with simpler implementation, broader applicability, and better overall performance.
- PPO performs each policy update with multiple epochs of stochastic gradient ascent.
- PPO methods have the stability and reliability of trust-region methods while requiring only a few lines of code change to vanilla policy gradients.
- PPO applies in more general settings, including joint architectures for the policy and value function, and has better overall performance.
A Hyperparameters
The appendix lists PPO hyperparameters for the MuJoCo, Roboschool, and Atari experiments. The supplied values include the MuJoCo horizon and Adam stepsize, plus experiment-specific settings.
- MuJoCo: The MuJoCo benchmark uses horizon T = 2048 and Adam stepsize 3 × 10^-4.
- MuJoCo: Table 3 covers PPO hyperparameters for the MuJoCo 1 million timestep benchmark.
- Roboschool: Table 4 covers PPO hyperparameters for the Roboschool experiments, with Adam stepsize adjusted using a target KL-divergence value.
- Atari: Table 5 covers Atari hyperparameters, with α linearly annealed from 1 to 0 during learning.
B Performance on More Atari Games
The paper extends Atari evaluation to 49 games, comparing PPO and A2C through learning curves and mean final scores. Table 6 reports final performance after 40M game frames, or 10M timesteps.
- The expanded Atari evaluation covers 49 games and compares PPO with A2C.
- Figure 6 shows learning curves for each of three random seeds.
- Table 6 reports mean final scores over the last 100 episodes after 40M game frames, equivalent to 10M timesteps.