PaperlayerView exact revision on arXiv

Source-linked AI summary

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov

arXiv:1707.06347v2cs.LG

TL;DR

Existing reinforcement learning methods leave room for improved scalability, data efficiency, and robustness. PPO alternates policy sampling with repeated first-order optimization of a clipped surrogate objective. It outperforms comparators on almost all tested continuous-control environments, while achieving better Atari sample complexity than A2C and similar sample complexity to ACER.

  • Problem

    Existing reinforcement learning methods leave room to improve scalability, data efficiency, and robustness across problems without hyperparameter tuning.

    Source: §1 · p. 1

  • Method

    PPO alternates policy sampling with multiple epochs of first-order optimization of a surrogate objective that clips policy probability ratios.

    Source: §1 · p. 1

  • Results

    PPO outperforms comparators on almost all continuous-control environments and achieves better Atari sample complexity than A2C and similar sample complexity to ACER.

    Source: §1 · p. 1, §6.2 · p. 7

  • Takeaways & Limitations

    PPO provides a simpler alternative to trust-region methods while retaining their stability and reliability and supporting joint policy–value architectures.

    Source: §7 · p. 8

    The adaptive KL-penalty variant occasionally yields policy updates whose KL divergence differs substantially from its target, though its penalty coefficient quickly adjusts.

    Source: §4 · p. 4

Abstract

from arXiv · show

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.

1 Introduction

PPO introduces a clipped-probability-ratio objective optimized using first-order methods over multiple epochs of sampled data. It outperforms compared algorithms on continuous control and improves Atari sample complexity over A2C while performing similarly to ACER.

1 Introduction

  • The paper targets scalable, data-efficient, robust learning, addressing Q-learning failures and vanilla policy gradients’ poor data efficiency and robustness [Mni+16].Q-learning with function approximation remains poorly understood. DQN works well in discrete-action game environments, but strong continuous-control performance was not demonstrated on benchmarks such as OpenAI Gym [Bro+16] and those described by Duan et al. [Dua+16].Source: §1 · p. 1
  • TRPO [Sch+15b] is relatively complicated and incompatible with architectures using noise, such as dropout, or parameter sharing between policies, value functions, and auxiliary tasks.These restrictions motivate a simpler optimization approach.Source: §1 · p. 1
  • PPO alternates policy sampling with several optimization epochs, using a clipped-probability-ratio objective that provides a pessimistic lower bound on policy performance.The algorithm seeks TRPO’s data efficiency and reliable performance while requiring only first-order optimization.Source: §1 · p. 1
  • PPO outperforms compared algorithms on continuous control; on Atari, its sample complexity is significantly better than A2C and similar to ACER, despite greater simplicity.Among the tested surrogate-objective variants, clipped probability ratios perform best.Source: §1 · p. 1

2 Background: Policy Optimization

Policy gradient methods alternate sampling and optimization, but repeatedly optimizing the same trajectories can produce destructively large policy updates. TRPO [Sch+15b] constrains update size, while fixed penalties are insufficient for the desired first-order alternative.

2.1 Policy Gradient Methods

  • Policy gradient methods feed estimated policy gradients into stochastic gradient ascent, alternating sampling and optimization.The estimator uses a stochastic policy and estimated timestep advantages, with expectations computed as finite-batch empirical averages. Automatic differentiation obtains the estimator by differentiating an objective function.Source: §2.1 · p. 2
  • Repeatedly optimizing the standard policy gradient objective on the same trajectory is not well-justified and often produces destructively large policy updates.The reported behavior was similar to or worse than the “no clipping or penalty” setting.Source: §2.1 · p. 2

2.2 Trust Region Methods

  • TRPO [Sch+15b] maximizes a surrogate objective subject to a constraint on policy-update size.Conjugate gradient approximately solves the problem after linearizing the objective and quadratically approximating the constraint.Source: §2.2 · p. 2
  • A fixed penalty coefficient β with SGD is insufficient to emulate TRPO’s [Sch+15b] monotonic improvement in a first-order algorithm; additional modifications are required.The theoretical penalty formulation follows from a surrogate using maximum rather than mean KL over states that lower-bounds policy performance. Choosing one β is difficult across problems and as learning changes within a problem.Source: §2.2 · p. 2

3 Clipped Surrogate Objective

PPO’s clipped surrogate takes the minimum of clipped and unclipped objectives, creating a pessimistic lower bound that discourages excessively large policy updates while retaining changes that worsen the objective.

3 Clipped Surrogate Objective

  • Unconstrained maximization of the conservative policy iteration surrogate can produce excessively large updates, motivating penalties when the new-to-old action-probability ratio moves away from 1.The probability ratio equals 1 at the old policy parameters.Source: §3 · p. 3
  • The clipped surrogate takes the minimum of unclipped and ratio-clipped objectives, removing incentives for large policy changes while retaining changes that worsen the objective.The clipping interval is [1 − ϵ, 1 + ϵ], with ϵ = 0.2 given as an example. This minimum forms a pessimistic lower bound on the unclipped objective.Source: §3 · p. 3
  • The clipped and unclipped objectives agree to first order at the old policy but diverge as the policy changes.For individual timestep terms, effective clipping occurs at 1 + ϵ for positive advantages and 1 − ϵ for negative advantages.Source: §3 · p. 3

4 Adaptive KL Penalty Coefficient

An adaptive KL-divergence penalty targets a specified divergence at each policy update and can replace or supplement the clipped surrogate objective. In experiments, the penalty performed worse than clipping but remains an important baseline.

4 Adaptive KL Penalty Coefficient

  • The adaptive KL penalty performed worse than the clipped surrogate objective in experiments, but is included as an important baseline.It can be used instead of the clipped surrogate objective or alongside it.Source: §4 · p. 4
  • Each policy update uses several epochs of minibatch SGD to optimize a KL-penalized objective, with the penalty coefficient adapted to target KL divergence dtarg.The updated coefficient β is used for the next policy update.Source: §4 · p. 4
  • Large deviations from the target KL divergence occur rarely, and β adjusts quickly.The heuristic parameters 1.5 and 2 have little effect on sensitivity. The initial β is another hyperparameter but is unimportant in practice because it adjusts quickly.Source: §4 · p. 4

5 Algorithm

PPO alternates fixed-length data collection with multiple epochs of minibatch surrogate optimization. Its actor-critic implementation combines policy and value learning, optionally adding an entropy bonus for exploration.

5 Algorithm

  • PPO requires only a minor policy-gradient implementation change: replace LPG with LCLIP or LKLPEN and perform multiple stochastic gradient-ascent steps.With automatic differentiation, the replacement surrogate can be constructed and differentiated directly.Source: §5 · p. 4
  • When policy and value networks share parameters, the objective must combine the policy surrogate with value-function error, optionally adding entropy to encourage exploration.The entropy bonus follows earlier work [Mni+16].Source: §5 · p. 5
  • Truncated generalized advantage estimation accumulates temporal-difference residuals within trajectory segments and reduces to the bootstrapped segment-return estimator when λ = 1.The recurrent-network-friendly implementation popularized in [Mni+16] collects T timesteps, much shorter than an episode, and requires advantage estimates that do not look beyond T.Source: §5 · p. 5
  • Each PPO iteration collects T timesteps from each of N parallel actors, then optimizes the surrogate over the NT samples for K epochs.Minibatch size M ≤ NT; Adam [KB14] usually performs better than SGD. After optimization, the updated policy becomes the old policy for subsequent collection.Source: §5 · p. 5

6 Experiments

Clipped PPO outperforms prior methods on almost all tested continuous-control environments. Further experiments examine surrogate objectives, showcase humanoid control, and compare algorithms on Atari.

6.1 Comparison of Surrogate Objectives

  • 7 simulated robotics tasks, each trained for one million timesteps, support a comparison of the clipped surrogate with alternative objectives and ablations.Removing clipping and penalties yielded a negative normalized score because HalfCheetah performed worse than the initial random policy. Fixed and adaptive KL penalties were considered, while log-space clipping offered no improvement.Source: §6.1 · pp. 5–6, Table 1 · p. 6

6.2 Comparison to Other Algorithms in the Continuous Domain

6.3 Showcase in the Continuous Domain: Humanoid Running and Steering

  • PPO is showcased on 3D humanoid tasks involving forward running, steering toward changing targets, and getting up while being pelted by cubes.The steering task changes the target every 200 timesteps or whenever the goal is reached.Source: §6.3 · p. 7

6.4 Comparison to Other Algorithms on the Atari Domain

  • PPO is compared with tuned A2C [Mni+16] and ACER [Wan+16] on 49 Atari games using learning-speed and final-performance metrics.The metrics are average reward per episode over the entire training period and over the last 100 training episodes. Game winners are determined by averaging each metric across three trials.Source: §6.4 · p. 8, Table 2 · p. 8

7 Conclusion

Proximal policy optimization (PPO) uses multiple epochs of stochastic gradient ascent per policy update, combining trust-region stability and reliability with simpler implementation, broader applicability, and better overall performance.

  • PPO retains the stability and reliability of trust-region methods while achieving better overall performance.Source: §7 · p. 8
  • PPO performs each policy update using multiple epochs of stochastic gradient ascent.Source: §7 · p. 8
  • PPO requires only a few lines of changes to a vanilla policy gradient implementation and applies in more general settings.These settings include using a joint architecture for the policy and value function.Source: §7 · p. 8

A Hyperparameters

PPO hyperparameters are specified for MuJoCo, Roboschool, and Atari experiments, including KL-based Adam stepsize adjustment in Roboschool and linear annealing in Atari.

  • 2048 is the horizon (T), and 3 × 10^-4 is the Adam stepsize for the MuJoCo 1 million timestep benchmark.Source: A Hyperparameters · p. 10, Table 3 · p. 10
  • Roboschool experiments adjusted the Adam stepsize based on the target KL divergence.Source: Table 4 · p. 10
  • 1 to 0 is the range over which α is linearly annealed during learning in the Atari experiments.Source: Table 5 · p. 10

B Performance on More Atari Games

PPO is compared with A2C on 49 Atari games through learning curves for three random seeds and mean final scores after 40M game frames.

B Performance on More Atari Games

  • Three random seeds are shown individually in the learning curves, complementing the mean-performance comparison.Figure 6 contains the learning curves, while Table 6 reports mean performance.Source: B Performance on More Atari Games · p. 10
  • 49 Atari games—all Atari games included in OpenAI Gym at publication—form the benchmark comparing PPO with A2C.Figure 6 presents the comparison across this collection.Source: B Performance on More Atari Games · p. 10, Figure 6 · p. 11
  • 40M game frames (10M timesteps) is the evaluation horizon for PPO and A2C’s mean final scores over the last 100 episodes.Table 6 describes this comparison, but the supplied passage contains no score values establishing which method performs better.Source: B Performance on More Atari Games · p. 10, Table 6 · p. 12

Related papers

Loading 1707.06347v2…