PaperlayerView exact revision on arXiv

Source-linked AI summary

Truly Proximal Policy Optimization

Yuhui Wang, Hao He, Chao Wen, Xiaoyang Tan

arXiv:1903.07940v2cs.LGcs.AIstat.ML

TL;DR

PPO’s optimization behavior is not fully proximal: it neither strictly restricts likelihood ratios nor enforces a well-defined trust region. The paper proposes Truly PPO, adding rollback and trust-region-based clipping, and reports improved policy performance and sample efficiency with guaranteed monotonic improvement of the ultimate policy performance.

  • Problem

    PPO may not strictly restrict likelihood ratios or enforce a well-defined trust-region constraint, despite the latter’s role in theoretical performance guarantees.

  • Method

    Truly PPO combines rollback with trust-region-based clipping to better confine policy differences while using first-order optimization.

  • Results

    Truly PPO’s resulting surrogate objective provides guaranteed monotonic improvement, while benchmark experiments report improved policy performance and sample efficiency.

  • Takeaways & Limitations

    The results highlight confining policy differences as necessary for stable policy improvement and indicate that KL-divergence-based constraints outperform likelihood-ratio-based ones.

  • Takeaways & Limitations

    In one reported evaluation, the method did not reach the reward threshold within the required timesteps on all seeds.

Abstract

from arXiv · show

Proximal policy optimization (PPO) is one of the most successful deep reinforcement-learning methods, achieving state-of-the-art performance across a wide range of challenging tasks. However, its optimization behavior is still far from being fully understood. In this paper, we show that PPO could neither strictly restrict the likelihood ratio as it attempts to do nor enforce a well-defined trust region constraint, which means that it may still suffer from the risk of performance instability. To address this issue, we present an enhanced PPO method, named Truly PPO. Two critical improvements are made in our method: 1) it adopts a new clipping function to support a rollback behavior to restrict the difference between the new policy and the old one; 2) the triggering condition for clipping is replaced with a trust region-based one, such that optimizing the resulted surrogate objective function provides guaranteed monotonic improvement of the ultimate policy performance. It seems, by adhering more truly to making the algorithm proximal - confining the policy within the trust region, the new algorithm improves the original PPO on both sample efficiency and performance.

1. Introduction

Policy-gradient methods must balance learning stability against learning speed because improper policy-update step sizes can cause severe degradation. PPO simplifies trust-region optimization through clipping, but the paper finds that PPO does not strictly restrict likelihood ratios or enforce a trust-region constraint; Truly PPO addresses both issues with rollback and trust-region-based clipping.

  • Motivation: Improper policy-update step sizes can cause severe policy degradation because training data depends strongly on the current policy.This makes the trade-off between learning stability and learning speed central for policy-gradient methods.
  • Prior approaches: TRPO controls KL divergence with a trust-region constraint, but its second-order optimization is computationally inefficient and difficult to scale.PPO reduces this complexity by using a clipping mechanism instead of imposing the hard constraint completely.
  • Research gap: PPO’s optimization behavior remains insufficiently understood, especially whether clipping restricts likelihood ratios and enforces a trust-region-like constraint.These questions matter because the trust-region constraint is theoretically linked to performance guarantees.
  • Findings: The paper finds that PPO neither strictly restricts likelihood ratios nor enforces a trust-region constraint.The likelihood-ratio issue arises from residual incentives to push the policy away, while the trust-region issue reflects differences between PPO’s and TRPO’s constraints.
  • Contribution: Truly PPO adds rollback and trust-region-based clipping, combining theoretical justification with first-order optimization and improving policy performance and sample efficiency on benchmark tasks.The expanded version further combines trust-region-based clipping with rollback on KL divergence and reports stronger theoretical and practical performance.

2. Preliminaries

The preliminaries define policy-gradient objectives, likelihood ratios, advantage values, and policy parameterizations before introducing TRPO’s KL-based trust region and PPO’s clipping mechanism. PPO’s minimum operation clips only when the clipped objective improves, while out-of-range ratios can otherwise lose gradient information.

  • MDP and performance: An MDP is specified by state and action spaces, transition probabilities, rewards, an initial-state distribution, and a discount factor.The policy performance is defined through the expected reward under the policy’s state and action distributions.
  • Policy gradients: Policy gradients update the policy using a surrogate objective based on the likelihood ratio between new and old policies and the old policy’s advantage.The likelihood ratio is rπold_s,a(π)=π(a|s)/πold(a|s), while the advantage measures the old policy’s advantage value.
  • Policy parameterization: Policies are represented by neural networks, with discrete policies outputting action distributions and continuous policies commonly modeled as Gaussian distributions.For continuous actions, neural networks output the Gaussian mean and covariance matrix.
  • Trust-region policy optimization: TRPO constrains KL divergence between old and new policies, and maximizing its surrogate bound guarantees non-decreasing performance of the new policy.The KL-based constraint is the trust-region constraint used to justify the performance guarantee.
  • Proximal policy optimization: PPO uses a clipping range (1−ϵ, 1+ϵ) to restrict the likelihood ratio, with 0<ϵ<1 controlling the range.Its objective takes the minimum of clipped and unclipped terms to form a lower bound on the unclipped objective.
  • Clipping behavior: The minimum operation clips only when the objective value improves, whereas direct clipping can lose gradient information after ratios leave the range.When the clipping condition holds, the gradient becomes zero, which can stop the ratio from moving outward.

3. Analysis of the “Proximal” Property of PPO

The analysis shows that PPO’s clipping mechanism does not strictly bound likelihood ratios and that likelihood-ratio bounds do not guarantee bounded KL divergence. These results motivate examining both ratio behavior and trust-region behavior as distinct properties of PPO.

  • Research questions: PPO is investigated through two questions: whether it bounds likelihood ratios within the clipping range and whether it enforces a trust-region constraint.The second question is tied to theoretical performance guarantees based on KL divergence.
  • Likelihood-ratio restriction: PPO can prevent likelihood ratios from exceeding the clipping range too much, but it cannot strictly bound them.The paper’s analysis identifies residual outward-driving incentives in the overall objective as the reason clipping is not a strict bound.
  • Outward movement: Under the theorem’s conditions, an already out-of-range likelihood ratio can be driven farther beyond the clipping range.Across benchmark tasks, the relevant condition occurred in 25% to 45% of more than 1 million samples.
  • Theoretical settings: The analysis considers discrete policies with at least three actions and continuous Gaussian policies, including one-dimensional and higher-dimensional action spaces.These settings support the paper’s examination of the relationship between ratio-defined regions and KL divergence.
  • Trust-region analysis: Bounding a likelihood ratio does not necessarily bound KL divergence, because the KL divergence within the ratio-defined sublevel set can grow without bound.Figure 2 illustrates this gap by comparing ratio sublevel sets with KL-divergence level sets.

4. Method

The method addresses PPO’s inability to strictly confine likelihood ratios or enforce a trust-region constraint by combining rollback with trust-region-based clipping. The resulting Truly PPO framework retains first-order optimization while providing a theoretically justified monotonic-improvement property.

  • PPO with Rollback (PPO-RB): PPO’s clipping function does not strictly confine likelihood ratios because the overall objective can continue pushing ratios beyond the clipping range.This motivates modifying the clipping function and adding a rollback mechanism.
  • PPO with Rollback (PPO-RB): The rollback objective reverses its slope when the likelihood ratio crosses the clipping range, rather than leaving the objective flat outside that range.The function uses α > 0 to determine rollback force.
  • PPO with Rollback (PPO-RB): PPO-RB introduces a negative incentive outside the clipping range to counteract the objective’s incentive to push likelihood ratios farther away.The rollback strength is controlled by α, and sufficiently large α can guarantee confinement within the clipping range.
  • Trust Region-based PPO (TR-PPO): Because bounding likelihood ratios does not suffice to bound KL divergence, TR-PPO replaces PPO’s ratio-based clipping trigger with a trust-region-based condition.Clipping is triggered when D_KL(π_old, π) ≥ δ, while first-order optimization avoids a difficult hard constraint.
  • Combining TR-PPO with Rollback (Truly PPO): Truly PPO combines trust-region-based clipping with rollback on KL divergence, applying a negative incentive outside the trust region.The method uses KL divergence rather than likelihood-ratio rollback because its goal is to restrict KL divergence.
  • Combining TR-PPO with Rollback (Truly PPO): The combined objective is reported to retain first-order optimization, improve robustness to the coefficient α, and provide guaranteed monotonic improvement.The clipping strategy applies the penalty only outside the trust region, reducing coefficient-adjustment difficulty without altering the monotonic-improvement property.

5. Related Work

Prior policy-optimization methods differ in how they restrict updates, the metric they constrain, and whether policy boundedness is theoretically guaranteed. PPO's ratio-based clipping is computationally simple but does not theoretically bound the policy, whereas rollback addresses this issue.

  • Restriction approaches: TRPO explicitly constrains policy updates using KL divergence, while PPO variants use clipping and PPO-penalty uses an adaptive KL penalty.TRPO requires second-order optimization; clipping- and penalty-based methods can use stochastic gradient descent.
  • Optimization behavior: Clipping-based methods seek a sub-optimal solution within a policy constraint because updates stop when the constraint is violated, despite often performing better in practice.The paper contrasts this behavior with constraint-based optimization within the feasible region.
  • Boundness: Constraint-based and penalty-based methods theoretically bound the policy, whereas original clipping-based methods suffer from an unbounded-policy problem.The paper states that incorporating rollback relieves this issue and theoretically guarantees boundedness.
  • Policy metric: PPO and PPO-RB use likelihood-ratio metrics, while other compared algorithms use KL divergence.KL divergence imposes a summation constraint over actions, whereas likelihood-ratio bounds are element-wise.
  • Policy metric: Bounding likelihood ratios at individual actions does not necessarily bound KL divergence, making KL divergence more theoretically justified for trust-region analysis.The paper also reports that ratio constraints can trap policies in bad local optima when initialization is poor.

6. Experiment

Experiments compare PPO, rollback and trust-region variants, and off-policy baselines on MuJoCo and Atari while tracking policy restriction and reward. PPO does not strictly enforce its clipping range, whereas Truly PPO improves restriction and often learning speed and final reward, especially on harder tasks.

  • Experimental setup: Experiments evaluate likelihood ratios, KL divergence, and episode rewards during training on MuJoCo and Atari benchmarks.MuJoCo uses 10 seeds and up to 1 or 20 million timesteps; Atari uses 4 seeds and 10 million timesteps.
  • 6.2 The Effect on Policy Restriction: PPO fails to strictly bound likelihood ratios: over 30% exceed the clipping range on Hopper, Reacher, and Walker2d, while maximum ratios exceed 4 on all tasks.The upper clipping range is 1.2, and maximum KL divergence grows with timestep.
  • 6.2 The Effect on Policy Restriction: Removing clipping or using the loose PPO-0.6 range produces larger likelihood ratios and KL divergences, with performance failing and fluctuating dramatically across tasks.These comparisons show that clipping still contributes to policy restriction and learning despite not strictly enforcing the ratio bound.
  • 6.2 The Effect on Policy Restriction: Rollback reduces out-of-range likelihood ratios and ratio magnitudes, while trust-region clipping reduces KL divergence relative to PPO.Truly PPO's KL divergence can be slightly larger than TR-PPO because it retains the likelihood-ratio term outside the trust region.
  • 6.3 The Effect on Policy Performance: Truly PPO outperforms PPO on hard high-dimensional tasks but is comparable on easier low-dimensional tasks.It needs about 60% and 50% of PPO's timesteps on Walker2d and Hopper, with about 15% and 24% higher final rewards; on Breakout it achieves almost twice PPO's final rewards.
  • Ablation results: Rollback variants outperform their non-rollback counterparts on most tasks, although PPO-RB's Atari improvements over PPO are not significant.Truly PPO performs better than TR-PPO on five of six MuJoCo and five of six Atari tasks.
  • Overall findings: Across compared methods, performance generally increases with stronger policy restriction, from unclipped A2C through loose and proper clipping to rollback methods.The reported ordering of restriction ability is A2C, PPO-0.6, PPO/TR-PPO, then PPO-RB/Truly PPO.
  • 6.3 The Effect on Policy Performance: Trust-region methods are generally more sample efficient and achieve higher rewards on most MuJoCo tasks than ratio-based methods.TR-PPO learns faster than PPO on almost all six MuJoCo tasks and obtains much higher reward on four of six tasks.

7. Conclusion

The paper identifies theoretical weaknesses in PPO’s policy-confinement mechanisms and proposes trust-region clipping with rollback to improve stability. Its conclusion emphasizes policy-difference confinement and the influence of constraint metrics on algorithmic behavior.

  • 7. Conclusion: PPO neither strictly bounds the likelihood ratio nor enforces a well-defined trust region constraint.The paper attributes these issues to residual incentives to push the policy away and differences between PPO and TRPO’s constraints.
  • 7. Conclusion: Trust region-based clipping provides stronger theoretical justification, while rollback improves the method’s ability to restrict policy changes.Together, the techniques improve policy confinement and training stability.
  • 7. Conclusion: The results highlight confining policy differences as necessary for stable policy improvement.The conclusion presents this as a central lesson from the proposed methods.
  • 7. Conclusion: KL divergence-based constraints outperform likelihood ratio-based constraints in the reported results.The paper calls for further study of how the selected policy metric affects algorithmic behavior.
  • 7. Conclusion: The proposed methods are described as simple to implement and tune as PPO while performing much better in practice.The authors suggest they may serve as useful alternatives to PPO.

Appendix A. Implementation Details

The appendix records the hyperparameters used for the proposed methods on Mujoco and Atari tasks.

  • Appendix A. Implementation Details: Table 4 lists hyperparameters for the proposed methods on Mujoco tasks.
  • Appendix A. Implementation Details: Table 5 lists hyperparameters for the proposed methods on Atari tasks.
Loading 1903.07940v2…