Source-linked AI summary
Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry
TL;DR
Deep RL performance is difficult to attribute because algorithms combine mechanisms whose individual effects are poorly understood. The paper compares PPO and TRPO through implementation analysis and controlled variants, finding that code-level optimizations account for much of PPO’s reward advantage and fundamentally alter algorithm operation. These findings support modular, fine-grained evaluation of RL methods.
Problem
Deep RL methods are brittle and difficult to reproduce, while the effects of their component mechanisms on agent behavior remain poorly understood.
Method
The paper studies PPO and TRPO by analyzing their implementations and controlling code-level optimizations in comparative algorithm variants.
Results
Code-level optimizations account for much of PPO’s improvement over TRPO and fundamentally change algorithm operation, including trust-region behavior.
Takeaways & Limitations
Reliable attribution of deep RL progress requires modular methods and fine-grained analysis of each component’s effects on training and behavior.
Takeaways & Limitations
Computational constraints limited the full ablation to the first four identified optimizations.
Abstract
from arXiv · showhide
We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at https://github.com/MadryLab/implementation-matters .
1 INTRODUCTION
Deep RL methods are brittle, difficult to reproduce, and poorly understood at the component level. This paper studies PPO and TRPO to identify how code-level optimizations affect reward, training dynamics, and algorithmic behavior.
- Deep RL methods remain brittle, hard to reproduce, unreliable across runs, and sometimes inferior to simple baselines.
- The paper asks how the many mechanisms in deep RL training algorithms affect agent behavior separately and jointly.
- The authors analyze PPO and TRPO using cumulative reward and finer-grained measures of algorithmic properties.
- Code-level optimizations account for much of PPO’s observed reward improvement and are instrumental to its performance.
- These optimizations fundamentally alter algorithm operation, including trust region enforcement, rather than merely improving auxiliary implementation details.
- PPO code-level optimizations matter more for final reward than the general algorithm choice between TRPO and PPO, complicating rigorous comparisons.
- The findings motivate modular RL methods whose components are evaluated for effects on behavior and performance.
2 RELATED WORK
Related work established policy gradients and trust-region methods while documenting brittleness, reproducibility problems, and strong dependence on implementation choices in deep RL.
- REINFORCE introduced gradient estimates for updating neural-network RL agents, and later work unified such methods under policy gradients.
- TRPO and PPO are prominent deep RL policy-gradient algorithms, with PPO motivated as a refinement of TRPO and both linked to trust-region ideas.
- Prior studies reported brittleness, reproducibility and experimental-practice issues, codebase-dependent rewards, and benchmark performance matched by simple randomized search.
3 ATTRIBUTING SUCCESS IN PROXIMAL POLICY OPTIMIZATION
The study investigates PPO implementations and finds that several non-trivial optimizations strongly affect performance and sensitivity, motivating a minimal PPO variant for attribution.
- Standard PPO implementations contain non-trivial optimizations that are absent or only barely described in the corresponding paper relative to TRPO.
- The examined optimizations include value-function clipping, reward scaling, orthogonal initialization, and learning-rate annealing.
- Value clipping replaces the standard value-network regression objective with a PPO-like objective that clips values around previous estimates.
- Reward scaling divides rewards by the standard deviation of a rolling discounted reward sum without subtracting and re-adding the mean.
- A full ablation over all configurations of the four optimizations, with learning-rate grid search and multiple seeds, found dramatic performance effects.
- PPO-M removes these code-level optimizations by using the standard value loss, no reward scaling, default initialization, and fixed-rate Adam.
- The results show that deep policy-gradient algorithms are highly sensitive to implementation choices beyond environmental brittleness.
- The full ablation was limited to the first four identified optimizations because of computational-resource constraints.
4 CODE-LEVEL OPTIMIZATIONS HAVE ALGORITHMIC EFFECTS
Code-level optimizations can alter the algorithmic behavior of PPO and TRPO beyond their effects on final reward, particularly in trust-region enforcement.
- Trust regions in TRPO and PPO: TRPO constrains mean KL divergence using a second-order KL approximation and natural-gradient descent over states observed in the current trajectory.The practical method replaces worst-case state constraints with an observed-trajectory mean-KL approximation.
- Trust regions in TRPO and PPO: PPO replaces TRPO’s KL-constrained update with a clipping objective that avoids explicit projection and simplifies parameter updates.This design was intended to make training faster and more sample-efficient than TRPO.
- Trust regions in TRPO and PPO: Trust regions are central to policy-gradient methods because update directions are guaranteed predictive only near the current policy.TRPO and PPO seek to constrain policy variation, but their trust-region properties are not directly observable from final rewards.
- Code-level optimizations have algorithmic effects: Optimizing PPO’s clipped objective makes the enforced trust region depend heavily on the optimization method rather than the objective alone.Because the initial policy ratio is one, the first update is identical to maximizing the unclipped surrogate; step size therefore depends on surrogate steepness.
- Code-level optimizations have algorithmic effects: All three algorithms maintain a KL-based trust region with optimal parameters, yet all fail to maintain the ratio-based trust region.This occurs despite PPO and PPO-M being trained directly with a ratio-clipping objective.
- Code-level optimizations have algorithmic effects: PPO and PPO-M enforce different KL dynamics despite sharing the same core algorithm: PPO peaks mid-training, whereas PPO-M rises with iterations.The results indicate that auxiliary optimizations can fundamentally alter agent behavior and the principles motivating the core algorithms.
5 IDENTIFYING ROOTS OF ALGORITHMIC PROGRESS
The study disentangles PPO’s clipping mechanism, TRPO/PPO step choices, and code-level optimizations, finding that implementation augmentations often matter more for reward than the core algorithm choice.
- Controlled comparison: The authors compare PPO and TRPO variants that combine each core step with or without PPO’s code-level optimizations.TRPO+ pairs the TRPO core with PPO-inspired optimizations, while PPO-M removes them; together, the variants cover all combinations.
- Relative contributions: Code-level optimizations contribute more to increased performance than switching between PPO and TRPO on benchmark tasks.After fixing whether optimizations are used, the core algorithm does not seem to have a significant reward impact across tasks.
- Relative contributions: 17% and 21% improvements occur for PPO and TRPO, respectively, on Hopper-v2 when equipped with code-level optimizations.These figures illustrate the contrast between optimization effects and the smaller apparent effect of core step choice.
- Relative contributions: AAI measures the maximal effect of switching step algorithms, whereas ACLI measures the maximal effect of adding optimizations for a fixed step algorithm.The metrics quantify algorithmic choice versus code-level optimization effects.
- Role of clipping: PPO-NOCLIP uses common code-level optimizations without PPO’s clipping mechanism and performs uniformly better than PPO-M.PPO-M uses the core PPO clipping mechanism without the code-level optimizations.
- Role of clipping: Table 3 reports little reward difference between PPO and PPO without clipping when both use code-level optimizations.The results indicate that clipping is often comparable to or less important than the code-level optimizations.
6 CONCLUSION
The paper identifies code-level optimizations as major determinants of deep policy-gradient performance and operation. It concludes that modular analysis is necessary for understanding and attributing algorithmic successes and failures.
- Findings: The study finds that code-level optimizations have a drastic effect on agent performance in TRPO and PPO.These augmentations appear only in implementations or as auxiliary details in algorithm presentations.
- Findings: PPO’s improvement over TRPO and stochastic gradient descent can be largely attributed to code-level optimizations.The optimizations fundamentally change algorithm operation in ways not predicted by the conceptual policy-gradient framework.
- Implications: The findings support designing deep RL methods modularly and measuring each component’s effects on performance and underlying training behavior.The paper states that careful component-level analysis is needed to attribute successes and failures in complicated deep RL systems.
A.1 EXPERIMENTAL SETUP
The experimental setup selects hyperparameters through grid searches and reports results with bootstrapped 95% confidence intervals across MuJoCo tasks.
- A.1 EXPERIMENTAL SETUP: Hyperparameters for PPO and TRPO were selected through grid searches over shared parameters and algorithm-specific controls.PPO grids the value-function learning rate, clipping constant, and learning-rate schedule; TRPO replaces the schedule with the KL constraint.
- A.1 EXPERIMENTAL SETUP: The appendix provides algorithm hyperparameter tables for Walker2d-v2, Humanoid-v2, and Hopper-v2.
- A.1 EXPERIMENTAL SETUP: 95% confidence intervals are obtained through bootstrapped sampling.
A.2 PPO CODE-LEVEL OPTIMIZATIONS
PPO includes a scaling optimization that normalizes rewards using running statistics of discounted returns.
- A.2 PPO CODE-LEVEL OPTIMIZATIONS: The PPO scaling optimization tracks the mean and standard deviation of discounted rewards through running statistics.
- A.2 PPO CODE-LEVEL OPTIMIZATIONS: Each reward is divided by the standard deviation of the running discounted-return statistics before being returned as the scaled reward.
A.3 TRUST REGION OPTIMIZATION
The trust-region analysis measures reward, probability-ratio behavior, and KL divergence for Humanoid agents using both collected and heldout state-action pairs.
- A.3 TRUST REGION OPTIMIZATION: Figure 3 measures per-step mean reward, maximum ratio, mean KL, and maximum-versus-mean KL on state-action pairs collected during training.Each line is a separate agent’s training curve, measured every twenty five steps; the black dotted line marks PPO’s 1 + ϵ ratio constraint.
- A.3 TRUST REGION OPTIMIZATION: Figure 4 measures the same quantities on state-action pairs from heldout trajectories.The figure uses separate-agent curves, measures every twenty five steps, and marks PPO’s 1 + ϵ ratio constraint with a black dotted line.