PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovered Policy Optimisation

Chris Lu, Jakub Grudzien Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, Jakob Foerster

arXiv:2210.05639v2cs.LGcs.AI

TL;DR

Handcrafted RL design is limited, while prior algorithm-discovery methods have struggled to combine generalisation, performance, and guarantees. The paper meta-learns a drift function within Mirror Learning and analyses it to construct DPO. In Brax, LPO and DPO outperform PPO, with LPO also transferring to unseen environments and hyperparameter settings.

  • Problem

    Handcrafted RL design is limited by human understanding, while prior discovery approaches have struggled to generalise beyond training tasks or outperform handcrafted algorithms with guarantees.

    Source: §1 · p. 1

  • Method

    The authors meta-train a neural-network drift function with evolution strategies in Mirror Learning, then analyse LPO’s learned features to formulate closed-form DPO.

    Source: §1 · p. 2, §7.1 · p. 8

  • Results

    LPO outperforms PPO in unseen environments and hyperparameter settings, while DPO matches LPO and outperforms PPO on the evaluated Brax environments.

    Source: §7.2 · p. 9, §8 · p. 10

  • Takeaways & Limitations

    DPO is a two-line analytic model of LPO that can be implemented as easily as PPO while matching the learned drift’s performance on evaluated environments.

    Source: §7.2 · p. 9

    The drift network uses probability ratios and normalized advantages, although other or domain-specific inputs may benefit specific applications.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Tremendous progress has been made in reinforcement learning (RL) over the past decade. Most of these advancements came through the continual development of new algorithms, which were designed using a combination of mathematical derivations, intuitions, and experimentation. Such an approach of creating algorithms manually is limited by human understanding and ingenuity. In contrast, meta-learning provides a toolkit for automatic machine learning method optimisation, potentially addressing this flaw. However, black-box approaches which attempt to discover RL algorithms with minimal prior structure have thus far not outperformed existing hand-crafted algorithms. Mirror Learning, which includes RL algorithms, such as PPO, offers a potential middle-ground starting point: while every method in this framework comes with theoretical guarantees, components that differentiate them are subject to design. In this paper we explore the Mirror Learning space by meta-learning a "drift" function. We refer to the immediate result as Learnt Policy Optimisation (LPO). By analysing LPO we gain original insights into policy optimisation which we use to formulate a novel, closed-form RL algorithm, Discovered Policy Optimisation (DPO). Our experiments in Brax environments confirm state-of-the-art performance of LPO and DPO, as well as their transfer to unseen settings.

1 Introduction

The paper argues that manual RL algorithm design is limited and existing algorithm-discovery approaches have struggled to combine generalisation, performance, and theoretical guarantees. It searches within Mirror Learning to learn a drift function, then uses insights from that process to construct DPO.

1 Introduction

  • Human-designed RL algorithms, including TRPO and PPO, remain limited by human understanding and can be brittle to hyperparameters or lack robustness guarantees.The paper presents these as possible flaws of handcrafted design.Source: §1 · p. 1
  • Existing algorithm-discovery methods either fail to generalise beyond their training distribution or do not outperform handcrafted algorithms while lacking theoretical guarantees.The authors contrast these approaches with a search space that retains Mirror Learning’s guarantees.Source: §1 · p. 1
  • The authors meta-train a neural-network drift function with evolution strategies inside Mirror Learning, producing the algorithm Learnt Policy Optimisation (LPO).Because LPO is in the Mirror Learning space, it inherits that framework’s theoretical convergence guarantees.Source: §1 · p. 2
  • Analysis of LPO identifies rollback—optimism toward actions that previously received low rewards—and informs the closed-form Discovered Policy Optimisation (DPO).In Brax, LPO and DPO outperform PPO and generalise to environments not used to train LPO.Source: §1 · p. 2

2 Related Work

The paper reviews handcrafted RL improvements and algorithm-discovery approaches, noting their limitations in robustness, generalisation, performance, or guarantees. It positions Mirror Learning as a structured search space for discovery.

2 Related Work

  • Handcrafted RL improvements address particular design issues, but the cited work does not fully mitigate sensitivity to hyperparameters and lack of domain generalisation.Examples include pessimistic Bellman targets [9], KL-penalised PPO design choices [15], and automatic entropy-temperature adjustment [13].Source: §2 · p. 2
  • Meta-RL methods such as RL2 and MAML [7] adapt across task distributions but generally use narrow domains and are not expected to generalise well to truly unseen environments..Source: §2 · p. 2
  • Prior algorithm-discovery methods show mixed results: EPG [14] has higher average performance than PPO but larger variance, while MetaGenRL [17] fails to improve on DDPG [21].The reviewed methods also had not produced novel analytical RL algorithms, and graph-based discovery methods lacked correctness guarantees.Source: §2 · p. 2
  • The authors position LPO in Mirror Learning, whose algorithms have convergence guarantees, and use its interpretable structure to derive the closed-form DPO.The paper presents this structure as making meta-training easier than learning algorithms from scratch.Source: §2 · p. 3

3 Background

This background introduces RL and meta-RL, Mirror Learning’s policy-update framework, and evolution strategies as a way to optimise algorithms without backpropagating through training.

3.1 Reinforcement Learning

  • RL models an agent interacting with an MDP to maximise discounted return, while meta-RL optimises an algorithm’s parameters for the return after K training iterations.Value functions assess expected returns, and the advantage compares action values across states.Source: §3.1 · pp. 3–4

3.2 Mirror Learning

  • Mirror Learning updates policies using a mirror objective with a drift function and neighbourhood operator; valid drift conditions yield monotonic improvement and convergence to optimal return.PPO is one instance, and its gradient-ascent steps implicitly constrain updates to an Euclidean neighbourhood.Source: §3.2 · p. 4
  • The framework can differ in sample complexity and wall-clock efficiency; this work optimises the return of the Kth iterate.The choice of objective reflects the paper’s stated optimisation target.Source: §3.2 · p. 4

3.3 Evolution Strategies

  • Evolution strategies estimate gradients from returns on noise-perturbed parameters, enabling meta-optimisation without backpropagation through the full training episode.Antithetic sampling is noted as a way to reduce estimator variance.Source: §3.3 · p. 4

4 Methods

The method meta-learns a drift function for policy optimization over a fixed episode length, using expected final return as its objective. The network operates on policy ratios and normalized advantages, and its training uses evolution strategies to optimize the final learner policy.

4 Methods

  • The drift function is meta-learned to maximize expected final return after policy optimization over a fixed episode length K.The expectation accounts for initial policy parameters and environment stochasticity.Source: §4 · p. 5, §4.2 · p. 5

4.1 Drift Function Network

  • The drift network maps policy-ratio and normalized-advantage features to updates, including nonlinear transformations of both inputs.Its input features combine 1 − r and log(r), their squares, and products with advantage A.Source: §4.1 · p. 5
  • The network architecture enforces drift-function validity through a bias-free design and a shifted ReLU output, while the main variant initializes near PPO.The PPO drift is added before the shifted ReLU; this initialization is reported to improve performance and generalization across environments.Source: §4.1 · p. 5

4.2 Meta-Training the Drift Function Network

  • Antithetic evolution strategies optimize the drift parameters by training policies for K iterations and estimating return from the final policy.Each generation samples parameter perturbations and initial policy parameters; the expectation includes environment stochasticity.Source: §4.2 · pp. 5–6
  • The experiments use Brax and vectorized policy optimization to train hundreds of agents per minute on accelerators.The implementation builds on Brax PPO while keeping its policy architecture and training hyperparameters unchanged.Source: §4.2 · p. 6

5 Empirical Studies

LPO, trained only on Ant, outperforms PPO in unseen environments and is robust to task-specific PPO hyperparameters. Learning from scratch yields LPO-Zero, which acquires some PPO-like behavior but largely underperforms LPO.

5.1 Learning drift functions from scratch

  • LPO-Zero learns PPO-like clipping only for negative advantages, but largely underperforms LPO and may require more training to catch up.LPO-Zero is learned from scratch across five Brax environments using a two-hidden-layer, 256-unit ReLU network.Source: §5.1 · p. 7

5.2 Learning with the PPO Initialisation

  • The PPO-initialized drift network adds PPO to the final hidden-layer output before shifted ReLU, making its initial drift similar to PPO's.This variant uses one hidden layer with 128 neurons, no biases, and tanh activation.Source: §5.2 · p. 7
  • Meta-training in a single environment is sufficient to produce drift functions that transfer to unseen tasks and generally share similar characteristics.The authors use the drift trained on Ant as LPO.Source: §5.2 · p. 7
  • LPO trained only on Ant outperforms PPO in unseen environments, despite Brax PPO using task-specific update epochs and total timesteps.The authors report that this also makes LPO robust to new hyperparameters.Source: §5.2 · p. 7

6 Analysis of LPO

LPO combines rollback for negatively advantaged actions with cautious optimism for positively advantaged actions, producing asymmetric updates and higher policy entropy than PPO. The analysis also finds that some learned gradient spikes did not improve the analytic model.

Rollback for negative advantage

  • For negative advantage and r < 1, LPO encourages rollback toward r = 1 − ϵ, preserving moderate action frequency rather than abandoning actions that currently seem poor.The authors interpret this as supporting continued exploration of those actions.Source: §6 · p. 7

Implicit entropy maximisation

  • LPO and DPO maintain significantly higher policy entropy than PPO throughout Ant training, although all three methods’ entropy declines.LPO’s two central features encourage probability mass to spread moderately across actions, which the authors describe as implicit entropy maximisation without an added entropy bonus.Source: Figure 4 · p. 7, §6 · p. 8

Cautious optimism for positive advantage

  • For positive advantage and r > 1, LPO slows updates and eventually stops chasing extreme advantages, which may reflect critic errors.Unlike PPO, which removes update incentives beyond r > 1 + ϵ, LPO takes a more graduated approach to positive advantages.Source: §6 · p. 8

Update asymmetry

  • LPO’s rollback and cautious optimism create asymmetric update incentives, unlike PPO’s symmetric clipping around the origin.The analysis distinguishes increasing action probability for positive advantage from decreasing it for negative advantage, which have different exploration implications.Source: §6 · pp. 7–8

Secondary Features

  • Gradient spikes around r = 1 did not improve the analytic model, and the authors speculate that they mainly reflect network parameterisation artifacts.The spikes appeared in LPO but not LPO-Zero.Source: §6 · p. 8

7 Discovered Policy Optimisation: A New RL Algorithm Inspired By LPO

DPO turns key features discovered in LPO into a closed-form Mirror Learning algorithm. In Brax, it matches LPO and outperforms PPO without retuning PPO’s hyperparameters.

7 Discovered Policy Optimisation: A New RL Algorithm Inspired By LPO

  • The paper introduces DPO as a new RL algorithm built from concepts discovered by analysing LPO.The authors describe the model as implementable in two lines on top of PPO and as having performance on par with their best learnt drift function.Source: §7 · p. 8, §7.2 · p. 9

7.1 The Discovered Drift Function Model

  • DPO is a parameter-free, closed-form drift model that combines LPO’s rollback and cautious optimism using different forms for positive and negative advantages.The authors report that α = 2 and β = 0.6 faithfully reproduce those LPO features.Source: §7.1 · pp. 8–9

7.2 Results

  • The paper evaluates DPO and LPO against PPO on Minatar using PPO-tuned hyperparameters reused for all three methods.Figure 7 reports mean evaluation return across 10 random seeds with standard-error bars; the supplied passages do not state the comparative outcome.Source: §7.2 · p. 9, Figure 7 · p. 10
  • DPO matches LPO and outperforms PPO on evaluated Brax environments without retuning PPO’s hyperparameters.The comparison covers most Brax environments, and the figure reports mean evaluation return over 10 seeds; both DPO and LPO are described as more consistent across runs.Source: §7.2 · p. 9, Figure 6 · p. 9

8 Conclusion

The paper restricts algorithm discovery to Mirror Learning, meta-learns a drift function to produce LPO, then uses analysis of LPO to formulate DPO. It reports strong performance and robustness in unseen settings, while noting possible sensitivity to implementation details.

Conclusion

  • LPO is meta-trained as a neural-network drift function using Evolution Strategies within the theoretically sound Mirror Learning framework.The authors report that LPO outperforms PPO in unseen environments and with unseen hyperparameter settings.Source: §8 · p. 10
  • Analysis of LPO’s learned features led to DPO, a closed-form approximation that matches LPO’s performance and robustness to hyperparameter settings.Source: §8 · p. 10
  • The authors identify possible overfitting to code-level implementation details as a weakness of the approach.They note that Brax’s PPO implementation makes design choices to optimise wall-clock time rather than sample efficiency.Source: §8 · p. 10
  • The authors plan to expand the drift function’s inputs and to meta-learn other Mirror Learning attributes.They expect this work to provide further policy-optimisation insights and ultimately yield more robust, better-performing RL algorithms.Source: §8 · p. 10

A Meta-Training Details

The appendix describes separate LPO-Zero and LPO meta-training setups. Both training accounts report that performance continued to increase with additional meta-training time.

A.1 LPO-Zero

  • LPO-Zero used a 256-unit drift network and distributed asynchronous meta-updates across 350 TPUv4i workers.Meta-training used antithetic Evolution Strategies and Adam at learning rate 0.006; the experiment used a uniform mixture of Ant, Walker2d, HalfCheetah, and Fetch, with 183 PPO epochs and 10,240 evaluation rollouts per worker training.Source: §A.1 · p. 14
  • LPO-Zero meta-training ran for two days and about 400 outer updates, with performance still increasing as training time increased.Source: §A.1 · p. 14

A.2 LPO

  • LPO used a 128-unit drift network and synchronous meta-training on one machine with four V100 GPUs.Training ran for two days and about 700 outer updates, with performance still increasing with additional meta-training time; Table 1 is identified as listing important LPO training parameters.Source: §A.2 · p. 14, Table 1 · p. 14

B LPO-Zero Results

The section compares PPO, LPO, DPO, and LPO-Zero in Brax, visualizes learned LPO objectives, and verifies conditions of the DPO drift. The verification concludes that the drift has zero gradient at the old policy for all advantages.

B LPO-Zero Results

  • Figure 8 compares PPO, LPO, DPO, and LPO-Zero by mean evaluation return across 10 seeds, with standard-error bars.The caption describes the comparison but does not state which method performs best.Source: Figure 8 · p. 15

C Visualisations of LPO

  • Figure 9 maps the LPO objective’s ratio derivative and shows slices at fixed advantage values; positive derivatives encourage updates toward the action.Source: Figure 9 · p. 16

D DPO Drift Verification

  • For all advantages, the DPO drift has zero gradient at r = 1, the old-policy ratio.The verification also checks that the drift is non-negative and equals zero at the old policy.Source: D DPO Drift Verification · p. 17

E Ablations on Drift Inputs

Meta-training the drift function with different input parameterisations produced good performance, although training was limited to 208 rather than 672 generations by computational constraints.

E Ablations on Drift Inputs

  • Across multiple input parameterisations, the meta-trained drift function performs well; computational constraints limited training to 208 rather than 672 generations.Source: Figure 10 · p. 18
Loading 2210.05639v2…