PaperlayerView exact revision on arXiv

Source-linked AI summary

ProMP: Proximal Meta-Policy Search

Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, Pieter Abbeel

arXiv:1810.06784v4cs.LGstat.ML

TL;DR

Meta-RL lacks a well-understood way to assign credit to pre-adaptation behavior, while existing gradient estimates can be difficult or high variance. The paper analyzes these issues, introduces the LVC surrogate objective and ProMP, and reports consistent gains over previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.

  • Problem

    Credit assignment to pre-adaptation behavior and reliable meta-policy gradient estimation remain insufficiently addressed in gradient-based Meta-RL.

  • Method

    The paper theoretically analyzes gradient-based Meta-RL, develops the low variance curvature surrogate objective, and builds ProMP with statistical-distance control for pre- and post-adaptation policies.

  • Results

    ProMP consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance across a diverse set of continuous control tasks.

  • Takeaways & Limitations

    The results support fully optimizing the pre-update sampling distribution while using lower-variance curvature estimates for efficient and stable Meta-RL.

  • Takeaways & Limitations

    The original RL-MAML implementation produced incorrect meta-gradient estimates because its score-function surrogate was unsuitable for higher-order automatic differentiation, although MAML is theoretically sound.

Abstract

from arXiv · show

Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-efficiency during meta-training as well as ineffective task identification strategies. This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL. Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difficulties in estimating meta-policy gradients. By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efficient and stable meta-learning. Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.

1 INTRODUCTION

Meta-RL seeks fast adaptation to unseen tasks from limited experience, but pre-adaptation behavior and its contribution to posterior performance remain poorly understood. ProMP addresses these credit-assignment and gradient-estimation challenges with a method that improves reported learning efficiency and performance.

  • Meta-learning aims to let agents succeed on unseen tasks with limited experience by learning how to learn.
  • Meta-RL uses early sampled interactions to adapt behavior to a given task, making pre-adaptation behavior instrumental for task identification.
  • Prior gradient-based Meta-RL methods either neglected credit assignment to the pre-update distribution or implemented it naively.
  • The paper formally analyzes pre-adaptation credit assignment, develops the LVC surrogate objective, and proposes ProMP for efficient and stable Meta-RL.
  • ProMP consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.

2 RELATED WORK

Related work divides meta-learning between flexible learned learning programs and methods that embed classical learning algorithms. Gradient-based Meta-RL is a successful embedded-algorithm approach, while many Meta-RL methods build on it.

  • Meta-learning seeks to acquire inductive bias so learning accelerates on unseen data or new problem settings.
  • One class learns a learning program using recurrent or memory-augmented models that process datasets and output parameters or predictions.
  • Another class embeds a classical learning algorithm and optimizes its parameters during meta-training.
  • Gradient-based meta-learning is a particularly successful embedded-algorithm approach in reinforcement learning.
  • Meta-RL applications include learned exploration, synthetic rewards, unsupervised policy acquisition, model-based RL, competitive environments, and modular policies.

3 BACKGROUND

The paper formulates reinforcement learning through MDPs and returns, then frames Meta-RL as learning rapid task-specific adaptation. Gradient-based methods optimize an initialization through one or a few policy-gradient steps.

  • An MDP specifies states, actions, transitions, initial-state distribution, rewards, and a finite horizon; RL maximizes expected trajectory return.
  • Meta-RL learns an algorithm that quickly approaches the optimal policy for tasks drawn from a task distribution, whose MDPs may differ in rewards or dynamics.
  • Gradient-based meta-learning learns policy parameters θ such that one or a few vanilla policy-gradient steps produce task-specific performance.
  • Formulation I, associated with MAML, performs a task-dependent update θ′ := U(θ,T) using an inner policy-gradient step and evaluates the post-update policy.
  • Formulation II, or E-MAML, treats U as a deterministic function of N sampled task trajectories and places the pre-update trajectory expectation outside U.

4 SAMPLING DISTRIBUTION CREDIT ASSIGNMENT

The analysis distinguishes post-update performance credit from credit assigned to the pre-update sampling distribution. Formulation I preserves the causal dependence of adaptation outcomes on pre-update behavior and therefore supports more informative meta-policy updates.

  • Figure 1 contrasts two stochastic computation graphs and marks credit flowing from post-update returns R′ to the pre-update policy πθ through ∇θJpre.
  • Both formulations optimize the same objective at zeroth order, but their differing computation graphs produce different gradients and optimization steps.
  • The post-update term is shared by both formulations and increases likelihood for post-update trajectories τ′ with higher returns.
  • Post-update credit alone does not optimize which pre-update trajectories τ produce better adaptation steps.
  • Formulation II treats the update function as unknown dynamics and uses post-update return R(τ′) to shift pre-update sampling toward better adaptation steps.
  • Formulation I accounts for causal dependence by maximizing the inner product of pre- and post-update policy gradients, encouraging larger returns, larger adaptation steps, and better gradient alignment.

5 LOW VARIANCE CURVATURE ESTIMATOR

The paper identifies biased or high-variance higher-order gradient estimates as obstacles to credit assignment in gradient-based Meta-RL and introduces LVC to improve estimation efficiency.

  • Motivation: MAML can provide superior meta-gradient updates, but correctly estimating those gradients is challenging because they involve the RL-objective Hessian.The update function’s gradient requires Hessian estimation, while standard surrogate differentiation can produce biased estimates.
  • Bias in standard estimates: Standard Monte Carlo surrogate differentiation drops Hessian terms H1 and H12+H12^⊤, thereby neglecting the influence of the pre-update sampling distribution.This omission corresponds to dropping the ∇θJpre contribution.
  • DiCE limitations: DiCE restores unbiased higher-order estimates for arbitrary stochastic computation graphs, but sequential action-probability dependence produces high-variance Hessian estimates.Empirically, noisy DiCE meta-gradients are associated with poor learning performance.
  • Low variance curvature: LVC removes sequential dependence within trajectories, reducing variance at the cost of a biased Hessian estimate.The resulting bias becomes negligible as θ approaches a local optimum under the stated conditions.
  • Empirical implication: Experiments show that LVC’s low-variance Hessian estimates significantly improve meta-learning sample-efficiency compared with DiCE.The paper presents this result as empirical support for its theoretical analysis.

6 PROMP: PROXIMAL META-POLICY SEARCH

ProMP combines the LVC objective with proximal policy optimization, accounting for pre-update distribution changes while constraining policy shifts through likelihood ratios and KL penalties.

  • Relation to prior optimization: TRPO may improve data efficiency and stability over VPG, but its Fisher information matrix estimate has cubic time complexity in the number of policy parameters.PPO offers a first-order alternative with comparable results to TRPO, motivating proximal components without explicit FIM computation.
  • Method: ProMP combines proximal policy optimization with the low variance curvature objective to perform meta-policy search.The method is presented as the paper’s novel meta-policy search algorithm.
  • Pre-update correction: The ProMP objective replaces stop-gradient importance weights with likelihood ratios, so derivatives at θo match LVC while accounting for changes in the pre-update action distribution.This design preserves the LVC derivative behavior at the reference policy.
  • Proximal constraint: A KL-penalty term between πθ and πθo creates a soft trust region that limits shifts in the pre-update state-visitation distribution.The paper motivates this constraint as necessary when reusing data from a recent policy for multiple meta-gradient steps.
  • Overall contribution: ProMP controls the statistical distance of both pre- and post-adaptation policies, promoting efficient and stable meta-learning.The paper reports consistent improvements over previous gradient-based Meta-RL algorithms in sample complexity, wall-clock time, and asymptotic performance.

7 EXPERIMENTS

Experiments evaluate ProMP and its LVC estimator across continuous-control Meta-RL benchmarks, gradient-estimator studies, and exploration tasks. ProMP improves sample efficiency and asymptotic performance, while LVC reduces gradient variance and supports effective task-identifying exploration.

  • Benchmark evaluation: Experiments evaluate ProMP on six continuous-control Meta-RL benchmark environments, averaging reported curves over at least three random seeds.Returns are estimated from sampled trajectories of adapted post-update policies and averaged over sampled tasks.
  • Benchmark evaluation: ProMP outperforms previous gradient-based meta-learning algorithms in all six MuJoCo environments.The comparison includes MAML-TRPO and E-MAML-TRPO, focusing on sample complexity and asymptotic performance.
  • Meta-gradient comparison: The LVC estimator consistently outperforms other meta-gradient estimators, while DiCE learns only slowly relative to the alternatives.The estimator comparison uses vanilla policy-gradient optimization.
  • Gradient estimator variance: 60% higher meta-gradient standard deviation is observed for DiCE on average compared with LVC, alongside poorer sample efficiency and asymptotic performance.The study reports relative gradient standard deviation and average return throughout learning in three meta-environments.
  • Initial sampling distributions: LVC learns consistent exploration across four regions and fully solves the 2D task, whereas MAML fails and E-MAML slightly misses task-specific targets.MAML assigns no credit to pre-update sampling, while E-MAML assigns credit only to batches of pre-update trajectories.
  • Formulation comparison: In a 1D environment, formulation I converges faster and more steadily than formulation II because the latter produces noisier gradient estimates.The comparison parameterizes the policy with two parameters and examines meta-gradient updates toward the optimum.

8 CONCLUSION

The paper analyzes how gradient-based Meta-RL assigns credit to pre-adaptation behavior and introduces ProMP with the LVC surrogate objective to improve meta-policy search. The analysis identifies formulation-dependent credit-assignment failures and motivates more stable, effective updates.

  • ProMP: ProMP fully optimizes the pre-update sampling distribution and uses the low variance curvature surrogate to produce low-variance meta-policy gradient estimates.The method is developed from the theoretical comparison of gradient formulations and curvature estimation.
  • Gradient formulations: Formulation II neglects the causal dependence of the post-update policy on the pre-update policy, weakening its pre-adaptation credit assignment.Its pre-update gradient term shifts sampling toward higher post-update returns but does not account for how the post-update policy depends on the pre-update policy.
  • Gradient formulations: Formulation I additionally rewards larger adaptation steps and better alignment between pre- and post-update policy gradients.The relevant contribution is represented by the inner product of inner and outer policy gradients.
  • Illustrative comparison: Formulation I converges faster and more steadily than formulation II in the illustrative 1D environment.The paper attributes formulation II’s noisier gradients and worse convergence properties to inferior credit assignment.
  • Curvature estimation: The RL-objective Hessian decomposes into four matrix terms, while prior MAML-style estimation can omit terms governing pre-update sampling influences.The paper links this omission to poor performance on tasks requiring pre-update exploration.

B.4 BIAS AND VARIANCE OF THE CURVATURE ESTIMATE

The paper contrasts unbiased but high-variance curvature estimation with the LVC estimator, which reduces variance by omitting difficult cross terms. This approximation becomes less biased near local optima under stated conditions.

  • DICE curvature: DICE provides an unbiased estimate of the inner-objective Hessian but uses trajectory-level products of importance weights.These products create high-variance curvature estimates.
  • DICE curvature: The DICE Hessian’s outer product of gradient sums is high variance, especially for the cross terms H12 + H12^⊤.The paper reports that these estimates require large batch sizes and impede sample-efficient learning.
  • LVC estimator: LVC approximates the state-visitation distribution as locally fixed with respect to policy parameters.This is a first-order Taylor approximation and is also used in monotonic policy improvement theory.
  • LVC estimator: Under certain infinite-horizon and policy-parameterization conditions, the omitted cross terms vanish around a local optimum.The paper therefore expects LVC bias to become negligible as parameters approach the local optimum.
  • Empirical implication: Experiments show that low-variance curvature estimates obtained through LVC improve Meta-RL sample efficiency by a significant margin.The result is presented as empirical confirmation of the theoretical argument.

C PROXIMAL POLICY SEARCH METHODS

Proximal policy search uses local policy updates that approximate policy performance while constraining changes from the current policy. PPO implements this proximity through adaptive KL penalties or likelihood-ratio clipping, with clipping reported to learn better.

  • Monotonic policy improvement theory: Monotonic policy improvement theory assumes infinite-horizon MDPs and uses state visitation frequencies to express trajectory expectations.The framework introduces value, action-value, and advantage functions for analyzing policy performance.
  • Monotonic policy improvement theory: Local policy search maximizes an expected advantage under the assumption that old and updated policies have identical state visitation frequencies.This replaces the difficult dependence of updated visitation frequencies on the new policy with a local surrogate objective.
  • Monotonic policy improvement theory: The surrogate objective matches the true objective to first order when the old and updated policies coincide.When the policies differ, an approximation error is introduced.
  • Trust-region methods: Trust-region policy optimization constrains the updated policy’s KL divergence from the current policy to define a local trust region.A quadratic KL approximation yields an update using the Fisher information matrix and conjugate-gradient approximation.
  • Proximal policy optimization: PPO either adapts the KL penalty coefficient to a target divergence or clips likelihood ratios to limit policy movement.The clipping option removes incentives to move too far from the original policy.
  • Proximal policy optimization: Empirical results report better learning performance for likelihood-ratio clipping than for the alternative PPO approach.Keeping policies close also permits multiple gradient steps without resampling trajectories, improving data efficiency over vanilla policy gradients.

D.2 ENVIRONMENT SPECIFICATIONS

The experiments use varied continuous-control environments with task-specific goals, dynamics, trajectory lengths, and numbers of adaptation steps. Tasks include directional locomotion, randomized physical parameters, target reaching, and point-mass navigation.

  • Experimental settings: Table 1 records the hyperparameter settings used for each algorithm, including environment-specific modifications.The reported settings include an inner learning rate of 0.05.
  • Environment tasks: Tasks span directional locomotion, randomized dynamics, target reaching, and point-mass navigation.Directional tasks vary between random XY directions or forward/backward motion; randomized-parameter tasks alter friction, joint mass, and inertia.
  • Environment tasks: Random-direction tasks require agents to run toward sampled XY directions, with rewards based on velocity minus control costs.This setup is used for AntRandDirec and HumanoidRandDirec.
  • Environment tasks: Target-reaching tasks specify locations or corners, requiring agents to locate and reach goals using directional forces or stopping behavior.PointEnv uses a random circular location, while another point-mass task assigns goals to area corners.

D.3 FURTHER EXPERIMENTS RESULTS

Additional experiments evaluate ProMP against other gradient-based methods in four new Mujoco environments and compare their computational time. ProMP is reported to improve sample complexity and asymptotic performance while keeping update time below environment-sampling time.

  • Additional environments: ProMP is compared with four other gradient-based meta-learning algorithms in four additional Mujoco environments.The corresponding meta-learning curves are presented in Figure 7.
  • Results: ProMP improves sample complexity and asymptotic performance in the additional experiments.The passage presents both improvements as advantages of ProMP.
  • Wall-clock time: MAML takes longer per iteration than ProMP and LVC-VPG because TRPO uses expensive conjugate-gradient steps.ProMP has longer outer updates than VPG because it performs multiple stochastic gradient-descent steps, but both update times remain a fraction of sampling time.
  • Wall-clock time: ProMP acquires better performance sooner, producing longer trajectories and fewer expensive environment resets.In parallelized environments, a reset blocks all environments, making reduced reset frequency consequential for sampling time.
Loading 1810.06784v4…