PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Based Reinforcement Learning via Meta-Policy Optimization

Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, Pieter Abbeel

arXiv:1809.05214v1cs.LGcs.AIstat.ML

TL;DR

Model-based reinforcement learning is sample efficient but depends on learned dynamics models that may not match reality, limiting asymptotic performance. MB-MPO meta-learns a policy over an ensemble of such models for one-step adaptation, matching model-free performance while using substantially less experience.

  • Problem

    Model-based reinforcement learning seeks data efficiency, but inaccurate learned dynamics can limit performance and create model-bias.

  • Method

    MB-MPO meta-learns a policy on an ensemble of learned dynamics models that can adapt to each model with one policy-gradient step.

  • Results

    MB-MPO matches model-free asymptotic performance, outperforms previous model-based methods, and remains effective with strongly biased models.

  • Takeaways & Limitations

    MB-MPO combines model-based sample efficiency with model-free performance and robustness to imperfect dynamics models.

Abstract

from arXiv · show

Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization (MB-MPO), an approach that foregoes the strong reliance on accurate learned dynamics models. Using an ensemble of learned dynamic models, MB-MPO meta-learns a policy that can quickly adapt to any model in the ensemble with one policy gradient step. This steers the meta-policy towards internalizing consistent dynamics predictions among the ensemble while shifting the burden of behaving optimally w.r.t. the model discrepancies towards the adaptation step. Our experiments show that MB-MPO is more robust to model imperfections than previous model-based approaches. Finally, we demonstrate that our approach is able to match the asymptotic performance of model-free methods while requiring significantly less experience.

1 Introduction

Model-free reinforcement learning is broadly effective but data intensive, whereas model-based methods are more sample efficient yet vulnerable to inaccurate dynamics models and model-bias. MB-MPO addresses this trade-off by meta-learning rapid adaptation across an ensemble of learned models, matching model-free asymptotic performance with substantially less experience.

  • Model-free algorithms generally achieve optimal performance and broad applicability, but their high sample complexity makes robotic data collection costly.
  • Model-based methods improve sample efficiency by optimizing policies against learned environment dynamics trained with off-policy data.
  • Inaccurate dynamics models can cause policy overfitting, suboptimal behavior, or catastrophic failures through model-bias.
  • MB-MPO meta-learns a policy that adapts to each ensemble model with one policy-gradient step instead of requiring highly accurate dynamics.
  • MB-MPO matches model-free asymptotic performance while substantially reducing experience requirements and outperforming previous model-based methods.

2 Related Work

Prior work addresses model inaccuracies through simpler models, replanning, robust optimization, differentiable trajectory methods, and model ensembles. These approaches face limits including restricted dimensionality, computational cost, conservatism, or gradient instability, motivating MB-MPO's adaptation-based strategy.

  • Model-Based Reinforcement Learning: Addressing Model Inaccuracies: Linear and Bayesian dynamics models have produced strong results but are limited to low-dimensional domains, whereas neural networks support complex high-dimensional control.
  • Model-Based Reinforcement Learning: Addressing Model Inaccuracies: MPC compensates for model imperfections through continual replanning but has limited credit assignment and high computational cost.
  • Model-Based Reinforcement Learning: Addressing Model Inaccuracies: Robust policy optimization seeks performance across models, but the resulting policies tend to be over-conservative.
  • Model-Based + Model-Free Reinforcement Learning: Differentiable trajectory optimization can suffer exploding and vanishing gradients because learned models are not explicitly trained to approximate first-order derivatives.
  • Model-Based + Model-Free Reinforcement Learning: Ensemble-based approaches reduce reliance on precise models by training policies to adapt when prediction uncertainty is high.

3 Background

The background defines reinforcement learning over finite MDPs and distinguishes model-based learning of transition dynamics from model-free learning. It then introduces meta-RL and MAML as frameworks for rapid policy improvement across tasks.

  • MDP and Reinforcement Learning: An MDP comprises states, actions, transition distribution, rewards, discount factor, initial-state distribution, and horizon; reinforcement learning maximizes expected trajectory return.
  • Model-based Reinforcement Learning: Model-based RL learns the environment transition distribution from observed transitions, often parameterizing it as ˆpφ(s′|s, a).
  • Meta-Reinforcement Learning: Meta-RL learns algorithms that quickly produce effective policies for MDPs drawn from a task distribution while sharing state and action spaces.
  • Meta-Reinforcement Learning: MAML trains a policy to improve on a new task using one or a few vanilla policy-gradient steps.

4 Model-Based Meta-Policy-Optimization

MB-MPO frames model-based reinforcement learning as meta-learning over an ensemble of learned dynamics models. It trains a policy to adapt quickly to each model, using real data for model learning and imagined trajectories for policy optimization.

  • 4 Model-Based Meta-Policy-Optimization: MB-MPO maximizes policy adaptation across a distribution of dynamics models rather than robustness of one policy across model disagreements.
  • 4 Model-Based Meta-Policy-Optimization: The method instantiates this framework with an ensemble of learned models and one policy-gradient adaptation step for each model.
  • 4.1 Model Learning: Ensemble models use different initializations and randomly selected data subsets, then are periodically retrained with new on-policy samples and warm starts.
  • 4.1 Model Learning: Each dynamics model is a deterministic feed-forward neural network trained to predict state change, with an ℓ2 one-step prediction objective and regularization techniques.
  • 4.2 Meta-Reinforcement Learning on Learned Models: The learned models define separate MDP tasks with shared rewards but differing dynamics, and the policy objective evaluates expected return under each estimated model.
  • 4.2 Meta-Reinforcement Learning on Learned Models: Imagined trajectories from each model support adaptation and meta-optimization, so the data-intensive meta-policy step does not use real-world data.
  • Algorithm: The algorithm iteratively collects real trajectories with adapted policies, retrains models, performs imagined adaptation, and updates the meta-policy until target performance.

5 Benefits of the Algorithm

MB-MPO uses meta-learning over an ensemble of dynamics models to adapt policies where predictions disagree, while exploiting agreement as a regularizing signal. The approach also supports targeted data collection, fast fine-tuning, and simpler model-based learning.

  • Regularization effect during training: One-step adaptation to each ensemble model regularizes policy learning by shifting model-discrepancy handling into the inner update.This regularization is strongest where model uncertainty is high and fades as models become more accurate.
  • Regularization effect during training: Model uncertainty correlates with policy plasticity, measured by the statistical distance between pre- and post-update policies.The paper reports a strong positive correlation between ensemble predictive variance and KL-divergence.
  • Tailored data collection for fast model improvement: Adapted policies collect diverse real-environment data, including regions where individual dynamics models are inaccurate.This targeted coverage accelerates correction of model imprecision.
  • Fast fine-tuning: Each ensemble model represents a possible dynamics belief, so meta-learning prepares the policy to fine-tune rapidly when predictions disagree.High ensemble discrepancy increases the likelihood that the real dynamics lie within the represented support.
  • Simplicity: MB-MPO avoids parameter-noise exploration, careful model-weight reinitialization, entropy handling, difficult probabilistic models, and explicit model-distribution-mismatch treatment.The paper presents this reduced dependency set as a simplicity advantage over previous methods.

6 Experiments

Across six MuJoCo control tasks, MB-MPO is evaluated against model-free and model-based baselines, while experiments probe uncertainty, imperfect models, and adaptation. It reaches model-free performance with much less data, outperforms prior model-based methods, and remains robust under biased predictions.

  • 6.1 Comparison to State-of-the-Art: Model-Free: Two orders of magnitude fewer samples match model-free asymptotic performance across six MuJoCo environments.Figure 1 compares MB-MPO with four state-of-the-art model-free algorithms using horizon 200.
  • 6.1 Comparison to State-of-the-Art: Model-Free: 10–100× less data achieves maximum performance than model-free methods across locomotion tasks.For ant, hopper, and walker2D, the reduction is two orders of magnitude; easier tasks require 20–50× less data.
  • 6.2 Comparison to State-of-the-Art: Model-Based: MB-MPO achieves better asymptotic performance and faster convergence than previous model-based methods across six MuJoCo environments.MB-MPC struggles with robust or medium/long-term planning, while ME-TRPO converges more slowly and can remain suboptimal.
  • 6.3 Model Uncertainty and Policy Plasticity: Model predictive variance and policy-update KL-divergence show a strong positive correlation across the 2-D Point state space.The relationship persists throughout training and across different hyper-parameter configurations.
  • 6.4 Robustness to Imperfect Dynamic Models and Compounding Errors: Under biased and noisy dynamics models, MB-MPO consistently outperforms ME-TRPO and still learns positive-velocity locomotion at bmax = 0.5 and bmax = 1.0.ME-TRPO catastrophically fails at those strong-bias settings.
  • 6.4 Robustness to Imperfect Dynamic Models and Compounding Errors: With adaptation, MB-MPO steadily converges to maximum performance across seeds, whereas α = 0 produces unstable and sometimes suboptimal learning.Without adaptation, some seeds get stuck or unlearn previously good behaviors.

7 Conclusion

The paper concludes that MB-MPO combines ensemble dynamics modeling with policy meta-optimization to achieve model-free-level performance at substantially lower sample complexity. It also reports stronger performance, faster convergence, and robustness to imperfect models than prior model-based approaches.

  • 7 Conclusion: MB-MPO meta-optimizes a policy for adaptation across an ensemble of learned dynamics models.The algorithm is presented as simple and generally applicable.
  • 7 Conclusion: MB-MPO reaches state-of-the-art model-free performance with substantially lower sample complexity and outperforms prior model-based methods.The conclusion also reports better convergence speed and robustness to imperfect models.

A.1 Tailored Data Collection

Tailored exploration collects data with policies adapted to individual dynamics models, targeting regions where those models are inaccurate. This improves performance in half-cheetah and walker2D.

  • A.1 Tailored Data Collection: Tailored exploration uses post-update policies adapted to each dynamics model to collect data where model predictions are inaccurate.These policies exploit model deficiencies, directing new data toward poorly modeled regions.
  • A.1 Tailored Data Collection: 12% improvement in half-cheetah and 11% improvement in walker2D result from tailored exploration.The comparison is against collecting data with the pre-update policy.

A.2 Hyperparameter Study

MB-MPO benefits from adaptation, larger model ensembles, and more meta-gradient steps, although additional computational cost eventually yields diminishing returns and excessive updates can harm learning.

  • A.2 Hyperparameter Study: Adaptation significantly improves performance compared with the non-adaptive case of α = 0.The study varies the inner learning rate, ensemble size, and meta-gradient steps.
  • A.2 Hyperparameter Study: Increasing the number of models and meta-gradient steps per iteration improves performance at a computational cost.The gains diminish as computational burden increases.
  • A.2 Hyperparameter Study: 60 meta-gradient steps can cause early convergence to a suboptimal policy by reducing exploration too early.Fixed policy variance or an entropy bonus can alleviate this problem.

A.3 Experiment Setup

The experiments evaluate MB-MPO across six MuJoCo control tasks using tailored policies, learned dynamics ensembles, policy-gradient meta-optimization, and specified data-collection and compute procedures.

  • A.3 Experiment Setup: Figure 7 varies the inner learning rate α, dynamics-ensemble size, and meta-gradient steps before new real samples and model refitting.These are the principal hyperparameters studied in half-cheetah.
  • A.3 Experiment Setup: Six continuous-control tasks are evaluated: swimmer, half-cheetah, walker2D, hopper, ant, and a 7-DoF torque-controlled PR2 arm.Five tasks involve robotic locomotion; PR2 reaches arbitrary end-effector positions.
  • A.3 Experiment Setup: MB-MPO uses a Gaussian policy whose neural-network weights and standard-deviation vector are learned during optimization.The network has two hidden layers of size 32 with tanh nonlinearities.
  • A.3 Experiment Setup: The policy uses generalized advantage estimation with γ = 0.99 and λ = 1 alongside a linear reward baseline.
  • A.3 Experiment Setup: An ensemble of five fully connected neural networks models dynamics in the experiments except Figure 7b.Models use weight normalization, ReLU nonlinearities, Adam optimization, validation splits, and individually stopped training.
  • A.3 Experiment Setup: Each iteration collects 4000 real environment transitions and samples 100000 imaginary transitions for meta-optimization.The real data comprises 20 trajectories of 200 time steps.
  • A.3 Experiment Setup: MB-MPO reaches maximum half-cheetah performance in 7 hours, using 20% more compute time than TRPO while reducing sample complexity 70×.Model training is the main computational bottleneck relative to model-free algorithms.
  • A.3 Experiment Setup: In real-world experiments, the bottleneck shifts toward data collection, making MB-MPO significantly faster than model-free approaches.
Loading 1809.05214v1…