PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Ensemble Trust-Region Policy Optimization

Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel

arXiv:1802.10592v2cs.LGcs.AIcs.RO

TL;DR

Model-free RL is sample-inefficient, while vanilla deep model-based RL can become unstable when policies exploit regions unsupported by sufficient model-training data. ME-TRPO addresses this with model ensembles, TRPO-based likelihood-ratio gradients, and ensemble-based validation, achieving comparable performance to state-of-the-art model-free methods with roughly 100× less data on challenging continuous-control tasks.

  • Problem

    Model-free RL has high sample complexity, while vanilla deep model-based RL is unstable because policies exploit regions with insufficient data for accurate model learning.

  • Method

    ME-TRPO combines an ensemble of learned dynamics models with TRPO likelihood-ratio optimization and ensemble-based policy validation.

  • Results

    100× reduction in sample complexity accompanies ME-TRPO’s reported performance matching state-of-the-art model-free algorithms on challenging continuous-control tasks.

  • Takeaways & Limitations

    Model ensembles regularize learning against model bias, while replacing BPTT with TRPO yields more stable learning and better final performance.

  • Takeaways & Limitations

    Future work is needed to use ensemble disagreement to guide exploration and to apply ME-TRPO to real-world robotics systems.

Abstract

from arXiv · show

Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity, which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and to date have succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks.

1 INTRODUCTION

Model-free RL is broadly applicable but sample-inefficient, while model-based RL can reuse learned dynamics for greater sample efficiency. The paper finds vanilla deep model-based RL unstable because policies exploit poorly modeled regions, and proposes ME-TRPO to address this with model ensembles and trust-region policy optimization.

  • Model-free RL methods are broadly applicable and easy to combine with deep neural networks, but their high sample complexity limits real-world use.
  • Model-based RL can potentially improve sample efficiency by learning an environment model and optimizing policies with it.
  • Vanilla model-based RL assumes that sufficient data will make the learned model accurate enough for policies optimized on it to perform well in the real environment.
  • On challenging continuous-control tasks, vanilla model-based RL becomes unstable because policy optimization exploits regions with insufficient data, causing catastrophic failures.
  • ME-TRPO uses an ensemble of deep neural networks to maintain model uncertainty and regularize policy learning through imagined stochastic roll-outs.
  • 100× reduction in sample complexity accompanies ME-TRPO’s reported performance matching state-of-the-art model-free algorithms.

2 RELATED WORK

Prior model-based RL work spans linear, Gaussian-process, Bayesian-neural-network, and latent-dynamics models. Deep neural networks have scaled model-free RL effectively, but their use in model-based RL has had limited success, especially beyond simple or low-dimensional domains.

  • Linear models enabled several impressive robotic learning applications despite their simple parameterization.
  • Gaussian-process models maintain prediction uncertainty and have flexible representations, but suffer from the curse of dimensionality and computational costs in policy updates.
  • Deep neural networks have scaled model-free RL to challenging scenarios, while their application to model-based RL has achieved only limited success.
  • Recent approaches use Bayesian neural networks or latent-variable dynamics models to seek more generally applicable model-based RL algorithms.

3 PRELIMINARIES

The paper formulates RL as optimizing expected cumulative reward in a discrete-time finite-horizon MDP with known rewards and unknown transition dynamics. Policies select actions stochastically from states, and the objective is expected return.

  • The setting is a discrete-time finite-horizon MDP with state space S, action space A, transition function f, reward function r, initial distribution ρ0, and horizon T.
  • A stochastic policy πθ(a|s) specifies action probabilities, and η(θ) denotes its expected return over trajectories.
  • The reward function is known, the transition function is unknown, and the learning goal is to maximize expected return.

4 VANILLA MODEL-BASED DEEP REINFORCEMENT LEARNING

Vanilla model-based deep RL learns a neural dynamics model from collected transitions, then optimizes a policy in the resulting approximate MDP. The approach uses imagined roll-outs and BPTT, but relies on model predictions throughout policy learning.

  • Model-based RL reuses collected data to train a dynamics model that serves as a simulator for policy training and gradient computation.
  • The transition dynamics are modeled with a feed-forward neural network that predicts state changes from states and actions.
  • The learned next-state approximator adds the neural network’s predicted state change to the input state.
  • The model is trained by minimizing an L2 one-step prediction loss on the dataset of experienced transitions.
  • Validation data, input-output normalization, and early stopping are used to reduce overfitting during supervised model learning.
  • The policy is optimized on an approximate MDP whose transitions are generated by the learned dynamics model.
  • The stochastic policy parameterizes Gaussian actions using state-dependent means and standard deviations with reparameterized noise.
  • BPTT estimates policy gradients through the learned model over time, with gradient clipping and Adam updates used for more stable learning.

5 MODEL-ENSEMBLE TRUST-REGION POLICY OPTIMIZATION

ME-TRPO addresses vanilla model-based RL instability by combining an ensemble of learned dynamics models with TRPO-based policy optimization and ensemble validation. The ensemble regularizes policy learning across alternative model predictions while validation stops updates when modeled performance ceases to improve.

  • Motivation: Vanilla model-based deep RL often exploits regions with scarce dynamics-model training data, causing erroneous predictions and unstable policy learning.Early stopping with validation initial states was insufficient because evaluation still used the same learned model, which could make consistent mistakes.
  • Model Ensemble: ME-TRPO fits multiple dynamics models on the same real-world data, differing by initialization and minibatch order, to represent model uncertainty.The ensemble is trained with standard supervised learning.
  • Policy Optimization: ME-TRPO uses TRPO to optimize the policy over imagined trajectories generated by randomly selected ensemble models.Likelihood-ratio methods replace backpropagation through time, using learned models as simulators rather than for direct gradient computation.
  • Policy Validation: Policy validation monitors the fraction of ensemble models on which the policy improves and terminates the current iteration when improvement falls below a threshold.In practice, validation occurs after every 5 gradient updates, with a 70% threshold and limited tolerance for further updates.
  • Policy Optimization: The ensemble regularizes policy learning by requiring robust performance across many possible alternative futures, despite potential bias in individual models.The paper reports that combining the models yields stable and effective policy improvement in its experiments.

6 EXPERIMENTS

The experiments evaluate ME-TRPO against state-of-the-art methods and analyze vanilla model-based RL, TRPO optimization, and model ensembles across six Mujoco continuous-control tasks.

  • Experimental questions: The experiments are designed to assess sample complexity, final performance, vanilla failure scenarios, and how ME-TRPO addresses those failures.An additional ablation study characterizes the effect of each algorithmic component.
  • Experimental setup: Experiments compare ME-TRPO with TRPO, PPO, DDPG, and SVG on six Mujoco continuous-control benchmark tasks.The tasks are Swimmer, Snake, Hopper, Ant, Half Cheetah, and Humanoid.
  • Comparison to state-of-the-art: ME-TRPO reaches model-free performance with ≈100× less real-world data.Prior model-based methods perform worse or are unstable over long horizons, whereas ME-TRPO also optimizes policies on Humanoid.
  • From vanilla to ME-TRPO: With one learned model, TRPO produces more stable learning and better final performance than BPTT, VPG, and related policy-gradient comparisons.The learned dynamics model acts as a simulator, so policy optimization does not require gradient information through the dynamics.
  • From vanilla to ME-TRPO: Using more ensemble models continually improves regularization and performance, with larger gains in challenging environments such as HalfCheetah and Ant.The comparison includes 5, 10, and 20 models versus a single model, while TRPO is used for policy optimization.

7 DISCUSSION

ME-TRPO is presented as a robust model-based algorithm for neural-network policies across challenging domains, combining TRPO and model ensembles to address instability and model bias.

  • Discussion: ME-TRPO learns neural-network policies across different challenging domains while significantly reducing sample complexity and matching state-of-the-art performance.The discussion contrasts this with vanilla model-based RL, which suffers from model bias and numerical instability.
  • Discussion: The authors identify TRPO and model ensembles as essential components for successful deep model-based reinforcement learning.They also describe model uncertainty as a principled way to reduce model bias.
  • Future work: Future work could use ensemble disagreement to guide exploration and collect data that resolves model disagreement.Applying ME-TRPO to real-world robotics systems is another proposed direction.

A.1.1 DATA COLLECTION

The implementation collects environment data, trains neural-network dynamics models, and optimizes neural-network policies with TRPO under environment-specific settings and validation procedures.

  • Data collection: Each outer iteration collects 3000 real-world timesteps per environment, except Humanoid, which uses 6000 timesteps.Policy noise is randomized at rollout starts and parameter perturbations are also applied.
  • Model learning: The dynamics model is a 2-hidden-layer 1024-1024 ReLU network trained with Adam at learning rate 0.001 and batch size 1000.Training stops after validation loss fails to decrease for 25 passes over the dataset.
  • Policy learning: Policies use feed-forward networks and are trained with TRPO using initial standard deviation 1.0, δKL 0.01, and batch size 50000.Humanoid uses a larger 100-50-25 policy architecture; failed validation for 25 updates triggers repetition of the process.
  • Environment representation: The evaluated environments omit contact information, making Half Cheetah, Ant, Hopper, and Humanoid effectively POMDPs.State inputs contain joint angles, joint velocities, and a part’s Cartesian center-of-mass position.
  • Baselines: Baseline implementations include TRPO, PPO, DDPG, and SVG, with their reported training schedules and hyperparameters specified separately.The benchmark environments are adopted from rllab.

B OVERFITTING

Using one model instead of an ensemble causes policy overoptimization: estimated performance rises while real performance falls during policy optimization. Most runtime is spent training the model ensemble, which can be parallelized across GPUs.

  • Estimated performance increases while real performance decreases at the end of policy optimization when the ensemble is replaced by one model.This behavior occurs in each outer iteration and is shown for Swimmer.
  • Replacing the ensemble with one model leads to policy overoptimization.
  • The majority of runtime is spent training the model ensemble.The authors note that ensemble training can be parallelized across multiple GPUs for potentially multiple-fold speed-up.

D ABLATION STUDY

The ablation experiments isolate the importance of ensemble validation and ensemble sampling, changing one component at a time while using five models.

  • The study evaluates the importance of ensemble validation and ensemble sampling as the algorithm’s two main regularization components.
  • Each ablation changes only one component of ME-TRPO at a time.
  • The ablation experiments use ME-TRPO with 5 models.

D.1 ENSEMBLE SAMPLING METHODS

The ablations compare ensemble sampling and validation strategies. Randomly sampling a model at each simulated step is reported as the most robust sampling method, while ensemble validation performs well across environments without extra hyper-parameters.

  • D.1 ENSEMBLE SAMPLING METHODS: Randomly sampling the next step from different ensemble models is the most robust method across environments.Mean and median predictions do not prevent overfitting, while gradient averaging leaves room for overfitting to one or more models.
  • D.1 ENSEMBLE SAMPLING METHODS: Sampling from model uncertainty is presented as a principled way to avoid overfitting learned models.
  • D.2 ENSEMBLE VALIDATION: The validation comparison includes real-performance oracle validation, TRPO-rollout averages, fixed stopping after 50 or 5 updates, one-model prediction, and ensemble validation.
  • D.2 ENSEMBLE VALIDATION: Ensemble validation has good performance across environments without adding extra hyper-parameters.Fixed-update stopping can perform well when the update count is correctly chosen, but adds another hyper-parameter; oracle validation is over-cautious and can limit exploration.
Loading 1802.10592v2…