Source-linked AI summary
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, Sergey Levine
TL;DR
Physical-control reinforcement learning is limited by costly and potentially unsafe real-world data, while simulated training is vulnerable to source–target model mismatch. The paper introduces EPOpt, which trains policies over an ensemble of simulated domains using adversarial sampling and adapts that ensemble with target-domain data. Experiments report robust, generalizable policies and improved target-domain policies from Bayesian model adaptation, with computational cost increasing as more physical parameters vary.
Problem
Deep RL for physical control faces high sample complexity and safety challenges, and simulated policies can be brittle under model mismatch.
Method
EPOpt trains robust policies over an ensemble of simulated models with adversarial sampling, alternating policy optimization and Bayesian adaptation of the source distribution using target-domain data.
Results
The ensemble approach produces highly robust and generalizable policies, while Bayesian model adaptation can yield better target-domain policies than standard maximum-likelihood estimation.
Takeaways & Limitations
Model ensembles provide robust direct-transfer performance, and adapting their distribution supports learning to better match the target domain.
Takeaways & Limitations
Bayesian adaptation is computationally intensive as the number of variable physical parameters increases and can require different sampling when the true model has low prior probability.
Abstract
from arXiv · showhide
Sample complexity and safety are major challenges when learning policies with reinforcement learning for real-world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model-based methods where the real-world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.
1 INTRODUCTION
Deep RL faces prohibitive sample-complexity and safety barriers in physical control, while simulated training can fail under source–target model mismatch. EPOpt addresses this mismatch by adversarially training policies over model ensembles and adapting the ensemble with target-domain data.
- 1 INTRODUCTION: Deep RL for physical control often requires many potentially dangerous real-world samples because expressive policies compound model-free learning times.The barriers are both high sample complexity and instability during sampling with partially learned policies.
- 1 INTRODUCTION: Model-based training reduces reliance on real data by learning with simulated source domains, but systematic source–target discrepancies impede transfer.The simulator may not capture all physical effects present in the target domain.
- 1 INTRODUCTION: Policies optimized for specific simulators can be brittle under model mismatch because deep networks exploit simulator regularities.Robustness is therefore important for transferring simulated-task performance to physical systems.
- 1 INTRODUCTION: EPOpt−ϵ learns robust policies by adversarially training over an ensemble of simulated models and alternating policy search with source-distribution adaptation.The method targets competence across model variation, including parametric errors and unmodeled effects, while target-domain data progressively refines the ensemble.
2 PROBLEM FORMULATION
The paper formulates transfer as learning a policy for a target MDP from a parameterized distribution of source MDPs. The desired policy should perform well across that distribution without observing the target model’s parameters.
- 2 PROBLEM FORMULATION: A parameterized MDP M(p) shares state and action spaces across models while allowing transitions, rewards, and initial-state distributions to vary with p.The parameter γ is the discount factor, and transitions are modeled as random processes.
- 2 PROBLEM FORMULATION: The source-domain distribution D represents beliefs about target-domain parameters and is parameterized by ψ.Examples of variable physical parameters include torso mass, ground friction, and joint damping.
- 2 PROBLEM FORMULATION: The objective is a robust policy that performs well across source models without explicit dependence on or knowledge of p.The target domain may include unmodeled effects outside the assumed model class.
3 LEARNING PROTOCOL AND EPOPT ALGORITHM
EPOpt alternates robust policy search over a source-model ensemble with target-domain data collection and source-distribution adaptation. It emphasizes poor trajectories through CVaR-based subsampling, while Bayesian updating refines the ensemble using observed target trajectories.
- 3 Learning Protocol: The learning protocol alternates target-domain interaction with robust-policy computation and updates both policy parameters θ_i and source-distribution parameters ψ_i.Simulator episodes produce policy-search data, while target-domain rounds provide data for adapting the source distribution.
- 3.1 Robust Policy Search: EPOpt samples models and trajectories from the source distribution, then updates the policy using only a subset of sampled trajectories.The subset is selected after evaluating sampled returns, forming the algorithm’s adversarial emphasis on poor-performing trajectories.
- 3.1 Robust Policy Search: CVaR optimization targets the expected return of the worst ϵ-percentile of models rather than average performance across the source distribution.ϵ controls the relaxation from a max-min objective, and the worst-performing models define the integration region.
- 3.1 Robust Policy Search: For small ϵ, starting with trajectory subsampling caused unstable learning, so EPOpt initially uses ϵ = 1 before lowering it.The authors associate this schedule with allowing initial exploration before reducing the probability of poorly generalizing trajectories.
- 3.2 Adapting the Source Domain Distribution: Bayesian adaptation updates the ensemble distribution from target trajectories using sampled model parameters, prior probabilities, and trajectory likelihoods.Weighted posterior samples can estimate a parametric model or represent the distribution as discrete weighted samples.
- 3.2 Adapting the Source Domain Distribution: The resulting model-adaptation rule supports robust policy search when the target model may differ substantially from the initially assumed source distribution.The likelihood explains target trajectories using stochastic state transitions, which also model sensor errors.
4 EXPERIMENTS
Experiments evaluate EPOpt on hopper and half-cheetah tasks under modeled and unmodeled physical discrepancies. Across these settings, adversarial ensemble training improves robustness and generalization, while Bayesian adaptation updates the source distribution using target-domain experience.
- Experimental setup: EPOpt is evaluated on 2D hopper and half-cheetah benchmarks with MuJoCo physics, where underactuation, high dimensionality, contact discontinuities, and parameter discrepancies challenge policy learning.The experiments use direct torque control and implement the batch policy-optimization subroutine with TRPO.
- Research questions: The experiments ask whether standard policy search degrades under systematic physical differences and whether ensemble and adversarial training improve robustness without substantially reducing performance.They also examine robustness to unmodeled effects and the efficiency of adapting a mismatched model ensemble with target-domain data.
- Robustness analysis: EPOpt(ϵ = 0.1) generalizes to a wider range of hopper models and is more robust than the compared policies in the model-configuration heat map.The comparison includes TRPO on mean parameters and EPOpt variants with ϵ = 1 and ϵ = 0.1.
- Comparison to standard policy search: Single-mass TRPO policies fail across target torso masses, whereas EPOpt(ϵ = 0.1), trained on a Gaussian mass distribution, succeeds almost uniformly across tested masses.The EPOpt policy shows almost no performance degradation for any tested mass setting despite adopting a more robust strategy.
- Robustness to unmodeled effects: When torso-mass variation is omitted from the source ensemble, EPOpt(ϵ = 0.1) remains robust across a broad range of target masses, although less robust than when mass is modeled.This tests robustness to an unmodeled physical effect.
- Model adaptation: During model adaptation, high-density regions of the Gaussian source distribution move toward the unknown target model while retaining probability mass over explanatory parameter ranges.Each iteration corresponds to one target-domain interaction round, with a corresponding learning curve reporting average and percentile performance.
5 RELATED WORK
The related work connects EPOpt to robust control, Bayesian reinforcement learning, risk-sensitive reinforcement learning, model ensembles, and parametrized skills. It distinguishes this work by targeting high-dimensional continuous control with unknown task parameters.
- Robust control: Robust control studies robust policies through worst-case analysis, but much of its theory focuses on linear systems or finite MDPs.These settings may not adequately model the complexities of real-world tasks.
- Bayesian reinforcement learning: Model-based Bayesian reinforcement learning maintains a distribution over possible models, providing a probabilistic alternative to approaches that do not assume source or target task distributions.The passage positions Bayesian model-based RL as related background for the paper’s model-distribution setting.
- Risk-sensitive reinforcement learning: Risk-sensitive reinforcement learning combines subjective model-belief priors with robustness-oriented guarantees and can avoid overly conservative policies.Its application to high-dimensional continuous control had not been sufficiently explored in the cited work.
- Model ensembles: Prior model-ensemble work used trajectory optimization with a small finite model set, whereas this paper uses sampling-based direct policy search over a continuous distribution.Standard model-based control commonly estimates a maximum-likelihood target model before policy optimization.
- Parametrized skills: Parametrized-skill methods typically assume task parameters are revealed at test time, while this work addresses target parameters such as friction that remain unknown.The related work also discusses methods aimed at reducing sample complexity when a baseline policy is available.
6 CONCLUSIONS AND FUTURE WORK
The paper concludes that EPOpt trains robust, generalizable policies through source-domain ensembles and supports Bayesian adaptation using limited target-domain experience. It also identifies computationally intensive parameter-space sampling as a limitation of the adaptation procedure.
- Conclusions: EPOpt-ϵ trains robust policies on source-domain ensembles and supports adversarial training intended to provide good direct-transfer performance.The authors report highly robust and generalizable policies in complex simulated robotic tasks.
- Conclusions: Bayesian model adaptation can update the source-domain ensemble with a small amount of target-domain experience and produce better target policies than maximum-likelihood estimation under unmodeled effects.This comparison is particularly emphasized when the target contains effects absent from the initial model description.
- Future work and limitations: The adaptation algorithm is computationally intensive because it samples the parameter space, with cost increasing as the number of variable physical parameters grows.The authors note that adaptive prior sampling is fast and reliable when the true model does not have very low prior probability.
- Future work and limitations: When the true model has very low prior probability, adaptation requires a different sampling distribution capable of producing samples from all regions of the parameter space.The paper identifies this as a general drawback of Bayesian adaptation methods and proposes alternative sampling and parameterization schemes.
A.1 DESCRIPTION OF SIMULATED ROBOTIC TASKS CONSIDERED IN THIS WORK
The experiments use simulated 2D hopper and half-cheetah tasks with torque-controlled articulated bodies. Their dynamics and contact challenges make them useful settings for studying robustness under parameter uncertainty.
- Hopper: The hopper has three joints, four body parts, a 12-dimensional state space, and a 3-dimensional torque action space.Its objective is to hop forward as fast as possible.
- Hopper: The hopper source domain varies torso mass, ground friction, armature, and foot damping across simulated models.These parameters represent the physical uncertainty used to construct the source-domain distribution.
- Half-cheetah: The half-cheetah has eight body links, an 18-dimensional state space, and a 6-dimensional joint-torque action space.Its objective is to run forward as fast as possible.
- Half-cheetah: The half-cheetah source domain varies torso and head mass, ground friction, damping, and foot-joint armature.The task therefore exposes policies to uncertainty across several physical parameters.
- Task challenges: Underactuation and contact discontinuities, combined with parameter uncertainty, can dramatically degrade policy quality when robustness is not explicitly considered.The paper illustrates the hopper and half-cheetah models as representative experimental challenges.
A.2 HYPERPARAMETERS
The experiments use a two-hidden-layer tanh policy network, TRPO updates, a 0.01 KL trust-region constraint, and 240 length-1000 rollouts per iteration. Results in Figures 1 and 2 use 150 and 200 TRPO iterations, respectively.
- The policy network has two hidden layers with 64 units each and tanh nonlinearities, with updates implemented using TRPO.
- Successive TRPO policy updates are constrained to a maximum KL divergence of 0.01.
- Each iteration samples N = 240 models, performs one rollout per model, and uses trajectories of length 1000.
- Figures 1 and 2 use 150 and 200 TRPO iterations, respectively, with 240 trajectories per iteration.
A.3 WORST-CASE ANALYSIS FOR HOPPER TASK
The hopper worst-case analysis evaluates the 10th percentile of return distributions, finding that EPOpt(ϵ = 0.1) outperforms the alternatives and is highly robust and reliable.
- The 10th percentile of the return distribution is used as a proxy for worst-case performance in the hopper task.The analysis motivates this statistic as important for robust control policies.
- EPOpt(ϵ = 0.1) clearly outperforms the other approaches on the hopper task’s 10th-percentile return.
- EPOpt(ϵ = 0.1)’s 10th-percentile return nearly overlaps with its expected return, indicating highly robust and reliable policies.
A.4 ROBUSTNESS ANALYSIS FOR HALF-CHEETAH TASK
The half-cheetah robustness analysis examines model-instance performance and training choices, showing robust generalization, lower variance with smaller ϵ, and important effects from baselines and policy-gradient subroutines.
- Robustness across model instances: The adversarially trained policy is robust and generalizes well across half-cheetah models in the source distribution.
- Different settings for ϵ: Decreasing ϵ decreases performance variance while causing a small decrease in average performance, thereby enhancing robustness.
- Importance of baseline: A baseline is important for policy gradients to work well in practice when optimizing expected return.
- Importance of baseline: For the CVaR objective, a baseline is crucial: without one, performance degrades very quickly after switching to ϵ < 1.
- Training comparisons: The learning curves compare EPOpt(ϵ = 1) with and without baselines and compare average-return and CVaR objectives after 100 iterations of pre-training.
- Alternate policy-gradient subroutines: TRPO produces significantly better performance than REINFORCE when used as the BatchPolOpt policy-gradient subroutine.