Source-linked AI summary
Robust Reinforcement Learning for Continuous Control with Model Misspecification
Daniel J. Mankowitz, Nir Levine, Rae Jeong, Yuanyuan Shi, Jackie Kay, Abbas Abdolmaleki, Jost Tobias Springenberg, Timothy Mann, Todd Hester, Martin Riedmiller
TL;DR
Policies trained in one environment can be sensitive to perturbed transition dynamics, motivating robustness for continuous-control RL. The paper adapts MPO using worst-case and soft-robust objectives with entropy-regularized Bellman operators. Robust and soft-robust methods outperform their non-robust counterparts across nine Mujoco domains, with improved robust performance also reported on a simulated dexterous hand.
Problem
RL policies trained in a single environment can be sensitive to test-time environment perturbations, which may significantly reduce performance.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The framework is suited to continuous-control algorithms that learn a value function, and the paper reports robust counterparts outperforming non-robust methods across nine Mujoco domains.
Source: §6 · pp. 8–9Evaluation uses manually selected uncertainty sets and unseen test environments, a choice practitioners commonly make using domain knowledge.
Source: §4 · p. 4
Abstract
from arXiv · showhide
We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at https://sites.google.com/view/robust-rl.
1 INTRODUCTION
The paper addresses how policies trained in one environment can remain robust when transition dynamics are perturbed. It adapts robustness to continuous-control RL, then evaluates the resulting methods across Mujoco domains, a simulated robotic hand, and investigative experiments.
Motivation
- Training in one environment can leave RL policies sensitive to test-time perturbations, including dynamics changes, sensor noise, actuator noise, and action delays.The paper notes that this mismatch can hurt performance, including when a policy moves from simulation to the real world.Source: §1 · pp. 1–2
Framework
- The framework incorporates robustness to transition-dynamics model misspecification into continuous-control RL algorithms.The paper uses “model misspecification” for training in one environment and performing in a perturbed version.Source: §1 · p. 2
Contributions
- The authors incorporate robustness into MPO to create R-MPO and also test an adaptation to Stochastic Value Gradients (SVG).The paper identifies extensive experimental analysis of robustness performance as its most significant contribution.Source: §1 · p. 2
- Robust and soft-robust entropy-regularized Bellman operators are shown to be contractions, and RE-MPO and SRE-MPO perform at least as well as R-MPO, sometimes significantly better.The paper motivates entropy regularization as supporting exploration and helping prevent early convergence to sub-optimal policies.Source: §1 · p. 2
- RE-MPO, SRE-MPO, and R-MPO outperform E-MPO and MPO, respectively, across nine Mujoco domains.The paper also reports improved robust performance over the non-robust MPO baseline on the simulated Shadow hand.Source: §1 · p. 2
Investigative experiments
- Investigative experiments examine uncertainty sets, data augmentation, domain randomization, and entropy regularization; learned transition models used as uncertainty sets can improve performance over R-MPO.The paper also compares methods with and without entropy regularization.Source: §1 · p. 2
2 BACKGROUND
The background defines the MDP and return-based value quantities, then describes robust MDPs and MPO. Robust MDPs optimize against an uncertainty set of transitions, while MPO alternates policy evaluation and improvement.
MDPs
- An MDP specifies state and action spaces, a bounded reward function, a discount factor, and transition distributions; RL seeks a policy maximizing expected return.The value and action-value functions express expected discounted returns from a state and from a state-action pair.Source: §2 · p. 2
Robust MDPs
- A robust MDP replaces a single transition model with an uncertainty set and evaluates a policy by its worst-case expected return.The robust Bellman operators are contractions under a rectangularity assumption that lets the worst-case transition be chosen independently for each state and action.Source: §2 · pp. 2–3
MPO
- MPO performs policy evaluation and policy improvement; its improvement step constructs a proposal close to the current policy, then fits a parameterized policy to that proposal.The proposal is constrained by a KL-divergence bound and is proportional to the current policy weighted by exponentiated action values.Source: §2 · p. 3
3 ROBUST MPO
Robust MPO learns a worst-case value function during policy evaluation, replacing MPO’s usual evaluation step. Entropy-regularized robust and soft-robust Bellman operators support corresponding MPO variants and are contraction mappings.
Robust MPO
- Robust MPO replaces the usual policy-evaluation objective with a worst-case squared TD error over the uncertainty set for each state-action pair.The policy-evaluation step applies the Bellman operators, and the robust Bellman operator converges to a unique fixed point for the current policy.Source: §3 · p. 3
Entropy-regularized MPO
- The robust entropy-regularized Bellman operator is a contraction, supporting convergence of robust policy evaluation.The paper also extends the operator to a soft-robust entropy-regularized version.Source: §3 · p. 3
- RE-MPO and SRE-MPO perform at least as well as R-MPO and sometimes significantly better.These variants pair robust or soft-robust entropy-regularized operators with corresponding MPO algorithms.Source: §3 · p. 4
4 EXPERIMENTS
The experiments compare robust and soft-robust MPO with non-robust baselines across perturbed continuous-control tasks. They report broad gains, including on a simulated dexterous hand, and investigate training choices, alternative algorithms, and learned uncertainty sets.
4 EXPERIMENTS
- Robust and non-robust agents train in the nominal environment; robust updates take the infimum across three perturbations, while soft-robust updates average their next-state values.The perturbation set is predefined, and test environments use unseen perturbations.Source: §4 · p. 4
4.1 MAIN EXPERIMENTS
- RE-MPO improves over E-MPO in all nine domains, while SRE-MPO improves over E-MPO in all but Cheetah; non-entropy-regularized robust variants also outperform MPO.Figure 1 shows three domains; the remaining domains and non-entropy-regularized results are reported in the appendix.Source: Figure 1 · p. 5, §4.1 · p. 5
- On the simulated Shadow hand's 79-dimensional state and 20-dimensional action task, RE-MPO outperforms E-MPO, especially as the cube shrinks on the unseen holdout set.The hand is trained to rotate a cube to a target orientation.Source: §4.1 · p. 5
4.2 INVESTIGATIVE EXPERIMENTS
- Threefold longer training does not improve non-robust performance; robust SVG variants outperform E-SVG on two tasks, and entropy-regularized robust return does no worse than robust expected return.The entropy-regularized objective performs significantly better in some cases, such as Cheetah.Source: §4.2 · pp. 6–7
- RE-MPO performs significantly better than limited domain randomization; even Full-DR with 100 perturbations does not outperform RE-MPO using three.The comparison uses Cartpole Balance and Pendulum Swingup for limited DR, and Pendulum Swingup for Full-DR.Source: §4.2 · p. 6
- Across a wider test sweep, robust agents maintain higher performance than non-robust agents, while soft-robust performance initially leads but degrades as perturbations grow.Robust and soft-robust agents remain competitive with the non-robust agent in the nominal environment.Source: §4.2 · pp. 6–7
- Robust agents remain at least as good under alternative nominal-model choices, and Pendulum performance rises when a training perturbation nears the unseen evaluation lengths.The Pendulum evaluation lengths are 1.5, 1.6, and 1.7 meters.Source: §4.2 · p. 7, Figure 4 · p. 7
- DDR-MPO performs poorly with small offline-data batches, can outperform R-MPO as batches grow, especially under larger perturbations, and converges toward R-MPO with still more data.Transition models are trained on batches from 100 to one million datapoints.Source: §4.2 · p. 8, Figure 5 · p. 8
5 RELATED WORK
The paper extends robust Bellman methods to entropy-regularized robust and soft-robust settings. It distinguishes its uncertainty-set formulation from related robust MDP work and contrasts robustness with domain randomization.
5 RELATED WORK
- The paper extends robust Bellman operators to entropy-regularized robust and soft-robust formulations, modifying MPO optimization accordingly.Unlike Grau-Moya et al. (2016), its uncertainty set has no KL constraint and maps to distributions over next states rather than model parameters.Source: §5 · p. 8
- Prior work applies robustness to options, actor-critic methods, and DQN, while soft-robust actor-critic methods seek a less conservative solution.Mankowitz et al. (2018a) learn robust options; Di Castro et al. (2012) introduce robust DQN.Source: §5 · p. 8
- Domain randomization averages learning errors over perturbed trajectories, whereas the paper characterizes it as data augmentation producing behavior averaged across the data.The paper contrasts this with its worst-case robust formulation elsewhere.Source: §5 · p. 8
6 CONCLUSION
The conclusion presents a framework for robustness to transition-dynamics perturbations in value-function continuous-control algorithms, developed chiefly for MPO and its entropy-regularized form. It reports gains across nine MuJoCo domains and a simulated dexterous hand, alongside improved performance using transition models learned from offline data.
6 CONCLUSION
- The framework addresses transition-dynamics perturbations, termed model misspecification, in continuous-control algorithms that learn a value function.The conclusion names actor-critic setups as an example.Source: §6 · pp. 8–9
- Robust and soft-robust entropy-regularized Bellman operators support policy evaluation for the MPO variants and are derived to ensure convergence.The conclusion also reports a robustness experiment with SVG.Source: §6 · p. 9
A BACKGROUND
- Entropy regularization adds a KL penalty relative to a reference policy, encouraging exploration and helping prevent early convergence to sub-optimal policies.The objective regularizes the policy by keeping it close to the base policy.Source: A BACKGROUND · p. 12
B ROBUST ENTROPY-REGULARIZED BELLMAN OPERATOR
The section extends robust Bellman operators with entropy regularization and establishes contraction properties. Its soft-robust version uses an average transition model over the uncertainty set.
B ROBUST ENTROPY-REGULARIZED BELLMAN OPERATOR
- The robust entropy-regularized Bellman operator is a max-norm contraction, enabling a standard value-iteration error bound.The section also states that the optimal robust operator is a contraction.Source: B ROBUST ENTROPY-REGULARIZED BELLMAN… · p. 12
C Soft-ROBUST ENTROPY-REGULARIZED BELLMAN OPERATOR
- The soft-robust operator uses an average transition model distributed over the uncertainty set and is also a gamma-contraction.The average model is defined as the expectation of transition models under a distribution over the uncertainty set.Source: C Soft-ROBUST ENTROPY-REGULARIZED… · p. 12
D ROBUST ENTROPY-REGULARIZED POLICY EVALUATION
The policy-evaluation updates incorporate entropy regularization and robustness into MPO. Robust updates use a worst-case transition, while soft-robust updates use an average transition.
D ROBUST ENTROPY-REGULARIZED POLICY EVALUATION
- Policy evaluation is modified to optimize entropy-regularized expected return and incorporate robustness into the return and TD update.The approach defines entropy-regularized and robust entropy-regularized expected returns.Source: D ROBUST ENTROPY-REGULARIZED POLICY… · pp. 12–13
- The robust TD update uses an infimum over the uncertainty set, whereas the soft-robust update removes that infimum and uses the average transition model.The robust entropy-regularized Bellman operators replace MPO's policy-evaluation step.Source: D ROBUST ENTROPY-REGULARIZED POLICY… · p. 13
- The robust and soft-robust policy-evaluation processes converge to unique fixed points, using the previous policy as the reference policy.Pseudocode for R-MPO, RE-MPO, and SRE-MPO is provided in Appendix I.Source: D ROBUST ENTROPY-REGULARIZED POLICY… · p. 13
E PROOFS
The proofs adapt contraction arguments to entropy-regularized robust operators and derive a value-iteration error bound. The soft-robust proof follows a parallel argument for arbitrary value functions.
E PROOFS
- The robust contraction proof adapts Tamar et al. and Iyengar's arguments to account for entropy regularization.Its estimates use arbitrarily close choices from the uncertainty set and extend to the optimal operator through sup and inf arguments.Source: E PROOFS · pp. 13, 15
- The greedy policy's value-function error is bounded by an approximation-error term and a discounted term depending on the initial value function.The stated approximation error is the maximum Bellman update residual over iterations 0 through N.Source: E PROOFS · p. 16
F SOFT-ROBUST ENTROPY-REGULARIZED BELLMAN OPERATOR
- The soft-robust proof considers arbitrary value functions and uses a parallel sup-operator argument to establish its result.The proof treats both orderings of the operator values.Source: F SOFT-ROBUST ENTROPY-REGULARIZED… · p. 17
G ENTROPY-REGULARIZED POLICY EVALUATION
The section outlines entropy-regularized and robust policy-evaluation updates, then gives implementation details for SVG and MPO and describes the uncertainty-set perturbation table.
G ENTROPY-REGULARIZED POLICY EVALUATION
- Policy evaluation first modifies the TD update for entropy-regularized expected return, then adds robustness to the value function and update.For the robust case, analogous steps yield a robust entropy-regularized TD update.Source: G ENTROPY-REGULARIZED POLICY EVALUATION · pp. 18–19
H.1 ADDITIONAL DETAILS ON THE SVG BASELINE
- The SVG baseline combines entropy regularization with gradient ascent using a reparameterized policy gradient.Its policy standard deviation is computed from the network output with a softplus activation; the reparameterized function uses a standard multivariate normal sample.Source: §H.1 · p. 20
H.2 EXPERIMENT DETAILS FOR MPO AND SVG
- MPO and SVG experiments use two-layer feed-forward networks with 50 neurons to output the mean and diagonal covariance of a Gaussian policy.The covariance's diagonal factors are made positive with a softplus transform; separate hyperparameter tables are listed for SVG and MPO.Source: §H.2 · p. 20, Table 1 · p. 20, Table 2 · p. 21
H.3 UNCERTAINTY SET PARAMETERS
- The uncertainty-set table lists chosen training and holdout perturbations by domain and identifies the perturbed parameter.The text identifies the table's final column as the perturbed parameter.Source: §H.3 · p. 20, Table 3 · p. 21
H.4 MAIN EXPERIMENTS
The section presents performance comparisons for robust, soft-robust, and non-robust MPO variants across nine Mujoco domains, using both bar plots and training curves.
H.4 MAIN EXPERIMENTS
- Figure 6 compares RE-MPO, SRE-MPO, and E-MPO performance in bar plots across nine Mujoco domains, while Figure 8 presents bar plots for R-MPO, SR-MPO, and MPO.Figure 6 uses blue, green, and red bars for RE-MPO, SRE-MPO, and E-MPO, respectively.Source: §H.4 · p. 21
- Figures 7 and 9 show the corresponding agents’ performance as a function of evaluation steps during training.Figure 7 covers RE-MPO, SRE-MPO, and E-MPO; Figure 9 covers R-MPO, SR-MPO, and MPO.Source: §H.4 · p. 21
- The section’s plots compare robust and soft-robust MPO variants with their non-robust counterparts across all nine domains.Source: §H.4 · p. 21
H.5 INVESTIGATIVE EXPERIMENTS
The investigative experiments compare entropy-regularized and non-entropy objectives, sample budgets, domain randomization, uncertainty sets, and nominal models. The algorithms section outlines the policy-evaluation, proposal-distribution, and policy-update steps for robust MPO variants.
H.5 INVESTIGATIVE EXPERIMENTS
- Entropy-regularized agents perform no worse on average than non-entropy-regularized agents, with Cheetah showing a considerable improvement.Figure 11 compares the objectives; the text identifies Cheetah as an example of a larger gain.Source: §H.5 · pp. 21, 24, Figure 11 · p. 24
- Figures 6–9 show three method variants across all nine domains; Figures 7 and 9 show the variants as a function of evaluation steps during training.Source: Figure 6 · p. 22, Figure 7 · p. 22, Figure 8 · p. 23
- Three times more training samples for non-robust E-MPO still yield significantly inferior performance to the robust and soft-robust versions in Cartpole Balance and Pendulum Swingup.Figure 12 shows the additional-sample comparisons.Source: §H.5 · p. 24, Figure 12 · p. 25
- RE-MPO uses a limited set of perturbations better than Limited-DR in Cartpole Balance and Pendulum Swingup; even with 100 perturbations, DR does not outperform RE-MPO in Pendulum Swingup.The 100-perturbation DR setting uses approximately 30 times more perturbations.Source: §H.5 · p. 24, Figure 13 · p. 25
- Robustness is strongest when training perturbations approximate unseen evaluation conditions: moving them farther away can reduce performance, though the agents remain better than the non-robust baseline in the reported cases.For Pendulum Swingup, performance rises as the final training perturbation approaches unseen evaluation lengths.Source: §H.5 · pp. 24–25
- Changing the nominal model leaves the methods comparable for Cartpole when the median or largest perturbation is closer to evaluation, while robust MPO significantly outperforms non-robust MPO for Pendulum when the median remains far away.Source: §H.5 · p. 25, Figure 15 · p. 26
I ALGORITHM
- The robust MPO algorithm evaluates the current policy, builds a proposal distribution from action values, then updates the policy by minimizing its KL divergence from that proposal.The corresponding algorithms are provided for robust entropy-regularized and soft-robust entropy-regularized MPO.Source: I ALGORITHM · p. 27