Maximum Entropy RL (Provably) Solves Some Robust RL Problems

Benjamin Eysenbach; Sergey Levine

Source-linked AI summary

Maximum Entropy RL (Provably) Solves Some Robust RL Problems

Benjamin Eysenbach, Sergey Levine

arXiv:2103.06257v2Mar 10, 2021cs.LGcs.RO

TL;DR

The paper addresses how RL policies can retain performance under disturbances to dynamics or reward functions, a setting where robust methods often add optimization machinery. It proves that MaxEnt RL optimizes a lower bound on a robust RL objective, characterizes the associated robust sets, and validates the result empirically. The results suggest that standard MaxEnt RL can provide robustness to certain disturbances without additional modifications.

Problem
RL applications need guarantees of good performance under disturbances to dynamics or reward functions, while existing robust RL methods typically add adversarial optimization and hyperparameters.
Method
The paper theoretically analyzes MaxEnt RL as a lower-bound optimization method for a robust RL objective defined over perturbed dynamics and reward functions.
Results
MaxEnt RL is formally shown to optimize a lower bound on a robust RL objective, with experiments validating robustness to dynamics and reward perturbations.
Takeaways & Limitations
MaxEnt RL by itself can serve as a simple robust RL method for certain disturbances, without additional modifications.
Takeaways & Limitations
The analysis uses a robust set different from standard H∞ and KL-divergence constraint sets, and constructing the augmented reward may require knowledge of the dynamics.

Abstract

from arXiv · show

Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, to the best of our knowledge our work provides the first rigorous proof and theoretical characterization of the MaxEnt RL robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm. In contrast, our results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modifications. While this does not imply that MaxEnt RL is the best available robust RL method, MaxEnt RL is a simple robust RL method with appealing formal guarantees.

1 INTRODUCTION

The paper argues that MaxEnt RL can provide robustness to certain disturbances without modifying existing RL methods. It theoretically characterizes this robustness and illustrates it empirically in disturbed environments.

Motivation: RL applications need policies that retain high reward under environmental, dynamics, modeling, or reward disturbances.Such disturbances may arise from modeling errors, non-stationarity, human biases, or adversaries.
Motivation: MaxEnt RL offers a simple route to robustness using existing methods rather than adversarial optimization and extra hyperparameters.The paper motivates leveraging established MaxEnt RL algorithms for robust RL problems.
Empirical intuition: MaxEnt RL learns stochastic policies that use multiple routes, so some routes can remain successful after an obstacle is added.In the robot-pushing example, standard RL repeatedly follows one route and collides with the new obstacle.
Theoretical contribution: The paper proves that the MaxEnt RL objective is a nontrivial lower bound on a robust RL objective for a particular robust set.The robust objective uses a different reward function, and the corresponding robust set covers dynamics perturbations and reward perturbations.
Theoretical contribution: Robustness is obtained only for relatively large entropy coefficients, according to the paper’s analysis and experiments.The experiments are presented as validation of the theoretical characterization.

2 RELATED WORK

Related work distinguishes robust RL from robust control, domain randomization, and transfer learning by its focus on worst-case reward under perturbed environments.

Robust control and robust RL: Robust RL methods can learn policies without knowing the true dynamics, but standard RL policies often fail under environmental disturbances.This motivates methods designed specifically for robustness.
Robust control and robust RL: Robust control seeks stability independent of reward, whereas robust RL estimates policy returns across similar MDPs.The paper places robust RL in relation to distance-metric approaches for MDPs.
Related objectives: Domain randomization maximizes average reward across environments, unlike robust RL’s focus on worst-case performance.The distinction separates average-case multi-environment training from adversarial or worst-case evaluation.
Related objectives: Transferable and generalizable RL work emphasizes average-case performance on new environments rather than worst-case performance.The paper identifies this as a related but distinct objective.

3 PRELIMINARIES

The preliminaries define standard and maximum-entropy RL through trajectory distributions, rewards, and policies. They then formulate robustness as worst-case evaluation over adversarially selected dynamics or rewards.

Standard RL formulation: A trajectory consists of states and actions generated by an initial-state distribution, transition dynamics, and policy over T steps.The trajectory distribution under policy π is denoted pπ(τ).
Standard RL formulation: The standard RL objective evaluates expected cumulative reward under the trajectory distribution induced by policy π.Discounting is omitted because it can be represented through an absorbing-state modification of the dynamics.
Maximum-entropy RL: MaxEnt RL maximizes expected reward plus conditional action entropy under a policy, dynamics p, and reward function r.The entropy coefficient α balances the reward and entropy terms; the analysis uses α = 1.
Robust RL formulation: Robustness is measured by evaluating a policy under a new reward function or dynamics function selected adversarially from specified sets.The robust objective is formulated as a two-player zero-sum game whose goal is a Nash equilibrium.
Robust RL formulation: The paper seeks to show that MaxEnt RL with a different reward function optimizes a lower bound on this robust objective.The analysis characterizes the dynamics and reward sets for which the bound holds.

4 MAXENT RL AND ROBUST CONTROL

The paper proves that MaxEnt RL maximizes a lower bound on robust RL objectives for particular reward and dynamics perturbation sets. Robustness to dynamics requires applying MaxEnt RL to a different, pessimistic reward that incorporates dynamics entropy, while the resulting robust set is governed by an entropy-based budget.

Robustness to adversarial rewards: MaxEnt RL is equivalent to a robust RL objective for a policy-dependent set of adversarial reward functions.The adversary cannot decrease rewards too much and may target states and actions frequently visited by the policy.
Robustness to adversarial dynamics: The robust dynamics set is defined through a divergence that penalizes adversarial transitions assigning low probability to likely training-environment transitions.The adversary selects a dynamics function independently at each time step, and the proof derives dynamics robustness from reward-perturbation robustness.
Robustness to adversarial dynamics: MaxEnt RL can robustly optimize one reward across similar dynamics, but it must be applied to a pessimistic reward function.The pessimistic reward includes a log-reward term and a dynamics-entropy term; computing it may require knowledge of the dynamics.
Main result: MaxEnt RL provides a lower bound on the robust RL objective for the combined reward and dynamics robust sets.This connects the MaxEnt RL objective to a robust objective while using distinct robust and training reward functions.
Size of the robust set: The robust-set budget is at least as large as the policy’s entropy, so increasing the entropy coefficient can enlarge the guaranteed robust set.For example, a policy with 10 bits of entropy gives ϵ ≥10; the perturbation constraint holds in expectation, allowing concentrated or distributed changes.
Worked examples: In a two-armed bandit example, the robust RL objective and MaxEnt RL objective are equivalent, and reward perturbations of the optimal action scale as ∆a = O(√ϵ).The worked examples are intended to provide intuition for the robust set’s interpretation in simple problems.
Limitations of analysis: The analysis uses a robust set different from standard H∞ and KL-divergence constraint sets, and estimating the augmented reward is difficult in general MDPs.The dynamics-entropy term can be ignored in special MDPs with constant stochasticity, but may be challenging to estimate otherwise.

5 NUMERICAL SIMULATIONS

Numerical simulations test MaxEnt RL against standard RL across perturbations to dynamics and rewards. The results support robustness to environmental, adversarial, and reward disturbances, with entropy influencing tolerance to larger disturbances.

MaxEnt RL and standard RL are compared in environments with perturbed dynamics, including altered obstacles, goal locations, external forces, and adversarial disturbances.The simulations evaluate whether learned policies continue solving tasks after environmental changes.
MaxEnt RL learns multiple routes or poses, whereas standard RL repeatedly uses one strategy that can fail after environmental perturbations.In the manipulation task, MaxEnt RL often navigates around newly added obstacles while standard RL almost always collides.
MaxEnt RL remains robust to larger mid-episode disturbances when trained with the largest entropy coefficient, supporting the claim that entropy controls robust-set size.All methods perform well without disturbances, but only the largest-entropy method handles larger disturbances.
Under adversarial hole-location perturbations, both methods approach 100% success for small disturbances, but only MaxEnt RL succeeds at 1.5cm or 2cm.For larger perturbations, neither method solves the peg insertion task.
On four continuous-control tasks, only MaxEnt RL succeeds at maximizing reward under the adversarial reward function specified by Equation 2.Standard RL and MaxEnt RL both maximize cumulative reward in the unperturbed evaluation, but only MaxEnt RL maximizes worst-case reward.

6 DISCUSSION

The paper formally characterizes MaxEnt RL's robustness, showing it optimizes a bound on a robust RL objective and performs competitively with adversarial robust RL methods. The analysis also identifies assumptions and practical limitations.

MaxEnt RL algorithms optimize a bound on a robust RL objective.
The robust RL objective uses a different reward function than the MaxEnt RL objective.For robustness across dynamics, MaxEnt RL should use a pessimistic version of the reward function.
The analysis characterizes robust sets for both reward and dynamics perturbations.
Simple MaxEnt RL algorithms perform competitively with, and sometimes better than, recently proposed adversarial robust RL methods.The comparison uses benchmarks proposed by those prior methods.
MaxEnt RL still requires choosing an entropy coefficient, and its robust set is not always simple.
The analysis relies on KKT conditions for convex optimization over the reward function, without requiring convex policies or dynamics.

A.3 WORKED EXAMPLE OF REWARD ROBUSTNESS

The worked example computes the robustness penalty using a Gaussian integral, yielding the reported expression.

2 log(2π) + log(20) is the penalty computed for Example 1.The penalty is obtained using a Gaussian integral.

A.4 PROOF OF THEOREM 4.2

The proof of Theorem 4.2 converts dynamics perturbations into reward deviations, relaxes the constrained problem, and identifies the resulting penalty with action entropy.

The proof proceeds by converting dynamics variation into reward variation, relaxing the robust objective, and showing the penalty equals action entropy.
Jensen's inequality provides a lower bound that analyzes adversarially chosen dynamics as reward perturbations.The rewards are assumed positive so the logarithms are well defined.
The dynamics deviation is parameterized as ∆r(st+1, st, at) = log p(st+1 | st, at) − log ˜p(st+1 | st, at).
The constraint ensuring ∆r represents a probability distribution does not affect the optimization value and can be ignored without loss of generality.
The final derivation applies the Fenchel dual relationship between the penalty and action entropy.

A.5 ROBUSTNESS TO BOTH REWARDS AND DYNAMICS

The paper extends its robustness analysis to simultaneous reward and dynamics perturbations, quantifies scope boundaries, and studies how reward-set robustness changes with temperature.

Appending rewards to observations makes robustness to dynamics equivalent to robustness to rewards, establishing simultaneous robustness to both perturbation types.
The proof establishes a lower bound on the robust-set size because duality alone does not specify whether the radius ϵ is nonzero.
The robustness radius ϵ is derived from the optimal adversarial dynamics, with the proof showing the constraint is tight at optimality.
For continuous state spaces, the entropy-based result requires nonnegative differential transition entropy.
A.3 WORKED EXAMPLE OF REWARD ROBUSTNESS: The dynamics example's penalty is infinite when adversarial and true dynamics have equal variance, while smaller perturbation regions reduce the penalty.
A.8 TEMPERATURES: Higher temperatures produce a strict subset of the reward-function robust set associated with lower temperatures.
A.8 TEMPERATURES: At high temperature, robustness to a reward function extends mainly to similar or pointwise weakly better rewards, whereas low temperature covers more dissimilar rewards.
A.9 MAXENT SOLVES ROBUST CONTROL FOR REWARDS: For any reward-function collection, a corresponding MaxEnt reward function exists whose policy is an optimal robust-reward policy.Finding that corresponding reward function still requires robust optimization.

A.11 ANOTHER COMPUTATIONAL EXPERIMENT

The experiment evaluates whether LowerBound + MaxEnt can approach the optimal minimax policy on robust-reward bandit problems. It achieves performance close to optimal while outperforming simpler baselines.

The evaluation uses 10 five-armed bandit problems with five Gaussian reward functions per problem and normalized minimax reward.Each reward-function mean is drawn from a zero-mean, unit-variance Gaussian, and rewards are shifted to remain positive.
LowerBound + MaxEnt alternates convex reward optimization with computation of the optimal MaxEnt RL policy.The method solves a harder robust-reward problem over a larger reward-function set than the original one.
Fictitious play is treated as converging to the optimal minimax policy, whereas LowerBound + MaxEnt is not expected to attain that optimum because it solves the harder problem.Fictitious play receives access to the full reward function for fair comparison.
0.91 normalized minimax reward is achieved by LowerBound + MaxEnt, close to the optimal value of 1.The pointwise minimum and uniform policies obtain 0.56 and 0.60, respectively.

B.1 DYNAMICS ROBUSTNESS EXPERIMENTS (FIG. 4)

The dynamics-robustness experiments evaluate MaxEnt RL across manipulation, button-press, navigation, peg-insertion, and minimax-reward settings. The figure evidence reports that MaxEnt RL succeeds where standard RL does not on minimax reward and obstacle avoidance.

The experiments use standard Pusher-v2 and SawyerButtonPressEnv settings, including fixed entropy coefficients and a 10-episode average for button pressing.The button-press environment is perturbed by offsetting the button along the Y axis.
The navigation and peg-insertion experiments use modified environments with explicit reward and episode-length specifications.The navigation episodes last 48 steps, while peg insertion episodes last at most 200 steps and terminate when the peg enters the hole.
MaxEnt RL succeeds in maximizing minimax reward, while both RL and MaxEnt RL effectively maximize expected reward.This comparison is summarized in Figure 11.

C ROBUST RL ABLATION EXPERIMENTS

The ablation study tests whether larger networks, dual critics, or increased exploration improve robust-RL baselines. These modifications generally do not significantly improve performance, and dual critics can perform worse.

C ROBUST RL ABLATION EXPERIMENTS: Larger networks, dual critics, and more exploration do not significantly improve NR-MDP or PR-MDP performance.The ablations were designed to distinguish the effects of SAC, network capacity, critic design, exploration, and entropy regularization.
C ROBUST RL ABLATION EXPERIMENTS: The dual-critic ablation generally performs worse than the baseline.The authors hypothesize that using the pessimistic minimum of two Q functions when updating the adversary may produce a weaker adversary.
C ROBUST RL ABLATION EXPERIMENTS: On Hopper-v2, PR-MDP with larger networks is possibly the best method for large relative masses, forming an exception to the general ablation pattern.This exception is reported alongside the broader finding that the changes do not significantly improve the baselines.
C ROBUST RL ABLATION EXPERIMENTS: Incorporating dual critics into the action-robustness framework may require non-trivial design decisions.The concern follows from the observed weaker-adversary behavior under the dual-critic update.