PaperlayerView exact revision on arXiv

Source-linked AI summary

Action Robust Reinforcement Learning and Applications in Continuous Control

Chen Tessler, Yonathan Efroni, Shie Mannor

arXiv:1901.09184v2cs.LGstat.ML

TL;DR

The paper asks how RL policies can remain effective under uncertainty in the actions actually executed. It formalizes probabilistic and noisy action-robust criteria, develops tabular and deep-RL methods, and reports robust policies with improved unperturbed performance across MuJoCo experiments. The approach also exposes stability and worst-case-robustness limitations.

  • Problem

    RL commonly assumes identical training and testing domains, leaving policy behavior under small perturbations and action uncertainty unclear.

  • Method

    The paper defines PR-MDPs with probabilistic adversarial action replacement and NR-MDPs with direct action perturbations, then extends corresponding algorithms to deep reinforcement learning.

  • Results

    Across several MuJoCo environments, the approach produces robust policies and improves performance even in the absence of perturbations.

  • Takeaways & Limitations

    Action robustness can function as implicit regularization, while PR-MDP behavior is more stable than NR-MDP behavior for the reported gradient-based approach.

  • Takeaways & Limitations

    The approach requires only α or α(s) rather than an uncertainty distribution, but therefore cannot handle arbitrary worst-case perturbations; performance can also depend on domain-specific tuning.

Abstract

from arXiv · show

A policy is said to be robust if it maximizes the reward while considering a bad, or even adversarial, model. In this work we formalize two new criteria of robustness to action uncertainty. Specifically, we consider two scenarios in which the agent attempts to perform an action $a$, and (i) with probability $α$, an alternative adversarial action $\bar a$ is taken, or (ii) an adversary adds a perturbation to the selected action in the case of continuous action space. We show that our criteria are related to common forms of uncertainty in robotics domains, such as the occurrence of abrupt forces, and suggest algorithms in the tabular case. Building on the suggested algorithms, we generalize our approach to deep reinforcement learning (DRL) and provide extensive experiments in the various MuJoCo domains. Our experiments show that not only does our approach produce robust policies, but it also improves the performance in the absence of perturbations. This generalization indicates that action-robustness can be thought of as implicit regularization in RL problems.

1. Introduction

The paper addresses how RL policies can remain effective when real-world dynamics or selected actions are perturbed. It introduces action-robust criteria for abrupt and continuous disturbances and evaluates them in deep RL.

  • Motivation: RL commonly assumes identical training and testing domains, leaving policy generalization under small dynamics perturbations unclear.Examples include changes in mass, friction, traction, tire pressure, humidity, or road conditions.
  • Action-robust criteria: The PR-MDP models an adversary replacing the selected action with probability α, whereas the NR-MDP adds a perturbation directly to the action.These criteria target distinct forms of action uncertainty.
  • Applications and evaluation: The criteria correspond to abrupt interruptions such as sudden pushes and persistent interrupting forces, and are extended to DRL experiments across MuJoCo environments.The experiments report robust policies and improved performance even without perturbations.

2. Preliminaries

The paper formulates continuous-action discounted MDPs with continuous dynamics and rewards, then situates them relative to two-player zero-sum games. Standard MDPs admit stationary deterministic optima, while general zero-sum games may require stochastic policies.

  • MDP framework: The framework uses an infinite-horizon discounted MDP with finite states, compact convex actions, weakly continuous transitions, continuous rewards, and γ ∈ (0, 1).Policies map states to probability measures over the action space.
  • MDP framework: The objective is to find a policy maximizing expected discounted reward, with an optimal value defined for every state.The supplied formulation includes the policy-induced trajectory distribution and conditional expectation.
  • Zero-sum games: An optimal stationary deterministic policy exists for the standard MDP, whereas zero-sum games involve both players’ actions and may select optimal policies from stochastic policies.Game solutions are characterized through Nash equilibrium under the stated value inequalities.

3. Probabilistic Action Robust MDP

The PR-MDP treats action failures as a zero-sum interaction in which an adversary can control the action with probability α. Its policy-iteration schemes exploit PR-MDP structure to obtain tractable updates and convergence guarantees.

  • Definition and interpretation: The PR-MDP defines robustness against an adversary that takes control and performs the worst possible action with probability α.This formalizes possible inability to execute the intended action.
  • Relation to Robust MDPs: PR-MDP is a specific Robust MDP whose implicitly defined uncertainty set consists of convex mixtures of nominal and adversarial dynamics and rewards.The resulting uncertainty set is convex and rectangular.
  • Policy iteration: PR-PI alternates between computing an optimal counter-strategy for a fixed adversary and solving a one-step greedy policy problem.The counter-strategy stage may use any MDP solver.
  • Policy iteration: Soft PR-PI replaces hard adversary updates with Frank-Wolfe gradient updates, while η = 1 makes it completely equivalent to PR-PI.The update maintains feasible policies through convex mixtures.
  • Policy iteration: Unlike general two-player policy iteration, PR-PI solves a single-agent one-step greedy improvement problem, which is easier than a one-step max-min decision.This simplification follows from the specific structure of PR-MDP.
  • Convergence: Both policy-iteration algorithms converge to the unique optimal Nash-equilibrium value, with convergence established for the stated schemes.The supplied results also state that Soft PR-PI contracts toward the optimal value.
  • Caveats: Soft PR-PI may be less sensitive to errors but can undergo dramatic policy changes when the adversarial policy changes, leaving its error analysis for future work.The authors contrast this with the smoother changes associated with Conservative PI.

4. Noisy Action Robust MDP

NR-MDP models action-space perturbations by letting an adversary alter the agent’s selected action, producing a continuous-action robust decision problem. Unlike PR-MDP, its optimal policy can generally be stochastic, and its policy-iteration solution requires a harder min-max step.

  • Action-space robustness: NR-MDP evaluates actions after an adversary adds a bounded perturbation, with executed action (1 − α)a + αā.The adversary’s influence is controlled by α, and α = 0 recovers the standard MDP.
  • Theoretical differences: Unlike PR-MDP, NR-MDP generally has a stochastic optimal policy, and strong duality need not hold over deterministic policies.The paper connects this behavior to optimal stochastic strategies in non-cooperative matrix games.
  • Action-space robustness: The induced NR-MDP reward and dynamics average the original reward and transition kernel over agent and adversary actions.These induced quantities define the value of a policy pair in the noisy-action formulation.
  • Policy iteration: NR-MDP policy iteration alternates MDP solution with a one-step min-max problem that cannot generally be reduced to a single-agent decision problem.The min-max stage also cannot generally be solved with one gradient-oracle call.
  • Policy iteration: The proposed DRL treatment of NR-MDP is a heuristic, whereas the underlying Soft PR-PI algorithm guarantees convergence for PR-MDP in the error-free case.The distinction applies to the Section 6 approach for solving both PR and NR MDPs.

5. Related Work

Prior work studies robustness through uncertain transition models, deterministic control perturbations, and adversarial input perturbations. These approaches span theoretical RL, continuous control, and supervised learning, but differ in setting and solution method.

  • Robust RL: Robust MDP research analyzes uncertainty in transition probabilities, but uncertainty sets and extensions to neural-network function approximation remain unclear.Earlier results include tabular and linear-function-approximation settings.
  • Control: Robust control computes policies against deterministic perturbations using linear programming, quadratic programming, or explicit tree search.The present work instead focuses on policy-iteration and gradient-based schemes for a more specific perturbation setting.
  • Robust Supervised Learning: Adversarial supervised learning trains robust classifiers against small input perturbations, including perturbations generated through GAN-based approaches.This provides a related robustness perspective in which an adversary attempts to cause misclassification.

6. Experiments

The experiments adapt action-robust criteria to deep reinforcement learning and evaluate them across MuJoCo continuous-control domains. The results show robust policies, cross-domain transfer in most cases, and performance gains without perturbations, while revealing sensitivity to hyperparameters and training dynamics.

  • Deep RL approach: AR-DDPG trains actor and adversary networks with a critic estimating the joint-policy Q-function for PR-MDP and NR-MDP.The approach adapts Soft PR-PI to high-dimensional policies using alternating actor and adversary gradient updates.
  • Experimental setup: MuJoCo evaluation tests robustness to abrupt perturbations and mass uncertainty across continuous-control domains.The probabilistic operator models abrupt forces, while the noisy operator models persistent action disturbances associated with mass uncertainty.
  • Hyperparameter ablation: PR-MDP performance is most stable for α ∈[0.1, 0.15], whereas NR-MDP hyperparameter effects are less predictable under the gradient-based approach.For PR-MDP, performance deteriorates when α deviates from this interval; the selected configuration uses parameter-space noise with α = 0.1 and a 10:1 training ratio.
  • Hyperparameter ablation: PR-MDP can outperform the baseline without additional exploration noise, indicating that its adversary supplies enough noise for exploration.The authors attribute this observation to the adversarial component of the PR-MDP criterion.
  • Testing on various MuJoCo domains: In most MuJoCo domains, both operators outperform the baseline in robustness and unperturbed performance, with parameters transferring without additional tuning.The PR-MDP advantage over NR-MDP in Hopper-v2 does not hold across all domains, and optimal parameters may differ between domains.
  • Testing on various MuJoCo domains: InvertedPendulum is a failure case: PR-MDP performs worse than the non-robust baseline, and a large α can prevent successful task completion.The authors suggest that hyperparameters tuned on Hopper-v2 may not suit every domain.
  • Diving deeper: NR-MDP’s operator drives performance improvement while adversarial exploration adds a small stability gain; PR-MDP requires both adversarial exploration and the operator.A stronger adversary experimentally degrades performance, potentially because the actor receives fewer updates or the adversary converges faster to a sub-optimal solution.

7. Summary

The paper introduces two action-robustness criteria and develops algorithms for solving them, including a deep reinforcement learning extension. The approach produces robust policies and improves performance even without perturbations.

  • The paper introduces Probabilistic and Noisy action Robust MDP criteria for modeling action uncertainty.The approach also discusses their theoretical differences and relations to real-world uncertainty.
  • The authors develop Soft PR-PI and a deep reinforcement learning method for solving the proposed criteria.The deep approach builds on the Soft PR-PI algorithm.
  • Action-robust training produces robust policies and improves performance even in the absence of perturbations.

A.1. Preliminaries

The preliminaries formulate discounted zero-sum Markov games with finite states and continuous actions, then establish Bellman-operator and policy-iteration properties. Under the stated assumptions, the game has a well-defined equilibrium value, and soft policy iteration contracts toward it.

  • Preliminaries: The Bellman operators and fixed-policy operators are defined over probability measures for both players, with attainable maxima and minima.Compactness and continuity justify replacing suprema and infima with maxima and minima.
  • Preliminaries: All principal Bellman operators are γ contractions in the sup-norm.The fixed-policy result follows from the stochastic-transition-matrix property, with analogous arguments for the other operators.
  • Preliminaries: The unique fixed point v* is the equilibrium value of the zero-sum Markov game, with stationary policies satisfying Nash-equilibrium inequalities.
  • Policy Iteration: Soft Zero-Sum Markov-Game Policy Iteration updates policies softly toward one-step greedy policies rather than using a fully greedy update.The paper presents this as a generalization of ordinary policy iteration to Markov games.
  • Policy Iteration: The sequence of soft policy-iteration values contracts toward v* at rate 1 −η + γη.Setting η = 1 yields the convergence rate for ordinary Zero-Sum Markov-Game Policy Iteration.
  • Probabilistic Robust MDP: The value of the probabilistic robust MDP corresponds exactly to the value of its induced zero-sum Markov game.This establishes equivalence between solving the two formulations.

B.2. Proof of Proposition 1

The proof maps probabilistic action robustness to a zero-sum Markov game whose adversary represents the action uncertainty. This equivalence allows the general soft game policy-iteration results to establish the probabilistic robust policy-iteration result.

  • The induced game decouples its Bellman operators into terms corresponding to the probabilistic action-robust formulation.
  • The optimal maximizing policy satisfies the Bellman optimality condition and can always be chosen deterministic.
  • The adversary can equivalently choose transition dynamics and rewards rather than an action that induces those changes.Thus, the probabilistic action-robust formulation is related to standard Robust MDPs through the induced game.
  • Soft Probabilistic Robust Policy Iteration is an instance of the general Soft Zero-Sum Markov-Game Policy Iteration scheme.Its convergence theorem follows as a corollary of the general game result.

C.2. Proof of Proposition 4

The proof shows that noisy action robustness can require stochastic maximizing policies. A quadratic single-state example demonstrates that randomized policies achieve a strictly larger robust value than deterministic policies, so deterministic-policy strong duality fails.

  • For the quadratic reward r(a) = a2 on [−1, 1], deterministic max-min and min-max values are not equivalent.
  • Strong duality therefore does not hold when both players are restricted to deterministic policies.
  • A maximizing policy that assigns probability 0.5 to each of −1 and 1 achieves robust value (1 −α)2.
  • The randomized-policy robust value is strictly greater than the deterministic-policy value in this example.
  • The noisy robust problem alternates between solving an MDP for a fixed adversary and solving a one-step greedy minimax problem over stochastic policies.
  • The proposed analysis does not establish that a simple Frank-Wolfe update works for the noisy robust policy-iteration problem.

D. Actor Gradients Proof

The paper derives action-robust actor-gradient updates for PR-MDP and NR-MDP, adapting deterministic policy gradients to actor–adversary mixtures and their assumptions. The resulting DDPG procedure updates actor, adversary, critic, and target networks using the corresponding mixture structure.

  • Assumptions: The derivation assumes continuity of transition, reward, policy, and gradient terms across states, actions, parameters, and auxiliary variables.Additional boundedness and Lipschitz conditions constrain initial-state, transition, reward, and gradient magnitudes.
  • Gradient derivation: The actor and adversary gradients are weighted by 1 −α and α, respectively, while the action-value gradient is taken with respect to the mixture action.This extends the deterministic policy-gradient structure to the PR-MDP setting.
  • Algorithm: Action-Robust DDPG executes actor or adversary actions in PR-MDP and a convex combination in NR-MDP, then stores perturbed transitions for replay.Exploration noise may be added at the action level or through parameter-space perturbations.
  • Algorithm: The critic differs from standard DDPG because it is trained on the policy expectation: an α-mixture policy in NR-MDP and a convex sum of Q values in PR-MDP.The actor and adversary are improved by backpropagating Q gradients through the critic.
  • PR-MDP formulation: PR-MDP uses a stochastic policy that selects the actor action with probability 1 −α and the adversary action with probability α.The criterion is constructed from two deterministic policies and represents adversarial control occurring probabilistically.

E. Empirical Results

The empirical section evaluates exploration and α choices for NR-MDP and PR-MDP, alongside robustness to model and mass uncertainty. The figures organize comparisons across noise, parameter, training-ratio, and uncertainty settings.

  • NR-MDP ablations: NR-MDP experiments ablate exploration and α, while separate plots examine α and training-ratio choices.These are presented in Figures 7, 8, and 10.
  • PR-MDP ablations: PR-MDP experiments likewise ablate exploration and α, with Figure 10 specifically identifying the PR-MDP α and training-ratio ablation.The figure labels distinguish no noise, OU noise, and parameter noise conditions.
  • Uncertainty evaluations: The robustness evaluations include model uncertainty, randomly sampled noise replacing the selected action, and mass uncertainty.Figures 11–13 define or label these uncertainty settings.
Loading 1901.09184v2…