PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy Gradient Method For Robust Reinforcement Learning

Yue Wang, Shaofeng Zou

arXiv:2205.07344v1cs.LG

TL;DR

Model mismatch between training and test environments can degrade reinforcement-learning performance. The paper develops robust policy-gradient and actor-critic methods, with global-optimality guarantees and an O(ε^-3) complexity result for the smoothed method.

  • Problem

    Model mismatch between training and test environments can significantly degrade reinforcement-learning performance.

    Source: §1 · p. 1

  • Method

    The paper develops robust policy gradients using a Fréchet sub-gradient, smooths the objective for convergence-rate analysis, and extends the approach to model-free actor-critic.

    Source: §1 · pp. 1–2, §6 · p. 6

  • Results

    O(ε^-3) steps suffice for the smoothed robust policy-gradient method to reach an ε-global optimum; under direct policy parameterization, robust policy gradient converges almost surely to a global optimum.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    In simulations, robust policy gradient performs much better than vanilla policy gradient when R > 0, while both perform the same when R = 0.

    Source: §7.1 · p. 7

    The analysis uses direct policy parameterization and an R-contamination uncertainty set; model-free approaches for other uncertainty sets remain future work.

    Source: §8 · p. 9

Abstract

from arXiv · show

This paper develops the first policy gradient method with global optimality guarantee and complexity analysis for robust reinforcement learning under model mismatch. Robust reinforcement learning is to learn a policy robust to model mismatch between simulator and real environment. We first develop the robust policy (sub-)gradient, which is applicable for any differentiable parametric policy class. We show that the proposed robust policy gradient method converges to the global optimum asymptotically under direct policy parameterization. We further develop a smoothed robust policy gradient method and show that to achieve an $\epsilon$-global optimum, the complexity is $\mathcal O(\epsilon^{-3})$. We then extend our methodology to the general model-free setting and design the robust actor-critic method with differentiable parametric policy class and value function. We further characterize its asymptotic convergence and sample complexity under the tabular setting. Finally, we provide simulation results to demonstrate the robustness of our methods.

1. Introduction

Model mismatch between training and test environments can degrade RL performance, while policy-gradient methods lacked robust optimality guarantees. This paper develops a robust policy-gradient approach with global-optimality and complexity analysis.

1.1. Related Works

  • Prior robust RL work includes value-based methods and empirical adversarial training, whose approaches lack theoretical robustness and optimality guarantees.The paper focuses instead on directly optimizing policies, including for uncertain transition kernels.Source: §1 · p. 1, §1.1 · p. 2

1. Introduction

  • Policy-gradient methods are widely used, but robust policy-gradient approaches with provable robustness and optimality guarantees remained largely open.The paper positions its method as the first such approach with complexity analysis.Source: §1 · p. 1
  • The method derives a robust policy subgradient for the worst-case value function, applicable to differentiable and Lipschitz policy classes.The worst-case maximum can make the robust value function nondifferentiable.Source: §1 · pp. 1–2
  • Under direct policy parameterization, the robust value function satisfies the PL condition and the method converges almost surely to a global optimum.The paper identifies the worst-case maximum over the uncertainty set as a challenge to the optimization landscape.Source: §1 · p. 2
  • The smoothed method reaches an ϵ-global optimum of the original nonsmooth robust RL problem with complexity O(ϵ^-3).Smoothing makes the corresponding objective differentiable.Source: §1 · p. 2
  • The paper extends the approach to a model-free robust actor-critic setting using centroid samples, with global optimality proved for tabular direct parameterization.The method is also described as applicable to differentiable parametric policies and value functions.Source: §1 · p. 2

2. Preliminaries

The preliminaries define the MDP and robust-MDP setting, then specify the worst-case objective and the R-contamination uncertainty model used in the paper.

2. Preliminaries

  • An MDP specifies states, actions, a transition kernel, a bounded cost function, and a discount factor; a policy maps states to action distributions.The value function is the expected accumulated discounted cost under that policy.Source: §2 · p. 3
  • In a robust MDP, nature may choose time-varying transition kernels from an uncertainty set, and the policy value is the worst-case expected discounted cost.The robust action-value function conditions additionally on the initial action, and the policy value averages it over actions.Source: §2 · p. 3
  • The paper uses R-contamination, which allows arbitrary transition perturbations with probability R and is suited to random perturbations, adversarial attacks, and sampling outliers.The text notes connections to total variation, KL-divergence, and Hellinger uncertainty sets, and says this model makes the analysis tractable.Source: §2 · p. 3

3. Robust Policy Gradient

This section formulates robust policy optimization and derives a policy subgradient that accounts for the worst-case value. It then gives an algorithm and an exploration measure for policy-gradient updates.

3. Robust Policy Gradient

  • The robust objective minimizes expected worst-case accumulated discounted cost over the chosen parametric policy class.Because the objective involves a maximum over nature’s choices, it may be nondifferentiable.Source: §3 · p. 3
  • The robust policy-gradient theorem gives the gradient where the objective is differentiable and a Fréchet subgradient at nondifferentiable parameters.The policy class is assumed differentiable and Lipschitz.Source: §3 · p. 4
  • The robust policy subgradient depends on the robust Q-function because the worst-case transition kernel depends on the policy.The passage contrasts this with a prior derivation that used the vanilla Q-function while ignoring that dependence.Source: §3 · p. 4
  • An optimization measure with positive mass on every state supports subgradient descent that can find an optimum for the performance measure.The performance measure is the initial-state distribution, while the optimization measure is used for the update objective.Source: §3 · p. 4
  • Algorithm 1 applies the robust policy-gradient update to differentiable, Lipschitz policy classes.The algorithm projects the update onto the parameter set.Source: §3 · p. 4

4. Global Optimality: Direct Parameterization

Under direct policy parameterization, the robust objective satisfies the PL condition, enabling Algorithm 1 to converge almost surely to a global optimum. The analysis handles the objective’s max-over-uncertainty structure, which makes global optimality harder to establish than in the non-robust case.

  • The robust objective Jρ satisfies the PL condition under direct policy parameterization.Source: §4 · pp. 4–5
  • The global-optimality analysis is challenging because the robust objective includes a max over κ and is non-differentiable.Source: §4 · pp. 4–5
  • For any optimization measure µ, a stationary point of Jµ is a global optimum of the performance objective Jρ.Source: §4 · p. 5
  • Algorithm 1 converges almost surely to a global optimum of Jρ under direct policy parameterization.Source: §4 · pp. 4–5

5. Smoothed Robust Policy Gradient

The section smooths the robust objective to address its non-differentiability, establishes global optimality under direct policy parameterization, and analyzes the convergence rate. The resulting method reaches an ε-global optimum in O(ε^-3) steps, while the stated non-robust case recovers O(ε^-2).

5. Smoothed Robust Policy Gradient

  • For convenience, the section assumes ρ = µ and ρmin > 0, while stating that the results extend to ρ ≠ µ.Source: §5 · p. 5
  • LogSumExp approximates the max while preserving a contraction with a unique fixed point, yielding a smoothed robust value function differentiable in θ.Source: §5 · p. 5

5.1. Global Optimality Under Direct Parameterization

  • Under direct policy parameterization, the smoothed objective satisfies the PL condition, and small gradients place θ within O(σ^-1) of a global optimum.Source: §5.1 · pp. 5–6

5.2. Convergence Rate

  • The convergence-rate analysis assumes a smooth policy class; σ trades objective smoothness against approximation error.Source: §5.2 · p. 6
  • O(ε^-3) steps suffice for Algorithm 2 to reach an ε-global optimum; with R = 0, its complexity reduces to O(ε^-2).Source: §5.2 · p. 6

6. Robust Actor-Critic

Robust actor-critic addresses the gap between estimating performance on the sample-generating kernel and optimizing worst-case performance. It combines robust TD value estimation with policy updates, with tabular convergence and sample-complexity guarantees.

6. Robust Actor-Critic

  • The analysis assumes exact robust gradients, while the model-free method uses training samples from the centroid transition kernel.The exact-gradient setting assumes full knowledge of robust value functions and visitation distributions.Source: §6 · p. 6
  • Robust TD estimates worst-case value from centroid-kernel samples, which Monte Carlo alone cannot estimate because it only evaluates the sample-generating kernel.The smoothed robust TD estimate reaches an ϵ_est-global optimum with O(ϵ^-2) samples.Source: §6 · p. 6
  • The robust actor-critic combines robust TD with policy-gradient descent and supports differentiable value-function approximations and parametric policy classes.Replacing max with LSE yields smoothed robust TD for estimating the smoothed value function.Source: §6 · p. 6
  • Convergence is proved for robust TD in the tabular setting; general function approximation may require additional regularity conditions.The convergence-rate theorem specializes the smoothed actor-critic to the tabular setting with direct policy parameterization.Source: §6 · p. 6
  • The overall sample complexity is O(ϵ^-7) to find an ϵ-global optimum under the stated tabular, direct-policy conditions.The theorem sets ϵ_est = O(ϵ^2), M = O(ϵ^-2), and T = O(ϵ^-3); robust TD uses T_c = O(ϵ^-4) samples.Source: §6 · p. 7

7. Numerical Results

Experiments compare robust and non-robust policy-gradient and actor-critic methods on Garnet and Taxi environments, including smoothed variants and RARL. The reported comparisons favor robust methods in worst-case performance, while smoothing approaches robust policy gradient as its smoothing parameter increases.

7. Numerical Results

  • The experiments use Garnet problems with randomly generated transitions and the Taxi environment, evaluating robust value across different R values.Garnet instances are specified by their state and action counts.Source: §7 · p. 7, §7.1 · p. 7

7.1. Robust v.s. Non-robust Policy Gradient

  • For R > 0, robust policy gradient performs much better than vanilla policy gradient; at R = 0, their performance is the same.The comparison plots worst-case objective against iterations on Garnet and Taxi environments.Source: §7.1 · p. 7, Figure 1 · p. 7

7.2. Smoothed Robust Policy Gradient

  • As σ increases, smoothed robust policy gradient behaves similarly to robust policy gradient, whereas small σ performs poorly.The experiment uses different σ values and compares objective values over iterations.Source: §7.2 · p. 8, Figure 4 · p. 8

7.3. Robust Actor-Critic

  • On Garnet G(30, 20), robust actor-critic achieves higher worst-case accumulated discounted reward than vanilla actor-critic.The experiment uses neural-network policy and critic approximations.Source: §7.3 · p. 8, Figure 5 · p. 8

7.4. Comparison with RARL

  • On Taxi, robust actor-critic achieves much higher accumulated discounted reward under the worst-case transition kernel than RARL.The comparison plots robust value against iterations.Source: §7.4 · p. 8

8. Discussions

The discussion emphasizes broad applicability to differentiable policy and value-function approximations, while identifying the R-contamination uncertainty set as the paper’s focus. The appendices establish Lipschitzness, a robust subgradient, and a PL condition under direct parameterization.

8. Discussions

  • The algorithms accommodate arbitrary differentiable value-function approximations and policy parameterizations, but the analysis focuses on direct policy parameterization and R-contamination uncertainty.Other uncertainty sets, including those defined by total variation, KL divergence, or Wasserstein distance, remain future work for model-free methods.Source: §8 · pp. 8–9

A.1. Robust Value Function Is Lipschitz

  • Under Assumption 3.2, robust value, action-value, and objective functions are Lipschitz in policy parameters.The value-function bound uses L_V = k_π|A|/(1 − γ)^2.Source: §A.1 · pp. 14, 16

A.2. Proof of Theorem 3.3: Sub-gradient of Robust Value Function

  • The derived statewise robust-value subgradient also yields a subgradient for the initial-distribution objective.The objective subgradient is the average of the statewise subgradients under ρ.Source: §A.2 · pp. 17, 19

A.3. Proof of Theorem 4.1: Global Optimality: PL-Condition under Direct Parametrization

  • Under direct policy parameterization, the robust-value subgradient satisfies the PL condition.The appendix states this condition as the basis of the direct-parameterization global-optimality analysis.Source: §A.3 · p. 20

A.4. Proof of Theorem 4.2: Convergence and Global Optimality

The proof connects robust policy-gradient updates to Clarke subgradient descent and establishes almost-sure convergence to a global optimum under its stated conditions.

A.4. Proof of Theorem 4.2: Convergence and Global Optimality

  • Where the robust objective is differentiable, its Fréchet subgradient equals its gradient; the method can also be viewed as Clarke subgradient descent.The proof invokes almost-everywhere differentiability of the objective.Source: §A.4 · p. 23
  • Any point with zero Clarke subgradient is globally optimal, so the algorithm’s sequence converges to the global optimum.The proof also views the robust policy-gradient update as Clarke subgradient descent.Source: §A.4 · pp. 23–24
  • The robust policy-gradient algorithm converges almost surely to a stationary point under the stated stochastic-algorithm conditions.The proof identifies stationary points using the Clarke subdifferential and requires the stationary-point objective values to have no interior.Source: §A.4 · pp. 23–24

B.2. Proof of Lemma 5.4: Smoothness of Jσ

The appendix analyzes smoothness and the PL condition for the smoothed objective, then gives convergence guarantees for smoothed policy gradient, tabular robust TD, and tabular actor-critic estimates.

B.2. Proof of Lemma 5.4: Smoothness of Jσ

  • The gradient of the smoothed robust objective is Lipschitz, supporting the smoothness analysis of Jσ.The proof bounds the relevant quantities using constants that depend on the smoothing parameter.Source: §B.2 · pp. 25, 28

B.3. Proof of Theorem 5.2: PL-Condition of Smoothed Robust Policy Gradient

  • The smoothed objective Jσ satisfies the PL condition, which is used in the convergence-rate analysis.The proof of the rate result invokes the PL condition established for the smoothed objective.Source: §B.3 · p. 28, §B.4 · p. 31

B.4. Proof of Theorem 5.5: Convergence Rate of Smoothed Robust Policy Gradient

  • The cited result states that Algorithm 2 finds a global ε-optimum of Jσ in O(ε^-3) steps.Source: §B.4 · p. 32

C.1. Convergence of Robust TD under Tabular Setting

  • Under the tabular setting, robust TD and smoothed robust TD converge almost surely to their respective action-value functions.The proof uses contraction of the smoothed robust Bellman operator for the smoothed algorithm.Source: §C.1 · pp. 32–33

C.2. Robust Actor-Critic under Tabular Setting

  • With the exact robust action-value function, the actor-critic update is an unbiased estimate of ∇Jσ.The actor-critic analysis also bounds the gradient-estimate bias when an estimated value function is used.Source: §C.2 · p. 33
  • With an estimated value function, the actor-critic gradient estimate can be biased; the proof bounds that bias in terms of estimation error and smoothing.The analysis characterizes the resulting update as projected stochastic gradient descent with bounded bias.Source: §C.2 · pp. 34, 37

C.3. Proof of Theorem 6.1: Global Convergence of Robust Actor-Critic under Tabular Setting

The convergence proof analyzes a projected stochastic-gradient mapping using the smoothed objective’s PL condition; a useful lemma relates active-function subgradients to the subgradient of their maximum.

C.3. Proof of Theorem 6.1: Global Convergence of Robust Actor-Critic under Tabular Setting

  • The proof defines a stochastic gradient mapping and uses smoothness and the PL condition in its convergence analysis.It also accounts for the stochastic-gradient error in the projected update.Source: §C.3 · pp. 37, 39, 40
  • The analysis considers an iterate selected uniformly from the first T iterates.The proof denotes the selected index by U ∼ Uniform(0, ..., T −1).Source: §C.3 · p. 39

D. Useful Lemmas

  • A subgradient of any active function is also a subgradient of the maximum of the functions.The lemma applies when the function is the pointwise maximum of finitely many functions.Source: D. Useful Lemmas · p. 41

E. Additional Experiments

Additional Garnet experiments compare robust and non-robust policy-gradient and actor-critic methods, reporting higher worst-case accumulated discounted reward for the robust methods.

E. Additional Experiments

  • Robust policy gradient achieves higher accumulated discounted reward under the worst-case transition kernel on Garnet problems G(12, 6) and G(20, 10).Figures 7 and 8 show the two problem settings.Source: E. Additional Experiments · p. 41, Figure 7 · p. 42, Figure 8 · p. 43
  • On G(40, 15), robust actor-critic is reported as more robust than vanilla actor-critic under model mismatch.The comparison uses a neural policy and a neural network to approximate robust value functions.Source: Robust Actor-Critic · p. 43, Figure 9 · p. 43
  • On G(30, 10), robust actor-critic achieves higher worst-case accumulated discounted reward than vanilla actor-critic.The experiment uses direct policy parameterization and a two-layer critic network with 20 hidden neurons.Source: Robust Actor-Critic · p. 43, Figure 10 · p. 44

Related papers

Loading 2205.07344v1…