PaperlayerView exact revision on arXiv

Source-linked AI summary

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing

Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, Lei Han

arXiv:2206.02829v3cs.LGcs.AIstat.ML

TL;DR

Offline RL must contend with distribution shift and limited robustness to observation deviations. RORL combines policy and value smoothing with conservative estimation, achieving strong benchmark performance and robustness to adversarial perturbations.

  • Problem

    Offline RL’s distribution-shift conservatism can impair robustness to observation deviations, while perturbed states may also create OOD observations and value overestimation.

    Source: §1 · pp. 1–2

  • Method

    RORL smooths policies and value functions near dataset states and conservatively estimates perturbed-state values using pessimistic bootstrapping.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    RORL achieves state-of-the-art D4RL performance with fewer ensemble Q networks and is considerably robust to adversarial observation perturbations across attack types.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    Robust training can improve performance in non-perturbed environments, as reported for the experiments.

    Source: §1 · p. 2

    Adversarial state sampling slows computation; the authors suggest smoothing or penalizing policy and Q-functions in latent spaces as future work.

    Source: §8 · p. 10

Abstract

from arXiv · show

Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are generally designed to be conservative in value estimation and action selection. However, such conservatism can impair the robustness of learned policies when encountering observation deviation under realistic conditions, such as sensor errors and adversarial attacks. To trade off robustness and conservatism, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique. In RORL, we explicitly introduce regularization on the policy and the value function for states near the dataset, as well as additional conservative value estimation on these states. Theoretically, we show RORL enjoys a tighter suboptimality bound than recent theoretical results in linear MDPs. We demonstrate that RORL can achieve state-of-the-art performance on the general offline RL benchmark and is considerably robust to adversarial observation perturbations.

1 Introduction

Offline RL’s distribution-shift conservatism leaves robustness to observation perturbations under realistic conditions as an open challenge; RORL addresses both concerns with conservative smoothing.

1 Introduction

  • Offline RL avoids costly online data collection, but distribution shift can cause OOD-action overestimation, while observation deviations raise an under-studied robustness concern.Sensor errors and model mismatch motivate robustness; perturbed states can also create OOD observations and extra value overestimation.Source: §1 · pp. 1–2
  • CQL’s non-smooth value function can degrade performance under even tiny observation perturbations, while simply smoothing existing methods may overestimate near support boundaries.That overestimation can direct the agent toward unsafe areas.Source: §1 · p. 2
  • RORL regularizes policies and value functions near dataset support, conservatively estimates perturbed-state values, and has a tighter suboptimality bound than previous work in linear MDPs.The bound claim is made under linear MDP assumptions.Source: §1 · p. 2
  • RORL achieves state-of-the-art D4RL performance with fewer ensemble Q networks than the current state-of-the-art approach and is more robust to adversarial observation perturbations.The paper reports consistently superior performance across several continuous-control tasks and attack types.Source: §1 · p. 2

2 Preliminaries

Offline RL learns from a fixed dataset, where distribution shift can make value estimates for rarely observed actions unreliable; prior approaches address uncertainty through pessimism or smoothness.

2 Preliminaries

  • Offline RL considers an episodic MDP and seeks an optimal policy using experiences from a fixed dataset; direct off-policy learning can extrapolate on rarely observed state-action pairs.The Bellman target’s greedy action may be out of distribution, causing extrapolation error.Source: §2 · p. 2
  • PBRL uses bootstrapped Q-functions to quantify uncertainty, penalizes high-uncertainty estimates, and samples OOD actions for additional regularization.Its uncertainty is the standard deviation among bootstrapped Q-functions; under linear MDP assumptions, it is equivalent to an LCB penalty.Source: §2 · pp. 2–3
  • SR2L regularizes policy and value outputs to change little under bounded state perturbations, and is reported to improve robustness to random and adversarial perturbations.Its perturbation set uses an ℓp distance and its regularizers maximize over the set.Source: §2 · p. 3

3 Robustness of Offline RL: A Motivating Example

The motivating example shows that CQL’s value function varies under adversarial state perturbations, while conservative smoothing improves value smoothness and policy robustness.

3 Robustness of Offline RL: A Motivating Example

  • The example attacks CQL policies by maximizing the Jeffrey’s divergence between original and perturbed-state policies within an ℓ∞-bounded perturbation set.In walker-medium-v2, perturbation scales include ε ∈ {0, 0.05, 0.1, 0.14}.Source: §3 · p. 3
  • As adversarial perturbation strength increases, CQL’s Q-curve deviates more from the unperturbed curve; with conservative smoothing, the value function becomes smoother.The comparison evaluates Q-values at adversarially perturbed states.Source: §3 · pp. 3–4
  • CQL-smooth significantly improves CQL’s robustness, especially at larger adversarial noise scales.The evaluation varies ε over [0, 0.15].Source: §3 · p. 4

4 Robust Offline RL via Conservative Smoothing

RORL combines smoothness near dataset states with uncertainty-based penalties on unfamiliar state-action pairs to balance robustness and conservative value estimation.

4 Robust Offline RL via Conservative Smoothing

  • RORL smooths policy and value functions near the dataset, then uses bootstrapped uncertainty to penalize perturbed states paired with OOD actions.The method targets overestimation outside dataset support while retaining smooth value estimates on OOD states.Source: §4 · pp. 4–5
  • RORL trains Q-functions on dataset pairs with an extended soft Q-learning target using the minimum target Q-value and entropy regularization.The target matches SAC-N’s learning target.Source: §4 · p. 4
  • For perturbed states paired with dataset actions, RORL minimizes Q-value differences under adversarial perturbations to smooth each Q-function.The perturbation is selected to maximize an inner objective; smoothing uses different weights for overestimated and underestimated perturbed values.Source: §4 · pp. 4–5
  • RORL applies smoothing to perturbed states with dataset actions, not OOD actions, because actions near the dataset should remain close to behavior actions.The perturbed-state smoothing loss is distinct from the uncertainty penalty used for OOD actions.Source: §4 · p. 5
  • The policy objective combines maximizing the ensemble’s minimum Q-value, entropy regularization, and a penalty for policy differences under adversarial state perturbations.The adversarial state is chosen within the perturbation set.Source: §4 · p. 5

5 Theoretical Analysis

The analysis studies RORL under linear MDP assumptions and shows that conservative smoothing yields a positive-definite covariance matrix and a tighter suboptimality bound than PBRL. It also establishes that RORL forms a valid uncertainty quantifier for pessimistic value iteration.

5 Theoretical Analysis

  • The analysis assumes linear state-action features and linear transition and reward functions, and solves RORL’s parameter in closed form using LSVI.The simplified objective contains an ordinary TD-error, a conservative smoothing loss, and an additional OOD-sampling loss.Source: §5 · p. 5
  • The OOD-sampling loss enforces conservatism for OOD states and actions, using perturbed states rather than dataset states and actions sampled from the policy.Source: §5 · pp. 5–6
  • RORL’s covariance matrix is positive-definite and dominates PBRL’s covariance matrix under the stated conditions.The additional conservative smoothing loss induces the covariance matrix used to address the coverage issue.Source: §5 · p. 6
  • RORL forms a valid ξ-uncertainty quantifier because its bootstrapped uncertainty estimates the LCB penalty and OOD sampling provides a covariance matrix bounded below by λI.This supports characterizing the optimality gap through pessimistic value iteration.Source: §5 · p. 6
  • RORL enjoys a tighter suboptimality bound than PBRL under the stated linear MDP conditions.The result is derived using pessimistic value iteration.Source: §5 · p. 6

6 Experiments

Experiments evaluate RORL on D4RL continuous-control benchmarks, under adversarial observation attacks, and through ablations and computational-cost comparisons. RORL performs better or comparably to key baselines on benchmark tasks and shows improved robustness under attacks, with runtime trade-offs.

6.1 Benchmark Results

  • RORL achieves better or comparable performance than EDAC with 10 Q-networks, outperforms EDAC-10 and PBRL, and improves on state-of-the-art results for walker2d and hopper tasks.Table 1 reports average normalized scores and standard deviations; the authors attribute the gains partly to balancing conservatism and robustness.Source: Table 1 · p. 7, §6.1 · p. 7

6 Experiments

  • The evaluation covers HalfCheetah, Hopper, and Walker2d across five dataset types, with small training perturbation scales and no observation perturbation at test time.The benchmark comparison includes BC, CQL, EDAC, PBRL, and SAC-10 baselines.Source: §6.1 · p. 7, §6 · p. 7

6.2 Adversarial Attack

  • Across five adversarial attack types on three medium-v2 datasets, RORL exhibits improved robustness over EDAC and SAC-10.Random attacks are not effective against ensemble-based offline RL algorithms, while mixed-order attacks cause larger performance drops than vanilla zero-order optimization.Source: Figure 4 · p. 8, §6.2 · p. 9

6.3 Ablations

  • On walker2d-medium-v2, the OOD loss is the most essential ablation component, policy smoothing matters especially above perturbation scale 0.2, and Q smoothing has the smallest impact.Each of the three losses contributes to performance under adversarial observation attacks.Source: Figure 5 · p. 9, §6.3 · p. 9

6.4 Computational Cost Comparison

  • RORL runs slightly faster than CQL and much faster than PBRL, but slower than SAC-10 and EDAC, while using 16.7% more GPU memory than comparable methods.The authors attribute PBRL's slower runtime to its 10 Q networks and OOD action sampling; RORL parallelizes its Q networks.Source: Table 2 · p. 9, §6.4 · p. 9

7 Related Works

Related work covers offline RL methods for controlling extrapolation error and robust RL methods for handling perturbations. RORL is positioned as an offline approach combining state-space smoothing with conservative treatment of perturbed states.

Offline RL

  • Model-free offline RL methods regularize policies near the dataset or penalize Q-values for OOD actions, while EDAC uses gradient-diverse Q ensembles and PBRL uses uncertainty-based value underestimation.RORL builds on ensemble networks and focuses on smoothness over the state space.Source: §7 · p. 10
  • Prior offline RL theoretical analyses generally support pessimism and conservatism, and RORL presents its theory as a robust extension of earlier results.The cited analyses make differing assumptions about the dataset.Source: §7 · p. 10

Robust RL

  • Robust RL includes methods for transition perturbations, online domain randomization, and action perturbations; unlike online SR2L, RORL addresses offline learning and potential overestimation on perturbed states.S4RL studies offline observation smoothing through data augmentation.Source: §7 · p. 10

8 Conclusion

RORL combines conservative smoothing with pessimistic value estimation to balance robustness and conservatism in offline RL. The authors report benchmark performance with fewer ensemble Q networks and robustness across adversarial attack types.

8 Conclusion

  • RORL combines conservative smoothing with pessimistic bootstrapping to trade off robustness and conservatism in offline RL.The method smooths policy and value functions for perturbed states and underestimates their values.Source: §8 · p. 10
  • RORL achieves comparable or better offline benchmark performance with fewer ensemble Q networks than previous methods.Source: §8 · p. 10
  • RORL is considerably robust to adversarial observation perturbations across different attack types.Source: §8 · p. 10
  • Adversarial state sampling slows computation; the authors identify smoothing or penalizing policy and Q functions in latent spaces as a future direction.Source: §8 · p. 10

Checklist

The checklist reports that the paper includes theoretical assumptions and proofs, experimental details and results, and citations for existing assets.

Checklist

  • The authors report stating assumptions and providing complete proofs for theoretical results in Appendix A.Source: Checklist · p. 15
  • The checklist reports that reproduction materials, training details, error bars, and compute-resource information are included or referenced.Source: Checklist · p. 15
  • The paper reports citing D4RL and EDAC for existing assets and including code through an anonymized link.Source: Checklist · p. 15

A Theoretical Analysis

The appendix analyzes RORL under linear MDP assumptions, deriving a closed-form LSVI solution and theoretical guarantees. It argues that RORL's uncertainty quantifier supports a tighter suboptimality bound than PBRL.

A.1 LSVI Solution

  • Under linear MDP assumptions, LSVI solves the value-function parameter in closed form from the offline dataset.The analysis assumes bounded rewards and known, bounded state-action features.Source: §A.1 · p. 16

A.2 RORL Solution

  • RORL's parameter is obtained by modifying the LSVI objective with Q-value smoothing and an additional OOD loss.The objective combines ordinary TD error, Q-value smoothing, and OOD loss.Source: §A.2 · p. 16

A.3 ξ-Uncertainty Quantifier

  • RORL's covariance matrix is positive definite under the stated conditions, and its bootstrapped uncertainty forms a valid ξ-uncertainty quantifier.The neural-feature discussion requires sufficient perturbed samples and useful variability in state-action features; it describes a bi-Lipschitz condition as one way to ensure this.Source: §A.3 · pp. 17–18

A.4 Suboptimality Gap

  • RORL enjoys a tighter suboptimality bound than PBRL in linear MDPs.The authors characterize the bound under the same conditions as their uncertainty result.Source: §A.4 · p. 20

B.1 Implementation Details

RORL adds smoothing and OOD-value underestimation to SAC-10, with perturbation sampling used in training. In the reported cost comparison, it uses 28.7% of PBRL's epoch time while retaining greater robustness than SAC-10 and EDAC under perturbations.

B.1 Implementation Details

  • RORL is implemented from SAC-10 with policy and Q smoothing plus OOD value underestimation; perturbed observations are sampled and selected by their policy divergence or smoothing loss.For halfcheetah tasks, the authors omit additional OOD loss because 10 ensemble Q networks already provide sufficient pessimism.Source: §B.1 · p. 21

B.2 Experimental Settings

  • Benchmark evaluations use clean environments and four random seeds, while adversarial evaluations average returns over four seeds across random, action-difference, and min-Q attacks.The adversarial environments perturb observations without changing internal transition dynamics.Source: §B.2 · p. 22

B.3 Visualization Settings of CQL

  • Figure 6 compares CQL and CQL-smooth Q-functions under min-Q mixed-order noise and evaluates their performance across 100 perturbation scales from 0.0 to 0.15.Source: Figure 6 · p. 22

C.1 Computational Cost Comparison

  • RORL requires only 28.7% of PBRL’s epoch time, while using 16.7% more memory and remaining slower than SAC-10 and EDAC.The comparison measures average epoch time and GPU memory on hopper-medium-v2; RORL's sampling procedures are implemented with GPU operations and parallelism.Source: §C.1 · pp. 22–23

C Additional Experimental Results

C.2 Ablations on Benchmark Results

These ablations examine which components and settings support RORL’s benchmark and adversarial performance. They also compare its robustness with alternative smoothing approaches and test how design choices affect performance and computational cost.

C.2 Ablations on Benchmark Results

  • The OOD loss most affects walker2d benchmark performance, while Q smoothing helps stabilize training and final performance in clean environments.With the OOD loss removed, performance drops close to SAC-N; setting ϵood = 0 also causes a significant drop, and ϵood > 0 contributes to value underestimation.Source: §C.2 · pp. 23–24, Figure 9 · p. 24

C.3 Robustness Measures

C.4 Ablations of Components in the Adversarial Experiments

  • RORL’s adversarial component ranking is OOD loss > ϵood > policy smoothing loss > Q smoothing loss, though halfcheetah may not need the OOD loss.The authors attribute that exception to sufficient pessimism from the SAC-10 framework.Source: §C.4 · p. 25

C.5–C.7 Ablations

  • Across adversarial ablations, K = 10 performs best as attack scale rises, τ = 0.2 slightly leads on four of five attack types, and more samples improve robustness at greater cost.K = 6, 8, and 10 have similar initial performance; n = 1 already outperforms SAC-10, making sample count a computational-budget choice.Source: §C.5 · p. 25, §C.6 · p. 25, Figure 10 · p. 26

C.8 Adversarial Attack with Different Q Functions

  • With the same Q functions, RORL’s margin over EDAC widens from 37.3 to 51.2 on walker2d-medium-v2 under the largest ‘min Q’ attack.EDAC’s value function may remain non-smooth, while RORL learns smooth value functions.Source: §C.8 · pp. 26–27, Figure 13 · p. 27

C.9 Comparison with EDAC+Smoothing

  • EDAC+Smoothing slightly improves robustness under large-scale attacks but significantly decreases overall performance, indicating smoothing without explicit OOD penalization can worsen robust scores.The reported large-scale range is 0.2–0.3.Source: §C.9 · pp. 27–28, Figure 14 · p. 28

C.10–C.11 Smoothing Comparisons

  • RORL shows higher robustness across tasks and attack types than PBRL+S4RL, while smoothing IQL helps on two tasks but has little effect on halfcheetah.RORL explicitly smooths value functions and underestimates OOD-state values; the IQL comparison suggests smoothing alone does not consistently improve offline performance.Source: §C.10 · pp. 28–29, §C.11 · p. 29, Figure 16 · p. 29

C.12 Comparing the ’max’ and the ’mean’ Operators in Smoothing

  • The ‘max’ operator yields a more conservative policy under small perturbations and higher robustness under large ones, at the same complexity as the ‘mean’ operator.The authors use ‘max’ by default; it is also a zeroth-order approximation to an inner optimization problem.Source: §C.12 · pp. 29–30, Figure 17 · p. 30

C.13 Comparing Different Optimization for Perturbation Generation during Training

These sections compare perturbation-generation and target choices, extend evaluation to Adroit and AntMaze, and offer customization guidance. RORL maintains the highest benchmark performance within a limited perturbation range, while the authors identify model-based robustness as future work.

C.13 Comparing Different Optimization for Perturbation Generation during Training

  • Zeroth- and first-order perturbation generation have very similar trained-policy performance, although mixed-order attackers outperform zeroth-order attackers during evaluation.Average epoch times are 29.6s for zeroth-order and 72.7s for first-order optimization.Source: §C.13 · pp. 30–31

C.14 Comparison of the “minus target” and the “min target”

  • The ‘min target’ uses fewer hyperparameters and is comparable on hopper-medium-expert-v2, but is unstable across tasks, so the authors default to the ‘minus target’.The ‘min target’ significantly overestimates values on walker2d-random-v2 and underestimates them on hopper-medium-v2 and walker2d-medium-v2.Source: §C.14 · p. 31, Figure 19 · p. 31

C.15 Experiments in Adroit Domains

C.16 AntMaze Tasks

  • RORL achieves the highest average score across six AntMaze tasks, while IQL+smoothing improves IQL on four of six tasks.RORL performs poorly on antmaze-large; its AntMaze variant uses only OOD loss and policy smoothing, with an independent target for Q functions.Source: §C.16 · pp. 32–33, Table 12 · p. 33

D Tips for Customizing RORL

  • For lower computation cost, the authors suggest disabling Q smoothing and using fewer sampled perturbed states; their tuning priority is OOD loss, policy smoothing, then Q smoothing.They note Q smoothing contributes least but consumes substantial computation.Source: D Tips for Customizing RORL · p. 33

E More Related Works

  • RORL focuses on model-free offline RL and leaves robustness for model-based methods to future work.The related-work discussion distinguishes adversarial attack strategies from the paper’s robustness setting.Source: E More Related Works · p. 34

C.17 Robustness of the Benchmark Results

  • On six benchmark tasks, RORL maintains the highest performance within a certain perturbation scale, while EDAC and SAC-10 decline faster for most tasks and attack methods.RORL was trained with smoothing scales in {0.001, 0.005, 0.01}; evaluation scales ranged from 0.00 to 0.05.Source: §C.17 · p. 33, Figure 20 · p. 35

Related papers

Loading 2206.02829v3…