Source-linked AI summary
Robust Reinforcement Learning using Offline Data
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, Mohammad Ghavamzadeh
TL;DR
Offline robust RL must learn a policy from data collected under a nominal model while seeking good performance against the worst models in an uncertainty set. This paper proposes RFQI, which uses offline data and function approximation to learn a robust policy. It proves approximate-optimality guarantees and reports superior performance on benchmark problems.
Problem
Offline robust RL must learn from nominal-model data while accounting for uncertainty-set models, a challenge involving data collection, optimization over models, and unbiased estimation.
Source: §1 · pp. 1–2Method
RFQI reformulates the robust Bellman operator and its model optimization, then uses offline-data estimates and function approximation in iterative fitted Q-updates.
Source: §1 · p. 3Results
Takeaways & Limitations
The paper demonstrates offline robust policy learning with function approximation for arbitrary large state spaces under its stated assumptions.
Source: §1 · p. 2, §6 · p. 9The work considers uncertainty sets defined using total variation distance; other divergences, including KL and chi-square, are left for future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
The goal of robust reinforcement learning (RL) is to learn a policy that is robust against the uncertainty in model parameters. Parameter uncertainty commonly occurs in many real-world RL applications due to simulator modeling errors, changes in the real-world system dynamics over time, and adversarial disturbances. Robust RL is typically formulated as a max-min problem, where the objective is to learn the policy that maximizes the value against the worst possible models that lie in an uncertainty set. In this work, we propose a robust RL algorithm called Robust Fitted Q-Iteration (RFQI), which uses only an offline dataset to learn the optimal robust policy. Robust RL with offline data is significantly more challenging than its non-robust counterpart because of the minimization over all models present in the robust Bellman operator. This poses challenges in offline data collection, optimization over the models, and unbiased estimation. In this work, we propose a systematic approach to overcome these challenges, resulting in our RFQI algorithm. We prove that RFQI learns a near-optimal robust policy under standard assumptions and demonstrate its superior performance on standard benchmark problems.
1 Introduction
Robust RL seeks policies that perform well under the worst model in an uncertainty set, but offline robust RL must learn from data generated by only a nominal training model. RFQI addresses this setting with function approximation and provable guarantees.
1 Introduction
- Training environments may differ from deployment through modeling errors, changing parameters, or disturbances, and current policies can fail under even mild train-test changes.Examples include differences in robot sensor noise, action delay, friction, and mass.Source: §1 · p. 1
- Prior robust RL work either focused on planning, used tabular or linear approximation with asymptotic guarantees, or lacked theoretical guarantees for deep methods.The introduction positions RFQI as the first offline robust RL work for arbitrary large state spaces using function approximation with provable guarantees.Source: §1 · p. 2
- RFQI targets offline robust RL with arbitrary large state spaces, arbitrary state-action sampling, function approximation, and provable performance guarantees.The closest prior work used tabular settings with generative-model data; this work directly learns robust values and policies.Source: §1 · p. 2
- Offline robust RL must use nominal-model data to account for all uncertainty-set models, while optimizing over those models and estimating the robust update pose further challenges.The robust Bellman operator requires minimization over models, and its approximate update depends on an approximate solution to that optimization.Source: §1 · p. 2
- RFQI reformulates the robust Bellman operator and model optimization, then uses empirical risk minimization and generalized least squares to build modified targets and iteratively learn a policy.The paper proves non-asymptotic approximate-optimality guarantees for the resulting algorithm.Source: §1 · p. 3
2 Preliminaries
The preliminaries frame robust RL as control under a set of possible transition models and offline RL as policy learning from fixed data. They set up the uncertainty set and the function-approximation context for the proposed method.
2 Preliminaries
- The paper considers finite but potentially arbitrarily large state and action spaces, with bounded rewards and policies mapping states to action distributions.Value and state-action value functions are defined for policies under a given model.Source: §2 · p. 3
- An RMDP evaluates a policy against the worst model in an uncertainty set, unlike an MDP with one transition model.The uncertainty set is assumed to satisfy state-action rectangularity; rewards are known and deterministic.Source: §2 · pp. 3–4
- Robust Q-Iteration repeatedly applies the robust Bellman operator, whose contraction yields the optimal robust value, but model-wise optimization makes tabular planning computationally intensive.The optimal policy is obtained by selecting the action with the highest optimal robust Q-value.Source: §2 · p. 4
- Offline Fitted Q-Iteration learns from a fixed dataset and approximates the Bellman update using a function class.The dataset comprises state-action-reward-next-state samples, with state-action pairs drawn from a data distribution.Source: §2 · p. 4
3 Offline Robust Reinforcement Learning
The offline robust RL setting uses data collected under a nominal training model, while seeking a policy robust to a neighborhood of possible testing models. The learning algorithm does not know the nominal model or the uncertainty set.
3 Offline Robust Reinforcement Learning
- The offline dataset is generated under a nominal model, but the learner has no access to data from other models in the uncertainty set.The nominal model itself is unknown to the learner, which only observes the dataset.Source: §3 · p. 4
- The function class F approximates robust Q-values, while G approximates dual-variable functions used in the reformulation.The analysis first treats these classes as finite but potentially exponentially large; covering numbers can replace log-cardinality for infinite classes.Source: §3 · p. 4
- The assumptions require data coverage of policy occupancies, approximate closure of F under the robust Bellman operator, and approximate realizability of the optimal robust Q-function.The concentratability condition bounds the occupancy-to-data distribution ratio; approximate completeness and concentratability are noted to imply realizability.Source: §3 · pp. 4–5
4 Robust Fitted Q-Iteration: Algorithm and Main Results
RFQI develops an offline approximation to robust Q-Iteration by reformulating its model optimization, estimating the resulting targets from nominal-model data, and iterating fitted updates. Under stated assumptions, the paper provides an approximate-optimality guarantee and a proof based on controlling estimation and regression errors.
4.1 Dual Reformulation of Robust Bellman Operator
- The robust Bellman update can be rewritten so its expectation is taken only under the nominal model, enabling estimation from offline data without high-variance importance sampling over uncertainty-set models.The resulting scalar inner optimization is convex; the derivation assumes a fail-state with zero reward and absorbing transitions.Source: §4.1 · p. 5
4.2 Approximately Solving the Dual Optimization using Empirical Risk Minimization
- The dual optimization becomes a functional optimization, replacing separate scalar optimizations for each state-action pair with an empirical risk minimization problem.The dual function class G must approximately represent the optimizer; ERM yields an approximate dual function and a bounded update error.Source: §4.2 · pp. 6–7
4.3 Robust Fitted Q-iteration
- RFQI alternates empirical-risk minimization for the dual function with least-squares regression for the robust Q-update, then outputs the greedy policy.Both the expectation and dual optimization are replaced by empirical estimates from the offline dataset.Source: §4.3 · p. 7
- Under Assumptions 1–4, RFQI has an approximate-optimality guarantee whose error includes the regression and dual-optimization approximation terms.The theorem is also interpreted as a sample-complexity result; robustness contributes ρ^-2 and log(|G|) factors relative to the stated non-robust comparison.Source: §4.3 · pp. 7–8
4.4 Proof Sketch
- The proof bounds policy suboptimality through Q-function error, then recursively controls that error using regression and empirical-risk-minimization bounds.Rademacher-complexity bounds control the dual ERM error, while generalized least squares bounds regression error for the modified robust targets.Source: §4.4 · p. 8
5 Experiments
Experiments evaluate RFQI on CartPole and Hopper under model-parameter changes and action perturbations. RFQI outperforms non-robust baselines in both environments, with results against soft-robust methods varying by environment.
5 Experiments
- Results average 20 seeded runs, with thick lines showing the mean and bands showing ±0.5 standard deviation.The vertical-axis quantity is averaged across the runs.Source: §5 · p. 8
- Across CartPole and Hopper, RFQI outperforms non-robust baselines; it is similar to soft-robust DQN on CartPole and superior to soft-robust DDPG on Hopper.The cited figures show the CartPole and Hopper comparisons.Source: Figure 3 · p. 9, Figure 1 · p. 9, Figure 2 · p. 9
- CartPole tests change force_mag and add action perturbations, while the Hopper test changes leg_joint_stiffness.The experiments use these changes to test robustness.Source: §5 · p. 9
6 Conclusion
The paper presents RFQI for robust MDPs with arbitrarily large state spaces, using offline data and function approximation, and reports provably optimal performance. Its uncertainty sets are limited to total variation distance.
6 Conclusion
- RFQI is presented as a robust RL algorithm with provably optimal performance for arbitrarily large state spaces, using offline data with function approximation.The conclusion also reports superior performance on standard benchmark problems.Source: §6 · p. 9
- The proposed algorithm is reported to perform better on standard benchmark problems.Source: §6 · p. 9
- The considered uncertainty sets use total variation distance only; other f-divergences, sample-complexity lower bounds, and relaxed assumptions are left for future work.The authors identify KL- and Chi-square divergence as examples for future uncertainty sets.Source: §6 · p. 9
Checklist
The checklist records the paper’s reporting of theoretical assumptions and proofs, experimental reproducibility details, and asset and ethics-related disclosures.
Checklist
- The checklist says the paper discusses its limitations and marks potential negative societal impacts as not applicable.Source: Checklist · p. 14
- The checklist reports that theoretical assumptions are stated and complete proofs are provided, with a proof sketch in the main paper and full proofs in the appendix.Source: Checklist · p. 14
- The checklist says code, data, and reproduction instructions are provided, training details are in the appendix, and error bars are reported in the paper and appendix.Source: Checklist · p. 14
- The checklist reports that existing assets are cited and their licenses are mentioned, with no new assets included.Source: Checklist · p. 14
- The checklist marks consent and personally identifiable or offensive-content disclosures as not applicable.Source: Checklist · p. 14
A Useful Technical Results
This section collects tools used in the analysis: concentration and ERM generalization bounds, a variational result for pointwise minimization, and a distributionally robust reformulation based on f-divergences.
A Useful Technical Results
- The section introduces Bernstein’s inequality, which uses second moments to obtain a tighter concentration inequality, and ERM generalization bounds involving Rademacher complexity.It also states an additional bound for finite hypothesis classes.Source: A Useful Technical Results · p. 15
- For decomposable spaces and normal integrands, integral minimization is characterized by pointwise minimization almost surely, provided the infimum is not −∞.Source: A Useful Technical Results · pp. 15–16
- The section gives f-divergence examples including total variation, KL, chi-square, and squared Hellinger, then uses a nominal-distribution expectation to derive a robust Bellman operator reformulation.Source: A Useful Technical Results · p. 16
B Proof of the Proposition 1
The proof specializes a dual representation to total-variation uncertainty and to the minimization in the robust Bellman update. It then bounds the scalar optimization to a finite interval.
Proof of Proposition 1
- For this divergence, the Fenchel conjugate is finite for s ≤ 1/2 and equals −1/2 when s ≤ −1/2.The proof separately treats s ∈ [−1/2, 1/2].Source: B Proof of the Proposition 1 · p. 16
- The proof represents the total-variation uncertainty set using the f-divergence generated by f(t) = |t − 1|/2.It adapts Proposition 2 to the TV set and to minimization rather than maximization.Source: B Proof of the Proposition 1 · pp. 16–17
- The scalar optimization can be restricted to η ∈ [0, 2ρ(1−γ)].The proof uses convexity to establish that η ≤ 0 is minimized at zero, then bounds the upper range.Source: B Proof of the Proposition 1 · p. 17
C Proof of Theorem 1
The theorem proof decomposes the robust policy-value gap and bounds its terms using dual-optimization and least-squares generalization results. It combines these bounds under the stated assumptions to complete the performance analysis.
Dual optimization error
- The dual optimization error is bounded with high probability for the empirical minimizer over the dual-function class.The proof uses the approximate dual realizability assumption and an empirical risk minimization bound.Source: C Proof of Theorem 1 · pp. 18–19
Least-squares generalization bound
- The least-squares generalization bound holds with probability at least 1 − δ for the solution returned by the algorithm.The bound is stated for the regression solution b_f^g and robust Bellman target T_g.Source: C Proof of Theorem 1 · p. 19
- The least-squares analysis compares the fitted robust Bellman target with the best function-class approximation.The comparison uses a dataset-independent best approximation and Bernstein's inequality with a union bound over F.Source: C Proof of Theorem 1 · pp. 19–20
Proof of Theorem 1
- The proof decomposes the gap between the robust optimal policy and RFQI's policy, then combines the resulting bounds to complete Theorem 1.The combination uses the dual optimization error and least-squares generalization bounds.Source: C Proof of Theorem 1 · pp. 21, 23
- The performance analysis invokes the concentratability assumption to bound errors under the relevant state-action distributions.The proof applies the bound to distributions induced by the learned policy and the optimal robust policy.Source: C Proof of Theorem 1 · p. 22
D Related Works
The related-work discussion distinguishes offline robust RL with function approximation from prior single-model offline RL and tabular robust RL. The accompanying experiment-details passage describes the practical RFQI implementation and its benchmark environments.
D Related Works
- Prior offline RL work assumes data from a single model, whereas this work targets data from a training model that may differ from testing dynamics.The stated objective is to learn a policy robust to an uncertainty set.Source: D Related Works · p. 23
- The related-work discussion also separates RMDP-based robustness from corruption-robust, risk-sensitive, adversarial, and classical robust-control approaches.It describes those approaches as using frameworks different from this work's robust MDP formulation.Source: D Related Works · pp. 23–24
- Unlike closest tabular robust offline RL work, this study uses arbitrary state-action sampling and function approximation for large state spaces.It learns robust values or policies directly rather than estimating a tabular uncertainty set and solving a planning problem.Source: D Related Works · p. 24
E Experiment Details
- The practical experiment details report results on Cartpole, Hopper, and Half-Cheetah OpenAI Gym MuJoCo environments.The implementation is based on BCQ and PQL architectures.Source: E Experiment Details · p. 24
E.1 RFQI Practical Algorithm
The practical RFQI description highlights its robust target and dual-variable optimization as additions to the baseline algorithms. The appendix also specifies shared hyperparameters and the offline datasets used for comparisons.
Practical algorithm
- RFQI adds dual-variable optimization and a robust state-action value target to the practical algorithm.The dual function and Q function are optimized on minibatches using ADAM.Source: §E.1 · p. 24
Implementation details
- The implementation uses the default VAE architecture from BCQ and PQL, with an encoder producing a Gaussian mean and standard deviation and a decoder mapping states and latent samples to actions.The VAE is trained using an evidence lower bound.Source: §E.1 · p. 24
Hyper-parameters
- Across algorithms, the experiments use τ = 0.005, K = 5 × 10^5 iterations, 10^6 offline samples, and minibatches of size 1000.The uncertainty radius ρ is grid-searched over {0.2, 0.3, …, 0.6}.Source: §E.1 · p. 25
Offline datasets
- The experiments compare FQI and RFQI on shared offline datasets, including Cartpole data, mixed MuJoCo data, and D4RL medium datasets.The D4RL datasets described are hopper-medium and halfcheetah-medium.Source: §E.1 · pp. 25–26
E.2 More Experimental Results
Across Cartpole and MuJoCo evaluations, RFQI generally retains reward better than non-robust FQI and other baselines under parameter or action perturbations. However, RFQI and FQI perform similarly for some D4RL perturbations.
Cartpole simulation results
- 75%: RFQI improves average cumulative reward over FQI at a 40% chance of Cartpole action perturbation.RFQI's performance is similar to soft-robust DQN, which uses online data from multiple models.Source: Figure 4 · p. 26, §E.2 · p. 26
Hopper simulation results
- At foot_joint_stiffness 15, RFQI achieves 3200 episodic reward versus FQI's 1400 on Hopper.The comparison is from the mixed dataset Dm evaluation.Source: Figure 5 · p. 26, §E.2 · p. 27
Hopper evaluation results
- At a 30% joint_damping change, Hopper RFQI achieves 3000 episodic reward versus FQI's 1400.RFQI's reward is described as almost the same as for the unperturbed model.Source: §E.2 · pp. 27–28, Figure 6 · p. 28
Half-Cheetah evaluation results
- For more than 50% change in a Half-Cheetah joint_stiffness value, RFQI stays around 5500 episodic reward while FQI drops to 4300.The evaluation uses D4RL dataset Dd.Source: Figure 7 · p. 28, §E.2 · p. 28
Similar-performance cases
- RFQI and FQI perform similarly on D4RL Hopper actuator_ctrlrange and thigh_joint_stiffness perturbations and Half-Cheetah joint_damping and joint_frictionloss perturbations.The authors note robustness can depend on available offline data; unrepresentative data may make some parameter perturbations difficult.Source: §E.2 · p. 28, Figure 8 · p. 28, Figure 9 · p. 28