PaperlayerView exact revision on arXiv

Source-linked AI summary

RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms

Xi Chen, Fengqi Hu, Zeyu Shen, Jiaying Xu, Shangding Gu, Pan Xu, Anqi Liu, Laixi Shi

arXiv:2610.32971v1cs.LGcs.AIcs.RO

TL;DR

Robust RL faces barriers to reuse and to understanding where algorithms remain effective under deployment shifts. RobustRLlib unifies representative algorithms and broad shift evaluation; its evaluation maps strengths and persistent gaps across 273 perturbation configurations.

  • Problem

    Robust RL algorithms are fragmented across implementations, and distinct evaluation settings leave their comparative capability boundaries unclear.

    Source: §1 · p. 3

  • Method

    RobustRLlib integrates 17 representative online, offline, and safe RL algorithms with a shared evaluation platform spanning diverse and compound shifts and advanced robotic tasks.

    Source: §1 · p. 3

  • Results

    Across 273 perturbation configurations, robust methods outperform standard RL references as a group, while individual and compound shifts reveal algorithmic strengths and capability boundaries.

    Source: §5 · p. 13

  • Takeaways & Limitations

    RobustRLlib provides a reusable platform for comprehensive evaluation and guidance on robust RL under diverse deployment shifts.

    Source: §5 · p. 13

Abstract

from arXiv · show

Robust reinforcement learning (RL) is critical for reliable real-world deployment, where policies may fail under diverse shifts and disruptions. Existing robust RL algorithms offer a rich set of mechanisms and principles, yet two major barriers limit their broader potential: limited algorithm reusability and an unclear empirical capability boundary. We introduce RobustRLlib, an algorithm-centric library and benchmark that addresses both challenges. It integrates 17 representative robust online, offline, and safe RL algorithms under a unified interface for substantially improved reusability, together with an enhanced evaluation platform spanning diverse real-world shifts and advanced humanoid and vision-language-action tasks. Systematic evaluation over 273 perturbation configurations provides an up-to-date capability map of current robust RL algorithms, revealing where they remain effective and where substantial robustness gaps persist. As an open-source, user-friendly platform, RobustRLlib offers a reproducible and extensible codebase for applying existing methods to new problems and developing new robust RL algorithms.

1 Introduction

Robust RL can address deployment disruptions, but fragmented implementations and inconsistent evaluations leave algorithm reuse and empirical capability boundaries unclear. RobustRLlib combines a reusable library with broad evaluation, and its 273-configuration study maps both strengths and persistent gaps.

1 Introduction

  • Existing robust RL methods are fragmented across implementations and training pipelines, leaving reuse and systematic cross-method comparison difficult.Methods are typically evaluated in distinct environments and shift scenarios, so performance beyond their original settings is poorly understood.Source: §1 · p. 3
  • RobustRLlib pairs reusable implementations with a comprehensive evaluation pipeline spanning tasks and diverse real-world shifts.It is designed to make robust RL methods systematically accessible, comparable, and extensible.Source: §1 · p. 3
  • The library integrates 17 representative robust online, offline, and safe RL algorithms, organized by learning method, target shift, and base algorithm.It is intended as a reusable foundation for applying existing methods and developing new robust RL algorithms.Source: §1 · p. 3
  • The disruptor toolbox covers observation, action, dynamics, reward/cost, latency, and semantic shifts, with stochastic, adversarial, parametric, isolated, and compound modes.Its interface extends to massively parallel Isaac Lab training and advanced vision–language–action robotic tasks.Source: §1 · p. 3
  • 273 perturbation configurations show robust methods outperform standard references on average, while both groups deteriorate as shifts intensify.The evaluation also finds that claimed target shifts do not reliably predict empirical strengths, and gains vary across tasks and shift sources.Source: §1 · p. 3

2 Related Work

Related work addresses robust RL across shifts in agent–environment interaction, algorithm design, benchmark types, and sim-to-real transfer. RobustRLlib combines a reusable algorithm library with unified evaluation that extends to latency, semantic shifts, and advanced robotic tasks.

Robust RL problems under diverse deployment shifts

  • Robust RL work covers observation, action, dynamics, reward/cost, latency, and semantic shifts, which may co-occur or evolve during deployment.Robustness to one shift does not necessarily extend to others or their combinations, motivating unified evaluation.Source: §2 · pp. 3–4

Robust RL algorithm taxonomy

  • Robust algorithms modify either the learner or training-data distribution; RobustRLlib integrates 17 online, offline, and safe methods for unified evaluation.The taxonomy also describes safe RL as jointly maintaining task performance and satisfying cost constraints under uncertainty.Source: §2 · p. 4

Related RL benchmarks

  • RL benchmarks include data-, task-, and algorithm-oriented types; RobustRLlib complements robust task benchmarks with a unified algorithm-oriented library and benchmark.It also extends considered shifts to latency and semantics and evaluation to advanced high-dimensional robotic tasks.Source: §2 · p. 4

Sim-to-real gap in robotics

  • Sim-to-real mismatch can degrade policies trained in simulation, and prior approaches include dynamics randomization, actuator and latency modeling, and online adaptation.Many existing approaches are task-specific and address particular mismatch sources; RobustRLlib provides 17 algorithms covering diverse tasks and sim-to-real gaps.Source: §2 · p. 5

3 RobustRLlib: An Algorithm-Centric Robust RL Benchmark

RobustRLlib unifies robust RL methods with a configurable shift toolbox and evaluation platform, then measures their strengths and limits across standard and advanced robotic tasks. The results show robust methods often outperform standard references, but effectiveness depends on the shift, task, and safety measure.

3.1 Robust RL algorithm library construction

  • RobustRLlib organizes methods by how learning changes, their claimed target shifts, and their base algorithms, with offline, online, and safe-RL tracks for reuse and comparison.The library integrates representative learner-centric, data-centric, environment-centric, and robust-safe methods.Source: §3 · p. 5, §3.1 · p. 5
  • Offline methods change learning objectives or reshape training data; online methods modify policy learning or collect rollouts in perturbed environments, while safe methods optimize return subject to costs under dynamics uncertainty.The library distinguishes these routes rather than treating robust RL as a single method class.Source: §3.1 · p. 5, §3.1.1 · p. 6, §3.1.2 · p. 7

3.2 Expandable disruptor toolbox

  • The toolbox covers observation, action, dynamics, reward/cost, latency, and semantic shifts, including appearance changes and goal displacement.Latency represents deployment timing mismatches, while semantic shifts can affect scenes, task structures, dynamics, or feedback.Source: §3.2.1 · pp. 7–8, Figure 2 · p. 8
  • Perturbation specifications make shifts reproducible and composable, while a shared disruptor interface extends through task-specific adapters to Isaac Lab and VLA robotic tasks.Modes include stochastic, adversarial, parametric, and time-varying disturbances; compatible shifts can be composed into multi-shift scenarios.Source: §3.2.2 · pp. 8–9, §3.2.3 · pp. 9–10, §4.3 · p. 12

4.1 Evaluation of the algorithm library

  • Across Q1–Q4, robust methods’ pooled normalized-score advantage over standard RL narrows from +10.3 to +5.1 as both groups decline; dynamics-targeting families lead under controlled dynamics shifts.DR-SAC and FWM are the strongest online and offline families, respectively, on those shifts.Source: Figure 4 · p. 10, §4.1 · pp. 10–11

4.2 Evaluation under diverse deployment shifts

  • Target-shift labels do not uniquely predict measured strengths: DR-SAC leads dynamics and latency, ATLA-SA-PPO leads observation and action, and compound shifts push both strongest methods below half their nominal scores.Online methods generally achieve larger gains over their base algorithms than offline methods; measured rankings also depend on task, shift mode, and perturbation.Source: §4.2 · pp. 11–12, Figure 7 · p. 12

4.3 Extension to advanced robotic tasks

  • On drawer manipulation, DR-PPO matches SPiDR-PPO’s 99.6% performance but has 0% versus 84% of episodes within the joint-limit cost budget; robust-safe methods keep both measures high across tasks.The comparison shows why task performance alone does not establish sim-to-real readiness.Source: §4.3 · p. 12, Figure 8 · p. 13
  • At the largest measured shift, VLA performance retains 0.95 of nominal under gravity but 0.33 under image blur; blur moves 13% of the image, while brightness moves frames seven times farther at little cost.The paper identifies semantic shifts as a core concern for visually grounded policies.Source: §4.3 · p. 12, Figure 8 · p. 13

5 Conclusion

RobustRLlib combines a reusable robust-RL algorithm library with evaluation across deployment shifts, including isolated and compound scenarios and advanced robotics tasks. Across 273 perturbation configurations, robust methods outperform standard references as a group, while evaluations expose algorithm-specific strengths and capability boundaries.

5 Conclusion

  • Across 273 perturbation configurations, robust methods as a group outperform standard RL references, and evaluations reveal algorithm-specific strengths and capability boundaries.The platform is intended to support comprehensive evaluation and guide future robust-RL method design.Source: §5 · p. 13

A.1 Benchmark Comparison

The comparison distinguishes benchmarks by their evaluated scope and extensibility, while positioning RobustRLlib as a shared-protocol combination of a structured robust-RL library and isolated and composed shift evaluations.

A.1 Benchmark Comparison

  • Table 2 distinguishes evaluated benchmark capabilities from potential extensibility, counting robust and safe algorithms and shifts across four MDP components.Its criteria also cover compound shifts and interfaces for adding tasks.Source: Table 2 · p. 19, §A.1 · p. 19
  • Related systems standardize different parts of robust-RL evaluation, including dynamics uncertainty, disrupted environments, practical stressors, and observation attacks.RLlib provides general algorithm infrastructure; RRLS and ODRL focus primarily on dynamics-related uncertainty, Robust-Gymnasium covers disrupted environments, RWRL Suite evaluates practical stressors, and RoAd-RL focuses on observation attacks and inference-time defenses.Source: §A.1 · p. 19
  • RobustRLlib combines structured online, offline, and safe-RL algorithms with isolated and composed shift evaluations under a shared protocol.Task-specific adapters demonstrate extensions from standard control to Isaac Lab and a VLA pipeline.Source: §A.1 · p. 19

A.2 Disruptor toolbox: shift modes, operators and implementations

The disruptor toolbox specifies each perturbation through its source, factor, mode, magnitude, resampling, optional schedule, and seed. It includes distinct latency operators and supports stochastic, adversarial, parametric, non-stationary, and composed interventions.

A.2 Disruptor toolbox: shift modes, operators and implementations

  • Each perturbation configuration records its source, factor, mode, magnitude, resampling rule, optional schedule, and seed.These fields specify where and how an intervention enters, its size and persistence, any time evolution, and reproducible stochastic draws.Source: §A.2 · pp. 19–20

A.2.1 Shift source: Latency

  • Latency shifts act through observation delay, action delay, or control-period variation, with operators implemented at different points in the decision loop.Observation histories are interpolated at the delayed time; action buffers execute earlier commands; variable control periods change how long commands are held.Source: §A.2.1 · p. 21

A.2.2 Operator details

  • Operator details distinguish per-step or episode-level stochastic draws, actor-dependent adversarial perturbations, specified parametric changes, scheduled events, and explicitly ordered composition.Each composition specification uses its own seeded random stream, making compound conditions reproducible.Source: §A.2.2 · p. 21

B.1 Metrics

The metrics normalize standard-control task scores using fixed task-level anchors, while advanced robotic tasks report performance and safety separately. The accompanying tables document perturbation configurations and normalization references.

B.1 Metrics

  • The standard-control metric is an unclipped normalized score based on mean raw evaluation return and fixed task-level reference scores shared across methods, seeds, and shift conditions.Normalization precedes cross-task aggregation; published D4RL references are used where available, otherwise task-defined or reference-policy anchors are used.Source: §B.1 · p. 21
  • Advanced robotic tasks report performance and safety separately, using task-specific success or normalized return measures and safety measures tied to survival or joint-limit costs.For drawer manipulation, safety is the fraction of episodes whose cumulative joint-limit cost stays below the per-episode budget.Source: §B.1 · pp. 21, 24
  • The perturbation tables enumerate individual and compound shift configurations, while Table 7 lists fixed normalization-anchor pairs for each environment.The compound-shift table applies every factor in a profile jointly; the individual-shift table groups configurations by shift levels.Source: Table 4 · p. 22, Table 5 · p. 23, Table 6 · p. 23

B.2 Evaluation grids

The benchmark uses three complementary grids to measure robustness breadth, shift-specific behavior, and robustness under interacting deployment factors. Advanced-task grids place evaluation conditions relative to a reference envelope or measured hardware settings.

B.2 Evaluation grids

  • The main grid varies one shift factor at a time across online and offline tasks, with 220 non-nominal conditions plus each factor’s nominal setting.The individual grid varies intervention modes within a shift source; the compound grid activates physical, control, sensing, and timing factors jointly in Hopper and Pusher.Source: §B.2 · p. 24
  • Isaac Lab evaluates G1 at 31 conditions, while Drawer uses 16 conditions; both position shifts relative to reference ranges.G1 includes six conditions inside, eighteen on the boundary, and five beyond the reference envelope; Drawer has two controls and fourteen conditions beyond reference ranges.Source: §B.2 · p. 24, Table 9 · p. 25
  • The VLA grid tests eight shift factors across five sources, with each factor’s Q1–Q4 levels ordered outward from nominal.Hardware logs place execution delay (83 ms) and command noise (0.020 rad) at Q2, so the sweep includes milder and more severe settings.Source: §B.2 · p. 24, Figure 9 · p. 25

C Benchmark tasks and datasets

The benchmark spans standard task interfaces and offline datasets, then organizes baseline and robust algorithms for comparison. Its algorithm descriptions distinguish standard references from learner-centric methods that modify learning targets or updates.

C Benchmark tasks and datasets

  • The benchmark covers locomotion, manipulation, navigation, and vehicle control, with additional semantic-channel and compound-profile tests in Pusher and DoorCausal.Table 10 gives task interfaces, while offline methods use available Minari datasets and D4RL ports; LunarLander uses a collected dataset.Source: C Benchmark tasks and datasets · pp. 25–26, Table 10 · p. 26, Table 11 · p. 26
  • Advanced robotic tasks extend evaluation to G1 locomotion, Franka drawer manipulation, and VLA bimanual battery insertion under perception and execution shifts.The G1 task measures return and survival; Drawer measures success and joint-limit cost, and the VLA task fine-tunes π0.5 on expert demonstrations.Source: C Benchmark tasks and datasets · p. 26, Figure 10 · p. 27, Figure 11 · p. 27

D Algorithms

  • The library organizes six standard baselines, thirteen robust RL algorithms, and two robust safe algorithms by the paper’s taxonomy.Standard offline references include IQL, TD3+BC, MOPO, and SynthER; SynthER augments logged transitions with diffusion-generated samples.Source: D Algorithms · p. 27, §D.1.1 · p. 27

D.1.2 Learner-centric algorithms.

  • Learner-centric offline methods alter value targets or policy learning, including RFQI’s worst-case Bellman backup, RORL’s local smoothing, and ATLA-IQL’s learned observation adversary.RSC-IQL instead rewrites logged transitions using a causal mask fitted from the dataset.Source: §D.1.2 · p. 28

D.1.3 Data-centric algorithms.

The paper describes robust methods through their data, model, and environment mechanisms, alongside standard online references and safe-RL approaches. Evaluation protocols freeze policies on common shift conditions, with separate training and evaluation arrangements for advanced tasks.

D.1.3 Data-centric algorithms.

  • RSC-IQL rewrites logged transitions by permuting or perturbing dimensions marked non-causal by a dataset-fitted causal mask.RAMBO instead augments data with learned-model rollouts and adversarially trains the model to lower policy value while retaining a maximum-likelihood fit to data.Source: §D.1.3 · p. 28

D.1.4 Generative model.

  • Generative methods use learned futures for critic training: FWM regresses an h-step return, while PLR-PVL selects per-sample futures from the model with the largest positive value loss.FWM uses flow matching and integrates an ODE from t = 0 to t = 1 in five midpoint steps for sampling.Source: §D.1.4 · p. 28

D.2.1 Standard RL baselines.

  • SAC and PPO are the off-policy and on-policy references, respectively; TRPO appears only as the base of RARL-TRPO.SAC uses twin critics and a stochastic entropy-regularized actor, while PPO uses a clipped probability-ratio surrogate and generalized advantage estimation.Source: §D.2.1 · p. 29

D.2.2 Learner-centric algorithms

  • Online learner-centric methods include adversarial observations, policy smoothing, and counterfactual transitions generated by intervening on non-causal variables.ATLA alternates a policy with an observation adversary; ATLA-SA adds a KL regularizer and recurrent policy history.Source: §D.2.2 · p. 29

D.2.3 Environment-centric algorithms.

  • Environment-centric methods train against perturbed environments: RARL uses adversarial external forces, while domain randomization samples simulator parameters from fixed ranges each episode.The library implements RARL-PPO, RARL-TRPO, DR-PPO, and DR-SAC.Source: §D.2.3 · p. 29

D.2.4 Robust safe methods.

  • Robust safe methods target worst-case return while satisfying cost constraints; RAMU applies risk measures to reward and cost targets, and SPiDR penalizes uncertain safety outcomes.SPiDR uses disagreement across next-state predictions from domain-randomized models.Source: §D.2.4 · p. 29

E.1 Training protocol

  • The default protocol trains on nominal tasks and evaluates frozen policies under shifts, except reward and latency shifts, which affect training feedback and are evaluated nominally.Isaac Lab uses three seeds and 128 episodes per condition, with training shifts for DR-PPO, ATLA-PPO, and SPiDR-PPO; the VLA task fine-tunes one π0.5 model on demonstrations.Source: §E.1 · p. 30

E.2 Training budget alignment

Training budgets are aligned to base algorithms while allowing specified official configurations to remain intact. The accompanying appendix supplies hyperparameter tables for online, offline, and robust safe methods.

E.2 Training budget alignment

  • PPO-based methods train for 2M environment steps and SAC-based methods for 1M; the additional PPO steps provide meaningful learning progress.On Walker2d, PPO at 2M steps takes 0.80 ± 0.29 hours versus SAC at 1M steps’ 1.60 ± 0.03 hours on H100 GPUs.Source: §E.2 · p. 30, Figure 12 · p. 30, Table 12 · p. 31
  • ATLA and ATLA-SA retain their official approximately 5M-step configuration, while methods sharing a base algorithm use the same budget.The PPO-versus-ATLA pairing matches the budgets under which the original ATLA results were established.Source: §E.2 · p. 30, Table 13 · p. 31, Table 14 · p. 31
  • Nine of twelve offline algorithm columns share a 1M-update budget, while RAMBO, RORL, and RFQI follow specified official-budget or checkpoint rules.RFQI uses its official budget and best checkpoint because forcing it onto the common protocol would break its mechanism.Source: §E.2 · p. 31, Table 14 · p. 31

F.1 Online algorithm hyperparameters

  • The appendix provides online hyperparameter tables for PPO, SAC, adversarial methods, domain randomization, and RSC-SAC.The tables specify algorithm hyperparameters; combined task columns are used only when listed values agree across those tasks.Source: §F.1 · p. 32, Table 16 · p. 32, Table 17 · p. 32

F.2 Offline algorithm hyperparameters

  • The tables report hyperparameters behind offline results for IQL, TD3+BC, MOPO, SynthER, and RAMBO; MOPO’s penalty type identifies the ensemble statistic used to discount reward.Source: §F.2 · p. 35, Table 24 · p. 35, Table 25 · p. 35

F.3 Robust safe algorithm hyperparameters

  • Robust safe algorithm hyperparameter tables cover PPO, DR-PPO, ATLA-PPO, RAMU-PPO, and SPiDR-PPO.The appendix also includes a PPO parameter table for the robust safe methods’ reference configuration.Source: Table 36 · p. 38, Table 37 · p. 38, Table 38 · p. 39

G.1 Analysis of robust methods’ gains

Feature-family comparisons separate robustness gains from base-learner strength, while task results vary by algorithm class and advanced Isaac Lab results foreground joint safety and performance.

G.1 Analysis of robust methods’ gains

  • Online families retain +5.5 and +6.6 base-relative gains, while offline gains are smaller; generative methods gain +9.8 over the standard pool but +2.5 over IQL.The generative family comprises three methods using IQL; its aggregate advantage reflects both the added mechanism and the strength of its base learner. The paired column includes only methods with a matched base learner, while the pooled column compares each family mean with six Standard RL references.Source: §G.1 · p. 39, Table 41 · p. 40

G.2 Analysis of task capability: On-policy vs Off-policy

  • Off-policy methods generally score higher on MuJoCo locomotion, whereas on-policy methods are stronger on PointMaze-UMaze and CarRace.The passage suggests replay-based reuse may contribute to the locomotion advantage, while recent on-policy rollouts may be more useful for goal discovery and visually coupled control.Source: §G.2 · p. 40

G.3 Results of advanced robotic tasks

  • Across Isaac Lab tasks, SPiDR-PPO most consistently balances performance and safety, satisfying Drawer’s cost budget and leading in return and survival under G1 action delays.The results motivate assessing constraint satisfaction alongside task performance rather than relying on return alone.Source: §G.3 · p. 40

G.4 Supplementary results

The supplementary figures cover online and offline comparisons, shift-isolated and compound-shift evaluations, and aggregated task and Isaac Lab results.

Supplementary results

  • Figures 15–16 present comparisons across 220 configurations and four task families, with separate supplementary online and offline results.The captions identify Figures 15 and 16 as supplementary online and offline results for Section 4.1, respectively.Source: §G.4 · p. 40, Figure 15 · p. 42, Figure 16 · p. 44
  • Figures 17–18 show shift-isolated evaluation across 44 distinct perturbation configurations.Figure 17 covers supplementary online results and Figure 18 supplementary offline results for Section 4.2.Source: §G.4 · p. 40, Figure 17 · p. 46, Figure 18 · p. 47
  • Figure 19 reports results for nine perturbation configurations that include a compound-shift scenario.The caption identifies Figure 19 as supplementary results for Section 4.2.Source: §G.4 · p. 40, Figure 19 · p. 48
  • Figure 13 aggregates on- and off-policy performance by task, while Figure 14 plots Isaac Lab safety success against performance by factor, with one point per algorithm and three-seed standard-deviation bars.Each Figure 14 panel repeats the comparison for one factor.Source: Figure 14 · p. 41, Figure 13 · p. 41

Related papers

Loading 2610.32971v1…