Source-linked AI summary
RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arnold, Jerry Li, Mohammad Norouzi, Matt Hoffman, Ofir Nachum, George Tucker, Nicolas Heess, Nando de Freitas
TL;DR
Offline RL needs standardized evidence because real-world deployment limits online interaction and existing comparisons use inconsistent protocols, datasets, and baselines. RL Unplugged addresses this with diverse benchmark domains, unified data access, and online/offline policy-selection protocols. Results are promising on some control and Atari tasks but lower on partially observable locomotion environments, while fixed-dataset policies remain bounded by their training data and reward function.
Problem
Offline RL seeks to learn policies from logged data without costly or unsafe environment interaction, but existing evidence lacks standardized protocols, datasets, and baselines.
Method
RL Unplugged provides diverse task datasets, a unified API, detailed domain-specific evaluation protocols, reference baselines, and open-source implementations.
Results
Performance is promising on some control-suite tasks and Atari games but lower on partially observable locomotion tasks; Atari results also show offline-selection sensitivity across methods.
Takeaways & Limitations
The suite is intended to make offline RL research more reproducible, systematic, accessible, and comparable across varied task conditions.
Takeaways & Limitations
Policies learned from fixed datasets may produce behavior reflecting the training data or reward function, so expected generalization must be considered carefully.
Abstract
from arXiv · showhide
Offline methods for reinforcement learning have a potential to help bridge the gap between reinforcement learning research and real-world applications. They make it possible to learn policies from offline datasets, thus overcoming concerns associated with online data collection in the real-world, including cost, safety, or ethical concerns. In this paper, we propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games (e.g., Atari benchmark) and simulated motor control problems (e.g., DM Control Suite). The datasets include domains that are partially or fully observable, use continuous or discrete actions, and have stochastic vs. deterministic dynamics. We propose detailed evaluation protocols for each domain in RL Unplugged and provide an extensive analysis of supervised learning and offline RL methods using these protocols. We will release data for all our tasks and open-source all algorithms presented in this paper. We hope that our suite of benchmarks will increase the reproducibility of experiments and make it possible to study challenging tasks with a limited computational budget, thus making RL research both more systematic and more accessible across the community. Moving forward, we view RL Unplugged as a living benchmark suite that will evolve and grow with datasets contributed by the research community and ourselves. Our project page is available on https://git.io/JJUhd.
1 Introduction
Offline RL learns policies from logged data without further environment interaction, addressing the cost, safety, and ethical constraints that limit real-world online RL. RL Unplugged standardizes diverse datasets and evaluation protocols to make offline RL comparisons more reproducible and practically informative.
- Offline RL learns new policies from logged data without further environment interactions, supporting pretraining, fixed-dataset evaluation, and connections to real-world applications.
- Existing offline RL comparisons are difficult because evaluation protocols, datasets, and baselines vary, while practical properties such as partial observability and stochasticity are under-represented.
- Reproducibility is also constrained by implementation and scaling difficulties reported for several offline RL algorithms, without implying that those algorithms fail.
- RL Unplugged introduces diverse task domains, associated datasets, and clear protocols spanning established benchmarks and challenging robotics and real-world RL tasks.
- The benchmark provides a unified dataset API, varied environments, evaluation protocols, and reference baselines, alongside open-source data and algorithms.
2 RL Unplugged
RL Unplugged combines diverse datasets with a unified API and evaluates offline RL under both online and offline policy selection. Its design spans environment, data, and evaluation properties that expose performance under realistic constraints.
- 2.1 Properties of RL Unplugged: A unified API makes the datasets easier to use with established pipelines and popular machine-learning frameworks, while the suite supplies compatible baselines.
- 2.1 Properties of RL Unplugged: The suite covers discrete and continuous actions, low-dimensional states and images, partial observability, varied exploration difficulty, and real-world challenges such as delays and stochasticity.
- 2.1 Properties of RL Unplugged: Datasets vary in behavior policy, state-action coverage, and size, including mixtures of policies on Atari and fixed sub-optimal policies for the RWRL suite.
- 2.2 Evaluation Protocols: Online policy selection isolates offline RL methods but can be optimistic because it chooses hyperparameters through environment interaction, whereas offline selection tests robustness without such validation.
- 2.2 Evaluation Protocols: The benchmark supports both online and offline policy selection, assigning task protocols explicitly and using a naive offline-selection approach as a baseline for future improvements.
3 Tasks
RL Unplugged spans diverse offline-RL task domains, standardized through shared datasets and evaluation protocols that cover varied difficulty, observability, action spaces, and real-world challenges.
- Task domains: The benchmark includes DM Control Suite, DM Locomotion, Atari 2600, and real-world RL Suite tasks, with datasets and protocols defined for each domain.The suite includes both established benchmarks and challenging domains involving high-dimensional control, perception, partial observability, and system constraints.
- 3.1 DM Control Suite: The DM Control Suite subset ranges from simple Cartpole swingup to difficult Humanoid run and Manipulator insertion tasks, with observation dimensions from 5 to 67.Manipulator insertion tasks had not been shown solvable in prior published work according to the paper.
- 3.2 DM Locomotion: DM Locomotion combines challenging high-degree-of-freedom continuous control with perception from rich egocentric observations.The tasks involve CMU Humanoid and virtual-rodent locomotion settings.
- 3.3 Atari 2600: The Atari release contains 46 games, reserving 9 for online policy selection and 37 for offline policy selection.The dataset records gameplay from online DQN training with five runs of 50 million transitions per game, using four-frame state stacks.
- 3.4 Real-world Reinforcement Learning Suite: Real-world RL Suite data uses no-challenge or easy combined-challenge settings, with datasets generated from fixed sub-optimal policies and all tasks assigned to online policy selection.The suite targets challenges including high-dimensional spaces, delays, constraints, multiple objectives, non-stationarity, and partial observability.
4 Baselines
The baseline study compares supervised, online, and offline RL methods under online and offline policy selection across continuous-control, locomotion, and Atari domains. Results vary with task difficulty and domain: BC and RABM are strong on difficult continuous-control tasks, while Atari favors robust RL methods over BC.
- Evaluation protocol: The evaluation compares online policy selection with offline policy selection, using domain-level hyperparameter choices for tasks where online interaction is unavailable.The baselines include BC, DQN, D4PG, IQN, and other published offline methods; Atari aggregates performance using the median across games [Agarwal et al., 2020, Hessel et al., 2018].
- 4.1 DM Control Suite: On difficult DM Control tasks, BC and RABM are among the best methods, although no algorithm reaches online-method performance.D4PG, BRAC, and RABM are good choices on simpler tasks where offline RL approaches online performance; difficult tasks include high-dimensional actions and hard exploration.
- 4.2 DM Locomotion: On DM Locomotion, BC and RABM perform best, whereas D4PG performs poorly across tasks designed around high-dimensional actions or hard exploration.BCQ and BRAC were excluded because the authors could not determine whether poor performance reflected the algorithms or implementation issues.
- 4.3 Atari 2600: On Atari, BCQ, REM, and IQN perform at least as well as the best policy in the training set, while BC performs poorly.DQN is not robust to hyperparameter choice, and off-the-shelf off-policy RL algorithms can surpass BCQ on offline policy selection tasks.
5 Related Work
Prior offline RL work spans novel algorithms and applications, but inconsistent environments and datasets make clear comparisons difficult. This paper instead establishes challenging simulated environments as standardized offline RL benchmarks.
- Prior work: Offline RL research has developed many algorithms, often evaluated on simple MDPs or fully observed environments, while real-world applications remain difficult to evaluate thoroughly because interaction is costly.The paper cites robots and dialog as difficult application domains with expensive environment interaction.
- Benchmark focus: RL Unplugged focuses on challenging simulated environments to provide consistent environments and datasets for comparing offline RL approaches.This focus addresses the comparison problem identified in prior work.
- Related benchmarks: Related benchmark work includes [Agarwal et al., 2020], but the supplied passage does not specify its scope or results.
6 Conclusion
The paper releases RL Unplugged as a diverse offline RL benchmark suite with a unified API and evaluation protocol. Results are promising on some control and Atari tasks but lower on partially observable locomotion environments, motivating continued expansion.
- Contribution: RL Unplugged releases diverse environments and datasets through an easy-to-use unified API, alongside a clear protocol for offline policy selection.
- Findings: Offline RL performance is promising on some control-suite tasks and Atari games but lower on partially observable locomotion environments.
- Future work: The authors plan to extend the suite with community-contributed environments and datasets to help close the gap between real-world applications and RL research.
Broader Impact
Offline RL learns policies from fixed logged datasets rather than requiring further environment interaction. This can improve safety, computational efficiency, and accessibility, but policies remain constrained by the training data and reward function.
- Offline learning: Offline methods learn from fixed datasets of previously logged environment interactions instead of requiring an agent to explore online.
- Benefits: Offline approaches are more suitable when free exploration is unsafe, reduce computational demands when exploration dominates cost, and can improve reproducibility for researchers with limited compute.
- Limitations: Offline RL may produce behavior reflecting its training data or reward function, so expected policy generalization must be considered within the fixed dataset’s scope.
Supplementary material
The supplementary material specifies architectures, hyperparameters, baseline algorithms, and evaluation settings for continuous-control and Atari experiments. It covers feature, pixel, sequential, recurrent, distributional, and behavior-constrained configurations.
- Continuous control: DM Control feature observations are encoded with an eight-layer residual MLP of width 1024 and instance normalization after every two layers.
- Continuous control: Locomotion pixel inputs are processed by a ResNet and combined with feature observations, while sequential rodent tasks add two LSTMs with hidden size 1024.
- Continuous control: Continuous-control experiments use shared hyperparameters, with separate settings for common agents and ResNet components used in locomotion experiments.
- Continuous baselines: Behavior Cloning maps observations to actions without reward, while D4PG, BCQ, BRAC, and RABM provide distributional, behavior-constrained, or actor-critic baselines.
- Continuous baselines: RABM extends ABM with distributional critics and recurrence for partially observable environments, and uses a Gaussian policy for its MPO-like update.
- Atari: Atari experiments use the Mnih et al. [2013] architecture and compare BC, DQN, IQN, and BCQ under shared and online-versus-offline hyperparameter settings.
B DM Control Suite results
This section reports detailed results for the DM Control and DM Locomotion suites, with results organized in separate tables.
- DM Control Suite results: DM Control Suite results are presented in Table 7.
- DM Locomotion results: DM Locomotion Suite results are presented in Table 8.
D Atari 2600 Results
The Atari 2600 results compare normalized and unnormalized evaluations across difficulty categories and policy-selection settings, with individual-game results shown in Figure 7.
- Atari 2600 results: Table 9 reports unnormalized Atari scores separately by difficulty level and policy-selection setting.
- Atari 2600 results: Figure 7 reports normalized performance for each Atari game and baseline.
- Atari 2600 results: The Atari suite excludes several games because they are outside Atari 57, pose very hard exploration challenges, or produced poor data-generating DQN performance.
H Atari Results Normalization and Environment Details
Atari performance is normalized relative to random and best online-DQN scores, while environment details, difficulty categories, and IQN sensitivity are reported separately.
- Atari Results Normalization and Environment Details: Atari normalized performance is defined relative to the random score and the best online-DQN score used to generate the dataset.
- Atari Results Normalization and Environment Details: Atari environment details, including sticky-action settings, are provided in Table 10.
- Atari Results Normalization and Environment Details: Figure 8 categorizes Atari games by comparing offline DQN performance with average and best dataset policies.
- Atari Results Normalization and Environment Details: 16 IQN τ samples achieve the best performance on online policy-selection games, whereas 8 samples perform better for online IQN.
J Atari BCQ Ablation Study
The BCQ ablation varies its discrete-action threshold hyperparameter and identifies 0.5 as the best setting under online policy selection.
- Atari BCQ Ablation Study: A threshold of 0.5 works best for discrete BCQ with online policy selection.
- Atari BCQ Ablation Study: The threshold controls when BCQ trusts the action selected by its generative model.
- Atari BCQ Ablation Study: Setting the threshold to 0 recovers offline DQN, which performs worse than BCQ.