Source-linked AI summary
D4RL: Datasets for Deep Data-Driven Reinforcement Learning
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine
TL;DR
Offline RL needs benchmarks that measure learning from fixed datasets representative of real-world data, rather than mainly online-RL training data. D4RL introduces such tasks and datasets, and finds that algorithms struggle on several realistic data properties that prior benchmarks underrepresent. The benchmark provides an accessible basis for identifying these shortcomings and comparing future methods.
Problem
Offline RL lacks accessible, standardized benchmarks that reflect heterogeneous real-world datasets and reliably differentiate algorithms.
Method
D4RL constructs simulated tasks and datasets around practical properties such as human demonstrations, passively logged data, multitask behavior, and mixed policies.
Results
Most algorithms succeed most on RL-trained-policy datasets but struggle on undirected data, mixture distributions, and other realistic benchmark settings.
Takeaways & Limitations
D4RL offers an open-source, accessible benchmark for exposing offline-RL shortcomings and guiding progress toward realistic applications.
Takeaways & Limitations
The benchmark is predominantly simulated and focused on robotics, autonomous driving, and traffic control, leaving stochasticity, large action spaces, and other domains less explored.
Abstract
from arXiv · showhide
The offline reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning. However, existing online RL benchmarks are not tailored towards the offline setting and existing offline RL benchmarks are restricted to data generated by partially-trained agents, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms. To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms, an evaluation protocol, and open-source examples. This serves as a common starting point for the community to identify shortcomings in existing offline RL methods and a collaborative route for progress in this emerging area.
1 INTRODUCTION
Offline RL learns policies from fixed, previously collected datasets, offering a bridge between reinforcement learning and supervised learning. D4RL addresses the lack of realistic, accessible benchmarks for measuring progress in this setting.
- Offline RL learns from previously collected datasets without continuing active data collection.This setting connects reward-driven sequential decision making with the benefits of large datasets.
- A useful benchmark should reflect real-world data challenges, be broadly accessible with reproducible evaluation, and span multiple difficulty levels.
- Existing offline RL benchmarks often use online-RL training data that does not reflect heterogeneous real-world datasets.Such benchmarks were also insufficient to distinguish simple baselines from newer algorithms.
- D4RL introduces tasks and datasets designed around practical offline-RL properties, benchmarks existing algorithms, and provides reference implementations.The suite includes settings such as partial observability, passively logged data, and human demonstrations.
2 RELATED WORK
Prior offline-RL studies largely rely on datasets generated by previously trained behavior policies. D4RL broadens evaluation toward collection procedures inspired by real-world applications rather than relying only on online-RL training runs.
- Prior offline-RL datasets commonly range from random initial policies to near-expert online-trained policies.This pattern appears across continuous control, navigation, industrial control, and Atari domains.
- Existing benchmarks remain limited because they rely on data collected during online-RL training runs.
- Real-world offline-RL applications demonstrate promise but often require private systems or expensive human labeling for evaluation.Those requirements are not scalable or broadly accessible for a benchmark.
3 BACKGROUND
Offline RL removes access to the environment after dataset collection, requiring policies to learn from fixed transition data. The resulting challenge includes distribution shift and behavior that may not fit the policy class.
- The MDP formulation specifies state and action spaces, transitions, rewards, initial states, and a discount factor for policy optimization.
- In standard episodic RL, algorithms can choose policies and iteratively collect trajectory samples.
- Offline RL instead presents a fixed transition dataset generated by an unknown behavior policy.
- Effective offline-RL algorithms must handle distribution shift and data-generation processes that may not be representable by the chosen policy class.
4 TASK DESIGN FACTORS
D4RL designs tasks and datasets around properties that challenge offline-RL algorithms in realistic applications. These include narrow, undirected, sparse, suboptimal, non-representable, and partially observed data, across diverse simulated domains.
- Narrow and biased data distributions: Narrow or biased datasets can cause offline-RL divergence, motivating methods that remain close to the behavior-data distribution.
- Undirected and multitask data: Stitching combines useful subtrajectories from undirected or multitask data to solve a task without relying on generalization outside the dataset.For example, trajectories from A-B and B-C can be combined to form an A-C trajectory.
- Sparse rewards: Sparse-reward datasets isolate credit assignment from exploration because offline RL uses fixed data without exploration.
- Suboptimal data: Suboptimal data challenges imitation-learning approaches that generally require expert demonstrations.
- Non-representable behavior policies, non-Markovian behavior policies, and partial observability: Human demonstrations and hand-crafted controllers may produce non-representable or non-Markovian behavior and partial observability.
- Realistic domains: Simulated environments balance accurate policy evaluation and broad reproducibility, but restrict task selection and omit some mature real-world domains.
- Realistic domains: D4RL covers locomotion, traffic management, autonomous driving, and robotics tasks spanning accessible to currently difficult problems.
5 TASKS AND DATASETS
D4RL assembles offline datasets and simulators spanning realistic data-collection challenges, including undirected, multitask, human-demonstration, controller-generated, and mixed-policy data.
- D4RL provides offline trajectory datasets, typically containing 10^6 steps, alongside simulators for policy evaluation.
- Navigation: Maze2D and AntMaze test stitching subtrajectories across undirected navigation data, with AntMaze adding an 8-DoF robot and sparse 0-1 rewards.
- Gym-MuJoCo: Gym-MuJoCo standardizes random, medium, medium-replay, and medium-expert datasets to examine suboptimal agents and mixtures of policies.
- Robotic manipulation: Adroit uses a 24-DoF Shadow Hand and narrow expert or human-demonstration datasets for sparse-reward manipulation tasks.
- Multitask demonstrations: FrankaKitchen uses complete, partial, and mixed human demonstrations to study multitask behavior and stitching in a 9-DoF kitchen robot.
- Driving and traffic: Flow and Offline CARLA use hand-designed or human-like controllers for traffic and visual driving tasks, including random and undirected navigation data.
6 BENCHMARKING PRIOR METHODS
Benchmark performance is strongest on familiar RL-generated datasets and some sparse-reward or controller-generated tasks, but undirected, mixed, and limited-data settings expose major weaknesses.
- Algorithms performed best on datasets generated by RL-trained policies, matching expert behavior and outperforming suboptimal behavior in Adroit and Gym-MuJoCo.
- Many methods outperformed online SAC on sparse-reward Adroit and AntMaze tasks, while conservative methods handled biased and narrow data in Flow and Gym-MuJoCo.
- Undirected Maze2D, FrankaKitchen, CARLA, and AntMaze tasks challenged existing methods, although smaller Maze2D and AntMaze instances remained tractable.
- Mixture distributions challenged every evaluated algorithm, with medium-expert MuJoCo data generally performing on par with medium data despite including expert demonstrations.
- Controller-generated Flow and carla-lane tasks saw partial success, whereas limited human demonstrations in Adroit and FrankaKitchen remained difficult.
7 DISCUSSION
D4RL broadens offline RL evaluation toward realistic dataset properties while showing that these challenges substantially test existing methods. The benchmark remains centered on simulated robotics and driving domains, with reliable real-world evaluation still unresolved.
- D4RL is an open-source benchmark motivated by narrow distributions and undirected logged behavior found in real-world applications.
- Experiments show that realistic dataset properties beyond RL-trained policy data substantially challenge existing offline RL methods.
- Simulation makes benchmark evaluation accessible, while the authors aim ultimately to apply offline RL to real-world datasets and domains.
- The benchmark does not explore several important real-world properties in depth, including environmental stochasticity and large action spaces.
- Its tasks predominantly cover robotics, autonomous driving, and traffic control, leaving finance, industrial, and operations-research benchmarks as further areas.
- Reliable real-world evaluation will require either system deployment or dependable off-policy evaluation methods, alongside standardized benchmarks.
A TASK PROPERTIES
The benchmark reports task-level statistics, including controller type and dataset sample counts, to characterize the data used by each task.
- Controller types identify whether data came from hand-designed planners, human demonstrations, or random or neural network policies.
- The number of samples records the environment transitions contained in each task's dataset.
B RESULTS BY DOMAIN
Results are summarized by domain using normalized and raw scores across offline RL algorithms and baselines. Normalized results are averaged over three seeds and scaled from random performance to 100.
- Normalized results are sorted from the best-performing algorithm to the worst for each domain, excluding CARLA because all algorithms performed poorly.
- Performance tables compare online and offline SAC, BEAR, BRAC variants, BC, AWR, BCQ, continuous-action REM, cREM, and AlgaeDICE.
- Normalized scores are averaged over 3 seeds and scaled between 0 for random performance and 100.
- Raw tables report the undiscounted return obtained from executing each policy in the simulator, averaged over 3 random seeds.
C TASK AND DATASETS
The benchmark combines newly proposed and borrowed domains and datasets, then separates tasks for hyperparameter tuning from held-out evaluation under a reproducible protocol.
- Table 4 includes new Maze2D and AntMaze domains alongside borrowed MuJoCo, Adroit, and FrankaKitchen datasets; remaining datasets are proposed here.
- The recommended protocol tunes hyperparameters on designated training tasks and evaluates algorithms without tuning on separate evaluation tasks.
- Experiments use default hyperparameters, minimal implementation changes, and 500K training iterations or gradient steps; continuous-action REM is an unofficial adaptation.
- AntMaze data follows the same high-level planner as Maze2D, supporting similar expected coverage despite more complex ant dynamics.
- Maze2D trajectories visualize visited states across umaze, medium, and large layouts, while AntMaze includes umaze, medium-play, large-play, and diverse variants.