D4RL: Datasets for Deep Data-Driven Reinforcement Learning

• Offline RL learns policies from previously collected datasets, potentially connecting RL with supervised learning and enabling reward-driven decisions over extended horizons.The introduction identifies robotics, autonomous driving, and healthcare as possible application domains.Source: §1 · pp. 1–2
• A useful offline RL benchmark should reflect real-world challenges, be accessible with clear reproducible protocols, and span difficulty levels that distinguish algorithms.The introduction presents these as criteria for addressing the shortage of standard evaluation benchmarks.Source: §1 · p. 2
• Existing benchmarks reuse online RL domains and training-run data, which may not reflect heterogeneous real-world data, differentiate algorithms, or support consistent comparisons.Wu et al. (2019) found existing benchmark datasets insufficient to distinguish simple baselines from newer algorithms; prior work also lacked a standard evaluation protocol.Source: §1 · p. 2
• D4RL uses high-quality simulations because real-world policy evaluation requires new data collection or off-policy evaluation, which is difficult to standardize or unreliable.The paper argues that simulation allows accurate evaluation of candidate policies while supporting a widely accessible, reproducible benchmark.Source: §1 · p. 2
• Most evaluated offline RL algorithms struggle with passively logged data, narrow data distributions, and limited human demonstrations.The authors say these challenges reveal deficiencies obscured by simpler benchmarks and datasets collected from partially trained RL agents.Source: §1 · p. 2
• D4RL introduces benchmark tasks and datasets designed to exercise practical offline RL challenges, including partial observability, passively logged data, and human demonstrations.The suite also includes reference implementations and an easy-to-use API for tasks, datasets, and benchmark algorithms.Source: §1 · p. 2

• Offline RL has used previously trained behavior-policy datasets across robotics, navigation, industrial control, and Atari, with benchmarks also proposed for discrete Atari (Fujimoto et al., 2019).Source: §2 · p. 2
• Existing benchmarks, including Gulcehre et al. (2020), rely on online-RL data; D4RL also targets human demonstrations, exploratory agents, and hand-coded controllers because performance depends on collection procedure (Wu et al., 2019).Gulcehre et al. (2020) use locomotion and manipulation tasks with perceptually challenging input and partial observability.Source: §2 · p. 3
• Offline RL has been applied to robotics, recommender systems, and dialogue, but real-world evaluation can require private systems or costly human labeling, limiting benchmark accessibility and scalability.Large-scale off-policy RL efforts have also used many robots for online data collection; D4RL instead focuses on a varied, accessible platform for algorithm development.Source: §2 · p. 3

3 BACKGROUND

• The RL problem is formalized as an MDP, with the goal of finding a policy that maximizes expected cumulative discounted rewards.The MDP tuple comprises state and action spaces, transition and reward functions, an initial-state distribution, and a discount factor.Source: §3 · p. 3
• In episodic RL, algorithms can collect trajectories for chosen policies, but omitting further data collection can cause off-policy methods to fail on expert data (Kumar et al., 2019).Off-policy methods may store trajectories in a replay buffer and optimize a policy with Q-learning, yet still iteratively collect additional data.Source: §3 · p. 3
• Offline RL learns from a fixed transition dataset without access to the MDP, whose generating behavior policy may differ from the learned policy.Effective methods must handle distribution shift and data-collection processes that the chosen policy class cannot represent.Source: §3 · p. 3

Narrow and biased data

• Narrow, biased data can cause offline RL algorithms to diverge or perform worse than the behavior policy, challenging them to handle diverse distributions safely.Such data can arise from deterministic policies, human demonstrations, or hand-crafted policies; a common response is to keep learned behavior close to the dataset distribution.Source: §4 · pp. 3–4

Undirected and multitask data

• Undirected, multitask data can support stitching: an agent combines trajectory pieces such as A-B and B-C to form a trajectory from A-C.This lets an agent use portions of existing trajectories rather than relying on generalization outside the dataset.Source: §4 · p. 4, Figure 2 · p. 4

Sparse rewards

• Fixed offline datasets with sparse rewards isolate algorithms’ credit-assignment ability from exploration, a major challenge in traditional RL.Traditional RL methods face sparse-reward difficulty from both credit assignment and exploration; offline RL removes exploration because the dataset is fixed.Source: §4 · p. 4

Suboptimal data

• Suboptimal datasets challenge imitation-learning approaches that generally require expert demonstrations, and prior work has predominantly used data with this property.Source: §4 · p. 4

Non-representable behavior policies, non-Markovian behavior policies, and partial observability

• Human or hand-crafted behavior may fall outside an algorithm’s model class, while non-Markovian policies and partial observability introduce modeling errors and bias.These issues especially affect methods that assume Markovian action probabilities, such as importance weighting.Source: §4 · p. 4

Realistic domains

• To balance realistic evaluation with accessibility and reproducibility, the benchmark uses established simulators and spans varied domains and task difficulty.It includes locomotion, traffic management, autonomous driving, and robotics, with some datasets from human demonstrations or mathematical models of human behavior; it also includes OpenAI Gym robotic locomotion tasks and similar prior datasets.Source: §4 · p. 4

Evaluation protocol

• Tasks pair typically 10^6-step offline datasets with simulators; hyperparameter tuning uses designated training tasks, while final performance is measured on evaluation tasks using scores normalized from random=0 to expert=100.Source: §5 · pp. 5, 7

Maze2D

• Maze2D tests whether offline RL can stitch collected subtrajectories into shortest paths across three mazes, using random goals and waypoint-following PD controllers that produce non-Markovian data.Source: §5 · p. 5

AntMaze

• AntMaze extends navigation stitching to an 8-DoF Ant with sparse rewards, using fixed-start, diverse random-start/random-goal, and play datasets across three maze layouts.Source: §5 · p. 5

Gym-MuJoCo

• Gym-MuJoCo standardizes Hopper, HalfCheetah, and Walker2d datasets—random, medium, medium-replay, and equal-part medium-expert mixtures—to probe suboptimal data and heterogeneous policy mixtures.Source: §5 · pp. 5–6

Adroit

• Adroit adapts 24-DoF Shadow Hand manipulation tasks to offline RL, with human demonstrations (25 trajectories per task), expert data, and cloned mixtures for sparse-reward, high-dimensional challenges.Source: §5 · p. 6

FrankaKitchen

• FrankaKitchen tests multitask stitching and generalization in complex 9-DoF robot trajectories: complete data solves tasks in order, partial includes solvable subsets, and mixed has no complete solutions.Source: §5 · p. 6

Flow

• Flow models traffic control on ring and merge roads, comparing random-acceleration data with “human” data from the hand-designed intelligent driver model (Treiber et al., 2000).Source: §5 · pp. 6–7

Offline CARLA

• Offline CARLA evaluates lane following and town navigation from first-person 48x48 RGB observations, using hand-designed controllers to create undirected data for stitching under partial observability.Source: §5 · p. 7

6 BENCHMARKING PRIOR METHODS

• Beyond simple tasks and data-collection strategies, algorithmic differences widened and deficiencies across methods became apparent.The evaluation included behavioral cloning and online and offline SAC alongside offline RL methods; online SAC was generally expected to outperform offline methods with equal data, except in exploration-challenging Adroit and maze tasks.Source: §6 · pp. 7–8
• On RL-policy datasets, algorithms had their greatest success, matching behavior policies with expert data and outperforming them with suboptimal data.This reflects the setting in which prior offline RL algorithms were predominantly benchmarked.Source: §6 · p. 8
• On sparse-reward Adroit and AntMaze tasks, many methods outperformed online SAC; conservative methods also handled biased, narrow data well on Flow and Gym-MuJoCo.The results indicate offline RL can help overcome exploration challenges.Source: §6 · p. 8
• Undirected data challenged existing methods in Maze2D, FrankaKitchen, CARLA, and AntMaze, though smaller Maze2D and AntMaze tasks were within reach; mixture data challenged all methods.Even the larger Maze2D instance posed a surprising challenge.Source: §6 · p. 8
• Adding expert data often failed to improve results: medium-expert MuJoCo datasets performed roughly like medium data except for hopper, and cloned Adroit data roughly matched limited demonstrations.The cloned Adroit datasets included additional data beyond the limited demonstration dataset.Source: §6 · p. 8
• Algorithms succeeded to some extent on controller-generated Flow and carla-lane data, but limited human demonstrations in Adroit and FrankaKitchen remained challenging.These limitations point to a need for more sample-efficient methods when large datasets are unavailable.Source: §6 · p. 8

7 DISCUSSION

• D4RL targets narrow, logged offline-RL data, while prior benchmarks largely use RL-trained robotic-control data and may overstate progress on harder real-world properties (Fujimoto et al., 2018a; Kumar et al., 2019; Wu et al., 2019; Dulac-Arnold et al., 2020).The authors argue these more challenging properties pose substantial difficulties for existing methods.Source: §7 · p. 8
• An accessible, representative simulation benchmark could accelerate offline RL by helping researchers leverage existing sequential data for flexible decision-making and real-world applications.Source: §7 · p. 8
• The benchmark leaves environmental stochasticity and large action spaces underexplored, and focuses mostly on robotics, autonomous driving, and traffic control rather than finance, industry, or operations research.Stochasticity is relevant to settings such as financial markets, healthcare, and advertising; recommender systems can have large action spaces.Source: §7 · pp. 8–9
• Moving offline RL from simulation to real-world logged data, including in recommender systems and medicine, requires reliable real-system or off-policy evaluation and standardized benchmarks across research groups.Source: §7 · p. 9

A TASK PROPERTIES

• The full carla-town dataset requires over 30GB of memory, so experiments used a subsampled version.The appendix notes that this subsampled version is provided alongside the full dataset.Source: A TASK PROPERTIES · p. 13
• Table 1 covers statistics for each benchmark task, including controller type and the number of recorded environment transitions.Controller categories are hand-designed navigation planners, human demonstrations, and random or neural network policies.Source: Table 1 · p. 13

B RESULTS BY DOMAIN

• The domain summaries exclude CARLA because all algorithms perform poorly there, and sort results from the best-performing algorithm to the worst.The stated exclusion applies to the domain performance tables.Source: B RESULTS BY DOMAIN · p. 14
• Table 2 compares online and offline SAC, BEAR, BRAC-p, BRAC-v, BC, AWR, BCQ, cREM, and aDICE using normalized scores.Scores average three seeds and are normalized from 0 for random to 100 for expert.Source: Table 2 · p. 15
• The normalized comparison includes online SAC and offline SAC as separate methods, alongside the listed offline algorithms.The table caption distinguishes SAC from SAC-off.Source: Table 2 · p. 15
• Table 3 reports raw, un-normalized scores for each task and algorithm as undiscounted simulator returns averaged over three random seeds.These raw scores accompany the normalized domain comparisons.Source: Table 3 · p. 16

• Table 4 lists the benchmark’s domains, dataset types, and sources for datasets originally proposed elsewhere.Source: Table 4 · p. 17
• Borrowed datasets include MuJoCo Expert, Random, and Medium; Adroit Human and Expert; and FrankaKitchen Complete, Partial, and Mixed.Source: Table 4 · p. 17
• Maze2D and AntMaze are new proposed domains, and all other datasets are proposed by this work.Source: Table 4 · p. 17

D TRAINING AND EVALUATION TASK SPLIT

• The recommended protocol allows hyperparameter tuning on designated training tasks and requires evaluation without tuning on separate evaluation tasks.Table 5 identifies the training/evaluation partition, but the supplied passage does not list its task assignments.Source: D TRAINING AND EVALUATION TASK SPLIT · p. 17, Table 5 · p. 17

E EXPERIMENT DETAILS

• Experiments used default hyperparameters, minimal implementation changes, and 500K training iterations or gradient steps; continuous-action REM was an unofficial implementation.Runs used Google Cloud Platform n1-standard-4 machines.Source: E EXPERIMENT DETAILS · p. 18

F ASSESSING THE FEASIBILITY OF HARD TASKS

• In carla-town and AntMaze, each lane or corridor was traversed multiple times, providing observed state-space coverage for these challenging tasks.The authors say few prior methods solved carla-town or the larger AntMaze tasks and use coverage as one feasibility check.Source: F ASSESSING THE FEASIBILITY OF HARD… · p. 18
• AntMaze data follows the Maze2D high-level planner, and both datasets cover most of the maze despite the ant’s more erratic motion.The authors note that the ant’s gait is relatively regular and that similar waypoints should yield coverage comparable to the 2D mazes.Source: F ASSESSING THE FEASIBILITY OF HARD… · p. 18

G MAZE DOMAIN TRAJECTORIES

• Maze2D trajectory plots show visited states as colored lines overlaid on the maze, with the goal marked by a white star.Figure 4 covers maze2d-umaze, maze2d-medium, and maze2d-large.Source: G MAZE DOMAIN TRAJECTORIES · p. 19, Figure 4 · p. 19
• AntMaze trajectory plots cover umaze, medium-play, large-play, and their diverse variants, with each dataset’s paths shown over the maze.Figure 5 arranges the three play variants across the top row and the diverse variants across the bottom row.Source: G MAZE DOMAIN TRAJECTORIES · p. 19, Figure 5 · p. 19