Source-linked AI summary
d3rlpy: An Offline Deep Reinforcement Learning Library
Takuma Seno, Michita Imai
TL;DR
Offline deep RL lacks standardized, reproducible implementations despite the need to learn from static datasets when active data collection is expensive. The paper introduces d3rlpy, a documented plug-and-play Python library supporting offline and online algorithms, and benchmarks its implementations on D4RL and Atari 2600 with accompanying scripts and results. The paper positions this combination of standardized access and faithful benchmarking as a reproducibility-oriented resource for offline RL research.
Problem
Offline RL needs reproducible, standardized implementations because active data collection is expensive, while existing implementations are scattered across repositories with inconsistent interfaces.
Method
The paper introduces d3rlpy, a Python library providing documented plug-and-play APIs for off-policy offline and online RL algorithms, together with faithful benchmarks on D4RL and Atari 2600.
Results
d3rlpy provides a standardized library of offline and online RL algorithms accompanied by a large-scale faithful benchmark, scripts, full tables, and released metrics.
Takeaways & Limitations
d3rlpy combines standardized access to offline RL algorithms with benchmark materials intended to support reproducible research and additional experiments.
Abstract
from arXiv · showhide
In this paper, we introduce d3rlpy, an open-sourced offline deep reinforcement learning (RL) library for Python. d3rlpy supports a set of offline deep RL algorithms as well as off-policy online algorithms via a fully documented plug-and-play API. To address a reproducibility issue, we conduct a large-scale benchmark with D4RL and Atari 2600 dataset to ensure implementation quality and provide experimental scripts and full tables of results. The d3rlpy source code can be found on GitHub: \url{https://github.com/takuseno/d3rlpy}.
1. Introduction
Offline RL addresses the challenge of expensive active data collection by learning policies from static datasets, but scattered implementations and nonstandard interfaces hinder adoption and reproducibility. d3rlpy responds with a documented, standardized library and a large-scale faithful benchmark.
- Offline RL learns policies from previously collected static datasets when active data collection is expensive.
- Scattered implementations and nonstandard interfaces make it difficult to incorporate recent offline deep RL algorithms into projects.
- d3rlpy provides documented, standardized, plug-and-play APIs for off-policy offline and online RL algorithms.
- A large-scale faithful benchmark is conducted with d3rlpy to address reproducibility in offline RL.
2. Related work
Existing deep RL libraries balance ease of use, flexibility, and reproducibility in different ways. d3rlpy combines a documented plug-and-play API with many offline RL algorithms and extensive benchmark results.
- Deep RL libraries variously emphasize plug-and-play usability, modular customization, general agent frameworks, or faithful reproduction results.
- d3rlpy is presented as the first library combining numerous offline RL algorithms with extensive benchmark results in this research field.
- The combination of a fully documented plug-and-play API and faithfully benchmarked offline algorithms distinguishes d3rlpy's offline RL research experience.
3. Design of d3rlpy
d3rlpy uses standardized components and scikit-learn-styled interfaces to support offline training, online training, customization, preprocessing, and offline-to-online fine-tuning. Its modular design separates user-facing training APIs from reusable algorithm and data components.
- 3.1 Library interface: d3rlpy uses a scikit-learn-styled API with dedicated MDPDataset and Algorithm components for offline and online training.Training methods such as fit and fit_online are implemented in Algorithm components.
- 3.1 Library interface: Neural network architectures are selected automatically from MLP and convolutional models based on observations, while custom architectures remain supported.
- 3.1 Library interface: Offline-trained policies can transition seamlessly to online fine-tuning through d3rlpy's support for both offline and online training.
- Algorithm: Algorithm components expose high-level training methods and lower-level update operations through a hierarchical design intended to increase algorithm reuse.
- MDPDataset: MDPDataset standardizes offline data from logged arrays, benchmark datasets such as D4RL and Atari 2600, or OpenAI Gym-style environments.
- EncoderFactory, QFunctionFactory, and scaler components support custom models, distributional Q-functions, and observation, action, and reward preprocessing.
4. Large-scale benchmark
d3rlpy's implemented algorithms are faithfully benchmarked on D4RL and Atari 2600 datasets to address reproducibility. The release includes benchmark scripts, full result tables, and logged metrics for further experiments.
- Large-scale benchmark: d3rlpy algorithms are faithfully benchmarked with D4RL and Atari 2600 datasets to address reproducibility.
- Large-scale benchmark: The benchmark release includes full Python scripts, complete tables in the appendices, and logged metrics in a GitHub repository.
5. Conclusion
d3rlpy is an offline deep reinforcement learning library that provides offline and online RL algorithms through a standardized plug-and-play API. A large-scale faithful benchmark addresses reproducibility in offline RL.
- d3rlpy provides offline and online RL algorithms through a standardized plug-and-play API.
- A large-scale faithful benchmark was conducted to address the reproducibility issue in offline RL.
- The library implements a broad set of named algorithms, including NFQ, DQN, SAC, CQL, AWAC, CRR, TD3+BC, and IQL.
Appendix A. Benchmark Results: D4RL
The D4RL benchmark evaluates implemented algorithms on OpenAI Gym MuJoCo tasks using standardized training and evaluation procedures, then compares normalized scores with reference results. The comparisons account for missing standard deviations and unavailable implementations or benchmark results.
- D4RL evaluation trains each algorithm for 500K gradient steps and evaluates every 5000 steps over 10 episodes.The procedure uses previously reported or author-recommended hyperparameters, with shared discount, target-update, and optimizer settings.
- The benchmark reports normalized scores and side-by-side comparisons with reference scores across D4RL tasks.Table 2 reports normalized scores, while Tables 3–11 provide algorithm-specific reference comparisons.
- CRR is excluded from side-by-side comparison because it lacks D4RL benchmarking and a publicly available author implementation.
- Performance discrepancies are considered trivial because some authors did not report standard deviations and many offline RL results used only 3 or 4 random seeds.
- BEAR and CQL reference scores were collected by executing author-provided implementations with suggested hyperparameters.
Appendix B. Benchmark Results: Atari 2600
The Atari 2600 benchmark evaluates algorithms on 1% subsets of open-sourced datasets using a fixed training procedure and reports raw scores with reference comparisons. Some comparisons are constrained by reference results that provide only mean scores.
- Atari 2600 evaluation uses 1% of transitions, or 500K datapoints, trains for 12.5M gradient steps, and evaluates every 125K steps over 10 episodes.
- NFQ and BCQ are excluded from side-by-side comparisons because suitable public datasets or directly applicable author implementations were unavailable.
- The benchmark reports raw Atari scores averaged over 10 random seeds and compares them with reference scores.
- The reported reference results for QR-DQN provide only mean scores, without standard deviations.
Appendix C. Benchmark Results: Fine-tuning
The fine-tuning evaluation tests AWAC and IQL on AntMaze datasets after static-dataset pretraining, using online experience for subsequent policy refinement. Results are reported with reference-score comparisons and averaged across repeated random-seed runs.
- Experimental setup: AWAC and IQL were evaluated in a fine-tuning scenario combining static-dataset pretraining with subsequent online experiences.The evaluation followed a previously described experimental procedure for algorithms designed to support fine-tuning.
- Experimental setup: Each policy was fine-tuned for 1M steps after pretraining, with experiments repeated using 10 random seeds.Reward values were shifted by subtracting 1, and original-paper hyperparameters were used.
- Benchmark results: The benchmark results are presented as having trivial performance discrepancies relative to the references because the original authors did not report standard deviations.This assessment accompanies the side-by-side comparisons in Tables 18 and 19.
- Benchmark results: Table 17 reports normalized AntMaze scores for pretrained and fine-tuned policies, averaged over 10 random seeds.The table places pretrained-policy scores on the left and fine-tuned-policy scores on the right.
- Benchmark results: Tables 18 and 19 compare the reproduced AWAC and IQL normalized scores with reference mean scores from the original work.The reference results provide means without standard deviations.