Source-linked AI summary
CORL: Research-oriented Deep Offline Reinforcement Learning Library
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, Sergey Kolesnikov
TL;DR
Offline RL libraries can make algorithm details harder to inspect, while offline-to-online methods and reliable evaluation remain comparatively underrepresented. CORL addresses this with single-file implementations, configuration files, tracking, and D4RL benchmarks; its observations identify strong average baselines and scope limitations in the reported experiments.
Problem
Existing abstractions can hinder understanding and modification of offline RL methods, while offline-to-online RL and reliable evaluation receive less coverage.
Method
CORL implements offline and offline-to-online algorithms as isolated single files, uses configuration files and experiment tracking, and benchmarks them on D4RL datasets.
Results
ReBRAC and IQL appear to be the strongest baselines on average across offline and offline-to-online results, while SAC-N and EDAC are strongest on Gym-MuJoCo and Maze2d but perform poorly on AntMaze and Adroit.
Takeaways & Limitations
CORL provides transparent implementations, comprehensive logs, and benchmark results intended as reusable references for offline RL research and applications.
Takeaways & Limitations
Weights&Biases is proprietary, and additional hyperparameter search is needed for some reported settings.
Abstract
from arXiv · showhide
CORL is an open-source library that provides thoroughly benchmarked single-file implementations of both deep offline and offline-to-online reinforcement learning algorithms. It emphasizes a simple developing experience with a straightforward codebase and a modern analysis tracking tool. In CORL, we isolate methods implementation into separate single files, making performance-relevant details easier to recognize. Additionally, an experiment tracking feature is available to help log metrics, hyperparameters, dependencies, and more to the cloud. Finally, we have ensured the reliability of the implementations by benchmarking commonly employed D4RL datasets providing a transparent source of results that can be reused for robust evaluation tools such as performance profiles, probability of improvement, or expected online performance.
1 Introduction
CORL presents a minimalistic offline RL library built around isolated single-file implementations, benchmark results, and experiment tracking. Its design aims to make algorithms easier to understand, modify, and evaluate reproducibly.
- The library addresses a tension between user-friendly abstractions and the difficulty of understanding or modifying new offline RL algorithms.
- CORL provides isolated single-file implementations of deep offline and offline-to-online reinforcement learning algorithms.The library supports optional experiment-tracking integration for analyzing algorithm results and behavior.
- CORL reports algorithm results and behavior rather than relying only on final performance values.
- The authors intend CORL’s implementations and D4RL results to support newcomer learning, rapid method modification, and reproducible evaluation.The results are also presented as reference points that can reduce repeated implementation and hyperparameter-tuning effort.
2 Related Work
CORL is positioned within a broader RL-library ecosystem while targeting offline and offline-to-online reinforcement learning. Unlike modular libraries, it emphasizes independent implementations with fewer abstraction layers and external dependencies.
- Existing RL libraries address varied perspectives, while offline and especially offline-to-online reinforcement learning remain less extensively covered.
- CORL focuses on offline RL methods through non-modular, independent algorithm implementations rather than additional abstractions.
- Compared with CleanRL, CORL targets offline and offline-to-online RL rather than online RL.
- CORL seeks to minimize requirements and external dependencies by implementing its components from scratch.
3 CORL Design
CORL’s design exposes implementation details through independent single-file experiments, predefined YAML configurations, and comprehensive experiment tracking. These choices target easier learning, prototyping, debugging, and performance analysis.
- Single-File Implementations: Single-file implementations expose environment details, algorithm hyperparameters, and evaluation parameters that modular abstractions can hide.The library includes examples such as AnyPercent BC, TD3-BC, and Decision Transformer implementations.
- Single-File Implementations: Separating offline and offline-to-online versions creates code duplication but supports a smoother learning curve, simpler prototyping, and faster debugging.The flat architecture keeps code and variables accessible during modification and inspection.
- Configuration files: Predefined YAML files collect environment and algorithm hyperparameters while retaining the familiar command-line interface.The authors report that this setup reduces the need to remember experiment-specific parameters.
- Experiment Tracking: CORL’s tracking integration automatically records source code, dependencies, hardware, environment variables, hyperparameters, training and system metrics, and logs.
- Experiment Tracking: Weights&Biases is proprietary, although Tensorboard and Aim can replace it with a few lines of code change.
- Experiment Tracking: Full metric logs support analysis beyond final or best performance, including performance profiles and probability-of-improvement analyses.
4 Benchmarking D4RL
CORL benchmarks offline and offline-to-online algorithms across D4RL domains, revealing strong average baselines alongside substantial domain, hyperparameter, and framework sensitivity. The evaluation reports both final and maximal performance, training behavior, and online-tuning outcomes.
- Offline: ReBRAC, IQL and AWAC are the strongest offline baselines on average.AWAC is often omitted in recent works.
- Offline: EDAC significantly outperforms other algorithms on Gym-MuJoCo, while SAC-N performs best on Maze2d.Neither result generalizes across all evaluated domains.
- Offline: SAC-N and EDAC cannot solve AntMaze and perform poorly on Adroit.This contrasts with their stronger performance on Gym-MuJoCo and Maze2d.
- Offline: CQL is extremely sensitive to hyperparameters and implementation details.AntMaze requires five critic-network hidden layers, whereas other tasks suffer with that depth; additional hyperparameter search is needed.
- Offline: Hyperparameters are not always transferable between deep-learning frameworks.Reference JAX parameters sometimes strongly underperform in PyTorch implementations, including IQL on Hopper and CQL on Adroit.
- Offline-to-Online: Considering offline and offline-to-online results, IQL and ReBRAC appear to be the strongest baselines on average.In offline-to-online evaluation, IQL and ReBRAC notably improve Adroit scores after tuning, while AWAC generally does not benefit.
5 Conclusion
CORL provides clear, reproducible implementations of offline and offline-to-online reinforcement learning algorithms, together with benchmarking and experiment-tracking support. Its D4RL results and logs are intended as reusable references for research and practice.
- CORL implements ten offline and six offline-to-online algorithms in a single-file library with configuration files and advanced metrics tracking.
- All implemented approaches were benchmarked on D4RL datasets, closely matching and sometimes exceeding available reference results.
- Comprehensive experiment logs allow researchers to access and reuse results without rerunning the experiments.
Checklist
The checklist reports that the paper addresses its claims, limitations, reproducibility materials, training details, error bars, compute, and asset documentation. Human-subject and personally identifiable-information items are marked not applicable.
- The authors report that the paper accurately reflects its contributions and scope and discusses its limitations.
- The paper reports no applicable potential negative societal impacts and confirms conformity with ethics review guidelines.
- The authors report releasing code, configurations, and in-depth reports and providing training details, error bars, and compute information.
- Existing assets are cited and licensed, while no new assets are included in the supplemental material or through a URL.
- Consent, personally identifiable information, offensive content, participant instructions, participant risks, and compensation are marked not applicable.
A.1 Offline
The offline appendix presents normalized-performance figures and training curves across Gym-MuJoCo, Maze2d, AntMaze, and Adroit datasets. It includes both last- and best-policy views and task-specific training-curve breakdowns.
- Figure 4 reports normalized performance of the last trained policy across Gym-MuJoCo, Maze2d, AntMaze, and Adroit datasets, averaged over 4 random seeds.
- Figure 5 reports normalized performance of the best trained policy across the same four D4RL dataset groups, averaged over 4 random seeds.
- Training curves are provided for HalfCheetah, Hopper, Walker2d, and Maze2d across their listed dataset variants.
- Additional training curves cover AntMaze datasets, including Umaze, play, and diverse variants.
- Adroit training curves cover Pen, Door, Hammer, and Relocate across Human, Colned, and Expert datasets.
A.2 Offline-to-online
The offline-to-online appendix reports normalized performance after online tuning and shows training curves for AntMaze and Adroit Cloned tasks. It also includes experiment-tracking interface screenshots.
- Figure 15 shows normalized performance of the last trained policy on AntMaze and Adroit datasets after online tuning, averaged over 4 random seeds.
- Figure 16 presents online-tuning training curves for six AntMaze dataset variants.
- Figure 17 presents online-tuning training curves for Pen, Door, Hammer, and Relocate in the Adroit Cloned task.
- Figure 18 shows screenshots of the Weights&Biases experiment-tracking interface.
D Experimental Details
The experiments use specified reward preprocessing, implementation sources, default hyperparameters where available, and substantial GPU compute. Configuration files and hyperparameter details are provided for reproducibility.
- AntMaze rewards are shifted by subtracting 1, except for CQL and Cal-QL, which map (0, 1) to (-5, 5).
- The implementations build on original code for TD3 + BC, SAC-N/EDAC, SPOT, and ReBRAC, alongside custom implementations of IQL and CQL/Cal-QL.
- Default hyperparameters are used for most algorithms and datasets when available, with configurations and additional hyperparameter details provided in the repository and subsection D.2.
- Experiments ran on V100 and A100 GPUs and consumed approximately 5000 hours of compute in total.
D.1 Number of update steps and evaluation rate
Training and evaluation schedules vary by algorithm, with SAC-N and EDAC receiving longer training, Decision Transformer using dataset-pass epochs, and other methods following a standard step-based schedule. Separate tables list hyperparameters for the evaluated algorithms.
- Number of update steps and evaluation rate: SAC-N and EDAC are trained for 3 million steps, except 1 million steps on AntMaze, and evaluated every 10000 steps.
- Number of update steps and evaluation rate: Decision Transformer is trained for 50 epochs per dataset and evaluated every 5 epochs.
- Number of update steps and evaluation rate: All other algorithms are trained for 1 million steps and evaluated every 5000 steps, or every 50000 steps on AntMaze.
- Hyperparameter configurations: Hyperparameter tables are provided for BC and BC-N%, TD3+BC, CQL and Cal-QL, IQL, AWAC, SAC-N and EDAC, Decision Transformer, and SPOT.
- Hyperparameter configurations: The CQL and Cal-QL hyperparameters used are noted as suboptimal on Adroit for the provided implementation.