Source-linked AI summary
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Sergey Levine, Aviral Kumar, George Tucker, Justin Fu
TL;DR
Offline reinforcement learning asks how to derive high-reward policies from fixed datasets when exploration and additional data collection are unavailable. This tutorial formulates the problem, surveys its challenges, methods, and applications, and discusses open problems; it concludes that model learning may be an important component of future effective methods.
Problem
Offline reinforcement learning must construct a policy with high cumulative reward entirely from a fixed dataset, while lacking the ability to improve exploration when high-reward transitions are absent.
Method
The tutorial develops the problem formulation, analyzes challenges associated with deep function approximation, surveys offline methods and applications, and presents perspectives on open problems.
Results
Model learning may be an important component of effective future offline reinforcement learning methods, while prior work has demonstrated good results with model-based methods, particularly for generalization with real-world data.
Takeaways & Limitations
Offline reinforcement learning offers a route to learning decision-making policies from large previously collected datasets without additional online interaction.
Takeaways & Limitations
Offline learning assumes the dataset adequately covers high-reward transitions, and model-based methods struggle with very high-dimensional observations and long-horizon prediction.
Abstract
from arXiv · showhide
In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.
1 Introduction
Offline reinforcement learning seeks to learn policies from fixed, previously collected datasets without further environment interaction, addressing the cost and danger of online data collection. This tutorial introduces its challenges, mitigation strategies, applications, and open problems.
- Online reinforcement learning can be impractical because collecting experience is expensive in robotics, education, and healthcare, and dangerous in autonomous driving and healthcare.
- Offline reinforcement learning aligns decision-making with data-driven learning by reusing previously collected data rather than requiring continual online interaction.
- Offline reinforcement learning trains a policy from a dataset collected by a potentially unknown behavior policy, without altering the dataset or interacting with the MDP during training.The policy is deployed only after training, making large previously collected datasets usable.
- Offline methods face distributional shift because expressive function approximators can become vulnerable when learning from complete offline datasets without on-policy interaction.
- Simulation-to-real-world transfer has become prominent in robotics and autonomous driving, highlighting deficiencies in effective offline reinforcement learning.
- The tutorial presents the offline problem formulation, surveys methods and applications, explains mitigation efforts, and discusses future work and open problems.
2 Offline Reinforcement Learning Problem Statement and Overview
Offline reinforcement learning seeks near-optimal policies from fixed previously collected data, requiring the learner to infer the environment and answer counterfactual questions without further interaction. Its central difficulties include limited dataset coverage, distributional shift, and error accumulation outside the training distribution.
- Offline reinforcement learning problem: Offline reinforcement learning learns policies from a fixed dataset and must infer the underlying dynamical system before deployment.The dataset functions as the policy’s training set, while the learned policy is ultimately evaluated through interaction with the MDP.
- Offline reinforcement learning problem: Unlike online reinforcement learning, offline learning cannot improve exploration when the dataset omits high-reward transitions.The tutorial therefore assumes that the dataset adequately covers high-reward regions to make learning feasible, while noting that formal sufficiency conditions remain an open problem.
- Challenges: Offline reinforcement learning requires counterfactual reasoning because improving on observed behavior entails executing action sequences that differ from those in the dataset.This conflicts with the i.i.d. assumptions underlying standard supervised learning and creates distributional shift during evaluation.
- Challenges: Behavioral cloning can accumulate an error bound that is quadratic in horizon H offline, compared with linear in H when additional on-policy data are collected.The cited bounds are C + H^2ϵ for offline training and C + Hϵ under DAgger-style on-policy data collection.
- Challenges: Distributional shift arises when the learned policy visits states unlike those represented by the behavior distribution, causing supervised generalization guarantees to fail.Once the policy enters out-of-distribution states, errors can persist for the remainder of a finite-horizon trial.
3 Offline Evaluation and Reinforcement Learning via Importance Sampling
Offline evaluation and policy improvement can reuse trajectories from a behavior policy through importance sampling, but estimator variance and policy mismatch limit reliable improvement. The section also describes approximate, marginalized, and confidence-bound methods, alongside off-policy gradient results and optimization approaches that avoid on-policy samples.
- Offline Evaluation and Reinforcement Learning via Importance Sampling: Importance sampling estimates policy returns or gradients from behavior-policy trajectories, extending off-policy evaluation methods to offline reinforcement learning.The section covers direct, weighted, per-decision, and policy-gradient estimators.
- Offline Evaluation and Reinforcement Learning via Importance Sampling: Unnormalized importance sampling can have potentially unbounded variance, while self-normalization introduces bias but may substantially reduce variance and remains strongly consistent.The same bias–variance trade-off appears for weighted policy-gradient estimators.
- Offline Evaluation and Reinforcement Learning via Importance Sampling: Per-decision estimators drop future importance weights because later states and actions do not affect the current reward, yet weighted versions can still have too much variance for practical use.This limitation persists even after exploiting the problem’s temporal structure.
- Offline Evaluation and Reinforcement Learning via Importance Sampling: Doubly robust estimators combine importance sampling with approximate Q-values and are unbiased if either the behavior policy is known or the model is correct.Confidence bounds can further support policy improvement with high-probability safety guarantees.
- Off-Policy Policy Gradients and Marginalized Importance Sampling: The approximate off-policy policy-gradient estimator is consistent as sample size grows, while related approaches preserve local optima only under restrictive conditions.Other offline approaches formulate convergent optimization problems under linear function approximation or estimate density ratios without on-policy samples.
- Challenges and Open Problems: Importance weights become degenerate when the learned policy differs substantially from the behavior policy, especially in high-dimensional or long-horizon tasks, limiting reliable improvement.The maximum improvement is also constrained by behavior-policy suboptimality, state-action dimensionality, and effective horizon.
4 Offline Reinforcement Learning via Dynamic Programming
Offline dynamic programming is challenged by distributional shift, especially when learned Q-functions evaluate actions outside the dataset. The article reviews policy constraints, uncertainty estimation, and conservative value functions as mitigation strategies, while emphasizing their tradeoffs and remaining limitations.
- Distributional Shift: Offline dynamic programming suffers from action distribution shift because Bellman targets evaluate actions sampled from the learned policy rather than only dataset actions.This creates erroneous Q-values for out-of-distribution actions and can produce the unlearning effect during training.
- Policy Constraints: Policy constraints limit the learned policy relative to the behavior policy, but their effectiveness depends on accurately estimating potentially multimodal behavior distributions.Support constraints can preserve optimal deterministic policies, whereas distribution constraints may force highly suboptimal behavior when the behavior policy strongly favors poor actions.
- Policy Constraints: Support constraints can avoid out-of-distribution actions without forcing the learned policy to match behavior-policy probabilities.For continuous actions, MMD can empirically approximate support constraint behavior in finite-sample settings because it operates on samples rather than specific densities.
- Conservative Value Functions: Conservative value methods penalize high Q-values for adversarially selected actions to discourage overestimation without explicitly modeling the behavior policy.Their conservative penalty pushes high values toward in-distribution actions, but small datasets can cause excessive underestimation for undersampled actions.
- Uncertainty Estimation: Uncertainty-based methods learn distributions or confidence sets over possible Q-functions and improve policies using conservative Q estimates.In practice, calibrated uncertainty estimates are difficult to obtain with high-capacity function approximators, limiting these methods relative to policy-constraint and conservative-value approaches.
- Remaining Challenges: Both constraint-based and conservative methods remain affected by the training-state distribution, which current offline methods do not fully account for.Function approximation can couple Q-values across states with substantially different densities under the behavior-policy state distribution.
5 Offline Model-Based Reinforcement Learning
Offline model-based reinforcement learning uses predictive models trained on fixed data for planning or policy learning, but must control model exploitation and distributional shift. Conservative and constrained approaches help, while accurate long-horizon modeling and the theoretical relationship to model-free methods remain open challenges.
- Offline Model-Based Reinforcement Learning: Predictive models estimate transition dynamics from offline data, enabling model-based methods to use supervised learning and large, diverse datasets for control.The model can support planning, including model predictive control, or policy training.
- Model Exploitation and Distributional Shift: Model exploitation occurs when policy optimization drives the learned policy into out-of-distribution states or actions where model errors produce falsely high predicted returns.This arises because the optimized policy may visit state distributions that differ from those represented in the offline data.
- Theoretical Analysis: Model-based policy-learning analyses bound errors from distributional shift in both the policy and transition model, with greater policy divergence leading to states farther outside the data distribution.The resulting value-estimation error contains contributions from policy and model distribution shifts.
- Approaches to Offline Model-Based RL: Offline model-based methods include naïve model training, uncertainty-aware planning, safe-region constraints, and conservative algorithms that penalize visits to states where the model is likely inaccurate.MoREL and MOPO modify the learned model or rewards to induce conservative behavior using estimates of model error.
- Challenges and Open Problems: Current fully offline model-based methods depend on imperfect uncertainty estimation, while high-dimensional observations and long horizons remain difficult to model accurately.Hybrid methods using short rollouts or avoiding full-observation prediction are identified as promising responses.
- Challenges and Open Problems: Whether model-based methods can theoretically outperform model-free dynamic programming remains open because both approaches solve prediction problems using different predicted quantities.Model-based methods predict future states, whereas model-free methods predict future returns and can be generalized to other return quantities.
6 Applications and Evaluation
Offline reinforcement learning is being evaluated across benchmarks and applied to robotics, healthcare, autonomous driving, and recommender systems, but standardized evaluation remains underdeveloped. Its appeal is strongest where online interaction is costly, dangerous, or impractical.
- Evaluation and Benchmarks: Benchmarking modern offline reinforcement learning remains less established than evaluation in individual application domains.A common approach uses data collected by standard online reinforcement learning algorithms, including entire replay buffers or offline datasets.
- Evaluation and Benchmarks: Offline RL can outperform individual training trajectories by composing subsequences into higher-reward behavior, a capability evaluated by D4RL tasks.The Maze2D example illustrates combining transitions through intermediate states to find shorter paths than those directly present in the dataset.
- Evaluation and Benchmarks: Offline RL evaluation may use simulators, off-policy evaluation, or human experts, depending on whether online testing is available and safe.Human labeling has assessed dialogue fluency and amicability, while domain experts have evaluated sepsis-treatment decisions.
- Applications in Robotics: Robotics benefits from offline RL because collecting online control data is expensive and impractical, especially for complex visual behavior.Prior work includes robotic grasping and model-based learning of manipulation skills from offline data, including datasets with more than 500,000 grasp trials.
- Applications in Healthcare: Healthcare applications face safety constraints and biased datasets, including records disproportionately representing serious outcomes.The MIMIC-III dataset contains approximately 60K ICU medical records and has supported offline RL for sepsis, ventilator use, and drug recommendations.
- Applications in Autonomous Driving and Recommendation: Offline RL is promising for autonomous driving and recommender systems because online exploration can cause catastrophic failures or substantial monetary losses.Real-world self-driving applications remain limited, while recommender-system evaluation commonly uses off-policy methods in contextual bandit formulations.
7 Discussion and Perspectives
Offline reinforcement learning reframes reinforcement learning as data-driven counterfactual inference, but distributional shift makes this difficult with high-dimensional models. The tutorial argues that advances in datasets, model learning, and statistical methods could broaden practical applications.
- Discussion and Perspectives: Offline reinforcement learning could bring reinforcement learning into a data-driven discipline that benefits from large datasets.Achieving this requires combining statistical methods with the fundamentals of sequential decision making.
- Open Problems: Standard off-policy approaches often struggle with deep networks, high-dimensional observations, and temporally extended tasks.Policy constraints and uncertainty estimation are proposed to address distributional shift between learned and behavior policies.
- Discussion and Perspectives: Offline RL is fundamentally a counterfactual inference problem: estimating consequences of decisions different from those recorded in the data.This challenge departs from the i.i.d. assumption that test-time queries follow the training-data distribution.
- Model-Based Offline RL: Model-based offline RL predicts resulting states for actions absent from the dataset, but out-of-distribution state-action tuples can make those predictions inaccurate.Prior work nevertheless reports good results, particularly for generalization with real-world data, motivating model learning as a future component.
- Perspectives: Advances in datasets may contribute as much as algorithmic advances to future reinforcement learning progress.Larger, more diverse, and more representative reusable datasets could support applications previously unsuited to reinforcement learning, including robotics and autonomous driving.