PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, Esther Luna Colombini

arXiv:2203.01387v3cs.LGcs.AIstat.ML

TL;DR

Offline RL addresses the cost and danger of environmental interaction by learning policies from static datasets, but distributional shift makes this setting challenging. The survey provides a taxonomy and unified review of methods, benchmarks, performance, and open problems, emphasizing evaluation without further interaction.

  • Problem

    RL remains inaccessible in domains where collecting online interaction is expensive or dangerous, while offline policies can encounter distributional shift beyond static-data coverage.

  • Method

    The survey proposes a taxonomy, reviews offline RL methods and benchmarks, summarizes method performance across dataset properties, and discusses open problems.

  • Results

    The survey reports a unified synthesis of offline RL methods, benchmark properties and shortcomings, performance patterns, and future research directions.

  • Takeaways & Limitations

    Offline RL offers a framework for applying RL to settings where online data collection is impractical, while robust offline policy evaluation remains essential.

Abstract

from arXiv · show

With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.

I. INTRODUCTION

Offline RL learns policies exclusively from static datasets, addressing the cost and danger of continued environmental interaction. This survey organizes the rapidly expanding field, reviews methods and benchmarks, compares performance across dataset properties, and identifies open problems.

  • Motivation: High-capacity deep RL methods remain difficult to apply in the real world because they typically depend on active data collection.
  • Motivation: Offline RL learns a policy from previously collected experiences in a static dataset without further environment interaction.
  • Challenges: Distributional shift occurs when learned policies select out-of-distribution actions or visit unseen states, causing prediction errors that can compound and destabilize behavior.
  • Methods: Offline RL methods address distributional shift through policy constraints, conservative value estimates, uncertainty penalties, and other specialized training procedures.
  • Contributions: The survey contributes a taxonomy, unified algorithmic review, benchmark assessment, performance summary, and discussion of open problems and future directions.

B. Reinforcement Learning

The paper frames RL through policies, value functions, and interaction with an MDP, then formalizes offline RL as policy learning from fixed behavioral data. Offline RL preserves the return-maximization objective but must handle limited coverage and the absence of online correction.

  • Reinforcement Learning: RL seeks an optimal policy that maximizes expected discounted return over trajectories induced by that policy.
  • Reinforcement Learning: State-value, action-value, and advantage functions quantify expected returns and action benefits for policy evaluation and improvement.
  • Offline Reinforcement Learning: Offline RL learns an improved policy from a fixed dataset generated by an unknown behavior policy, without further environment interaction.
  • Offline Reinforcement Learning: The offline objective remains return maximization, but arbitrary policy evaluation is invalid when distributional shift reaches states absent from the dataset.
  • Desirable Properties: Offline RL can improve on behavioral cloning through generalization, filtering good behaviors, and recombining useful parts across trajectories.

2) Challenges:

Offline RL is difficult because fixed datasets cannot support exploration or correct errors outside the training distribution. The survey addresses this challenge with a compositional taxonomy of algorithm structures and modifications.

  • Challenges: Offline RL cannot explore new states or actions, so distributional shift and compounding errors can make learned policies diverge from the training distribution.The Bellman objective is accurate only under matching action distributions, while policy-induced state distributions can also deviate.
  • Taxonomy: The survey proposes a high-level taxonomy for classifying offline RL algorithms while using mereology to represent modifications that can be combined.This distinction separates is-a class relationships from has-a relationships among algorithm components.
  • Algorithm structure: Offline RL algorithms can use an optionally filtered static dataset to learn a policy directly, learn dynamics, or learn a trajectory distribution for planning.Dynamics models may also generate synthetic samples for policy learning, while actor–critic and imitation learning provide alternative policy-learning routes.
  • Algorithm structure: Loss modifications such as policy constraints, uncertainty estimation, and regularization can be added to policy evaluation or policy improvement, including in model-based approaches.Table I summarizes modification types and their additional implementation requirements.
  • Policy constraints: Direct policy constraints estimate the behavior policy and constrain the learned policy using a divergence threshold between their distributions.These methods explicitly estimate πβ to compute and enforce the divergence constraint.
  • Policy constraints: Estimating the behavior policy is a major limitation because human, hand-designed, or mixed-policy data can be difficult to model; misspecification can cause dramatic failures.A unimodal estimate fitted to multimodal data is given as an example of problematic misspecification.

B. Importance Sampling

Importance sampling evaluates offline policies with behavior-policy data, while regularization and uncertainty estimation modify learning objectives to control conservatism and model confidence.

  • Importance Sampling: Importance sampling evaluates policy πθ with samples from behavior policy πβ by weighting off-policy policy gradients.The weights include products of importance ratios across time steps.
  • Importance Sampling: Importance weights grow exponentially with horizon H, creating high variance and motivating strategies that reduce this variance.The product weight w0:H is the specific source of the horizon-dependent growth.
  • Regularization: Policy regularization adds penalties to tune learned functions without directly restricting the policy to the behavior policy.Value regularization makes value estimates more conservative, while policy regularization can encourage properties such as stochasticity.
  • Uncertainty estimation: Uncertainty-based methods estimate uncertainty in a policy, value function, or model and relax policy constraints in low-uncertainty regions.For Q-functions, uncertainty can be estimated from the variance among Q-value predictions in an ensemble.
  • Model-based methods: Offline model-based methods learn transition and reward models from dataset D, then simulate transitions for planning, but cannot correct model mistakes through environment interaction.Conservative models can avoid transitions to out-of-distribution states by using uncertainty estimates and reward penalties.

F. One-Step Methods

One-step methods replace iterative policy evaluation and improvement with a single evaluation step followed by policy improvement, reducing exposure to iterative distributional shift. The survey places these methods within a broader taxonomy of offline RL approaches and tracks their development alongside other field milestones.

  • Motivation: Actor–critic methods alternate policy evaluation and policy improvement, but evaluating updated-policy targets on behavior-policy data creates distributional shift.The target values concern actions from πoff while the Q-function is trained on actions from πβ.
  • One-Step Formulation: One-step methods perform a single policy-evaluation step followed by a single policy-improvement step, using multiple state sweeps to estimate values accurately.
  • Imitation Learning: Imitation learning methods mimic the behavior policy, with behavioral cloning exactly copying it through supervised learning in its simplest form.The survey extends this category to methods using behavioral cloning at their core, including filtering undesirable behaviors or learning conditional policies.
  • Imitation Learning: Behavioral cloning can suffer compounding errors from unseen states, but may succeed when most dataset trajectories contain expert behavior.
  • Imitation Learning: Without expert data, imitation methods filter undesirable behaviors using value functions or heuristics, while conditional policies learn from task-dependent outcome functions.A central challenge for conditional policies is defining an appropriate outcome function for the data and task.
  • Trajectory Optimization: Trajectory optimization learns a joint state-action model of behavior-policy trajectories and plans actions from an initial state, while sequence anchors reduce selection of OOD actions.Large sequence models are compatible with offline RL because they avoid active data collection during training.
  • Survey Organization: The survey’s taxonomy classifies popular offline RL methods, reveals underexplored areas, and uses a timeline to show the development and popularity of method classes.

A. Policy Constraints

Policy-constraint methods limit learned policies’ deviation from dataset behavior, either directly through behavior-policy estimation or implicitly through regularization. The reviewed methods span direct constraints, divergence penalties, advantage weighting, and behavior-cloning regularization, with performance and fine-tuning behavior depending on the setting.

  • Direct Constraints: BCQ directly constrains the learned policy to choose actions similar to those selected by an estimated behavior policy.Its perturbation model and sampled actions determine how closely πθ remains tied to πβ.
  • Direct Constraints: Support matching seeks to avoid OOD actions without forcing the learned policy to reproduce poor behavior-policy actions.
  • Direct Constraints: BRAC penalizes either policy evaluation or policy improvement with a divergence term, and reports stronger results from value penalties than policy regularization.BRAC-v and BRAC-p use DKL(πθ∥πβ) in the respective objectives.
  • Direct Constraints: Fisher-BRC constrains an entropy-regularized learned policy with Fisher divergence and adds a gradient penalty to prevent the offset-function gradient from dominating.
  • Implicit Constraints: AWR and AWAC use advantage-weighted maximum log-likelihood objectives, with AWR estimating advantages from Monte Carlo returns and AWAC using a Q-function.AWAC’s Q-based estimate is intended to reduce variance and increase sample efficiency.
  • Implicit Constraints: TD3+BC adds a behavior-cloning regularizer to TD3, and achieves competitive results on D4RL Gym-MuJoCo datasets despite its simple modification.The regularizer penalizes mean squared error between actions sampled from the learned and behavior policies.
  • Offline-to-Online Fine-Tuning: Implicit policy constraints are promising for offline-to-online fine-tuning because new transitions can be appended without repeatedly reestimating the behavior policy.AWR and AWAC significantly outperform BEAR and BRAC in the cited suboptimal-data fine-tuning comparison.

B. Importance Sampling

Importance-sampling methods estimate policy values or gradients from behavior-policy data by weighting observed trajectories or decisions. Per-decision, doubly robust, and marginalized estimators reduce variance or avoid exponentially long products, but no single approach is consistently sufficient across settings.

  • Variance Reduction: Per-decision importance sampling rewrites trajectory-level policy gradients so each reward is weighted only by preceding importance ratios.
  • Variance Reduction: Per-decision weighting gives an unbiased gradient estimator with lower variance, while self-normalized weights trade additional bias for a large variance reduction.
  • Variance Reduction: Doubly robust estimation uses Q-function estimates as control variates and remains unbiased if either the behavior policy is known or the Q-function is correctly estimated.Importance-weight products remain exponential in the horizon for these estimators.
  • Marginalized Estimation: Marginalized importance sampling estimates state-marginal ratios directly, replacing long products of action ratios and reducing policy-gradient variance.The state-marginal ratio satisfies ρπθ(st)=w0:t under the stated cancellation of policy-independent terms.
  • Marginalized Estimation: State-marginal importance ratios can be estimated through temporal-difference updates using dataset samples without estimating the behavior-state distribution explicitly.
  • Marginalized Estimation: GenDICE extends marginalized importance sampling to state-action marginal ratios and optimizes a residual error subject to a normalization constraint.The method minimizes a divergence between the two sides of its modified Bellman equation.

C. Regularization

Regularization modifies policy or value objectives to improve stability and control errors from actions outside the dataset. The survey covers entropy, value, and uncertainty regularization, including methods whose advantages and limitations depend on task coverage and model uncertainty.

  • Policy Regularization: Entropy regularization controls optimal-policy stochasticity and can improve robustness and training stability by avoiding premature convergence of policy variance.Increasing its weight produces a more stochastic policy.
  • Value Regularization: Value regularization pushes down Q-values for learned-policy actions to limit overestimation of OOD actions.
  • Value Regularization: CQL learns a lower bound on the true Q-function by raising values for dataset actions and lowering values for unseen actions.For all states in the dataset, the survey reports VCQL(s) ≤ V(s).
  • Value Regularization: CQL’s saddle-point optimization can be unstable in practice because it simultaneously minimizes and maximizes the Q-function.
  • Value Regularization: CQL typically outperforms policy-constraint methods on challenging D4RL AntMaze and Kitchen tasks that require stitching together suboptimal behavior.
  • Uncertainty Regularization: Naive Q-function ensembles can underestimate uncertainty because disjoint-data training may produce little diversity, increasing susceptibility to OOD actions.
  • Uncertainty Regularization: REM can work well on high-coverage datasets relative to standard DQN, but still lags behind policy-constraint approaches.

E. Model-Based Methods

Model-based offline RL methods learn or use environment models to address distributional shift, often through uncertainty estimation, pessimism, or behavior regularization. The surveyed methods differ in how they constrain model rollouts and policy updates.

  • Uncertainty-aware methods: MOReL estimates epistemic uncertainty with an ensemble of dynamics models and penalizes decisions associated with model disagreement.Its uncertainty measure is used to constrain the learned model.
  • Uncertainty-aware methods: MOPO uses the maximum prediction uncertainty from an ensemble of Gaussian transition models to modify model-based offline RL.The ensemble models each transition with a mean and covariance, and uncertainty is computed from their disagreement.
  • Behavior regularization: BREMEN learns an ensemble of behavior-MDP models and implicitly constrains policy updates toward the behavior policy through trust-region updates.It initializes the training policy from an estimated behavior policy and generates rollouts using sampled ensemble models.
  • Conservative methods: COMBO combines a learned single-dynamics model with conservative value estimation, pushing down Q-values on model-rollout state-action tuples.The dataset-to-model-rollout sampling ratio f controls conservatism: larger f yields more conservative Q estimates.
  • Conservative methods: COMBO removes the need for uncertainty quantification by making model-generated states and actions unattractive through conservative regularization.This addresses the challenge that uncertainty quantification can be difficult and unreliable.

F. One-Step Methods

One-step methods simplify offline RL by using weak regularization and avoiding iterative policy improvement, while IQL retains iterative dynamic programming through a different value target. Their relative performance depends on dataset composition and coverage.

  • One-step framework: One-step methods can outperform multistep and iterative methods in several OpenAI Gym environments.The authors attribute this success mainly to weak regularization that lets function approximators fit the true Q-function more freely.
  • One-step framework: Increasing regularization improves multistep stability but produces overly conservative policies, whereas weak regularization can cause divergence.Multistep methods perform best at the smallest regularization weight that avoids divergence.
  • Limitations: One-step methods underperform multistep methods when datasets contain mostly suboptimal behavior or provide good state-action coverage.In these settings, multistep methods generally fit better policies and suffer less from iterative error exploitation.
  • Implicit Q-learning: IQL evaluates policies using a value-function approximator as the target instead of target actions sampled from the behavior policy.Its expectile regression objective estimates a value associated with strong actions within the dataset support.
  • Implicit Q-learning: IQL differs from the proposed one-step methods by performing iterative dynamic programming.Its value updates use Bellman optimality equations to improve the behavioral policy.

G. Imitation Learning

Imitation-learning methods filter or weight dataset actions and then apply supervised policy learning, while related approaches use learned behavior priors or conditioning variables. Their performance depends strongly on how suboptimal behavior and task structure are represented.

  • Best-action imitation learning: BAIL fits a value function, selects high-return state-action pairs from the dataset, and trains a policy by behavior cloning.It filters pairs whose returns exceed a chosen ratio of the estimated value function, using ρ = 0.25.
  • Behavior priors: ABM learns an advantage-weighted behavior prior before applying maximum a posteriori policy optimization to extract an improved policy.The prior is used as a KL-constrained reference during policy improvement.
  • Critic regularized regression: CRR aggressively filters below-average actions with an indicator function and uses a pessimistic advantage estimator.The approach is reported to outperform implicit policy-constraint methods on tasks mixing expert and suboptimal behavior.
  • Conditional policies: RvS learns goal-conditioned and reward-conditioned policies within a supervised-learning framework.The conditioning variable is crucial: goal conditioning performs well in AntMaze and FrankaKitchen, whereas reward conditioning performs poorly there but works well in Gym-MuJoCo.
  • Conditional policies: In some domains, RvS with increased network capacity and regularization matches or exceeds more complex methods such as CQL.This result is reported for the simple conditional-policy formulation.

H. Trajectory Optimization

Trajectory optimization methods model or predict return-conditioned trajectories with transformers, while offline policy evaluation methods estimate policy quality without new interaction. The survey emphasizes that reliable evaluation remains essential because real-world rollouts are often unavailable.

  • Trajectory Transformer: Trajectory Transformer models trajectories as sequences of returns-to-go, states, and actions, then uses beam search with return estimates for planning.Training maximizes the likelihood of each sequence token conditioned on preceding tokens.
  • Decision Transformer: Decision Transformer uses a similar return-conditioned trajectory representation but trains by minimizing mean squared error for predicted actions.Its planning strategy differs from Trajectory Transformer’s token-likelihood and beam-search procedure.
  • Offline policy evaluation: Model-based OPE fits dynamics and reward models from an evaluation dataset, then computes expected return under the trajectory distribution induced by the policy.This evaluates the policy without collecting new environment interactions.
  • Offline policy evaluation: Importance sampling estimates policy return by fitting the behavior policy and evaluating an importance-weighted objective.Weighted, doubly robust, and marginalized variants can reduce importance-sampling variance.
  • Offline policy evaluation: FQE performs surprisingly well despite its simplicity, and it has the best overall performance among OPE methods on benchmarks such as RL Unplugged and D4RL.Its use has also been reported as sufficient for offline hyperparameter selection when combined with policy constraints and value ranking.
  • Practical challenges: Reliable OPE is essential because real-world policies often cannot be rolled out for evaluation, leaving simulator reliance as a practical barrier.The survey therefore stresses OPE methods that work across diverse datasets.

A. Offline RL Benchmarks

Offline RL benchmarks aim to measure progress under realistic dataset properties, but current benchmarks still leave important real-world settings underrepresented. The section compares D4RL, RL Unplugged, and DOPE while highlighting limitations in behavior policies, evaluation protocols, and coverage.

  • Desired dataset properties: Offline datasets should include narrow and biased distributions, undirected and multitask data, and nonstationarity to reflect realistic applications.These properties test robustness to out-of-distribution states, trajectory stitching, and changing environments.
  • Benchmark coverage: D4RL and RL Unplugged together cover many task domains, including locomotion, mazes, manipulation, autonomous driving, and traffic simulation.D4RL includes Gym-MuJoCo, Adroit, FrankaKitchen, CARLA, and Flow; RL Unplugged adds a wide variety of tasks.
  • Benchmark coverage: RL Unplugged behavior policies are trained online, whereas D4RL includes deliberately nonrepresentable policies such as non-Markovian data sources.This makes RL Unplugged less guaranteed to represent realistic behavior-policy complexity.
  • Evaluation protocols: D4RL does not impose a particular evaluation protocol, while RL Unplugged separates online and offline validation environments.Offline validation requires off-policy evaluation for assessment and hyperparameter tuning, whereas online validation uses additional online samples.
  • Remaining gaps: Current benchmarks insufficiently cover stochastic dynamics, nonstationarity, and datasets designed to contain risky biases.These gaps are especially relevant to real-world domains such as economics, healthcare, and education.
  • OPE benchmarks: DOPE evaluates offline policy evaluation across tasks with varied difficulty and compares six baseline methods, including FQE, model-based, IS, DR, DICE, and VPM.The benchmark supports both absolute policy evaluation and policy-selection metrics such as absolute error, Regret@k, and rank correlation.

C. Method Performance

The survey compares offline RL methods and taxonomy classes across dataset properties using D4RL results. Recent methods generally perform better, while trajectory optimization is especially strong for sparse-reward and multitask data; evaluation and benchmark coverage remain open problems.

  • Method performance: Recent offline RL methods tend to outperform older methods across datasets, with TT and implicit Q-learning among the best-performing methods.Methods in the comparison are ordered from left to right by release date.
  • Performance evaluation: The performance comparison uses relative D4RL scores across dataset properties, with taxonomy-class scores aggregated from the best method in each class.Scores are based on average normalized D4RL results; brighter heatmap colors indicate higher relative performance.
  • Taxonomy-class performance: Trajectory optimization performs particularly well on sparse-reward and undirected multitask datasets.The survey identifies this result as evidence that planning can be compelling in offline RL, especially when combined with Q-functions trained by dynamic programming.
  • Open problems: Hyperparameter tuning and OPE still lack satisfying solutions because inaccurate evaluation or fixed training durations can leave policies suboptimal or overfit.The survey highlights policy validation and early stopping as unresolved needs.
  • Future directions: Unsupervised RL may extend offline RL by leveraging large unlabeled datasets, while incremental RL targets nonstationary data and offline-to-online fine-tuning.Reported examples combine suboptimal unlabeled data with limited high-quality labels or downstream reward relabeling.
Loading 2203.01387v3…