PaperlayerView exact revision on arXiv

Source-linked AI summary

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, Zhaoran Wang

arXiv:2202.11566v1cs.LG

TL;DR

Offline RL must learn without new environment interaction, but distributional shift makes value estimates unreliable for out-of-distribution actions. PBRL uses bootstrapped uncertainty, pessimistic value updates, and OOD sampling; experiments report stronger performance than several baselines, while neural-network OOD regularization remains a limitation.

  • Problem

    Offline RL suffers extrapolation errors when value functions evaluate out-of-distribution actions that are poorly covered by the dataset.

  • Method

    PBRL quantifies epistemic uncertainty with bootstrapped Q-functions, penalizes value estimates pessimistically, and samples OOD state-action datapoints to regularize learning.

  • Results

    PBRL outperforms several strong offline RL baselines on D4RL environments and produces reliable uncertainty quantifications with OOD sampling.

  • Takeaways & Limitations

    PBRL exploits optimal trajectories in suboptimal datasets and is less affected by the behavior policy than the reported policy-constraint baselines.

  • Takeaways & Limitations

    L2-regularization is not sufficient to regularize the OOD behavior of neural networks, unlike its role in linear value-function methods.

Abstract

from arXiv · show

Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolation error caused by the out-of-distribution (OOD) actions. Previous methods tackle such problem by penalizing the Q-values of OOD actions or constraining the trained policy to be close to the behavior policy. Nevertheless, such methods typically prevent the generalization of value functions beyond the offline data and also lack precise characterization of OOD data. In this paper, we propose Pessimistic Bootstrapping for offline RL (PBRL), a purely uncertainty-driven offline algorithm without explicit policy constraints. Specifically, PBRL conducts uncertainty quantification via the disagreement of bootstrapped Q-functions, and performs pessimistic updates by penalizing the value function based on the estimated uncertainty. To tackle the extrapolating error, we further propose a novel OOD sampling method. We show that such OOD sampling and pessimistic bootstrapping yields provable uncertainty quantifier in linear MDPs, thus providing the theoretical underpinning for PBRL. Extensive experiments on D4RL benchmark show that PBRL has better performance compared to the state-of-the-art algorithms.

1 INTRODUCTION

Offline RL avoids costly or dangerous environment interaction but suffers from distributional shift and extrapolation error. PBRL addresses these challenges with uncertainty-driven pessimism and novel OOD sampling.

  • Motivation: Offline RL learns from fixed datasets, avoiding the costly or dangerous interactions required by many successful DRL applications.The motivation includes navigation and healthcare settings where collecting samples under a possibly suboptimal policy can be costly and dangerous.
  • Motivation: Directly adapting off-policy algorithms to offline RL often causes performance drops because learned policies and behavior policies induce different data distributions.This distributional shift makes policy evaluation difficult and contributes to extrapolation error.
  • Prior approaches: Policy-constraint methods keep learned policies near behavior policies, whereas conservative methods penalize Q-functions for out-of-distribution actions.These are the two main categories of prior approaches described for addressing distributional shift.
  • PBRL: PBRL combines bootstrapped Q-functions for uncertainty quantification, uncertainty-based pessimistic updates, and OOD sampling that regularizes learned Q-functions.The method adds OOD datapoints containing training-buffer states, current-policy actions, and uncertainty-aware targets.
  • PBRL: PBRL assigns smaller penalties to near-support actions with low epistemic uncertainty, allowing high-value actions that policy-constraint and conservative methods may avoid.Experiments on D4RL report reasonable uncertainty quantification and better performance than state-of-the-art algorithms.

2 PRELIMINARIES

Offline RL evaluates policies using only an existing dataset, but distributional shift makes greedy target actions potentially out of distribution. The resulting extrapolation errors can propagate and destabilize training.

  • MDP formulation: An episodic MDP is defined by state space, action space, episode length, reward function, and transition distribution, with RL seeking a policy maximizing expected cumulative reward.The formulation includes a discount factor for episodic settings.
  • Online versus offline RL: In online RL, transitions are collected through environment interaction and used to train the Q-network.Training samples consist of state, action, reward, and next-state transitions.
  • Online versus offline RL: In offline RL, the agent cannot interact with the environment and instead samples experiences from a fixed offline dataset.This restriction prevents collecting new transitions during training.
  • Distributional shift: Distributional shift occurs when the learned policy visits state-action pairs differently from the behavior policy represented in the dataset.A greedy action selected by the target Q-network may be scarcely covered by the offline data.
  • Distributional shift: Out-of-distribution actions produce extrapolation errors in value estimates, which can amplify through repeated propagation and potentially cause divergence.PBRL addresses this challenge using uncertainty quantification and OOD sampling.

3 PESSIMISTIC BOOTSTRAPPING FOR OFFLINE RL

PBRL quantifies epistemic uncertainty with bootstrapped Q-functions, uses uncertainty-based pessimistic updates, and directly samples OOD state-action pairs for regularization. Its theoretical analysis connects this uncertainty to LCB penalties under linear MDP assumptions.

  • Uncertainty quantification: PBRL maintains an ensemble of bootstrapped Q-functions whose disagreement estimates epistemic uncertainty, especially where offline data are scarce.Ensemble members tend to agree in data-rich regions and diverge in poorly covered regions.
  • Uncertainty quantification: The uncertainty estimate measures how far state-action pairs deviate from offline-data support, enabling more refined conservatism than uniform OOD penalties.Low uncertainty corresponds to state-action pairs aligned with the offline-data support.
  • Pessimistic learning: PBRL penalizes in-distribution targets using next-state-action uncertainty and constructs OOD pseudo-targets with additional uncertainty penalties.The OOD penalty is smaller for OOD samples near the offline-data support and larger for distant samples.
  • Pessimistic learning: The critic minimizes both regular TD-error on offline data and pseudo TD-error on sampled OOD data, producing a smooth pessimistic value function.The two losses are combined during Q-function updates, while policy training uses the pessimistic ensemble values.
  • OOD regularization: OOD sampling directly regularizes neural-network value functions because L2-regularization alone is insufficient to control their OOD behavior.Under linear MDP assumptions, adding OOD datapoints supplies a covariate-matrix regularization analogous to the λ·I prior in LSVI.
  • Theoretical connections: In linear MDPs, bootstrapped uncertainty estimates the LCB penalty and can form a valid uncertainty quantifier under the stated theoretical construction.The connection is framed through an appropriately selected tuning parameter and the theorem’s OOD target.

4 RELATED WORKS

Related offline RL methods address distributional shift through policy constraints, conservative penalties, uncertainty-aware model-based methods, or theoretical assumptions. PBRL is situated among uncertainty-based approaches while emphasizing the distinct challenges of offline coverage.

  • Model-free offline RL: Model-free offline RL methods commonly constrain the learned policy with behavior cloning, divergence penalties, or advantage-weighted objectives.These approaches restrict policy behavior to reduce exposure to OOD actions.
  • Model-based offline RL: Model-based methods such as MOPO and MOReL use ensembles of learned dynamics models for uncertainty quantification and pessimistic value updates.Other model-based approaches additionally use behavior-cloning, advantage-weighted, conservative, or geometric policy constraints.
  • Uncertainty quantification: Uncertainty-based online RL methods include bootstrapped Q-networks, ensemble dynamics, Bayesian neural networks, and distributional value functions.The paper notes that uncertainty quantification is more challenging offline because the dataset provides limited coverage.
  • Theoretical foundations: Theoretical offline RL work studies provable efficiency under linear MDP and Bellman-completeness assumptions, which motivate PBRL’s analysis.PBRL follows the linear-MDP framework used in prior online and offline RL theory.

5 EXPERIMENTS

Experiments evaluate PBRL and PBRL-prior against offline-RL baselines on D4RL, showing strong performance, especially on non-optimal datasets, while uncertainty estimates distinguish in-distribution from OOD actions.

  • Experimental setup: PBRL and PBRL-prior are evaluated with SAC on D4RL Gym and Adroit tasks alongside policy-constraint, uncertainty-weighted, conservative, and model-based baselines.PBRL-prior adds fixed random prior networks to bootstrapped Q-functions, while both methods use SAC as their actor-critic architecture.
  • Gym results: In Gym, PBRL outperforms baselines on most of the 15 task setups, while PBRL-prior is slightly better and more stable across seeds.The evaluation uses normalized scores after one million training steps on the latest D4RL v2 datasets.
  • Gym results: PBRL has strong advantages on medium, medium-replay, and medium-expert datasets, where it is less affected by the behavior policy and exploits optimal trajectories in suboptimal data.Its early training performance is weaker than TD3-BC and CQL, but it later converges to better policies as uncertainty quantifiers improve.
  • Adroit results: In Adroit, PBRL outperforms baselines on most tasks, although CQL and BC achieve the best average performance among baselines.The Adroit setting is more challenging because of task complexity and human demonstrations in the offline data.
  • Uncertainty quantification: PBRL assigns high uncertainty to OOD actions, lowest uncertainty to offline actions, and smoothly increasing uncertainty between in-distribution and OOD actions.This pattern is observed for Walker2d across medium-replay and medium datasets.
  • Ablation study: Ablations find reasonable performance for K ≥6 bootstrapped Q-functions, best βin performance in [0.0001, 0.01], and better cross-task generalization from decaying βood.Using a zero target for OOD actions instead produces overly pessimistic value functions with suboptimal performance.

6 CONCLUSION

The paper proposes PBRL as an uncertainty-based model-free offline-RL algorithm combining bootstrapped uncertainty, provable pessimism, and OOD sampling. Experiments show strong D4RL performance and reliable uncertainty quantification.

  • Conclusion: PBRL combines bootstrapped uncertainty with provably efficient pessimism and a novel OOD sampling technique to regularize OOD actions.The method is closely related to provably efficient offline RL under the linear MDP assumption.
  • Conclusion: PBRL outperforms several strong offline-RL baselines in D4RL environments and is less affected by the behavior policy when exploiting optimal trajectories in suboptimal datasets.The paper also reports reliable uncertainty quantifications when OOD sampling is incorporated.

A THEORETICAL PROOF

The theoretical analysis connects PBRL's bootstrapped uncertainty to established uncertainty penalties in linear and tabular MDPs. It derives these connections through LSVI, Bayesian linear regression, and state-action feature counts.

  • Linear MDP formulation: Under linear MDPs, state-action features, transitions, rewards, and Q-functions are represented linearly, enabling linear value estimation.The analysis considers γ = 1 and uses a feature map φ over state-action pairs.
  • LSVI estimation: LSVI estimates Q-functions by solving for wt from a least-squares objective whose targets use the next-step estimated value function and rewards.The resulting estimate has the form Qt(st, at) ≈ bwt^T φ(st, at), with Λt accumulating training-buffer features.
  • LCB pessimism: The LCB penalty measures Q-function uncertainty from training data and is used to construct a pessimistic value function by penalizing Qt.Under linear MDPs, this pessimistic value iteration is information-theoretically optimal, while using the same quantity as an exploration bonus is provably efficient online.
  • Bayesian interpretation: A Bayesian linear-regression view uses Gaussian noise and a Gaussian prior to derive the posterior of the Q-function parameters from the offline dataset.The posterior analysis applies Bayes' rule to connect the LSVI objective with uncertainty quantification.
  • Bootstrapped uncertainty: The standard deviation of the bootstrapped Q-functions estimates the posterior uncertainty and is equivalent to the LCB penalty under the linear MDP assumptions.Bootstrapping therefore provides an empirical approximation to the theoretically motivated LCB uncertainty measure.
  • Tabular specialization: In tabular MDPs, bootstrapped uncertainty is approximately proportional to the reciprocal count of each state-action pair.One-hot state-action features make the covariance matrix diagonal, with entries determined by state-action visitation counts and regularization.

A.3 REGULARIZATION WITH OOD SAMPLING

OOD sampling regularizes Q-function extrapolation by adding synthetic state-action targets, while its construction supports theoretical uncertainty guarantees under linear MDP assumptions. The analysis also identifies target-selection trade-offs and practical sampling choices.

  • Regularization mechanism: OOD sampling regularizes estimated Q-functions on state-action pairs outside the offline dataset.The method uses OOD samples to regularize extrapolation behavior without relying on the L2 regularizer.
  • Target design: Setting OOD targets to zero imposes a Gaussian prior through the covariance matrix Λ^-1 but can make Q-functions overly pessimistic.Because the Q-network is smooth, the strong regularizer can drive values toward zero on both offline and OOD data.
  • Theoretical characterization: With coordinate-aligned OOD features, the OOD covariance becomes Λ_ood = λ · I, and the resulting objective is equivalent to the paper’s regularized formulation.In the tabular setting, OOD samples also correspond to modifying the state-action count used in the analysis.
  • Target design: Nonzero OOD targets preserve regularization but may bias value estimation and uncertainty quantification, requiring careful target design.The target must maintain consistent pessimistic value estimates while avoiding additional bias.
  • Theoretical characterization: Under linear MDPs, choosing the OOD target as T V_t+1(s_ood, a_ood) makes ensemble disagreement a valid ξ-uncertainty quantifier.The guarantee holds with probability at least 1 − ξ for all state-action pairs under the stated assumptions.
  • Practical target: The theoretically ideal OOD target is unavailable in practice, so PBRL uses a pessimistically penalized estimate based on the next Q-function and uncertainty.The practical target is y = Q_t+1(s_ood, a_ood) − Γ_t+1(s_ood, a_ood).
  • Practical target: The theory requires isotropic OOD embeddings, while experiments find that randomly generating actions alone provides reasonable empirical performance.Randomly generating states is described as more expensive than randomly generating actions.

B IMPLEMENTATION DETAIL

PBRL trains bootstrapped critics on both offline and OOD data, updates the actor using ensemble pessimism, and uses SAC-style settings with additional computational cost from ensembles.

  • Algorithm: PBRL initializes K bootstrapped Q-networks, target networks, a policy, and separate in-distribution and OOD penalty coefficients.Training alternates between offline critic updates, OOD critic updates, actor updates, and target-network smoothing.
  • Algorithm: Offline critic training estimates bootstrapped uncertainty from target networks and minimizes the resulting temporal-difference loss.The offline-data update uses transitions sampled from D_in.
  • Algorithm: OOD critic training samples OOD actions for each state, computes ensemble uncertainty, and optimizes a pseudo TD-loss using pessimistic targets.This explicitly incorporates OOD samples into critic training.
  • Algorithm: Actor training maximizes the minimum ensemble Q-value with entropy regularization, while target Q-networks are updated by exponential smoothing.The target update uses θ^- ← (1 − τ)θ^- + τθ.
  • Hyperparameters: PBRL follows SAC implementations for most hyperparameters and uses CQL settings for discounting, target smoothing, learning rates, and optimizers.Adroit uses different β_in and β_ood settings while retaining the other listed settings.
  • Experimental setup: The experiments use D4RL v2 datasets and an official CQL implementation, removing CQL’s behavior-cloning warm-up because it performed better without warm-up.The baseline implementations are specified as part of the experimental setup.
  • Computational cost: PBRL incurs extra computation relative to CQL because its ensemble-based design requires additional parameters, GPU memory, and runtime.The comparison measures these costs per epoch on Halfcheetah-medium-v2 using a single A100 GPU.
  • Pessimism design: Critic training uses separate β_in and β_ood factors, whereas actor training uses the minimum ensemble Q-function to avoid unusually large fitted values.The minimum is approximately related to a fixed standard-deviation penalty and is supported by the ablation discussion.

C ABLATION STUDY

The ablations examine how bootstrapping, uncertainty penalties, OOD sampling, and target choices affect PBRL. Results favor sufficient Q-function ensembles, next-Q penalization, decaying OOD regularization, and nonzero OOD targets.

  • Number of bootstrapped-Q: PBRL performance improves with more bootstrapped Q-functions, with K = 6, 8, and 10 yielding similar final scores.The implementation adopts K = 10.
  • Uncertainty of in-distribution target: Penalizing the next-Q value performs best among penalizing the reward, next-Q, or both for in-distribution targets.PBRL therefore adopts target-Q penalization.
  • Tuning parameter βood: A decaying βood strategy performs well across medium and medium-replay datasets, avoiding the excessive pessimism caused by a large constant βood.Large constant βood can make in-distribution Q-values overly pessimistic and produce sub-optimal performance.
  • Actor training: Actor training performs best when maximizing the minimum value among ensemble Q-functions rather than their mean or maximum.Using the mean can be unstable on medium-replay, while the maximum performs worse on both tasks.
  • Number of OOD samples: OOD sampling is necessary for strong performance: Nood = 0 performs poorly, while a small number of OOD actions improves results and performance remains robust to Nood settings.The ablation considers Nood ∈ {0, 2, 5, 10}.
  • OOD Target: Setting the OOD target to zero makes value functions overly pessimistic, whereas PBRL’s target performs well without large extrapolation errors.The zero-target setup drives Q-values close to zero for both in-distribution and OOD state-action pairs.

D REGULARIZATION FOR PBRL

The regularization study compares OOD sampling with alternative regularizers for controlling extrapolation in offline RL. OOD sampling is the only tested method with reasonable performance, although pessimistic initialization and spectral normalization may benefit from further tuning.

  • Compared regularizers: The study compares no regularization, L2 regularization, spectral normalization, pessimistic initialization, and OOD sampling.The experiments evaluate normalized performance, Q-values, and uncertainty along training-policy trajectories.
  • Empirical comparison: OOD sampling is the only tested regularization method with reasonable performance, whereas L2 regularization and spectral normalization perform poorly in offline RL.These methods can work well in supervised learning but do not transfer effectively here.
  • Empirical comparison: Pessimistic initialization and spectral normalization can score early on medium-replay, but their performance drops quickly during training.The authors conjecture that additional parameter tuning and algorithm design may make them effective.
  • Conclusion: The study concludes that prior deep-learning and reinforcement-learning regularizers are insufficient for distribution shift in offline RL alone.Combining them with policy constraints and conservatism methods is left for future research.

E EXPERIMENTS IN ADROIT DOMAIN

The Adroit experiments evaluate offline RL on challenging robotic-hand tasks using human, expert, and cloned datasets. PBRL slightly outperforms CQL and behavioral cloning in the reported average excluding expert data, while PBRL and CQL outperform the best baselines overall.

  • Adroit domain: Adroit tasks control a 24-DoF robotic hand across hammer, door-opening, pen-twirling, and ball-moving tasks.The domain includes human demonstrations, expert RL-policy data, and cloned mixed data.
  • Dataset characteristics: Human and cloned datasets are challenging because human trajectories may violate the Markov property, reflect unobserved information, and vary across demonstrators.These characteristics distinguish them from machine-generated expert data.
  • Results: PBRL slightly outperforms CQL and behavioral cloning on the average score excluding the expert dataset.CQL and behavioral cloning have the best average performance among the baselines.
  • Results: PBRL and CQL outperform the best baselines in the Adroit average normalized-score comparison.The table reports averages over 3 seeds.

F RELIABLE EVALUATION FOR STATISTICAL UNCERTAINTY

The paper uses robust aggregate and distributional metrics with bootstrap confidence intervals to address statistical uncertainty in offline RL evaluation. Across both evaluation views, PBRL and PBRL-prior outperform other methods with small variability.

  • Reliable evaluation: The evaluation uses stratified bootstrap confidence intervals, performance profiles, and aggregate metrics including mean, median, IQM, and optimality gap.IQM averages the middle 50% of runs, while optimality gap counts runs below η = 50.0.
  • Results: PBRL and PBRL-prior outperform other methods in both aggregate-metric and performance-profile evaluations with small variability.The comparisons are reported in Figures 18 and 19.
  • Aggregate metrics: Figure 18 reports aggregate metrics from 15 tasks and 5 seeds per task, with higher mean, median, and IQM and lower optimality gap preferred.The figure uses 95% confidence intervals estimated by stratified percentile bootstrap.
  • Performance profiles: Figure 19 evaluates score distributions and average score distributions using pointwise 95% confidence bands.The median corresponds to the profile’s intersection with y = 0.5, and area under the profile corresponds to the mean.
Loading 2202.11566v1…