PaperlayerView exact revision on arXiv

Source-linked AI summary

Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Aviral Kumar, Justin Fu, George Tucker, Sergey Levine

arXiv:1906.00949v2cs.LGstat.ML

TL;DR

Off-policy RL struggles to learn robustly from fixed datasets because Bellman backups can bootstrap from actions outside the training distribution. The paper analyzes this error, constrains backup actions using the data distribution, and proposes BEAR, which learns across random, medium-quality, and expert datasets while retaining limitations during long runs and under mixture data.

  • Problem

    Practical off-policy RL methods can fail on fixed datasets because Q-learning backups evaluate actions outside the training distribution, producing accumulating errors without new data.

  • Method

    The paper analyzes bootstrapping error and develops BEAR, which constrains backup actions to have non-negligible support under the training distribution using an approximate MMD constraint.

  • Results

    BEAR-QL learns well across random, medium-quality, and expert data, and consistently outperforms naïve off-policy RL across a range of tasks and dataset compositions.

  • Takeaways & Limitations

    Constraining action selection in Bellman backups can reduce bootstrapping-error accumulation and support robust learning from static off-policy datasets.

  • Takeaways & Limitations

    Performance can still degrade during long learning runs, and constrained-action selection may be overly conservative for datasets collected from mixtures of policies.

Abstract

from arXiv · show

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.

1 Introduction

Off-policy RL could use static datasets, but practical methods remain sensitive to data distribution and may become unstable when Bellman backups evaluate out-of-distribution actions.

  • Static datasets are plentiful in domains such as autonomous driving and recommender systems, but RL commonly depends on expensive active data collection.
  • Off-policy algorithms can perform sample-efficiently on complex tasks yet may fail on arbitrary fixed datasets, even when data comes from effective expert policies.
  • Bootstrapping from out-of-distribution actions can introduce Q-function errors that accumulate through Bellman backups, causing unstable or divergent training without new data.
  • The paper analyzes this error accumulation and develops a practical way to constrain backups in off-policy, value-based RL.

2 Related Work

The paper builds on error-propagation analyses and prior action-constrained methods to address instability from previously unseen actions, while targeting stronger performance under approximation errors and suboptimal data.

  • Approximate dynamic programming studies how Bellman errors accumulate and propagate through bootstrapping under sampling, distributional-shift, and function-approximation errors.
  • Prior work constrains policy updates or uses learned action priors, motivating distribution-aware restrictions for off-policy learning.
  • BEAR extends action-constrained backup analysis beyond tabular error-free settings by providing results under approximation errors and a suboptimality bound.
  • BEAR outperforms BCQ on suboptimal-policy data and achieves stronger, more consistent results across datasets and environments, while BCQ can remain too conservative.

3 Background

The paper frames RL as finding a policy that maximizes discounted return in an MDP, then describes Q-learning, function approximation, actor-critic methods, and static off-policy datasets.

  • An MDP is specified by state and action spaces, transitions, rewards, an initial-state distribution, and a discount factor.
  • RL seeks a policy maximizing expected cumulative discounted rewards, or return, with µπ(s) denoting its discounted state marginal.
  • Q-learning estimates the optimal state-action value function, from which an optimal policy selects the maximizing action.
  • With function approximation, Q-iteration projects Bellman targets into a hypothesis class by minimizing mean squared Bellman error under a behavior-policy occupancy measure.
  • For large action spaces, actor-critic methods learn a policy that maximizes the Q-function; here, learning uses static transitions collected under an unknown behavior policy.

4 Out-of-Distribution Actions in Q-Learning

Naïve Q-learning on static off-policy data can evaluate actions outside the training distribution, producing bootstrapping errors that accumulate through Bellman backups. Distribution-constrained backups trade off reduced error propagation against suboptimality bias by restricting backup policies to supported actions.

  • Bootstrapping error: Naïve Bellman backups maximize learned Q-values over out-of-distribution actions, where the estimator is unreliable and can produce pathological values.These errors accumulate through bootstrapping and are not necessarily fixed by increasing the static dataset size.
  • Bootstrapping error: Q-learning error combines current Bellman error with discounted errors propagated from successor state-action pairs, making OOD regions a source of instability.Errors on OOD state-action pairs are high because training does not directly minimize them.
  • Distribution-constrained backups: A distribution-constrained backup restricts maximization to policies supported where the behavior policy assigns probability above a threshold, avoiding OOD action evaluation without fixing action probabilities.This is less restrictive than forcing the learned policy to remain close to the behavior policy.
  • Distribution-constrained backups: Theoretical analysis separates suboptimality bias from distribution shift and bounds performance using the constraint set’s suboptimality and concentrability properties.The concentrability coefficient measures how far visitation distributions from constrained policies are from the training distribution.
  • Distribution-constrained backups: Expanding the constrained policy set increases concentrability while decreasing suboptimality, creating a tradeoff that careful support-set selection can balance.The gridworld visualization contrasts unconstrained backups, behavior-policy evaluation, and an intermediate distribution-constrained backup.
  • Support-set construction: Support-based policy sets provide a single threshold that simultaneously trades off concentrability and suboptimality, motivating an effective practical construction.The paper reports that this construction provides theoretical guarantees and a simple implementation approach.

5 Bootstrapping Error Accumulation Reduction (BEAR)

BEAR reduces bootstrapping-error accumulation by constraining backup and actor actions to the training distribution’s support while using conservative Q-value estimates. It approximates this constraint with sampled MMD and policy optimization over supported actions.

  • Core approach: BEAR uses distribution-constrained backups to reduce accumulation of bootstrapping error.The method searches for policies sharing the training distribution’s support while limiting accidental error accumulation.
  • Core approach: BEAR-QL combines K Q-functions, minimum-Q policy improvement, and a support constraint over the policy set Πϵ.These components modify policy improvement to favor conservative estimates within the supported policy class.
  • Support constraint: Sampled MMD constrains the actor to the unknown behavior policy using only action samples, without requiring either distribution’s density.Universal kernels, including Laplacian and Gaussian kernels, work in the experiments.
  • Implementation: The policy improvement constraint uses an approximately chosen threshold ε = 0.05 in the experiments.The threshold appears in the constrained optimization problem used for actor updates.
  • Backup computation: The practical algorithm approximates distribution-constrained backups by sampling actions in the behavior-policy support and empirically maximizing over them.Exact maximization over Πϵ is intractable in practice; the algorithm instead uses sampled supported actions.

6 Experiments

BEAR is evaluated on static datasets generated by random, medium-quality, and optimal policies across continuous-control tasks. It consistently performs well across dataset compositions, including settings where competing methods fail or imitate the data.

  • Performance on Medium-Quality Data: BEAR-QL consistently outperforms BCQ and naïve off-policy RL by large margins on one-million-transition medium-quality datasets.This setting represents imperfect demonstrations or experience from a mediocre prior policy.
  • Performance on Random and Optimal Datasets: BEAR achieves good results on both random and near-optimal data, exceeding average random-dataset return and matching optimal-policy return.The comparisons average performance over 5 seeds.
  • Performance on Random and Optimal Datasets: BEAR-QL is the only evaluated algorithm capable of learning in both random and optimal-data scenarios.Naïve RL cannot handle optimal data, while BCQ fails on random data because its constraint is too strict.
  • Performance on Random and Optimal Datasets: BEAR-QL is robust to dataset composition, whereas KL-control and DQfD can be unstable in the random and optimal-data settings.The results support the paper’s hypothesis that out-of-distribution actions are a major error source in static-dataset off-policy learning.
  • Additional benchmark: BEAR outperforms the considered prior methods in Humanoid-v2 on medium-quality and random data.This result extends the comparison to a challenging continuous-control environment.

7 Discussion and Future Work

The paper identifies bootstrapping from out-of-distribution actions as a major source of off-policy instability and introduces BEAR-QL to constrain backups to supported actions. BEAR-QL performs well across tasks and dataset qualities, including Humanoid-v2, though long training runs and action constraints remain limitations.

  • Results: BEAR-QL outperforms prior methods on Humanoid-v2 using both medium-quality and random data.The comparison includes BEAR-QL, BCQ, naïve RL, and behavioral cloning.
  • Findings: Bootstrapping from out-of-distribution actions causes error to accumulate through Bellman backups in static off-policy reinforcement learning.The experiments identify this source of error as a primary issue afflicting off-policy RL.
  • Method: BEAR-QL constrains backup actions to have non-negligible support under the data distribution without imposing an overly conservative constraint on the learned policy.This design targets bootstrapping error while retaining broader policy selection within the supported action set.
  • Results: BEAR-QL learns well from random, medium-quality, and expert datasets across a range of continuous-control tasks.The reported robustness spans different dataset compositions rather than a single data quality level.
  • Limitations: Performance can still degrade during long learning runs, and constrained-action selection may be overly conservative for datasets collected from mixtures of policies.The paper proposes early stopping and direct state-distribution constraints as future directions.

Appendices

The constrained Bellman backup can be interpreted as an ordinary Bellman backup in a modified MDP. This reduction transfers standard fixed-point and convergence results to the constrained operator.

  • Modified MDP: The constrained Bellman backup is reduced to a standard Bellman backup in a modified MDP.The modified MDP retains the original state space, discount, and initial-state distribution while treating policies as actions.
  • Modified MDP: The modified MDP uses policies π ∈ Π as actions and transitions by executing one policy step under the original dynamics.Its transition distribution averages the original transition kernel over the selected policy.
  • Modified MDP: QΠ(s, π) represents executing policy π for one step before following the original MDP policy that maximizes future discounted value.This gives the constrained backup an ordinary MDP interpretation.
  • Guarantees: Because TΠ is mathematically the Bellman operator of the modified MDP, it has a fixed point and repeated application converges to it.These are standard MDP-theoretic properties transferred through the reduction.

B Error Propagation

The error-propagation analysis separates Bellman approximation error from distribution and support effects, using a policy-constrained backup and modified-MDP arguments to obtain bounds. The policy set is restricted according to behavior-policy support, which controls distribution shift but introduces approximation and suboptimality considerations.

  • Error bounds: Approximate distribution-constrained value iteration assumes a bound δ(s, a) on the Bellman error across iterations.The theorem frames the propagation analysis in terms of the maximum Bellman error over supported actions.
  • Error decomposition: The proof introduces VΠ as the fixed point of TΠ and decomposes value error relative to both VΠ and the optimal value V*.The triangle-inequality decomposition separates error from the constrained fixed point and the remaining optimality gap.
  • Error decomposition: The analysis bounds the value error using the maximum expected Bellman error and separates terms associated with constrained-backup error from suboptimality.The proof explicitly identifies maxπ Eπ[δ(s, a)] as an upper bound for one component.
  • Support constraint: The support-constrained policy class Πϵ contains policies that assign zero probability to actions with behavior-policy probability below ϵ.The analysis assumes the data distribution is generated by behavior policy β and studies its induced state distributions.
  • Distribution shift: The proof bounds the relationship between behavior-policy and constrained-policy state distributions using total variation and minimum visitation assumptions.The stated bound depends on γ, ϵ, and a positive minimum visitation function f(ϵ).
  • Context: The appendix connects the constrained-backup analysis to support matching and prior error-propagation arguments.It presents the constrained operator as the central object for controlling propagation under approximation.

C.1 Why can we choose actions from Πϵ, the support of the training distribution, and need not restrict action selection to the policy distribution?

The analysis argues that supported actions can be selected without restricting the policy to the exact behavior distribution: with expressive function approximation, error is expected to depend primarily on support membership. BEAR-QL therefore searches broadly within supported actions, while MMD-based support matching and finite-sample estimation introduce practical considerations.

  • Support versus density: With expressive Q-function approximators, Bellman error is expected to depend more on whether an action lies within training support than on its exact data-distribution density.This motivates selecting actions from Πϵ rather than only from the behavior-policy distribution.
  • Support versus density: When support remains unchanged, propagated error is expected to be insensitive to distribution changes within that support.Policies in Πϵ may incur similar propagated error while differing in suboptimality bias.
  • Objective choice: Inverse behavior-density weighting is not used because it has not substantially improved vanilla RL performance in the relevant off-policy setting.The paper therefore retains the unmodified Bellman-error objective.
  • Assumption: The support-based argument assumes bounded Bellman error on in-support actions.Under this assumption, the analysis treats bounded δk(s, a) as reasonable for supported action distributions.
  • Algorithmic choice: BEAR-QL maximizes expected Q-value over policies while enforcing in-support actions through an MMD constraint.The maximization step is important in high-dimensional action spaces because directly finding the best supported action may require exponentially more samples.
  • MMD behavior: Finite-sample MMD estimates can be larger between a distribution and itself than between a uniform distribution over its support and that distribution.This effect is observed for smaller sample sizes and appropriately chosen α values.

D Additional Experimental Details

Additional experiments specify dataset generation, kernel choices, evaluation procedures, and implementation settings used to assess BEAR.

  • Data collection: Datasets contain 1e6 samples generated from random, converged optimal, or partially trained behavior policies.Random data uses a uniform random policy; optimal and mediocre datasets come from SAC-trained policies.
  • MMD analysis: MMD experiments compare Gaussian and uniform-support approximations across sample sizes and support widths.For small n ≈1−10, uniform-support MMD can be slightly lower than self-comparison; enlarging support increases the approximation’s MMD.
  • Choice of kernels: Laplacian and Gaussian kernels use environment-specific or shared bandwidth settings, with σ=20 working broadly across environments.Kernel choice is important, and high-dimensional distance measurements remain a future-work concern.
  • Evaluation: Evaluation uses greedy maximization over sampled actions from the behavior-policy support rather than unrolling the learned actor.This evaluation strategy performs better than executing the learned actor directly in the environment.
  • Other hyperparameters: The Lagrange-multiplier learning rate is 1e-3, its log is clipped to [−5,10], and average Q can outperform minimum Q for Walker2d.Baselines use official BCQ and TD3 implementations plus a VAE-based behavior-cloning baseline; evaluation uses 10 separate episodes.

E Additional Experimental Results

Additional results compare MMD and KL constraints, examine Q-value behavior and importance sampling, and report greater stability for MMD-constrained BEAR-QL.

  • Q-value diagnostics: BEAR-QL is more well behaved than BCQ in Q-value versus Monte-Carlo-return trends on medium-quality data.BCQ diverges negatively in Walker2d-v2, while BEAR-QL is slightly lower in Ant-v2, indicating less overestimation there.
  • Q-value diagnostics: BCQ Q-values are more unstable than BEAR-QL, especially diverging negatively in Walker2d-v2.The comparison tracks Q-values over gradient steps in three environments.
  • Constraint comparison: MMD-constrained optimization is more stable and produces better returns than the KL alternative in ablation studies.The experiments compare constraints on medium-quality data and include an extensively tuned KL constraint.
  • Constraint comparison: Even after extensive hyperparameter tuning, the KL constraint performs worse than the MMD constraint.Plain dual descent with KL produced unstable solutions, and the tuned comparison is summarized in Figure 11.
  • Importance sampling: Importance sampling does not provide substantial practical benefit when combined with BEAR-QL on medium-quality data.The experiment reweights Bellman error using the inverse fitted behavior-policy density.
Loading 1906.00949v2…