PaperlayerView exact revision on arXiv

Source-linked AI summary

Conservative Q-Learning for Offline Reinforcement Learning

Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine

arXiv:2006.04779v3cs.LGstat.ML

TL;DR

Offline RL must learn from static datasets despite distributional shift that can make standard methods overestimate values for out-of-distribution actions. CQL adds conservative Q-value regularization to lower-bound policy values and supports policy learning with improvement guarantees. Across offline RL benchmarks, it often substantially outperforms prior methods, especially on complex and multi-policy datasets, while deep-network theory remains incomplete.

  • Problem

    Offline RL needs to learn effective policies from static datasets, but distributional shift between behavior and learned-policy actions causes overestimated values.

  • Method

    CQL adds a conservative Q-value regularizer to Bellman-error learning so the expected learned value under a policy lower-bounds its true value.

  • Results

    CQL outperforms prior offline RL methods across complex control and raw-image tasks, with final returns often exceeding prior methods by 2-5x.

  • Takeaways & Limitations

    CQL is a simple, broadly applicable offline RL framework that performs high-confidence safe policy improvement over the behavior policy.

  • Takeaways & Limitations

    Rigorous CQL theory for deep neural networks remains future work, and offline RL methods remain liable to overfitting.

Abstract

from arXiv · show

Effectively leveraging large, previously collected datasets in reinforcement learning (RL) is a key challenge for large-scale real-world applications. Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction. However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. In this paper, we propose conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations. On both discrete and continuous control domains, we show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return, especially when learning from complex and multi-modal data distributions.

1 Introduction

Offline RL seeks to learn from static datasets without costly environment interaction, but distributional shift can cause overestimated values. CQL addresses this by learning conservative Q-functions that lower-bound policy values.

  • Motivation: Offline RL learns policies from previously collected datasets without further environment interaction.This can reduce the cost, danger, and limited data availability associated with online interaction.
  • Challenge: Standard off-policy methods can fail offline because out-of-distribution actions and overfitting produce erroneously optimistic value estimates.The resulting overestimation reflects distributional shift between dataset actions and learned-policy actions.
  • Approach: CQL learns a less conservative Q-function whose expected value under the policy, rather than every individual action value, is lower-bounded.This avoids extra underestimation associated with point-wise lower bounds.
  • Contribution: CQL regularizes Q-values during training and is designed for policy evaluation and offline policy learning.The framework adds a simple regularizer to standard value-based RL algorithms.

2 Preliminaries

Offline RL uses Bellman-based value learning with data generated by a behavior policy, but learned-policy actions create distributional shift. Without environment interaction to correct errors, out-of-distribution actions can receive spuriously high Q-values.

  • RL formulation: RL seeks a policy maximizing expected discounted cumulative reward in a Markov decision process.The MDP is specified by state and action spaces, transition dynamics, rewards, and a discount factor.
  • Notation and assumptions: The behavior policy is represented by πβ(a|s), while D denotes the dataset and dπβ(s) its discounted marginal state distribution.Rewards are assumed bounded by |r(s, a)| ≤ Rmax.
  • Off-policy RL: Q-learning updates a parametric Q-function with the Bellman optimality operator and recovers a greedy policy through maximization.Actor-critic methods instead alternate policy evaluation with separate policy optimization.
  • Offline difficulty: Offline Bellman backups use actions from the learned policy, while the Q-function is trained only on behavior-policy actions in the dataset.This action distribution shift can bias the policy toward out-of-distribution actions with high learned Q-values.

3 The Conservative Q-Learning (CQL) Framework

CQL augments Bellman-error learning with conservative Q-value penalties so policy values are lower-bounded despite offline distribution shift. Its analysis supports tighter expected-value bounds, gap expansion, and high-confidence safe improvement over the behavior policy.

  • Framework: CQL learns Q-functions whose expected value under a policy lower-bounds that policy’s true value.The framework is usable for off-policy evaluation or as part of a complete offline RL algorithm.
  • Conservative evaluation: The tighter CQL update minimizes values under a chosen distribution and adds a maximization term over the data distribution.With μ(a|s)=π(a|s), the expected learned Q-value is lower-bounded even when the Q-function is not pointwise conservative.
  • Theory: When α is sufficiently large, CQL lower-bounds Qπ under the theorem’s assumptions, including sampling and function-approximation conditions.The exact theoretical results are extended beyond tabular evaluation to linear and selected nonlinear approximators.
  • OOD robustness: CQL’s gap-expanding update increases the separation between in-distribution actions and over-optimistic out-of-distribution actions.This makes the policy derived from Q-values closer to the dataset distribution and more robust to OOD errors.
  • Policy learning and guarantees: CQL optimizes a penalized empirical-MDP objective while constraining the learned policy through a penalty based on its divergence from the behavior policy.The policy obtained this way is a high-confidence ζ-safe improvement over the behavior policy, with sampling error affecting the guarantee.

4 Practical Algorithm and Implementation Details

The practical CQL methods modify standard actor-critic and Q-learning algorithms by applying the CQL objective to neural-network Q-functions. The implementation adds a small amount of code while retaining standard training structures and hyperparameters.

  • Variants: CQL provides actor-critic and Q-learning variants built on standard deep reinforcement-learning algorithms.The Q-function uses the CQL(H) or CQL(R) objective, with Bellman operators chosen for the corresponding variant.
  • Training loop: The algorithm repeatedly trains the Q-function and, for actor-critic, improves the policy with SAC-style entropy regularization.Q-learning uses the optimality operator, while actor-critic uses the current policy’s Bellman operator.
  • Implementation: 20 lines of code are added to standard SAC and QR-DQN implementations for continuous and discrete control experiments.The tradeoff factor α is automatically tuned for continuous control and fixed for discrete control.

5 Related Work

Prior offline RL methods address distribution shift through policy constraints, behavior-policy bootstrapping, uncertainty estimation, or robust-MDP formulations, while related analyses study error propagation and safe improvement.

  • Off-policy evaluation methods use importance sampling or dynamic programming, but dynamic-programming approaches can also suffer from out-of-distribution actions.
  • Offline Q-learning methods constrain policies toward behavior data or use behavior-policy actions to reduce errors from out-of-distribution actions.
  • Uncertainty-based approaches have not generally performed well offline because they require high-fidelity uncertainty estimates.
  • Robust MDPs provide a popular theoretical abstraction for offline RL, while prior theory also analyzes safe improvement and approximate dynamic-programming error propagation.

6 Experimental Evaluation

CQL is evaluated across continuous and discrete domains, varied dataset compositions, and image inputs. It generally matches or exceeds prior methods and shows its largest gains on difficult, multi-policy, sparse-data, and high-dimensional tasks.

  • Gym domains: 2-3x gains: CQL outperforms prior methods by large margins on D4RL datasets combining multiple policies.On single-policy datasets, CQL roughly matches or slightly exceeds the best prior methods.
  • Adroit tasks: 2-9x scores: CQL variants are the only methods improving over behavioral cloning on Adroit tasks with human demonstrations.CQL(ρ) outperforms CQL(H) on several higher-dimensional tasks because CQL(H) importance weights have higher variance.
  • AntMaze: Non-zero returns: CQL is the only method making meaningful progress on the harder medium and large AntMaze tasks.
  • Kitchen tasks: Over 40% success rate: CQL is the only method outperforming behavioral cloning on all Franka kitchen tasks.
  • Offline RL on Atari games: 36x and 6x returns: CQL exceeds the best prior method on Q*bert and Breakout, respectively, in the harder 1% Atari-data condition.
  • Analysis of CQL: Negative differences: CQL is the only evaluated method whose predicted policy values lower-bound actual returns on three D4RL datasets.CQL(H) is much less conservative than the CQL variant using Equation 1.

7 Discussion

CQL is presented as an offline RL framework that lower-bounds policy values and outperforms prior methods across diverse benchmark settings, while theoretical and methodological challenges remain.

  • CQL learns a lower bound on policy value and outperforms prior offline RL methods across complex control and raw-image benchmark tasks.
  • 2-5x higher final returns: CQL often substantially exceeds the best-performing prior methods, especially on complex and multi-modal data distributions.
  • Deep-neural-network theory remains incomplete: rigorous analysis of CQL with deep neural nets is left for future work.The paper also identifies offline-RL overfitting and the lack of simple effective early-stopping methods as future challenges.

Appendices

The appendices develop several CQL variants by choosing different regularizers or action distributions, including entropy-, KL-, and variance-based formulations.

  • Variants: The CQL family includes CQL(H), CQL(ρ), and CQL(var), with CQL(var) connected to distributionally robust optimization.
  • CQL(H): CQL(H) replaces the regularizer with entropy and solves the optimization over the auxiliary distribution in closed form.
  • CQL(ρ): CQL(ρ) uses a KL-divergence regularizer instead of entropy, producing an auxiliary distribution proportional to ρ(x) exp(f(x)).
  • CQL(var): CQL(var) penalizes variance in Q-function predictions across actions under a chosen action-conditional distribution.
  • CQL(var): The variance-regularized variant can use an inverse empirical action distribution or a uniform action distribution for ˆP(a|s).

B Discussion of Gap-Expanding Behavior of CQL Backups

CQL expands the gap between in-distribution and out-of-distribution Q-values, while policy-constraint methods can exhibit the opposite behavior under function approximation. Empirically, CQL avoids the training-time unlearning effect observed for policy constraints.

  • Gap-expanding behavior: CQL backups expand the gap between expected Q-values for in-distribution and out-of-distribution actions.The CQL update increases this difference, a property attributed to its regularizer.
  • Function-approximation effects: Narrow dataset distributions can produce incorrect high Q-values for out-of-distribution actions despite large offline datasets.These errors persist without active data collection to correct them.
  • Policy-constraint methods: Policy constraints may fail to prevent policy effects from incorrectly high out-of-distribution Q-values.Function-approximation coupling can make out-of-distribution actions appear promising even when target values are constrained.
  • Empirical evidence: CQL generally yields negative ˆ∆k values, whereas policy-constraint methods yield positive values that can increase during training.The comparison is shown on hopper-expert and hopper-medium datasets.
  • Empirical evidence: On hopper-expert, policy-constraint performance eventually deteriorates through unlearning, while CQL does not show this effect after equally many training steps.Performance on hopper-medium fluctuates but does not suddenly deteriorate.
  • Empirical evidence: Continuously growing ˆ∆k values may precede an unlearning effect, especially for methods without Q-function regularization.The summarized empirical evidence associates positive and increasing gaps with absent gap expansion and later performance deterioration.

C Theorem Proofs

The proofs establish that CQL induces conservative value estimates, with guarantees extending from tabular evaluation to function approximation. The tighter regularizer lower-bounds expected policy value while expanding the gap between in-distribution and out-of-distribution actions.

  • Sampling and approximation: For suitable α, CQL’s lower-bound guarantees account for sampling error and function approximation, with required α decreasing as dataset coverage increases.In the infinite-data limit, extremely small α can suffice for a lower bound.
  • Policy evaluation: CQL policy evaluation with its simplified objective produces a point-wise lower bound on the Q-function.The result holds with an appropriate support condition on the minimizing distribution and high probability under sampling error.
  • Expected-value lower bounds: The dataset-distribution maximization term tightens the guarantee from a point-wise Q-function bound to a lower bound on expected policy value.Theorem 3.3 shows that maximizing Q-values under the dataset distribution is sufficient for this expected-value guarantee.
  • Gap expansion: The CQL update expands the difference between in-distribution and out-of-distribution Q-values, with the expansion controlled by the free parameter α_k.The regularizer maximizes Q-values under the dataset distribution and minimizes them otherwise.
  • Function approximation: Under linear approximation, CQL lower-bounds the corresponding LSTD-Q estimate, while an NTK analysis extends lower-bound guarantees to nonlinear approximators.The nonlinear result assumes an invertibility condition and a suitable α_k for each gradient step.

D.3 CQL with Empirical Dataset Distributions

The empirical-dataset analysis characterizes how sampling error affects CQL and derives a penalized empirical-MDP objective. Concentration-based bounds then support a high-probability safe policy improvement guarantee relative to the behavior policy.

  • Empirical Bellman updates: CQL’s practical implementation optimizes a sample-based version of its regularized Bellman objective using empirical rewards and transitions.The empirical Bellman operator is computed from dataset samples, while expectations under the policy can be evaluated exactly.
  • Sampling error: The empirical Bellman operator can overestimate the actual operator when finite data omit reachable next states and retain an optimistic transition.This issue is especially relevant in stochastic MDPs.
  • Penalized empirical objective: CQL optimizes a penalized reinforcement-learning objective in the empirical MDP and uses this formulation to analyze policy improvement.The objective subtracts a conservatism penalty based on the learned policy’s divergence from the behavior policy.
  • Safe policy improvement: The safe-improvement analysis compares the learned policy’s actual-MDP return with the behavior policy’s return while accounting for empirical-MDP sampling error.The theorem bounds the learned policy’s performance using concentration arguments and a CQL penalty term.
  • Sampling-error bound: The return gap between the actual and empirical MDPs is bounded by separately controlling reward, transition, and marginal-state-distribution discrepancies.The proof uses concentration inequalities and a total-variation bound on state distributions.

E Extended Related Work and Connections to Prior Methods

CQL is connected to uncertainty-based and policy-constraint approaches through its treatment of overestimation under distribution shift. Its conservative objective can incorporate uncertainty information, while its dataset-distribution choice is necessary for the stated lower-bound guarantee.

  • Uncertainty estimation: Uncertainty-based offline RL methods estimate epistemic uncertainty and optimize lower-confidence Q-values, but uncertainty alone has not been sufficient to prevent OOD actions.Such methods are often augmented with additional mechanisms.
  • Relation to uncertainty methods: CQL strengthens point-wise lower-bound approaches by targeting the expected policy value, the quantity directly needed for policy evaluation and improvement.A naïve CQL regularizer yields a uniform Q-function lower bound, whereas the refined objective provides a tighter expected-value bound.
  • Uncertainty-aware CQL: CQL can be modified to depend on epistemic uncertainty under statistical concentration assumptions on rewards and dynamics.The modification weights Bellman-error updates using state-action visitation counts.
  • Finite-data guarantees: For finite datasets, choosing α proportional to state-action counts can preserve a lower bound by offsetting sampling-induced overestimation.In the infinite-data limit, the required conservatism tends toward zero.

F Additional Experimental Setup and Implementation Details

CQL is implemented with standard RL frameworks using fixed or automatically tuned penalties, while experiments use specified domain-dependent hyperparameters and largely standard settings.

  • Implementation: CQL was implemented on top of SAC for D4RL benchmarks and QR-DQN for discrete-action Atari tasks.The implementations retained standard algorithmic components and used different penalty-selection strategies across domains.
  • Penalty selection: The Lagrange version adjusts α toward larger penalties when the expected Q-value difference exceeds threshold τ.When the difference is below τ, α approaches zero; larger differences lead to more aggressive Q-value penalization.
  • Penalty selection: The experiments used τ = 10.0 for D4RL Gym MuJoCo, τ = 5.0 for Franka Kitchen and Adroit, and fixed Atari penalties varying by data condition.Atari used α = 1.0 with 10% data, α = 4.0 with 1% data, and α = 0.5 for Figure 1.
  • Implementation: CQL(H) computes the discrete-action log-sum-exp exactly and estimates it in continuous tasks using importance sampling with sampled actions.Continuous-task estimates used ten action samples at each state from a uniform distribution and the current policy.
  • Evaluation: D4RL results report normalized, smoothed average undiscounted return over four seeds, with CQL built on an SAC implementation and a reduced policy learning rate.The policy learning rate was selected as 3e-5 or 1e-4 for continuous control tasks.
  • Hyperparameters: Preliminary experiments found no significant Q-function learning-rate difference between 1e-4 and 3e-4, while 3e-5 was generally effective for policy learning.The default Q-function learning rate was 3e-4, and the policy-learning-rate range included 3e-5, 1e-4, and 3e-4.
  • Hyperparameters: For MuJoCo, τ = 5.0 or τ = 10.0 produced stable Q-value curves, whereas τ = 10.0 failed to prevent overestimation in some other cases.Some unstable cases involved Q-values diverging above 1e+6; threshold choices were made without online evaluation.

G Ablation Studies

The ablations compare conservative-objective variants and penalty-selection strategies, showing that dataset maximization and adaptive α can matter substantially in particular data or task settings.

  • CQL variants: Across three D4RL MuJoCo environments, CQL(H) generally outperformed CQL(ρ), which uses the previous policy in its objective.The comparison is summarized in Table 5 using average return.
  • Objective terms: Omitting the dataset Q-value maximization term generally decreased CQL(H) performance, especially for datasets generated from a single policy such as hopper-medium.This ablation corresponds to Equation 1 and has a theoretically weaker lower bound than the full formulation.
  • Penalty selection: The Lagrange version generally outperformed the non-Lagrange formulation, with differences reaching 30% of maximum achievable performance in some AntMaze cases.On gym MuJoCo tasks, the benefit was smaller but remained clear for settings such as hopper-mixed.
  • Penalty selection: The Lagrange and fixed-α versions were generally comparable except in AntMaze, where adaptive α greatly outperformed a single chosen value.Table 7 reports average returns for CQL(H) and its automatically tuned variant.
Loading 2006.04779v3…