PaperlayerView exact revision on arXiv

Source-linked AI summary

COMBO: Conservative Offline Model-Based Policy Optimization

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, Chelsea Finn

arXiv:2102.08363v2cs.LGcs.AIcs.RO

TL;DR

Offline RL suffers from distribution shift, while practical model-based methods rely on uncertainty estimation that can be challenging and unreliable with complex models. COMBO avoids explicit uncertainty quantification by penalizing Q-values on out-of-support state-action pairs, and it provides lower-bound and safe policy-improvement guarantees while matching or outperforming prior methods across offline RL benchmarks.

  • Problem

    Offline RL requires methods that address distribution shift, but prior model-based approaches rely on uncertainty estimation that can be challenging and unreliable with complex models.

  • Method

    COMBO learns a dynamics model and conservatively penalizes value estimates for out-of-support state-action pairs generated through model rollouts, without explicit uncertainty quantification.

  • Results

    COMBO matches or outperforms prior offline model-free and model-based methods, achieving the best performance in 9 of 12 D4RL settings and in three tasks requiring adaptation to unseen behaviors.

  • Takeaways & Limitations

    COMBO provides a conservative lower-bound estimate of policy value and guarantees policy improvement beyond the behavior policy in the offline setting.

  • Takeaways & Limitations

    The theoretical guarantees rely on concentration assumptions for the analyzed backups and sufficiently large penalty parameter β.

Abstract

from arXiv · show

Model-based algorithms, which learn a dynamics model from logged experience and perform some sort of pessimistic planning under the learned model, have emerged as a promising paradigm for offline reinforcement learning (offline RL). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating pessimism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. We theoretically show that our method optimizes a lower bound on the true policy value, that this bound is tighter than that of prior methods, and our approach satisfies a policy improvement guarantee in the offline setting. Through experiments, we find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods on widely studied offline RL benchmarks, including image-based tasks.

1 Introduction

Offline RL trains policies from static datasets, but distribution shift between the dataset and learned policy makes standard online RL algorithms perform poorly. COMBO addresses this with a model-based method that avoids explicit uncertainty quantification.

  • Motivation: Offline RL trains policies using static, previously collected datasets for data reuse and safe policy learning.The setting is relevant to healthcare, autonomous driving, robotics, and recommendation systems.
  • Motivation: Distribution shift between the offline dataset and learned policy is a primary reason online RL algorithms perform poorly offline.
  • Prior approaches: Model-free methods can become overly conservative because they learn only on states present in the offline dataset.
  • Prior approaches: Prior model-based methods use pessimistic dynamics models but rely on model-error oracles or heuristic uncertainty estimates that can be difficult or unreliable with deep models.
  • Contribution: COMBO develops a principled model-based offline RL algorithm intended to match or exceed prior methods on benchmark tasks without explicit uncertainty quantification.

2 Preliminaries

Offline RL seeks a high-return policy from a fixed behavior-policy dataset, but direct dynamic programming or actor-critic learning suffers from distribution shift and Q-value overestimation. Prior approaches address this with model-free regularization or model-based uncertainty-aware pessimism.

  • Offline RL setting: The RL objective is to maximize discounted long-term cumulative reward under the learned policy.
  • Offline RL setting: Offline RL uses a fixed transition dataset collected by a behavior policy and seeks the best policy obtainable from that dataset.
  • Prior approaches: Model-free offline methods regularize policy or value training without learning dynamics, whereas model-based methods learn dynamics to support policy search.
  • Challenges: Directly extending approximate dynamic programming or actor-critic methods offline performs poorly because of distribution shift and Q-function overestimation bias.
  • Prior approaches: CQL penalizes Q-values for dataset states and unobserved actions, producing conservative value estimates that mitigate overestimation and distribution shift.
  • Prior approaches: MOReL and MOPO use uncertainty quantification to construct and optimize lower bounds on policy performance, often relying on model-error estimation.

3 Conservative Offline Model-Based Policy Optimization

COMBO combines learned-model rollouts with offline data and conservatively penalizes Q-values on out-of-support state-action tuples. It replaces uncertainty-based penalties with conservative policy evaluation and retains a policy-improvement guarantee.

  • Algorithm: COMBO samples synthetic rollouts from the learned model, starting from states in the offline dataset, and adds them to a model-data replay buffer.
  • Algorithm: COMBO trains a probabilistic dynamics model on the offline dataset and alternates model rollouts, conservative policy evaluation, and policy improvement.Its practical implementation uses an actor-critic procedure with neural-network dynamics.
  • Conservative policy evaluation: The critic pushes down Q-values on potentially out-of-support model-generated tuples while pushing up values on trusted offline state-action pairs.
  • Conservative policy evaluation: COMBO mixes real and model-generated data in Bellman backups, using an interpolation between offline and synthetic rollout distributions.
  • Guarantees: Unlike uncertainty-based model-based methods, COMBO obtains a lower-bounding critic without explicit uncertainty estimation.
  • Policy improvement: The policy is improved using the conservative critic, with the resulting policy theoretically guaranteed to improve over the behavior policy.

4 Theoretical Analysis of COMBO

COMBO theoretically optimizes a lower bound on policy value by conservatively training its critic on model-generated state-action tuples. The analysis establishes when this bound is valid, compares COMBO’s conservatism with CQL, and gives safe policy-improvement guarantees over the behavior policy.

  • 4.1 COMBO Optimizes a Lower Bound: The COMBO critic penalizes value estimates using distributions that include model-generated state-action tuples, producing a penalty determined by the critic-training distributions.The analysis characterizes this penalty through the distributions ρ, d_f, and the dataset-related distribution in the critic objective.
  • 4.1 COMBO Optimizes a Lower Bound: COMBO’s learned Q-function lower-bounds the actual policy value under the initial-state distribution, with high-probability guarantees under stated concentration assumptions.The guarantee holds for sufficiently large β; when sampling error or model bias is small, a smaller β can suffice with an appropriate choice of f.
  • 4.1 COMBO Optimizes a Lower Bound: COMBO is less conservative than CQL because it penalizes Q-values on generated state-action tuples rather than forcing underestimation at every dataset state.The comparison depends on how closely learned-policy and behavior-policy action probabilities align under the generated distribution versus dataset states.
  • 4.2 Safe Policy Improvement Guarantees: COMBO provides ζ-safe policy improvement over the behavior policy when β is sufficiently large and ν(ρπ, f) − ν(ρβ, f) exceeds a positive constant.With probability at least 1 − δ, the resulting policy satisfies J(π̂out, M) ≥ J(πβ, M) − ζ.
  • 4.2 Safe Policy Improvement Guarantees: The bound’s ζ combines limited-data error, learned-model bias, and an improvement term; the first decreases with dataset size, while the second reflects model suboptimality.A suitable β can make the improvement term offset the data and model-error terms when the distributional improvement condition is positive.
  • 4.2 Safe Policy Improvement Guarantees: Choosing f lets COMBO trade off sampling error and model bias, combining model-free and model-based components without requiring an oracle uncertainty estimator.Near-accurate models can reduce sampling error, while greater reliance on the model-free component can reduce model bias.

5 Experiments

COMBO is evaluated against prior offline model-free and model-based methods on generalization, image-based, and standard benchmark tasks. It performs strongly across these settings, including tasks requiring adaptation to unseen behaviors and diverse dataset types.

  • Experimental setup: The experiments compare COMBO with model-free and model-based baselines across compact-state, image-based, and standard offline RL domains.Comparisons include BEAR, BRAC, CQL, MOPO, MOReL, SAC-off, BC, LOMPO, LMBPO, and SLAC-off.
  • Tasks requiring generalization: COMBO is tested on halfcheetah-jump and ant-angle, where behavior policies solved different original tasks, and on sawyer-door-close.These environments require generalization beyond the behavior policy’s solved task.
  • Tasks requiring generalization: Approximately 8%, 4%, and 12% improvement over MOPO, MOReL, and CQL respectively is reported on ant-angle.COMBO also significantly outperforms these methods on halfcheetah-jump and sawyer-door-close.
  • Uncertainty estimation: Max Var is unable to accurately predict true model error on halfcheetah-jump and ant-angle, while COMBO avoids explicit uncertainty quantification.The analysis uses fitted linear regression between normalized uncertainty estimates and normalized model error.
  • Image-based tasks: COMBO is evaluated on walker-walk and sawyer-door using image observations, with datasets including medium, medium-replay, medium-expert, and expert.Walker uses four datasets, while sawyer-door uses medium-expert and expert because of sparse rewards.
  • D4RL benchmark: COMBO achieves the best performance in 9 of 12 D4RL settings and a comparable result in hopper medium-replay.The reported pattern indicates strong performance across random, medium, medium-replay, and medium-expert dataset types.

6 Related Work

Related offline RL methods use model-free or model-based approaches with explicit or implicit regularization. COMBO retains model-based generalization while avoiding explicit policy regularization and uncertainty quantification.

  • Offline RL: Offline RL learns policies from static datasets and has been studied with both model-free and model-based algorithms.Regularization plays a major role in both classes of methods.
  • Model-free offline RL: Model-free methods regularize policies or value functions but learn only on dataset states, which can produce overly conservative algorithms.Examples include regularized importance sampling, offline actor-critic, uncertainty-based Q-value methods, and conservative Q-learning.
  • Model-based offline RL: Model-based methods learn dynamics models to support broader generalization but rely on uncertainty quantification or direct policy constraints.Uncertainty quantification can be difficult for deep network models.
  • COMBO: COMBO penalizes Q-values on out-of-support state-action pairs and retains model-based generalization without explicit policy regularization or uncertainty quantification.This distinguishes COMBO from prior model-based methods described in the related-work discussion.

7 Conclusion

The paper presents COMBO as a model-based offline RL algorithm that combines conservative Q-value penalization with theoretical and empirical guarantees. It performs well across generalization, vision, and D4RL evaluations, while leaving hyperparameter selection and function selection as challenges.

  • Contribution: COMBO penalizes Q-values on out-of-support state-action pairs and removes the need for uncertainty quantification.The method is designed for model-based offline RL.
  • Theory: COMBO achieves less conservative Q-values than prior model-free offline RL methods and guarantees safe policy improvement.These are theoretical claims stated in the conclusion.
  • Empirical results: COMBO achieves the best generalization performance in 3 tasks requiring adaptation to unseen behaviors.The conclusion also reports scaling to vision-based locomotion and robotic manipulation tasks.
  • Limitations: The paper identifies offline hyperparameter selection and automatic selection of the function f as remaining challenges.The conclusion does not provide a uniform hyperparameter-selection scheme across datasets.
  • Scope: The theoretical statements are proven for finite state and action spaces.The proof section explicitly states the assumption |S| < ∞ and |A| < ∞.

A.1 A Useful Lemma and Its Proof

The lemma establishes that COMBO’s expected penalty is nonnegative, increases with interpolation weight, and vanishes exactly when the policy distribution matches the dataset distribution or the weight is zero.

  • Setup: The proof defines d_f as an f-interpolation between the dataset distribution d and ρ.The interpolation is d_f(s, a) := f d(s, a) + (1 − f)ρ(s, a).
  • Lemma: The expected penalty ν(ρ, f) is nonnegative for every distribution ρ and interpolation weight f.This supports the penalty’s role in combating overestimation from the behavior-policy backup.
  • Lemma: For fixed ρ, ν(ρ, f) is monotonically increasing in f.The proof uses the positivity of the derivative with respect to f.
  • Lemma: ν(ρ, f) equals zero if and only if ρ(s, a) equals d(s, a) for every state-action pair or f equals zero.When the distributions differ and f > 0, the penalty is strictly positive.

A.2 Proof of Proposition 4.1

The proof establishes that COMBO’s critic can lower-bound true policy value in expectation for sufficiently large β, while allowing statewise overestimation unlike CQL. The guarantee depends on sampling and model errors, and becomes attainable with smaller β when those errors are small.

  • Lower-bound guarantee: Sampling error and model bias contribute positive error terms that the conservative penalty, controlled by β, must offset.The proof bounds empirical-MDP error and learned-model Bellman error before choosing β large enough.
  • Lower-bound guarantee: COMBO’s learned Q-function lower-bounds the true policy value in expectation under the initial state distribution for sufficiently large β.The result is stated with high probability for the asymptotic critic under the learned model.
  • Lower-bound guarantee: When sampling error or model bias is small, a small β can suffice with an appropriate choice of f.This includes the large-sample regime and cases with small model bias ϵ_m.
  • Comparison with CQL: COMBO’s guarantee is only in expectation over the initial state distribution, so its value estimate may exceed the true value at individual states.The proof contrasts COMBO’s expected bound with CQL’s statewise lower bound.
  • Comparison with CQL: COMBO can overestimate states frequent in the dataset but infrequent under the policy’s learned-model visitation distribution.The expected penalty under the dataset distribution can be negative, implying overestimation at least at some dataset states.

A.3 Proof of Proposition 4.2

The proof analyzes when COMBO is less conservative than CQL by decomposing its penalty terms and identifying a condition under which the additional term is negative. For f = 1, the state distribution simplifies to the learned-model distribution, enabling the comparison.

  • Comparison with CQL: The proof derives a condition under which COMBO is less conservative in estimated value than CQL.It specifically analyzes the additional term in COMBO’s penalty relative to the CQL term.
  • Real-data fraction: For f = 1, COMBO’s real-data fraction makes the induced state distribution equal to d^πβ(s).This simplification is used to derive the comparison condition.
  • Comparison with CQL: The comparison follows by adding and subtracting d^πβ(s)π(a|s), isolating the CQL term and analyzing the remaining term.The proof notes that the CQL term appears on the right-hand side and the inequality holds when the second term is negative.

A.4 Proof of Proposition 4.3

The proof bounds COMBO’s return in an interpolated MDP relative to the actual MDP, then combines this bound with conservative policy optimization to establish safe improvement conditions.

  • Step 1: Bounding the return in the actual MDP: Equation 4 is interpreted as optimizing an f-interpolant MDP formed from empirical and learned-model Bellman backups, with a conservative penalty on learned-model state-action distributions.The return is denoted J(M, c M, f, π), and the penalty uses the marginal state-action distribution ρπ in the learned model.
  • Step 1: Bounding the return in the actual MDP: For any policy, the interpolated-MDP return lies within [J(π, M) − α, J(π, M) + α], where α captures reward and dynamics discrepancies.The proof separates reward differences into ∆R and dynamics differences involving the divergence between actual and learned transition models.
  • Policy improvement guarantee: If ν(ρπout, f) − ν(ρβ, f) ≥ C > 0, COMBO satisfies J(πout, M) ≥ J(πβ, M) − ζ with probability at least 1 − δ.This is the theorem’s safe policy improvement guarantee in the actual MDP.
  • Step 2: Incorporate policy improvement in the f-interpolant MDP: The policy-improvement step lower-bounds COMBO’s improvement over πβ using conservative-penalty separation, sampling error, and divergence between the actual and learned dynamics.The proof applies upper and lower interpolated-MDP return bounds to πout and πβ before bounding the resulting terms.
  • Interpretation of Proposition 4.3: When ζ is negative, πout improves over πβ in the original MDP; choosing f near 0 favors model-based improvement when the learned model is accurate.The analysis states that f near 1 recovers a model-free bound, while f near 0 recovers a model-based bound.

B.6 License of datasets

The paper states that its datasets use the MIT license and provides an anonymous download link for datasets from the generalization environments.

  • License: All datasets used in the paper use the MIT license.
  • Dataset access: Datasets from the generalization environments are available through an anonymous Google Drive link.
  • Related appendix material: Table 8 reports COMBO and CQL+MBPO results on out-of-distribution generalization tasks, averaged over 6 random seeds with 95%-confidence intervals.

C Comparison to the Naive Combination of CQL and MBPO

COMBO differs from the naive CQL+MBPO combination by regularizing Q-values on model-generated states as well as unseen actions, and it performs better on generalization experiments.

  • Method distinction: CQL+MBPO regularizes Q-values on unseen actions at dataset states, whereas COMBO penalizes Q-values on states generated by the learned model.
  • Empirical comparison: CQL+MBPO performs quite a bit worse than COMBO on generalization experiments, suggesting that considering the state distribution is crucial.The comparison averages results across 6 random seeds with 95%-confidence intervals.
Loading 2102.08363v2…