PaperlayerView exact revision on arXiv

Source-linked AI summary

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning

Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, Sergey Levine

arXiv:2303.05479v4cs.LGcs.AI

TL;DR

Offline RL can initialize policies for online fine-tuning, but existing methods may improve slowly or initially unlearn the offline policy. Cal-QL calibrates conservative Q-values against a reference policy, and it matches or outperforms prior methods across the evaluated fine-tuning tasks.

  • Problem

    Existing offline RL methods often struggle to provide an initialization that both uses offline data effectively and enables efficient online fine-tuning.

  • Method

    Cal-QL learns conservative Q-functions calibrated against a reference policy, such as the behavior policy, so estimated values remain on a useful scale.

  • Results

    Cal-QL matches or outperforms the best methods on all evaluated tasks and achieves the best fine-tuned performance in 9 out of 11 tasks.

  • Takeaways & Limitations

    Calibrated value functions facilitate fast online fine-tuning while avoiding initial unlearning and retaining effective asymptotic performance.

  • Takeaways & Limitations

    The theoretical analysis uses an idealized finite-horizon bilinear-model setting and simplifies aspects such as incorporating pessimism.

Abstract

from arXiv · show

A compelling use case of offline reinforcement learning (RL) is to obtain a policy initialization from existing datasets followed by fast online fine-tuning with limited interaction. However, existing offline RL methods tend to behave poorly during fine-tuning. In this paper, we devise an approach for learning an effective initialization from offline data that also enables fast online fine-tuning capabilities. Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also being calibrated, in the sense that the learned Q-values are at a reasonable scale. We refer to this property as calibration, and define it formally as providing a lower bound on the true value function of the learned policy and an upper bound on the value of some other (suboptimal) reference policy, which may simply be the behavior policy. We show that offline RL algorithms that learn such calibrated value functions lead to effective online fine-tuning, enabling us to take the benefits of offline initializations in online fine-tuning. In practice, Cal-QL can be implemented on top of the conservative Q learning (CQL) for offline RL within a one-line code change. Empirically, Cal-QL outperforms state-of-the-art methods on 9/11 fine-tuning benchmark tasks that we study in this paper. Code and video are available at https://nakamotoo.github.io/Cal-QL

1 Introduction

The paper asks how offline RL can provide an effective initialization that also supports fast online fine-tuning with limited interaction. Cal-QL calibrates conservative value functions to improve fine-tuning performance and reduce regret.

  • Motivation: Offline pre-training followed by online fine-tuning could reduce the active samples needed for online RL.The approach adapts the pre-training and fine-tuning recipe to sequential decision-making using static offline datasets.
  • Motivation: Existing offline RL methods often struggle to acquire an initialization that also enables efficient fine-tuning.Some methods improve slowly, while others initially degrade and spend online interaction recovering their offline performance.
  • Contribution: Cal-QL learns conservative value functions calibrated against a reference policy to facilitate online fine-tuning.Its calibration objective is designed to provide a useful value scale while retaining conservatism.
  • Contribution: Cal-QL matches or outperforms the best methods on all evaluated tasks, in some cases by 30-40%.The evaluation spans benchmark tasks including robotic manipulation and navigation.

2 Related Work

Prior approaches use offline data to improve online RL through replay, behavioral cloning, or skill extraction. However, offline RL methods based on pessimism or policy constraints can slow fine-tuning or cause initial unlearning.

  • Using Offline Data: Prior methods incorporate offline data through replay buffers, behavioral cloning losses, or high-level skill spaces.These approaches improve the sample efficiency of online RL from scratch.
  • Offline RL Fine-Tuning: Offline RL methods commonly use policy constraints or pessimism before continuing training with offline and online data.This includes methods that retain the offline objective during fine-tuning.
  • Offline RL Fine-Tuning: Pessimism or constraints can slow fine-tuning or produce initial unlearning.Prior methods therefore either improve less quickly than online RL or lose the offline initialization.

3 Preliminaries and Background

The paper formalizes RL objectives, value functions, offline datasets, and cumulative regret for online fine-tuning. It builds on CQL, whose conservative regularizer penalizes estimated values for out-of-distribution actions while compensating for dataset actions.

  • RL Setup: RL seeks a policy maximizing cumulative discounted value in an MDP defined by states, actions, dynamics, rewards, an initial-state distribution, and a discount factor.The policy is evaluated through its discounted value function.
  • Value Functions: A Q-function gives the value of taking an action and then following a policy, while Qθ denotes its neural-network estimate.The estimate is parameterized by network parameters θ.
  • Offline-to-Online Setup: Offline training uses a static dataset collected by a behavior policy before online interaction begins.Fine-tuning aims to obtain the optimal policy using as few online samples as possible, formalized through cumulative regret.
  • CQL Background: CQL adds a conservative regularizer that penalizes Q-values for out-of-distribution actions while increasing values for actions observed in the dataset.The regularizer minimizes values under a policy and counterbalances this by maximizing values under the behavior policy.

4 When Can Offline RL Initializations Enable Fast Online Fine-Tuning?

Offline initializations can fail during fine-tuning through slow improvement or initial unlearning. The paper attributes CQL's unlearning to overly low Q-values and proposes calibration against a reference policy to keep values on a useful scale.

  • Empirical Challenges: Existing offline RL initializations can perform poorly during online fine-tuning despite strong offline performance.The section studies conservative methods to explain this failure and motivate calibrated Q-learning.
  • Empirical Challenges: At roughly 0.5 normalized offline return, TD3+BC, IQL, and AWAC improve slowly while CQL initially unlearns its offline initialization.CQL then spends substantial online interaction recovering offline performance before improving further.
  • Why CQL Unlearns: CQL's offline Q-values are much smaller than their ground-truth values and adjust toward the true range when fine-tuning begins.The period of Q-value adjustment coincides with performance recovery.
  • Calibration: Calibration requires conservative Q-values to be lower-bounded by the ground-truth value of a suboptimal reference policy.This prevents conservatism from producing overly small values and reduces the need to unlearn and recover the initialization.
  • Calibration: Cal-QL enforces calibration using a reference policy whose value can be reliably estimated without bootstrapping error, such as the dataset behavior policy.The policy optimizer is then discouraged from replacing the learned policy with one worse than the reference policy.

5 Cal-QL: Calibrated Q-Learning

Cal-QL builds on CQL by calibrating conservative Q-values against a reference policy, preventing low-valued dataset actions from being displaced during online fine-tuning.

  • Cal-QL constrains the learned Q-function to produce Q-values larger than those of a reference policy, typically the behavior policy.
  • The method modifies CQL by masking its downward push on out-of-distribution actions whenever the learned Q-function is not calibrated.
  • With sufficiently large α, the modified objective upper-bounds the reference Q-function for state-action pairs where the learned value is smaller, although function approximation prevents universal enforcement.
  • Cal-QL estimates reference-policy values from return-to-go targets, and a simple one-line objective change improves prior fine-tuning results.
  • Erroneous peaks on suboptimal actions can make online-updated CQL abandon high-reward dataset actions, whereas Cal-QL uses a reference value function to prevent worse actions appearing optimal.

6 Theoretical Analysis of Cal-QL

The analysis studies Cal-QL’s online-fine-tuning regret under an idealized finite-horizon bilinear model and decomposes regret into miscalibration and overestimation. Calibration controls the former while conservatism controls the latter, yielding favorable guarantees and empirical fine-tuning behavior.

  • The theory analyzes Cal-QL regret in an idealized finite-horizon setting under the bilinear model, simplifying aspects such as pessimism.
  • The regret decomposition separates miscalibration from overestimation, with conservatism expected to reduce overestimation and calibration controlling the other term.
  • A high-value reference policy close to optimal lets calibration control the miscalibration term, so regret depends on balancing calibration with conservative Q-learning.
  • With high probability, Theorem 6.1 gives a bound on total regret accumulated during online fine-tuning.
  • When the reference policy is near-optimal, Cal-QL can achieve a tighter regret guarantee than Song et al.; with broad coverage and high suboptimality, it reverts to that prior guarantee.
  • Cal-QL consistently matches or exceeds the speed and final performance of the best prior method across the benchmark fine-tuning tasks shown in Figure 6.

7 Experimental Evaluation

The evaluation compares Cal-QL with offline-to-online baselines across diverse benchmark tasks, measuring normalized performance before and after fine-tuning, cumulative regret, high-UTD training, and controlled behavior studies. Cal-QL achieves strong fine-tuning gains, particularly on narrow datasets, while reference-value estimation errors have limited effect when the approximator is reasonable.

  • Evaluation setup: Cal-QL is evaluated on AntMaze, FrankaKitchen, Adroit, and vision-based robotic manipulation benchmarks, alongside prior offline-to-online methods and online SAC.The protocol measures performance before and after online fine-tuning and includes higher-dimensional raw-image tasks.
  • Empirical results: 9 out of 11 tasks achieve the best fine-tuned performance with Cal-QL, which improves over its offline initialization by 106.9% in aggregate.Its offline initialization is comparable to methods such as CQL and IQL, but its improvement during fine-tuning is larger.
  • Empirical results: Cal-QL attains the smallest cumulative regret on 8 out of 11 tasks, with average regret 0.22, a 42% improvement over IQL.The regret metric is averaged over fine-tuning steps, where smaller values are better and 1.00 is worst.
  • Cal-QL With High Update-to-Data (UTD) Ratio: With UTD=20, Cal-QL improves over UTD=1 and training from scratch while generally matching or exceeding RLPD asymptotically with smaller cumulative regret.The comparison uses three seeds and incorporates design choices from RLPD.
  • Understanding the Behavior of Cal-QL: Cal-QL is most effective on narrow datasets, where calibration corrects initially underestimated Q-values and avoids the unlearning seen with naïve conservative methods.With high-coverage datasets, naïve methods may already be calibrated, reducing the need for explicit calibration.
  • Understanding the Behavior of Cal-QL: Replacing Monte-Carlo reference returns with a neural-network reference-value approximator leaves Cal-QL performance largely unchanged when the approximator is reasonable.The experiment indicates that reference Q-function errors do not significantly affect performance under this condition.

8 Discussion, Future Directions, and Limitations

Cal-QL provides conservative offline initializations for fast online fine-tuning by calibrating value functions against a reference policy. The paper reports theoretical and experimental benefits, while leaving cross-task pre-training and fine-tuning mismatches for future work.

  • Discussion and contribution: Cal-QL constrains conservative value functions to be larger than a reference policy’s value function, helping avoid initial unlearning during fine-tuning.The approach is intended to retain the asymptotic performance of conservative methods while enabling faster online adaptation.
  • Limitations and future directions: A limitation is that the study does not consider fine-tuning setups where the pre-training and fine-tuning tasks differ.The paper identifies this setting as an avenue for future work.

Appendices

The appendices provide implementation details for Cal-QL and describe benchmark settings, evaluation metrics, mixing-ratio experiments, and baseline hyperparameters.

  • Implementation: The appendices include pseudocode and code listings for training the Q-function and policy.The listings show critic and actor optimization procedures alongside Algorithm 1.
  • Implementation: Cal-QL modifies CQL by replacing policy-action Q-values with max(q_pi_is, mc_return) in the critic update.The implementation labels this as Cal-QL’s modification.
  • Benchmark settings: Antmaze experiments use 1M offline pre-training steps followed by 1M online environment steps across medium and hard mazes.The study covers large-diverse, large-play, medium-diverse, and medium-play D4RL datasets.
  • Benchmark settings: Visual Manipulation uses 50K offline pre-training steps and 100K online environment steps, with five gradient steps per environment step.The domain is a multitask pick-and-place setting with distractor objects.
  • Evaluation: Normalized scores for sparse-reward visual-manipulation, Adroit, and Antmaze tasks are computed as goal-achievement or task-success rates.The Adroit door-binary metric is the success rate for opening the door.
  • Ablations: The mixing ratio m controls the offline-data fraction in online fine-tuning batches, while m = -1 activates an alternative buffer-setting option.For m = 0.25, each batch contains 25% offline data and 75% online data.
  • Hyperparameters: The appendices report hyperparameter procedures for CQL, Cal-QL, IQL, AWAC, SAC variants, and related baselines.IQL’s visual-manipulation sweep selected τ = 0.7 and β = 10; CQL variants use domain-dependent action-sample counts.

D D4RL locomotion benchmark

This appendix section presents tables for normalized D4RL locomotion scores before and after online fine-tuning and for cumulative regret.

  • Normalized scores: Table 5 reports normalized scores before and after online fine-tuning on D4RL locomotion tasks.
  • Cumulative regret: Table 6 reports cumulative regret on D4RL locomotion tasks, where smaller values are better and 1.00 is the worst.

E Extended Discussion on Limitations of Existing Fine-Tuning Methods

The discussion analyzes why existing fine-tuning methods improve slowly, finding that policy constraints and update choices can preserve or disrupt offline initializations.

  • IQL analysis: Changing IQL’s online temperature β has little to no effect on sample efficiency.The temperature controls how closely the learned policy matches the behavior policy.
  • IQL analysis: Increasing IQL to five gradient steps per collected sample does not improve its asymptotic performance, although it improves CQL.This indicates that update frequency alone does not explain IQL’s slow improvement.
  • Policy unlearning: A more aggressive IQL policy update often induces policy unlearning, resembling CQL’s failure mode.The analysis links slow learning to the tension between policy constraints and preserving the offline initialization.

F Initial Unlearning of CQL on Multiple Tasks

Across several domains, CQL initially unlearns its offline policy while Cal-QL mitigates this behavior; the appendix also states the assumptions underlying its theoretical analysis.

  • Initial unlearning: Cal-QL mitigates CQL’s initial unlearning and quickly recovers performance across Franka Kitchen, Adroit, and visual-manipulation tasks.For Antmaze, the default high-coverage dataset does not show initial unlearning, but narrowing the dataset distribution reveals a similar phenomenon.
  • Mixing-ratio ablation: Larger mixing ratios, representing more offline data, generally produce slower online-fine-tuning improvement on Adroit door-binary.The ablation compares Cal-QL and IQL.
  • Theoretical algorithm: The theoretical version alternates online data collection with conservative least-squares fitting on aggregated offline and online data.A reference policy is set for calibration during the procedure.
  • Assumptions: The analysis assumes a pessimistic function class containing the reference policy’s Q-function and closed under the relevant Bellman operation.The function class is constrained to conservative estimates relative to the reference policy.
  • Assumptions: The reference-policy class has bilinear rank dref no greater than the ambient bilinear rank d.This assumption supports the theoretical treatment of reference policies.
  • Theoretical scope: The paper states that the exact theoretical connection between pessimistic realizability and regularized Bellman consistency remains beyond scope.The assumption restricts the functional class and yields smaller rank and concentrability terms in the analysis.
  • Proof structure: The proof decomposes total regret into offline and online components and uses generalization, bilinear-model, and elliptical-potential arguments.The stated dependencies include the pessimistic realizability and reference-policy rank assumptions.

H.6 Our Results

The paper establishes a high-probability cumulative-suboptimality guarantee for Algorithm 2 against any comparator policy under bilinear-rank and function-class assumptions. The proof combines performance-difference arguments, Bellman-error bounds, bilinear-rank control, and inverse-covariance bounds.

  • Theorem H.5: With probability at least 1 −δ, Algorithm 2 obtains a cumulative-suboptimality bound with respect to any comparator policy πe.The formal result assumes the relevant function-class and bilinear-rank conditions.
  • Proof strategy: The proof adapts Hy-Q’s analysis, with major changes arising from Assumption H.1, Assumption H.3, and Definition H.4.The authors also restate Hy-Q’s major theoretical results for completeness.
  • Proof strategy: A telescoping decomposition and the performance-difference lemma connect cumulative suboptimality to bounds on offline and online Bellman errors.The proof explicitly invokes Lemma I.4 and the performance-difference lemmas.
  • Assumptions: Bilinear-rank assumptions control the relevant error terms, using rank d for the underlying MDP and rank dref for the reference-policy function class.The proof applies these assumptions on complementary events and combines their resulting bounds.
  • Proof conclusion: Setting moff = K and mon = 1 completes the theorem’s proof after substituting the offline and online error bounds.The final simplification uses subadditivity of the square-root function and K1, K2 ≤K.
Loading 2303.05479v4…