Source-linked AI summary
Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, Jinwoo Shin
TL;DR
Offline-to-online fine-tuning can suffer severe bootstrap error when online state-action distributions differ from offline data, destroying a strong initial policy. The paper combines balanced replay with multiple pessimistic offline Q-functions, and reports stable, more sample-efficient, higher-performing fine-tuning across locomotion and manipulation tasks.
Problem
State-action distribution shift during online fine-tuning can make Q-values inaccurate and bootstrap errors destructive to the initial offline-RL policy.
Method
The method combines balanced replay of online and near-on-policy offline samples with an ensemble of pessimistically trained offline Q-functions.
Results
The method achieves stable fine-tuning while outperforming considered baselines in final performance and sample-efficiency across locomotion and manipulation tasks.
Takeaways & Limitations
The framework leverages offline samples for both offline training and online fine-tuning across continuous-control robotic tasks.
Takeaways & Limitations
The distribution-shift discussion distinguishes offline data from online data generated by a mixture of policies observed during fine-tuning.
Abstract
from arXiv · showhide
Recent advance in deep offline reinforcement learning (RL) has made it possible to train strong robotic agents from offline datasets. However, depending on the quality of the trained agents and the application being considered, it is often desirable to fine-tune such agents via further online interactions. In this paper, we observe that state-action distribution shift may lead to severe bootstrap error during fine-tuning, which destroys the good initial policy obtained via offline RL. To address this issue, we first propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples from the offline dataset. Furthermore, we leverage multiple Q-functions trained pessimistically offline, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase. We show that the proposed method improves sample-efficiency and final performance of the fine-tuned robotic agents on various locomotion and manipulation tasks. Our code is available at: https://github.com/shlee94/Off2OnRL.
1 Introduction
Offline RL can produce strong robotic agents, but fine-tuning through online interaction is difficult when state-action distributions shift. The paper addresses this with balanced replay and pessimistic Q-ensembles, reporting stable, sample-efficient improvements across locomotion and manipulation tasks.
- Offline RL may produce agents that outperform dataset behavior policies, yet suboptimal datasets or deployment environments motivate online fine-tuning.
- Distribution shift exposes the Q-function to unfamiliar state-action regimes, causing inaccurate bootstrap estimates and destructive policy updates.
- Balanced replay prioritizes online samples while selecting near-on-policy offline samples to improve value propagation and policy evaluation in novel regimes.
- A pessimistic Q-ensemble constrains early fine-tuning near the behavior policy and provides higher-resolution pessimism against unfamiliar actions.
- Experiments on MuJoCo locomotion and vision-based manipulation tasks show stable fine-tuning with improved final performance and sample-efficiency over considered baselines.
2 Background
The paper frames reinforcement learning around discounted return and stationary state-action distributions, then introduces SAC and CQL as relevant off-policy methods. CQL adds pessimistic value estimation through regularization of unseen and seen actions.
- An RL agent observes states, selects actions under a policy, receives rewards, and seeks to maximize discounted expected return.
- The stationary state-action distribution dπ(s, a) aggregates discounted occupancy under a fixed policy.
- SAC is an off-policy actor-critic algorithm that learns a soft Q-function and a Gaussian stochastic policy through alternating critic and actor updates.
- CQL is an offline RL algorithm that pessimistically evaluates the current policy and learns a lower bound in expectation of the ground-truth Q-function.
- CQL regularization decreases Q-values for unseen actions while increasing them for seen actions, and its policy-improvement step follows SAC.
3 Fine-tuning Offline RL Agent
Offline-to-online fine-tuning suffers when online data differs from the offline distribution, because inaccurate values can destabilize policy updates. The paper studies replay sampling and offline Q-function pessimism as remedies.
- 3.1 Distribution Shift in Offline-to-Online RL: Offline and online buffers have different state-action distributions, with VAE log-likelihoods showing a clear shift between their samples.
- 3.1 Distribution Shift in Offline-to-Online RL: Unseen online regimes can produce inaccurate Q-values, erroneous policy evaluation, and arbitrary updates that destroy the initial offline-RL policy.
- 3.2 Sample Selection: Online-only replay can destabilize fine-tuning, whereas uniform mixing can underuse online samples and slow value propagation, motivating balanced replay.
- 3.3 Choice of Offline Q-function: Pessimistic and non-pessimistic Q-functions perform similarly on the random dataset, whose random rollouts provide good action-space coverage.
- 3.3 Choice of Offline Q-function: On the medium-expert dataset, the non-pessimistic Q-function loses the initial policy and reaches zero average return at one point.
- 3.3 Choice of Offline Q-function: Pessimistic Q-values remain conservative around initially encountered novel states, while balanced replay lets pessimism gradually diminish as online data accumulates.
4 Method
The framework combines balanced replay with a pessimistic Q-ensemble to address state-action distribution shift during offline-to-online fine-tuning. Balanced replay prioritizes online-like transitions, while pessimistic CQL initialization limits overoptimism on unseen actions.
- Overview: The method has two components: balanced experience replay and a pessimistic Q-ensemble.Together, they target state-action distribution shift during fine-tuning.
- Balanced Experience Replay: Balanced replay prioritizes transitions by estimated online-ness, increasing sampling of online transitions and relevant near-on-policy offline samples.Samples are stored with density-ratio priorities and drawn proportionally to those priorities.
- Balanced Experience Replay: The priority is the density ratio w(s, a) := d_on(s, a)/d_off(s, a), estimated without explicit likelihood modeling from offline and online samples.The estimator uses a network trained solely on samples from B_off and B_on; d_off and d_on correspond to the offline behavior distribution and the observed online-policy mixture.
- Balanced Experience Replay: Density-ratio training uses a variational representation of the Jensen-Shannon divergence and applies self-normalization for more stable estimates.The estimator models a non-negative ratio w_ψ(x) using a neural network.
- Pessimistic Q-Ensemble: The pessimistic ensemble begins with N CQL agents pretrained offline, then updates the ensemble actor and critic during fine-tuning.The ensemble policy is formed from the component policies, while the parameters are updated using the fine-tuning rules.
- Pessimistic Q-Ensemble: Pessimistic Q-functions constrain behavior around the offline policy and remain pessimistic about unseen actions at online states during initial fine-tuning.Multiple pessimistic Q-functions provide higher-resolution pessimism and help protect the initial policy from bootstrap error.
5 Related work
Prior offline RL methods address action-distribution shift through behavior-policy constraints or pessimistic value functions, while ensemble methods target overestimation and bootstrap-error propagation. The paper builds on CQL and applies its approach to online fine-tuning, evaluated on MuJoCo locomotion tasks.
- Offline RL: Offline RL methods address state-conditional action-distribution shift by constraining the policy near the behavior policy or training pessimistic value functions.CQL is cited as a pessimistic offline RL method used for robotic manipulation and serves as a foundation for this work.
- Experimental Context: Figure 3 evaluates online fine-tuning performance on D4RL MuJoCo locomotion tasks using means and standard deviations across four runs.The figure provides the experimental setting for comparing offline-to-online methods and an online method trained from scratch.
- Ensemble Methods: Model-free RL ensembles have been studied to reduce Q-function overestimation, improve exploration, and reduce bootstrap-error propagation.The cited prior work mainly focuses on online RL and estimating ground-truth Q-functions.
6 Experiments
Experiments evaluate the method on MuJoCo locomotion and vision-based manipulation tasks, testing balanced replay, pessimistic Q-ensembles, dataset relevance, and baseline performance. The method generally provides stable, sample-efficient fine-tuning across the evaluated settings.
- Experimental setup: The evaluation covers MuJoCo locomotion tasks and three sparse-reward, pixel-based manipulation tasks with scripted and random exploratory data.Locomotion uses random, medium, medium-replay, and medium-expert datasets; manipulation includes pick-place, grasp-closed-drawer, and grasp-blocked-drawer.
- Comparative evaluation: The method outperforms the considered baselines on most locomotion tasks in both sample-efficiency and final performance.Baselines include AWAC, BCQ-ft, SAC-ft, and SAC trained from scratch.
- Balanced replay analysis: Offline samples become obsolete quickly on random data but remain useful throughout medium-expert fine-tuning because they contain relevant expert-level transitions.The replay ratio therefore depends on how well offline data matches the current policy.
- Ablation studies: Balanced replay improves fine-tuning, whereas Uniform and Online only sampling produce slower and less stable improvement.Balanced replay samples near-on-policy transitions while retaining online data.
- Ablation studies: Larger ensemble sizes improve walker2d-random fine-tuning performance, consistent with higher-resolution pessimism and more stable policy updates.The comparison evaluates ensemble sizes N ∈{1, 2, 5}.
- Robotic manipulation tasks: On manipulation tasks, the method consistently learns within 40K to 80K additional environment steps, while CQL-ft fails in some seeds and has high variance.Balanced replay also filters task-irrelevant random data as fine-tuning proceeds.
7 Conclusion
The paper identifies state-action distribution shift as the major obstacle in offline-to-online reinforcement learning and presents balanced replay with a pessimistic Q-ensemble. Experiments show strong performance across continuous-control locomotion and manipulation tasks.
- Conclusion: The framework combines balanced experience replay with a pessimistic Q-ensemble to address state-action distribution shift in offline-to-online reinforcement learning.The authors expect it to improve sample-efficient robotic-agent training by reusing offline samples during both offline and online learning.
- Conclusion: The proposed method performs well across many continuous-control robotic tasks, including locomotion and manipulation.
Supplementary Material
The supplementary material gives pseudocode for initializing pessimistic CQL agents, collecting online transitions, updating the agent, and maintaining replay priorities. It also notes the ensemble’s computational cost and possible parallelization.
- Pseudocode: The training procedure initializes N independent CQL agents with an offline dataset before forming an ensemble.The pseudocode then trains each agent through repeated minibatch updates.
- Pseudocode: During fine-tuning, the procedure collects transitions online, updates the density-ratio estimator, samples replay minibatches, and updates agent and priority values.Offline transitions are initially inserted into the priority buffer with a default priority.
- Computational complexity: A naive ensemble implementation increases training and inference time by N×, although network parallelization can improve computational efficiency.Parallelization was used for the MuJoCo locomotion tasks.
C Training Details for Concept Experiments
The concept experiments compare uniform and online-only replay strategies and evaluate likelihood-based sample relevance and offline Q-function choices. These procedures motivate balanced replay and pessimistic initialization for fine-tuning.
- Sample selection analysis: Uniform replay samples offline and online transitions agnostically from one buffer, while Online only uses online samples exclusively.Uniform replay can underuse online samples when the offline dataset is large.
- Choice of offline Q-function: Concept experiments train a CQL agent and a non-pessimistic FQE Q-function on halfcheetah random and medium-expert datasets for comparison.
D.1 Locomotion Tasks
The locomotion experiments use D4RL tasks and four dataset compositions, with specified offline and online training procedures for fine-tuning.
- Task detail & offline dataset: The locomotion benchmark includes halfcheetah, hopper, and walker2d, using random, medium, medium-replay, and medium-expert datasets.The datasets range from random or medium-policy rollouts to mixtures of medium- and expert-level data.
- Training details for offline RL: Offline CQL agents use two-layer MLP value and policy networks, except halfcheetah-medium-expert, which uses three-layer MLPs.The agents are built on the publicly available SAC implementation and follow the CQL training setup.
- Training details for online RL: Fine-tuning is evaluated for 250K timesteps across four random seeds, using ensemble size N = 5 for the proposed method.The method uses Adam, selected policy learning rates, a value learning rate of 3e−4, and extra updates after the first 1000 online samples.
- Training details for balanced replay: Balanced replay estimates sample priorities with a two-layer density-ratio network trained on equal-sized offline and online batches.The density-ratio estimator uses batch size 256 for each source and learning rate 3e−4.
- Training details for balanced replay: Self-normalization uses temperature T = 5.0 to stabilize density ratios across random minibatches.The priority computation treats x as (s, a) and P as Boff.
- Training details for balanced replay: The replay buffer initializes offline samples at priority 1.0 and assigns online samples a high default priority so the first 1000 online samples receive probability ρ of being sampled.After a sample is used, its priority is updated using the estimated density ratio; later, the default priority tracks the maximum observed priority.
- Baselines: Baselines include AWAC, BCQ and BCQ-ft, and SAC implementations drawn from publicly available author or benchmark code.BCQ-ft incorporates additional online samples during training.
D.2 Robotic Manipulation Tasks
The manipulation experiments evaluate the method on three sparse-reward pixel-based tasks with structured offline data and specified convolutional offline agents.
- Task detail & offline dataset: The manipulation benchmark contains pick-place, grasp-closed-drawer, and grasp-blocked-drawer tasks.The tasks require object placement or grasping from drawer configurations, including a blocked bottom-drawer handle.
- Task detail & offline dataset: The offline manipulation data contain highly structured transitions from randomized scripted behaviors and drawer, grasping, and placement behaviors.The dataset is rarely characterized as broad, unconstrained interaction data.
- Training details for offline RL: Offline CQL agents use the official implementation with convolutional and fully connected value networks.The architecture includes three convolutional layers, pooling layers, and fully connected hidden sizes of 1024, 512, and 256.
- Training details for online RL: Main fine-tuning results cover 1K episodes and 8 random seeds, using Adam with policy learning rate 3e−5 and value learning rate 1e−4.The density-ratio estimator uses the value-network architecture with smaller fully connected layers and learning rate 3e−4.
- Baseline: The CQL-online baseline applies CQL regularization during online fine-tuning while using online samples exclusively for updates.Its learning rates match those used during offline training.
E.1 Locomotion
The locomotion analyses find that balanced replay and pessimistic Q-ensembles stabilize fine-tuning, with benefits across replay, ensemble, regularization, and dataset-composition studies.
- Effects of CQL Regularization on Fine-tuning: Removing CQL regularization enables faster fine-tuning, whereas retaining it prevents improvement through online interaction; narrowly distributed data can still produce initial instability without regularization.
- Balanced replay analysis: Balanced replay automatically adjusts offline-sample use according to relevance to the current agent across locomotion environments.
- Q-ensemble analysis: The Q-ensemble shows superior AUROC discrimination between real and random-action transitions, supporting stable fine-tuning by limiting distribution shift.
- Ablation on balanced replay: Balanced replay significantly outperforms Online only and Uniform sampling, especially on hopper and walker2d.
- Ablation on ensemble size: Performance improves as ensemble size increases from N ∈{1, 2, 5}, while N = 5 balances computational overhead and performance.
- Dataset composition and performance: On halfcheetah after 1 million steps, the method reaches 15k average return on random data versus 12k on medium and medium-replay data.The authors conjecture that dataset composition explains this counterintuitive ordering.
- Dataset composition and performance: Initial pessimism does not harm asymptotic online performance, while preventing initial degradation that affects FQE-init on medium and medium-expert tasks.
- Exploring different density ratio schemes: The alternative density ratio don/don∪off yields similar performance across most setups and may trend toward 1.0 asymptotically.