PaperlayerView exact revision on arXiv

Source-linked AI summary

Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

Sami Nur Islam, Johan Obando-Ceron, Doina Precup, Aaron Courville, Pablo Samuel Castro

arXiv:2610.36247v1cs.LGcs.AI

TL;DR

When the offline dataset disappears, behavioral cloning can refit the behavioral model to the agent’s limited, nonstationary online experience rather than preserve offline behavior. The paper freezes the pretrained expressive prior while continuing online value and decision learning, and reports substantial gains across IFQL, FQL, and QC on seven OGBench domains. Its findings also identify a boundary: extreme environmental shifts may eventually require prior adaptation.

  • Problem

    When offline data is unavailable after pretraining, it remains unclear how an agent should be fine-tuned without retaining that dataset, which may be costly or infeasible to keep.

    Source: §1 · p. 1

  • Method

    The approach stops online behavioral fitting to preserve the pretrained prior while continuing critic and algorithm-specific decision improvement from online experience.

    Source: §2 · p. 3

  • Results

    Across seven OGBench domains (35 tasks), preserving the pretrained prior substantially improves no-retention fine-tuning for IFQL, FQL, and QC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    No-retention fine-tuning requires an expressive, multimodal behavioral prior and a decoupled decision mechanism that lets online value learning exploit it.

    Source: §6 · p. 10

    The frozen-prior approach is highly effective when pretraining and fine-tuning share underlying dynamics, while extreme environmental shifts may eventually require prior adaptation.

    Source: §6 · p. 10

Abstract

from arXiv · show

Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.

1 INTRODUCTION

The paper asks how offline-pretrained agents can improve when their offline dataset is unavailable, and identifies continued behavioral cloning as an actor-side failure in this setting. It freezes the expressive behavioral prior while online learning continues, substantially improving results across the evaluated methods and domains.

1 INTRODUCTION

  • No-retention fine-tuning asks how an offline-pretrained agent can improve through interaction after its original dataset is no longer available.The data may be costly or infeasible to retain because of storage, privacy, licensing, or deployment requirements.Source: §1 · p. 1
  • Prior work emphasizes critic recalibration under distribution shift, but leaves open what should happen to the behavioral model itself.The online transition changes the state–action distribution, while the critic must recalibrate from a small online replay buffer.Source: §1 · p. 1
  • Across seven OGBench domains (35 tasks), preserving the pretrained prior substantially improves no-retention fine-tuning for IFQL, FQL, and QC.Stopping BC alone is insufficient with a simple Gaussian prior, for which performance collapses.Source: Figure 1 · p. 2, §1 · p. 2
  • With offline data gone, BC fits the agent’s scarce, correlated, policy-dependent, nonstationary online actions rather than preserving the offline behavior distribution.Behaviors not revisited early may be undervalued and underused before improved value estimates can highlight them again.Source: §1 · pp. 1–2
  • The proposed principle freezes the expressive offline prior while allowing the critic and algorithm-specific decision improvements to keep learning online.IFQL uses a fixed flow policy for proposals; FQL and QC also retain their priors while updating value-related components.Source: §1 · p. 2

2 NO-RETENTION FINE-TUNING WITH EXPRESSIVE PRIORS

The section formalizes fine-tuning without offline data and distinguishes the behavioral prior from the decision policy that uses it. Its method freezes the pretrained prior while online critic and algorithm-specific improvement updates continue.

2 NO-RETENTION FINE-TUNING WITH EXPRESSIVE PRIORS

  • No-retention fine-tuning sets ρ_t = 0: after pretraining, updates use only the accumulated online data D_t^on.With retained data, training may instead mix offline and online state–action distributions.Source: §2 · p. 2
  • The behavioral prior models the conditional action distribution, while the decision policy may separately improve that prior using the critic and trainable parameters.Thus, fixing the prior does not require fixing the decision rule.Source: §2 · pp. 2–3
  • Flow matching fits a state-conditioned flow prior to replay actions; IFQL selects the highest-value sampled proposal, while FQL distills a one-step actor toward the flow.This separates behavioral flow fitting from value-guided actor improvement in FQL.Source: §2 · p. 3
  • BC-OFF freezes the pretrained prior but continues online critic and algorithm-specific improvement updates, including FQL actor distillation and QC chunk selection.BC-ON denotes continued behavioral fitting on online replay; BC-OFF denotes freezing the prior.Source: §2 · p. 3
  • The primary evaluation covers seven OGBench domains with five single-task variants each, using 1M offline gradient steps and 1M online environment steps.The offline buffer is discarded at the transition, and evaluations use three independent seeds per task.Source: §2 · p. 3

3 FREEZE THE PRIOR, IMPROVE THE DECISION RULE

This section tests how to fine-tune an offline-pretrained agent after its data are discarded. Across expressive policy classes, stopping BC while retaining the prior supports stronger online learning than continuing to refit behavior.

Stopping BC improves no-retention fine-tuning

  • Stopping BC consistently improves no-retention fine-tuning across IFQL, FQL, and action-chunked RL.The comparison covers five OGBench domains and 25 tasks; the full seven-domain plot is in Appendix B.4.Source: §3 · p. 3, Figure 3 · p. 4

Attenuating the BC penalty cannot prevent collapse

  • Scaling the BC weight from 0.1× to 10× fails to prevent collapse; only removing the loss enables sustained online improvement.Continued fitting on online-only replay progressively erases the offline prior.Source: §3 · p. 4, Figure 2 · p. 4

Can a large offline warmup rescue continued BC?

  • Seeding replay with 25,000 online transitions prevents BC-ON from flatlining, but BC-OFF still achieves superior online adaptation.The reported comparison covers IFQL and FQL on cube-double and antsoccer-arena.Source: §3 · pp. 4–5, Figure 4 · p. 5

Dissecting IFQL: Isolating the mechanisms of recovery

  • A sufficiently expressive prior matters: IFQL still improves with a Bellman double critic, while a Gaussian actor and ReBRAC without BC both collapse.The results indicate that stopping BC alone does not rescue a unimodal policy.Source: §3 · p. 5, Figure 4 · p. 5

Dissecting FQL: Distillation is not an optional regularizer

  • FQL collapses when its one-step actor is disconnected from the frozen flow prior, because distillation is its pathway for transferring offline behavior.Unlike IFQL, FQL does not sample directly from the flow.Source: §3 · pp. 5–6

Can BC-OFF recover performance from low-quality offline data?

  • With low-quality offline data, a frozen flow prior still supports online improvement above the initial offline baseline.The experiments use cube-double-noisy and antmaze-large-explore datasets.Source: Figure 4 · p. 5, §3 · p. 6

Generalization to Human Teleoperation Demonstrations

  • On Robomimic Multi-Human Can and Lift demonstrations, BC-ON degrades while BC-OFF prevents collapse and enables effective fine-tuning.The datasets contain 300 demonstrations from six operators of varying proficiency.Source: §3 · p. 6, Figure 5 · p. 6

Generality Across Generative Priors

  • Turning BC off also enables online adaptation with diffusion priors: Cube Double improves beyond the offline checkpoint and retained-data baseline, while Ant Soccer makes steady progress.With full buffer retention in Ant Soccer, continuing to train the prior instead degrades performance toward zero.Source: §3 · pp. 6–7, Figure 6 · p. 6

4 DIAGNOSING ONLINE COLLAPSE

With offline data discarded, continued BC fits the actor to narrow online replay and erases its pretrained behavioral prior. This triggers a feedback loop from actor collapse and lost exploration to critic degeneration; freezing the prior breaks that chain.

Catastrophic forgetting of the behavioral prior

  • BC-ON on restricted online replay rapidly raises offline action MSE for IFQL and FQL, while BC-OFF preserves the pretrained task manifold.The increase appears within the first 100k online steps.Source: §4 · p. 7

Actor collapse and data drift

  • Actor feature-rank collapse under BC-ON reduces cross-state and within-state action diversity, restricting rollouts to repetitive failure modes and narrow state-space coverage.On cube-double, temporal coherency also drops as broad exploration degenerates into localized jitter.Source: §4 · pp. 7–8, Figure 8 · p. 8, Figure 9 · p. 8

Data starvation breaks the critic

  • The actor’s restricted coverage starves the critic of diverse transitions, collapsing its feature rank and Q-value separation between top- and bottom-return trajectories.BC-OFF preserves actor diversity and supplies data that support critic features and continued policy improvement.Source: §4 · p. 8, Figure 8 · p. 8

Drift accumulation and prior degradation

  • The immediate-freeze optimum (τ = 0) contrasts with collapse under persistent BC (τ = 1M), while intermediate termination yields non-monotonic recovery.The output-adjacent layer diverges within 10^3–10^4 steps; deeper layers drift more slowly and can retain enough generative capacity for rescue.Source: §4 · pp. 8–9

Drift and optimizer statistics

  • Under BC-ON, parameter drift spreads from the output-adjacent layer and Adam’s second-moment estimate falls far below BC-OFF.Clearing optimizer state yields zero performance recovery across all six Discard+BC-ON configurations, ruling out stale statistics as the failure mechanism.Source: Figure 10 · p. 9, §4 · p. 9

5 RELATED WORK

Prior offline-to-online methods commonly retain offline supervision or add mechanisms to preserve its influence, while no-retention work has largely emphasized value learning. This paper instead examines continued behavioral fitting and whether expressive priors can remain useful when frozen.

Behavior regularization in offline-to-online RL

  • Offline behavior regularizers rely on the dataset to define the actions to preserve, an assumption challenged when that supervision disappears during fine-tuning.The cited approaches include BCQ, BEAR, BRAC, advantage-weighted methods, TD3+BC, and ReBRAC.Source: §5 · p. 9
  • No-retention methods such as WSRL and PORL address critic preservation or reconstruction, whereas this work retains the pretrained actor and critic to study continued behavioral fitting.WSRL uses pretrained-policy rollouts; PORL learns a new critic from interaction.Source: §5 · p. 9

Expressive policies for data-driven RL

  • Expressive diffusion and flow policies separate behavior-distribution modeling from value-based action selection or improvement, and Q-chunking extends expressive modeling to action sequences.The paper asks whether such a behavioral model remains useful as a fixed prior after its training data and fitting objective are removed.Source: §5 · pp. 9–10

6 LESSONS AND OPPORTUNITIES

No-retention fine-tuning calls for preserving a rich behavioral prior while allowing online value learning to exploit it through a decoupled decision mechanism. The authors note that hard freezing may need to give way to selective adaptation under extreme distribution shifts.

Lessons

  • When offline data disappear, BC imitates narrowing online experience and cannibalizes the prior, so successful online-only learning pairs an expressive multimodal frozen prior with a decoupled decision mechanism.Critic-guided selection or distillation can let online value learning exploit the preserved behavior manifold.Source: §6 · p. 10

Opportunities

  • A strictly frozen prior is highly effective when pretraining and fine-tuning share underlying dynamics, but extreme environmental shifts may eventually require adaptation.The authors suggest generative replay, uncertainty-aware plasticity, or selective behavioral updates as future directions.Source: §6 · p. 10

AI USE STATEMENT

The paper reports AI assistance in writing, related-work discovery, and code work, with authors reviewing the final content. Its related-work discussion distinguishes its intervention—stopping behavioral fitting after offline data vanish—from prior work on value stability, replay, and online policy adaptation.

AI USE STATEMENT

  • The authors used generative AI to polish writing, refine introduction structure, find related work, and assist with code and early section drafts.They state that early drafts were substantially rewritten and that they reviewed AI-assisted work and code.Source: AI USE STATEMENT · pp. 10–11

A ADDITIONAL RELATED WORK

  • Offline RL behavior constraints depend on dataset supervision, while many offline-to-online methods retain offline transitions or add mechanisms to preserve their influence.No-retention studies include WSRL, which uses pretrained-policy rollouts, and PORL, which starts from a policy-only initialization with a newly learned critic.Source: A ADDITIONAL RELATED WORK · p. 16
  • Replay-composition methods change which transitions the learner sees, whereas this work changes whether behavioral fitting continues after its supervision changes.Without the offline buffer, replay manipulation alone cannot restore the original behavior distribution unless the model has preserved it.Source: A ADDITIONAL RELATED WORK · p. 16
  • Diffusion, flow, and action-chunked policies provide expressive behavior models, while recent methods also explore adapting expressive policies online.This paper instead studies preserving the pretrained expressive prior and continuing improvement through the critic and decision rule.Source: A ADDITIONAL RELATED WORK · pp. 16–17

B EXPERIMENT DETAILS

The experiments reused shared offline-pretrained checkpoints for online fine-tuning. The setup also reduced redundant computation and avoided continued behavioral-model optimization.

B.1 COMPUTATIONAL RESOURCES

  • Each algorithm, task, and seed used a shared checkpoint after 1M offline gradient steps for all subsequent online fine-tuning setups.This decoupled offline pretraining from online fine-tuning and ensured matched starting conditions.Source: §B.1 · p. 17
  • Runs used NVIDIA H100 GPUs, multiplexing six experiments per GPU under a 12-hour budget, with 12 CPU cores and 124 GB memory.The study comprised several hundred online training runs across FQL, IFQL, ReBRAC, and QC.Source: §B.1 · p. 17
  • Checkpoint reuse avoids redundant offline computation, while freezing the behavioral prior removes continued behavioral-model optimization and may reduce computational overhead.The authors link these efficiency choices to accessible and reproducible deep RL research.Source: §B.1 · p. 17

B.2 IMPLEMENTATION AND HYPERPARAMETERS

The experiments use fixed reference implementation settings and evaluate OGBench tasks alongside Robomimic demonstrations from human operators.

B.2 IMPLEMENTATION AND HYPERPARAMETERS

  • Base optimization hyperparameters remained fixed across experiments to isolate the studied components, though independent tuning could change results at substantial computational expense.Implementations used official FQL and QC repositories and recommended task-specific settings without retuning the base parameters.Source: §B.2 · p. 17
  • FQL used four-layer, 512-unit MLPs, Adam at 3×10−4, 256-sample minibatches, and flow matching with 10 Euler steps.Its two Q-functions are averaged for the actor objective and, on these OGBench tasks, the critic target.Source: §B.2 · pp. 17–18
  • IFQL uses IQL expectile κ = 0.9 and selects the highest-valued of N sampled actions; QC uses H = 5 and selects the highest-valued of N sampled action sequences.ReBRAC retains default architecture and optimization settings, with actor and critic BC coefficients α1 and α2 as its method-specific hyperparameters.Source: §B.2 · p. 18

B.3.1 OGBENCH

  • The OGBench evaluation covers five single-task environments per domain across manipulation and locomotion, with semi-sparse rewards.The described tasks include cube and puzzle manipulation, soccer and maze navigation, and humanoid maze navigation.Source: §B.3.1 · p. 19

B.3.2 ROBOMIMIC

  • Robomimic evaluation uses Multi-Human teleoperation datasets with 300 successful demonstrations per task from six operators spanning three proficiency levels.The listed tasks are Lift and Can; demonstrations include natural variance, suboptimal paths, and multimodal execution strategies.Source: §B.3.2 · pp. 19–20

B.4 EVALUATION ACROSS ADDITIONAL DOMAINS

The evaluation includes two additional OGBench domains, with Figure 11 covering 35 tasks across seven domains for IFQL, FQL, and QC.

B.4 EVALUATION ACROSS ADDITIONAL DOMAINS

  • Figure 11 covers performance across 35 tasks—five per domain—for IFQL, FQL, and QC across all seven OGBench domains.The additional domains incorporated are Scene Play and Ant Maze Large.Source: Figure 11 · p. 20, §B.4 · p. 20

C MECHANISTIC ANALYSIS METRIC DEFINITIONS

The mechanistic analysis evaluates actor and critic representations and action behavior post hoc on a fixed offline validation set, separating network capacity from online replay quality.

C MECHANISTIC ANALYSIS METRIC DEFINITIONS

  • All metrics are evaluated post hoc on a fixed offline validation set to decouple network capacity from the evolving quality of online replay.The analysis tracks representational and behavioral collapse in both actor and critic.Source: C MECHANISTIC ANALYSIS METRIC… · p. 20
  • Actor feature rank counts the singular values needed to capture 99% of feature energy; higher rank indicates richer representations rather than a low-dimensional collapse.The metric is computed from the actor's penultimate-layer activations on 1,024 offline validation states.Source: C MECHANISTIC ANALYSIS METRIC… · p. 20
  • Within-state action diversity averages action standard deviations across validation states, distinguishing multiple proposals from deterministic collapse.Different random noise seeds generate candidate actions for each state.Source: C MECHANISTIC ANALYSIS METRIC… · p. 20
  • Between-state action diversity measures mean pairwise distance between predicted mean actions; a value near zero indicates a constant action across states.The metric uses 128 distinct validation states.Source: C MECHANISTIC ANALYSIS METRIC… · pp. 20–21
  • Critic feature rank applies the 99% threshold stable-rank measure to penultimate-layer activations from 1,024 offline state-action pairs.It measures whether the Q-function maintains expressive, high-capacity features for value prediction.Source: C MECHANISTIC ANALYSIS METRIC… · p. 21
  • Critic separation gap subtracts mean minimum Q-values for the lowest-return 20% of transitions from those for the highest-return 20%.A large positive gap indicates higher values for high-return states; a gap near zero indicates value distortion.Source: C MECHANISTIC ANALYSIS METRIC… · p. 21

D EXPERIMENTS

Resetting Adam’s state at the offline-to-online switch does not rescue the DISCARD+BC-ON collapse. The results indicate that stale optimizer statistics are not its cause; continuing behavioral fitting is implicated instead.

D EXPERIMENTS

  • The experiments reset Adam’s first- and second-moment estimates before online fine-tuning while keeping pretrained parameters and other hyperparameters unchanged.Source: D EXPERIMENTS · p. 21
  • The results indicate that stale optimizer statistics do not explain the collapse, which is associated with continued behavioral fitting on online-only data.Source: D EXPERIMENTS · p. 21
  • All six DISCARD+BC-ON cells show exactly zero change after resetting Adam, while BC-OFF effects are small and inconsistent.The figure compares the reset with no reset at the online switch.Source: D EXPERIMENTS · p. 21, Figure 12 · p. 22

Related papers

Loading 2610.36247v1…