PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation

Yuheng Huang, Yunpeng Qing, Yixiao Chi, Yilun Kong, Changqing Zou

arXiv:2609.33127v1cs.LGcs.AI

TL;DR

O2O RL methods often treat the offline-trained policy as a given initialization, leaving open whether it remains plastic enough for online adaptation. The paper studies this issue and proposes REFIT, which transfers offline behavior to a freshly initialized, partially frozen student. Across D4RL and OGBench, REFIT achieves higher aggregate performance than corresponding O2O plug-in methods; the authors note that interactions with other O2O mechanisms remain to be studied.

  • Problem

    Existing O2O methods primarily focus on value calibration while treating the offline-trained policy as given, and the role of its plasticity in subsequent online adaptation is underexplored.

    Source: §1 · pp. 1–2

  • Method

    REFIT distills an offline teacher into a freshly initialized student while temporarily freezing a random subset of student units to preserve plasticity and transfer offline behavior.

    Source: §1 · p. 2

  • Results

    REFIT achieves higher aggregate performance than corresponding O2O plug-in methods across D4RL and OGBench for IQL and Cal-QL.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings identify policy plasticity as an important factor in O2O adaptation and support explicitly restoring it at the O2O transition.

    Source: §6 · p. 10

    How plasticity restoration interacts with other O2O mechanisms, and how complementary mechanisms can be combined, remain open questions.

    Source: G LIMITATIONS AND DISCUSSION · p. 19

Abstract

from arXiv · show

Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online interactions. Existing O2O methods primarily address the transition through value calibration, while generally treating the offline-trained policy as a given initialization. We instead study O2O adaptation from the perspective of network plasticity, asking whether the offline-trained policy remains sufficiently adaptable for online learning. Controlled experiments show that prolonged optimization on static offline data progressively reduces network plasticity even after offline performance has largely saturated, and that lower plasticity is associated with weaker subsequent online improvement. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT), a lightweight model-level method for the O2O transition. Before online fine-tuning, REFIT distills the offline policy into a freshly initialized student while temporarily freezing a random subset of student units, transferring the learned offline behavior to a more plastic policy initialization. Extensive experiments on D4RL and OGBench demonstrate that REFIT consistently achieves higher aggregate performance than existing O2O plug-in methods across both Cal-QL and IQL backbones, while plasticity diagnostics and ablations provide further evidence of restored network plasticity.

1 INTRODUCTION

O2O RL combines offline pretraining with online adaptation, but existing methods largely treat the offline policy as a fixed initialization and focus on value correction. The paper identifies policy plasticity as another constraint and introduces REFIT to transfer offline behavior to a more adaptable initialization.

1 INTRODUCTION

  • O2O RL pretrains on offline data and then adapts through online interaction, reducing reliance on costly exploration while permitting adaptation beyond offline data coverage.Online interaction can be costly and unsafe, while fixed datasets have limited coverage and pose OOD-action challenges.Source: §1 · p. 1
  • Offline-trained policies can lose plasticity, limiting their ability to move beyond suboptimal offline behavior during online fine-tuning.In a controlled bandit, IQL stays near the suboptimal peak, while IQL+Expert improves slowly; low actor FAU suggests data coverage alone is not the only constraint.Source: §1 · p. 2
  • REFIT distills the offline policy into a freshly initialized student and temporarily freezes random student units to preserve plasticity while transferring offline behavior.The method aims to retain the offline prior while supporting adaptation during online fine-tuning.Source: §1 · p. 2
  • REFIT outperforms existing plug-in O2O methods across D4RL and OGBench, with stronger performance across varied settings.The introduction reports results across two O2O backbones.Source: §1 · p. 2

2 RELATED WORKS

Prior O2O work has focused on transition challenges such as value estimation and distribution mismatch, while offline-induced policy plasticity loss remains comparatively underexplored. The paper positions REFIT as a plasticity-restoration approach that complements those existing lines of work.

2.1 OFFLINE-TO-ONLINE REINFORCEMENT LEARNING

  • Existing O2O methods address value underestimation and distribution mismatch, while offline-induced plasticity degradation has received comparatively limited attention.REFIT explicitly targets plasticity during O2O adaptation.Source: §2.1 · p. 3

2.2 PLASTICITY IN REINFORCEMENT LEARNING

  • REFIT’s plasticity focus is presented as complementary to methods targeting other O2O bottlenecks, including inaccurate value estimation.The paper identifies systematic study of interactions and joint incorporation of complementary mechanisms as future work.Source: §2.1 · p. 3, §2.2 · p. 3
  • Prior plasticity research examines degradation symptoms and mitigation strategies, but direct assessment and restoration of policy plasticity in the O2O pipeline remain underexplored.Prior approaches include resets, normalization, regularization, and architectural designs; REFIT distills an offline teacher into an initialized, partially frozen online student.Source: §2.2 · p. 3

3 PRELIMINARIES

The preliminaries frame reinforcement learning as policy optimization in an MDP and describe O2O RL as offline pretraining followed by online optimization. They define network plasticity as adaptability to new data distributions and use active-unit fraction as a diagnostic.

3 PRELIMINARIES

  • An MDP is specified by state and action spaces, transition dynamics, rewards, initial-state distribution, and a discount factor; RL seeks a policy maximizing expected discounted return.The O2O setup uses an offline dataset to pretrain value functions and a policy that initialize online optimization.Source: §3 · p. 3
  • O2O RL pretrains value functions and a policy from a behavior-policy dataset, then uses them to start online interaction and optimization.The preliminaries describe this as combining offline initialization with subsequent online learning.Source: §3 · p. 3
  • Network plasticity is the capacity to adapt continually to new data distributions, and FAU measures the fraction of hidden units with nonzero post-ReLU activation.FAU is computed for hidden layers under a state distribution.Source: §3 · pp. 3–4

4 UNDERSTANDING AND MITIGATING PLASTICITY LOSS IN O2O RL

Controlled evidence links offline over-optimization to declining plasticity and weaker online adaptation; REFIT transfers offline behavior to a fresh student through partial freezing and distillation.

4.1 EMPIRICAL ANALYSIS: PLASTICITY LOSS IN O2O RL

  • Across comparable offline performance levels, FAU declines with continued offline optimization, including after offline performance largely saturates.The study uses IQL on D4RL antmaze-medium-play with four offline-training budgets and 20 random seeds.Source: Figure 2 · p. 4, §4.1 · pp. 4–5
  • Higher-FAU policies show larger online gains, while online improvement diminishes as offline training increases from 0.25M to 2M steps.The 1M- and 2M-step policies show little further online improvement; Figure 2b shows a positive correlation between mean FAU and online performance gain.Source: Figure 2 · p. 4, §4.1 · p. 5
  • The findings motivate restoring plasticity at the O2O transition while preserving the offline policy’s learned behavior.REFIT transfers the offline teacher’s behavioral prior to a freshly initialized student for subsequent online adaptation.Source: §4.1 · p. 5

4.2 REFIT FOR PLASTICITY RESTORATION

  • REFIT distills the offline teacher into a fresh student while freezing a random subset of units during distillation.Only the policy network is refitted; after distillation the freeze mask is removed and all student parameters can be trained online.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

Experiments compare REFIT with O2O plug-in methods across D4RL and OGBench, then test plasticity diagnostics, design ablations, and robustness to offline training budgets. REFIT improves aggregate performance, restores actor FAU at the O2O transition, and remains effective across the tested settings.

5.1 MAIN PERFORMANCE RESULTS

  • REFIT improves aggregate scores over vanilla IQL by 35.4% and Cal-QL by 20.3%, outperforming corresponding plug-in methods across the evaluated benchmarks.Training curves show pronounced gains on OGBench and sparse-reward AntMaze tasks, where base algorithms and plug-in methods often plateau.Source: Table 1 · p. 6, §5.1 · p. 7, Figure 4 · p. 7

5.2 PLASTICITY DIAGNOSTICS

  • Actor FAU declines during offline pre-training and stays low with vanilla online fine-tuning, while REFIT sharply restores FAU at the O2O transition.Complementary weight-norm and SRank metrics show consistent trends.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 ABLATION ANALYSIS

  • Only the complete REFIT configuration achieves the highest overall score; fresh initialization, policy distillation, and partial freezing provide complementary benefits.The component ablation compares the baseline, reset, reset-plus-distillation, and full REFIT.Source: §5.3 · p. 8, Table 2 · p. 8
  • Intermediate freeze ratios achieve comparable aggregate performance, while both extremes perform notably worse; the authors use ρ = 0.5 by default.Without freezing, plasticity gains may be reduced; freezing the whole student prevents it from inheriting the offline behavioral prior.Source: §5.3 · p. 8, Table 3 · p. 8
  • Actor-only distillation largely avoids the performance drop seen with critic-distillation variants at the start of online fine-tuning.REFIT therefore distills only the actor and lets the critic adapt freely online.Source: §5.3 · pp. 8–9, Figure 6 · p. 8

5.4 ROBUSTNESS TO OFFLINE TRAINING BUDGET

  • Across offline training budgets from 0.25M to 2M steps, REFIT outperforms the baseline and maintains higher FAU during online fine-tuning.The baseline increasingly plateaus as its offline budget grows, while REFIT continues improving.Source: §5.4 · p. 9, Figure 8 · p. 9

6 CONCLUSION

The conclusion frames policy plasticity as an important factor in O2O adaptation and presents REFIT as a way to restore it at the O2O transition. The paper’s reported experiments cover simulated continuous-control, navigation, and robotic-manipulation benchmarks, and it provides supplementary code and configuration files.

6 CONCLUSION

  • Prolonged optimization on static offline data can reduce network plasticity even after offline performance largely saturates, and higher plasticity is associated with stronger subsequent online adaptation.REFIT transfers an offline teacher’s behavior into a freshly initialized student while temporarily freezing a random subset of units.Source: §6 · pp. 9–10
  • Experiments across D4RL and OGBench show that REFIT improves aggregate performance for IQL and Cal-QL and exceeds their corresponding O2O plug-in methods.The conclusion presents explicit plasticity restoration at the O2O transition as an effective intervention.Source: §6 · p. 10

ETHICS STATEMENT

  • The experiments use simulated continuous-control, navigation, and robotic-manipulation benchmarks, without human subjects, personal data, or safety-critical systems.The authors identify no specific ethical concerns beyond those common to general RL research.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors provide supplementary code and hyperparameter configuration files, with implementation details, hyperparameters, and hardware specifications documented in Appendix B.Source: REPRODUCIBILITY STATEMENT · p. 10

B.2 REFIT IMPLEMENTATION

REFIT inserts policy distillation between offline pre-training and online fine-tuning, with a randomly selected subset of student units temporarily frozen. The experiments use IQL and Cal-QL backbones and specify the training budgets and freeze-ratio settings.

B.2 REFIT IMPLEMENTATION

  • At the O2O transition, REFIT distills a fresh student policy from the offline teacher while freezing a random subset of actor units, then unfreezes them for online fine-tuning.The critic and value networks are inherited from offline training; the mask applies to kernel columns and biases in actor Dense layers, including the policy output head.Source: §B.2 · p. 14
  • Across environments and backbones, the implementation uses 10^6 offline steps, 10^5 distillation updates, and 3 × 10^5 online fine-tuning steps.The default freeze ratio is ρ = 0.5, with ρ = 0.1 for Cal-QL on HalfCheetah and Hopper.Source: §B.2 · p. 14

B.3 BACKBONE IMPLEMENTATION

  • REFIT is evaluated on IQL and Cal-QL, against OPT for IQL and PARS for Cal-QL.The implementations build on the respective public repositories.Source: §B.3 · p. 14

C COMPUTATIONAL COST

The supplementary sections quantify REFIT’s computational overhead, compare it with CPR, and examine additional ablations. They report modest sustained overhead and show that distillation helps retain the offline behavioral prior beyond what direct resetting provides.

C COMPUTATIONAL COST

  • IQL+REFIT takes 2298.9 seconds versus 2163.4 seconds for IQL, a 6.26% wall-clock increase.After a one-time JAX compilation offset near the end of distillation, the curves progress approximately in parallel.Source: C COMPUTATIONAL COST · p. 14, Figure 9 · p. 14

D ADDITIONAL COMPARISON WITH CPR

E SUPPLEMENTARY ABLATION ANALYSIS

  • The supplementary ablations examine distillation, network reinitialization, distillation loss, standard-deviation transfer, and unfreezing during online adaptation.Source: E SUPPLEMENTARY ABLATION ANALYSIS · p. 15

E.1 ABLATION OF POLICY DISTILLATION

  • Direct reinitialization beats standard fine-tuning on aggregate but falls short of REFIT, which combines structured reinitialization with distillation to preserve the offline behavioral prior.The comparison uses IQL, IQL From Scratch, and IQL+REFIT.Source: §E.1 · p. 15, Figure 10 · p. 15

E.2 ABLATION OF POLICY REINITIALIZATION

The ablations test which REFIT design choices support online adaptation and define additional metrics used to diagnose plasticity. They find that fresh initialization and online unfreezing matter, while performance is comparable across distillation losses and standard-deviation settings.

E.2 ABLATION OF POLICY REINITIALIZATION

  • 46.3 to 14.7 on halfcheetah-r when directly unfreezing offline parameters, versus default REFIT, supports fresh initialization for effective online improvement.The comparison spans four representative environments, with the largest degradation noted on tasks requiring significant adaptation.Source: Figure 11 · p. 16, §E.2 · p. 16

E.3 ABLATION OF DISTILLATION LOSS FUNCTION

  • All three distillation objectives yield comparable aggregate performance, with per-task differences largely within reported standard deviations.The objectives are MSE, forward KL divergence, and reverse KL divergence; MSE is adopted for computational simplicity.Source: §E.3 · p. 16, Table 6 · p. 16

E.4 ABLATION OF STANDARD-DEVIATION TRANSFER

  • Copy Std performs comparably to default REFIT across evaluated environments, indicating gains do not rely on increased exploration from resetting standard deviation.Aggregate scores differ by only a small margin; the authors attribute gains to plasticity restoration from structured reinitialization and distillation.Source: §E.4 · pp. 16–17, Table 7 · p. 16

E.5 ABLATION OF ONLINE UNFREEZING

  • Retaining the frozen-unit mask during online fine-tuning limits improvement and lowers asymptotic performance; unfreezing restores the student's full optimization capacity.Default REFIT removes the mask and updates the entire policy during online adaptation.Source: Figure 12 · p. 17, §E.5 · p. 17

F.1 DEFINITIONS OF ADDITIONAL PLASTICITY METRICS

  • Weight Norm tracks the L2 magnitude of all actor parameters, while SRank counts leading activation singular values needed to retain a fraction 1 − ϵ of total singular-value mass.SRank uses post-ReLU activations from sampled states; its relative criterion is invariant to global rescaling, and reported values are capped at 256.Source: §F.1 · p. 17

F.2 ADDITIONAL DIAGNOSTIC RESULTS

Additional diagnostics show that REFIT changes parameter scale at the online transition and preserves or expands representational rank on several tasks, with task-dependent results.

F.2 ADDITIONAL DIAGNOSTIC RESULTS

  • REFIT's actor weight norm drops sharply at the start of online fine-tuning, while the baseline maintains a relatively high norm across the O2O transition.The authors interpret this alongside FAU and SRank as complementary evidence for plasticity restoration, not as a standalone measure.Source: Figure 13 · p. 18, §F.2 · p. 18
  • REFIT sharply increases SRank on several tasks and is comparable on others, while baseline representations can stagnate; the relationship is task-dependent.Higher-rank representations suggest retained capacity to capture novel dynamics during online exploration.Source: Figure 14 · p. 18, §F.2 · p. 18

G LIMITATIONS AND DISCUSSION

REFIT addresses O2O adaptation through network plasticity, a perspective largely orthogonal to methods targeting inaccurate value estimation.

G LIMITATIONS AND DISCUSSION

  • How plasticity restoration interacts with other O2O mechanisms, including value-estimation corrections, remains an open direction for more systematic study.The authors identify joint incorporation of complementary mechanisms as future work.Source: G LIMITATIONS AND DISCUSSION · p. 19
Loading 2609.33127v1…