PaperlayerView exact revision on arXiv

Source-linked AI summary

Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learning

Everest Yang, Skye Thompson, George D. Konidaris

arXiv:2609.18167v1cs.ROcs.LG

TL;DR

Continual model-based RL must adapt to changed robot dynamics without discarding older replay that may become useful again. The paper compares full-history and recent-window replay, characterizes their trade-off using change magnitude and age–staleness AUC, and evaluates interaction-data-based selection. Recent replay improves return after large permanent changes, whereas older replay is more useful when dynamics recur.

  • Problem

    The paper asks when recent transitions should replace full replay history, because older data can either slow adaptation after shifts or regain value when dynamics recur.

  • Method

    The study compares replay histories and recent windows across dynamics patterns, morphologies, algorithms, and benchmark perturbations, while estimating change magnitude and age–staleness from actuator responses.

  • Results

    Recent replay improves return after large permanent changes, but recurrence makes older data useful again and causes fixed recency windows to lose return relative to Passive replay.

  • Takeaways & Limitations

    Replay retention should depend on both change magnitude and how dynamics evolve over time, rather than on magnitude alone.

Abstract

from arXiv · show

Adapting to changes in robot dynamics requires learning from new data without discarding experience that may still be useful. In continual model-based reinforcement learning (RL), replay collected before a dynamics change can slow adaptation, while removing it unnecessarily reduces available training data and can be especially costly if earlier dynamics return. We study when recent transitions are preferable to the full replay history. Two quantities characterize this trade-off: change magnitude and age-staleness area under the curve (AUC), measuring how well transition age separates stale from fresh data. Forgetting stale data helps after large permanent shifts but hurts when dynamics recur and older data becomes useful again. Choosing a replay strategy therefore depends on predicting when older data will help or hurt. We test these effects across two locomotion morphologies, two model-based RL algorithms, and Real-World RL benchmark perturbations. Because ground-truth staleness labels are unavailable on deployed robots, we evaluate whether an estimator built from interaction data can still provide the quantities needed to choose a replay strategy after permanent changes. Our results show that replay retention depends on change magnitude and on how the dynamics evolve.

I. INTRODUCTION

The paper studies when older replay transitions help or hinder continual model-based RL under changing robot dynamics. It characterizes this trade-off using change magnitude and age–staleness, then evaluates replay choices and interaction-data-based selection.

  • I. INTRODUCTION: Replay usefulness depends on both the magnitude and temporal structure of dynamics changes.Older transitions can remain mismatched after lasting changes but become useful again when earlier dynamics recur.
  • I. INTRODUCTION: The study compares full-history replay with fixed windows of recent transitions to assess how staleness affects adaptation.Age–staleness AUC measures how well transition age separates outdated from current data.
  • I. INTRODUCTION: Experiments vary permanent changes, recurring changes, and no-change controls across two morphologies, two model-based RL algorithms, and RWRL perturbations.The study also varies replay-window size and buffer capacity.
  • I. INTRODUCTION: The paper reports that the same dynamics change can favor forgetting when persistent but retention when it recurs, so magnitude alone is insufficient.Its retrospective actuator-response analysis estimates change magnitude and staleness without simulator dynamics parameters.
  • II. RELATED WORK: Prior replay methods include local forgetting, informative-transition prioritization, diverse long-term replay, adaptive windows, and distribution-based weighting.This work focuses specifically on when transition age is useful for deciding retention.

III. PROBLEM FORMULATION

The formulation models continual model-based RL as ongoing world-model and policy updates after a dynamics change, with replay strategy determining which stored transitions remain available. It defines Passive and Recency through their replay populations and compares them using post-shift return differences.

  • A. Continual Learning and Retention: The agent continues updating its world model and policy after the environment dynamics change without resetting the learned model.At each step it acts, stores a transition, and samples replay for further updates.
  • A. Continual Learning and Retention: The replay strategy determines which stored transitions remain available for training.This makes retention a direct part of post-change adaptation.
  • A. Continual Learning and Retention: Passive replay samples from the full buffer, which has capacity one million transitions.The formulation contrasts this with a recent-data window.
  • A. Continual Learning and Retention: Recency restricts replay to the newest w = 10,000 transitions.The window defines the recent-data alternative to full-history replay.
  • A. Continual Learning and Retention: Js denotes mean episodic return for strategy s, averaged over evaluations during the 90k frames following the dynamics change.Each evaluation uses ten episodes.
  • A. Continual Learning and Retention: The primary comparison is the paired post-shift return difference between replay strategies.The comparison is defined after the dynamics shift.
  • A. Continual Learning and Retention: D > 0 favors Recency, and D is called the Recency advantage.The sign convention makes positive differences favor the recent-data strategy.

B. Staleness Labels and Age Ranking

Controlled experiments label transitions by the dynamics regime in which they were collected, making staleness observable from regime mismatch. Age–staleness AUC then measures whether older transitions are more likely to be stale, with recurrence requiring evaluation over time.

  • B. Staleness Labels and Age Ranking: A transition collected at τ is stale at t when its collection regime differs from the regime active at t.These controlled experiments therefore provide ground-truth staleness labels.
  • B. Staleness Labels and Age Ranking: Transition age is defined as At(τ) = t − τ and is used to rank stale against fresh data.Age-based forgetting is useful only when age is informative about staleness.
  • B. Staleness Labels and Age Ranking: Age–staleness AUC is the area under the receiver operating characteristic curve for separating stale and fresh transitions by age.It operationalizes the ranking quality of transition age.
  • B. Staleness Labels and Age Ranking: The AUC equals the probability that a random stale transition is older than a random fresh transition.An AUC of 1 indicates perfect ordering, while 0.5 corresponds to chance ranking.
  • B. Staleness Labels and Age Ranking: Under recurring dynamics, age–staleness AUC is averaged over the post-change evaluation window because transitions can alternate between stale and useful.Episodes are assigned to the regime active when they end.

IV. CHARACTERIZING REPLAY RETENTION UNDER DYNAMICS SHIFT

The paper frames replay retention as a bias–variance trade-off: forgetting can remove mismatch bias but incurs a variance cost from fewer samples. It combines this analysis with retrospective actuator-response replay selection based on estimated change and age–staleness.

  • A. Bias–Variance Trade-offs in Replay Retention: Retaining old data mainly reduces variance when dynamics mismatch is small, whereas larger changes lower the optimal weight on old data toward zero.Thus, larger changes favor forgetting under the model.
  • A. Bias–Variance Trade-offs in Replay Retention: Forgetting is beneficial when the bias reduction from removing stale data outweighs the variance cost of using fewer samples.The trade-off compares stale fractions in the full buffer and recent window.
  • A. Bias–Variance Trade-offs in Replay Retention: Retrospective Replay Selection takes an interaction trajectory, estimates actuator response, and detects stale segments before choosing a replay strategy.The procedure uses interaction-derived signals rather than simulator dynamics parameters.
  • A. Bias–Variance Trade-offs in Replay Retention: The selection rule chooses Recency when AUCage ≥ 0.87 and estimated change magnitude is at least 0.40; otherwise it chooses Passive.The rule maps estimated staleness separation and change magnitude to replay retention.
  • A. Bias–Variance Trade-offs in Replay Retention: Age–staleness AUC measures how well age distinguishes stale from fresh transitions but does not directly determine the stale fraction inside a window.Recency is useful only when stale-data removal outweighs reduced sample availability.
  • A. Bias–Variance Trade-offs in Replay Retention: The model varies change magnitude, recurrence, window size, and buffer capacity separately to test replay trade-offs across conditions.It abstracts away policy updates, nonlinear function approximation, and changing training data distributions.

B. Estimating Regime Changes from Actuator Response

The estimator infers dynamics changes from actuator responses measured through the agent’s actions and joint velocities, then uses estimated staleness and change magnitude to guide replay decisions.

  • B. Estimating Regime Changes from Actuator Response: Actuator-response estimation uses the agent’s actions and measured joint velocities to infer dynamics changes without privileged regime labels.The response signal can vary with contacts, state visitation, and policy behavior, so it is evaluated across joint-averaged, large-action, and full-vector variants.
  • B. Estimating Regime Changes from Actuator Response: Response channels are standardized by their median and median absolute deviation before PELT segments episode-level series into candidate regimes.PELT uses squared-error cost, a minimum segment length of five episodes, and penalty 3d log n.
  • B. Estimating Regime Changes from Actuator Response: Earlier segments are labeled stale when their difference from the final segment produces a two-sample z-statistic greater than three, yielding estimated AUC and change magnitude from segment medians.The threshold is set before evaluation rather than tuned on the evaluated runs.
  • B. Estimating Regime Changes from Actuator Response: The replay selector uses estimated age–staleness AUC and change magnitude, with thresholds of 0.87 and 0.40 for the evaluated 10k-transition window.The AUC threshold corresponds to a sampled recurrence condition with uncertain Recency advantage, while the magnitude threshold lies between gain factors where Recency hurt and helped.

A. Setup and Dynamics Changes

The study evaluates Passive and Recency replay under controlled gain changes and external perturbations, varying recurrence, magnitude, replay capacity, and window size across agents, algorithms, and checkpoints.

  • A. Setup and Dynamics Changes: The experiments compare Walker and Cheetah locomotion with DreamerV3 and TD-MPC2, including controlled gain changes and two Real-World RL perturbations.The benchmark perturbations are cyclic Walker joint damping and friction decreasing from 0.7 toward 0.01.
  • A. Setup and Dynamics Changes: Table I reports Walker post-shift return and age–staleness AUC over five seeds, with Recency helping after permanent changes but hurting under recurring dynamics and no change.Figure 3 tracks evaluation return before and after permanent and recurring gain reductions, while Figure 4 examines recurrence, magnitude, and window-size controls.
  • A. Setup and Dynamics Changes: All replay strategies within a seed branch from the same pre-shift checkpoint, using five paired seeds and reporting means with standard errors.Controls involving window size, capacity, and gain 0.6 use independently trained pre-shift checkpoints.
  • A. Setup and Dynamics Changes: The main Recency window holds 10,000 transitions, and a 10k reservoir buffer separates the effect of recentness from capacity by matching storage size with a different age distribution.A 20k Recency window tests sensitivity to window size.

VI. RESULTS

Recency improves adaptation after permanent dynamics changes but underperforms Passive replay when dynamics recur or remain unchanged.

  • VI. RESULTS: 146 ± 38 and 172 ± 42 return gains favor Recency after permanent gain reduction and permanent actuator damage, respectively.Both conditions have age–staleness AUC of 1.00, meaning stale transitions are older than fresh transitions in the buffer.
  • VI. RESULTS: 191 ± 17 and 201 ± 25 return deficits favor Passive replay under recurring dynamics and no dynamics change, respectively.Under recurrence, older transitions can become useful again, while Recency falls below Passive even before the change once its window is applied.
  • VI. RESULTS: Graded replay stays close to Passive under recurrence, avoiding the large loss produced by the fixed Recency window, while WMAR has the highest mean return in the Walker gain-reduction condition.These comparisons show that fixed recent-only replay is especially costly when older dynamics return.

B. Recurrence, Magnitude, and Window Size

Replay retention depends on both how much dynamics change and whether earlier dynamics recur: forgetting can help after permanent shifts, but retaining older data is safer when regimes return.

  • Recurrence: Recency’s advantage changes with recurrence: as age–staleness AUC falls from 1.00 to 0.53, its advantage moves from +146 to −287.At AUC 0.87, the advantage is −19 ± 14, so the sign is uncertain; recurrence period also changes training time under each regime.
  • Magnitude: Recency’s advantage is negative at g = 1.0, 0.9, and 0.7 but positive at g = 0.5, while g = 0.3 is near the task floor.Separately trained checkpoints narrow the transition between harmful and useful forgetting to between g = 0.6 and g = 0.5.
  • Magnitude: At g = 0.6, Recency is −154 ± 36, whereas at g = 0.5 it is +68 ± 26; actuator responses estimate magnitudes of 0.35 and 0.43, respectively.These estimates provide a magnitude signal separating the two cases for the selector’s gate.
  • Window size: Under recurrence, larger replay windows reduce Recency’s loss because older transitions become useful when previous dynamics return.When recurrence is likely, retaining older data or widening the window is preferable.
  • Replay controls: Recency improves after a permanent gain change, but a 10k reservoir performs worse than Passive under both permanent and recurring dynamics.Because both retain 10k transitions, the result indicates that which transitions are retained matters beyond replay capacity.
  • Transfer: Cheetah reproduces the Walker pattern, with Recency +184 ± 35 after damage and −126 ± 12 under recurrence; TD-MPC2 is −123 ± 13 under recurrence but only +8 ± 15 after damage.The recurrence cost transfers across morphology and algorithm, while the permanent-shift benefit is weaker for TD-MPC2.
  • External benchmark: On RWRL, Recency is −222 ± 32 under cyclic damping, while its +94 ± 58 drifting-friction advantage is inconclusive because the paired bootstrap interval includes zero.Across settings, forgetting is consistently costly under recurrence, whereas its benefit under nonrecurring changes depends on conditions.

D. Retrospective Replay Decisions

An interaction-data detector identified permanent actuator-response gain changes within three episodes, while the selector often chose the better replay strategy across evaluated histories.

  • Across all 24 histories, the selector chose the replay strategy that performed better in 19 cases.For permanent changes at g = 0.5, it selected Recency in three of five runs.

A. Implications for Replay Design

Replay retention should follow both the magnitude and temporal evolution of dynamics changes. Recent-only replay can help after lasting shifts but discard useful experience when dynamics recur, and interaction-based detection alone does not reliably select the strategy.

  • 191 ± 17 lower return under recurrence shows that fixed recent-data windows can discard experience that later becomes useful.Without a dynamics change, Recency’s return is also 201 ± 25 below Passive.
  • Detecting a dynamics shift does not determine whether older replay should be discarded; selection additionally requires estimated change magnitude and age–staleness separation.The detector identified the gain change in every seed, but the selector chose the better strategy in only three of five runs.
  • Age–staleness AUC ranks stale versus fresh transitions but does not indicate how much stale data lies within a particular replay window.Validated staleness labels were unavailable for the Real-World RL perturbations, so AUC was not reported there.
  • The actuator-response analysis is retrospective, with thresholds and evaluation histories drawn from Walker; an online version would require past-only decisions and unseen-condition evaluation.Its estimates can also vary with policy behavior and state visitation.
  • Recent-only replay improves return after large permanent changes, whereas older transitions become useful again when dynamics recur.This pattern also appears across Cheetah, TD-MPC2, and an external benchmark perturbation.
Loading 2609.18167v1…