PaperlayerView exact revision on arXiv

Source-linked AI summary

Changepoint-Aware World Models: Detecting Dynamics Shifts and Recovering by Forgetting Stale Replay in Model-Based RL

Everest Yang

arXiv:2609.18950v1cs.LG

TL;DR

Abrupt changes can make a world model wrong, while stale replay slows an agent’s recovery. CAWM detects shifts from its own prediction error and forgets stale replay while retaining the learned representation. Across two robot-relevant shifts, it recovers faster than passive retraining and model respawning, with comparable asymptotic performance.

  • Problem

    When dynamics shift abruptly, a world model trained on the old regime becomes wrong, and stale replay can hold back recovery; the paper asks whether an agent can detect and respond to such shifts using its own signals.

    Source: §I · p. 1

  • Method

    CAWM detects shifts with an online CUSUM test on world-model prediction loss against a rolling baseline, then caps replay to forget stale data while retaining the learned representation.

    Source: §I · p. 1

  • Results

    Across two robot-relevant shift types and three seeds, forgetting recovers faster than passive retraining and model respawning, while reaching comparable asymptotic performance to respawning.

    Source: §I · p. 1

  • Takeaways & Limitations

    Forgetting is most beneficial when a shift is severe enough to make old data obsolete, while the learned encoder can still generalize across the shift.

    Source: §V · p. 4

Abstract

from arXiv · show

A robot's learned model of its own dynamics is only valid until those dynamics change: actuators wear, payloads shift, and joints stiffen. A model-based agent that keeps training as if nothing happened adapts slowly, dragged back by a replay buffer full of stale experience. We present Changepoint-Aware World Models (CAWM), a DreamerV3 agent that detects an abrupt dynamics shift from its own internal prediction error, using an online CUSUM test against a rolling baseline that fires only on abrupt change rather than on slow learning drift. It then forgets stale replay, keeping the learned representation while flushing obsolete data. On simulated locomotion under two robot-relevant shifts, doubled gravity and halved actuator gain, CAWM recovers substantially faster than passive retraining. It also beats a strong baseline that respawns a fresh dynamics model on detection, the deep-world-model analogue of model-bank methods. With the response triggered at the shift, CAWM gains +95 to +153 return in the first 30k post-shift frames over three seeds, while matching that respawn at asymptote. Running the detector in closed loop reproduces this gain on the gravity shift. The benefit holds across both shift types, and is largest when the shift is severe enough that old data is genuinely obsolete.

I. INTRODUCTION

The paper asks how an agent can detect abrupt dynamics shifts from its own signals and respond without being held back by stale replay. It proposes detecting shifts with an online CUSUM test and forgetting obsolete data while retaining the learned representation; recovery is faster than respawning a fresh dynamics model, with comparable asymptotic performance.

I. INTRODUCTION

  • The work separates two questions: whether an agent can detect a shift using signals it already computes without an external oracle, and how it should respond once it knows.Detection is studied first, then the response.Source: §I · p. 1
  • CAWM applies an online CUSUM test to world-model dynamics-prediction loss against a rolling baseline, then caps replay while retaining the encoder, actor, and critic.Only the data used to train the model is refreshed.Source: §I · p. 1
  • Forgetting stale experience avoids the respawn baseline’s sharp re-adaptation transient and second crash, while both approaches reach comparable asymptotic performance across two shift types and three seeds.The comparison is with an MBCD-style fresh dynamics model.Source: §I · p. 1
  • The authors report that forgetting helps most when a shift is severe enough to make old data obsolete, and that it outperforms passive retraining and model respawning at recovery.The reported comparison covers a gravity shift and actuator-wear shift.Source: §I · p. 1

II. RELATED WORK

Prior work studies world models largely under stationary dynamics, while changepoint and robot-adaptation methods use different detection, response, or policy mechanisms. CAWM combines detection and data refresh within a deep world model.

II. RELATED WORK

  • World-model RL methods such as DreamerV3 typically assume stationary dynamics, and model-disagreement exploration targets uncertainty in a fixed environment rather than changing dynamics.DreamerV3 is the base agent used in this work.Source: §II · p. 1
  • MBCD uses an ensemble and model-bank switch, while CAWM’s deep-world-model analogue respawns its dynamics model; the proposed forgetting path instead retains the encoder and flushes stale replay.The figure depicts internal prediction-error monitoring, CUSUM detection, and the replay-cap response.Source: §II · p. 1, Figure 1 · p. 2
  • Rapid Motor Adaptation addresses payload, wear, and terrain changes through online latent-context inference in a legged policy, whereas CAWM detects changes and refreshes world-model data.The approaches share a robot-adaptation motivation but operate through different mechanisms.Source: §II · p. 2

III. METHOD

The method models an abrupt transition change at an unknown changepoint, detects it from world-model prediction loss, and responds by replacing stale replay while preserving learned components. It also compares this response with reinitializing the dynamics model.

III. METHOD

  • Post-shift performance is measured by episode return in transient [0, 30k) and asymptote [30k, 90k) frame windows, plus their cumulative window as a return-based proxy for adaptation regret.The transition dynamics switch abruptly at an unknown changepoint and then remain fixed.Source: §III · p. 2
  • A rolling, lagged baseline standardizes the dynamics-prediction loss for a one-sided CUSUM, so sustained positive deviations trigger detection while slow learning drift is absorbed.The statistic fires above threshold h; slack k and threshold h trade off detection delay against false alarms.Source: §III · p. 2
  • On detection, CAWM caps replay to recent transitions so stale data is flushed within roughly Nkeep frames, while leaving the encoder, actor, and critic intact.The world model then trains on post-shift experience; unlike prioritized replay, obsolete stored transitions are discarded.Source: §III · p. 2
  • The controlled respawn baseline reinitializes the latent dynamics and prediction heads and resets the model optimizer, while keeping the encoder and policy.It isolates discarding the model versus discarding only the data after a shift.Source: §III · p. 2

IV. EXPERIMENTS

Across two robot-relevant shifts, CAWM recovers faster than passive retraining and an MBCD-style respawn, with its advantage concentrated in early recovery. The online detector also preserves the benefit on a gravity shift, while forgetting helps when the shift makes old data obsolete.

Setup

  • The walker_walk experiments compare passive training, forgetting, and respawn after gravity ×2.0 and actuator-gain ×0.5 shifts, using oracle-timed responses and three seeds.The online-detector gravity comparison is reported separately from the oracle-timed response comparisons.Source: §IV · p. 2

Recovery

  • +153 on gravity and +95 on motor-wear in the first 30k frames over the MBCD-style respawn; asymptotic gains (+18 and +27) are within seed variation.The cumulative-window gains are +63 and +50, respectively.Source: Table I · p. 3, §IV · p. 3
  • CAWM beats passive retraining across all windows, with gains reaching +294 at asymptote under motor-wear.Passive retraining remains dragged down by its stale replay buffer.Source: §IV · p. 2, Table I · p. 3, Figure 2 · p. 3

Consistency across shift types

  • Across gravity and motor-wear, the same detect-and-forget response works without per-shift tuning, despite the shifts changing different physical properties.Fig. 3 shows CAWM’s transient advantage over respawn shrinking to a tie at asymptote on both shifts.Source: §IV · p. 3, Figure 3 · p. 4

Closed-loop detection

  • With online detection on gravity ×2.0, CUSUM fired 200 to 1000 frames after the shift without pre-shift false alarms, and forgetting beat passive by +203 ± 18 at asymptote.Detector-triggered forgetting recovered to the same level as the oracle-timed response.Source: §IV · p. 3

Robustness across seeds

  • CAWM wins the first-30k window in all three seeds for both shifts, while its asymptotic ordering is not stable across seeds.On motor-wear, the asymptotic margins shrink across seeds to +2, supporting a tie rather than a higher asymptotic-performance claim.Source: §IV · p. 3

When does forgetting help?

  • Forgetting helps when shifts make old data obsolete, but under gravity ×1.5 passive retraining reaches roughly 900 while forgetting plateaus near 716.The result makes the benefit dependent on shift magnitude rather than establishing that forgetting is universally useful.Source: §IV · p. 3

Scope

  • The detector assumes a roughly converged pre-shift baseline; broader task coverage, gradual drift, and fully independent seeds remain future work.The compared seed arms share their pre-shift checkpoint and vary post-shift stochasticity.Source: §IV · p. 4

V. DISCUSSION

The discussion argues that, after an abrupt dynamics change, discarding stale data while retaining the learned representation can speed recovery. That benefit is a transient one and appears when the shift makes old data obsolete.

Discussion

  • The learned encoder can generalize across the shift because observations map to states the same way, even as state transitions change and the dynamics model is refit.The recovery advantage is transient rather than an asymptotic ceiling, and appears when the shift makes old data obsolete.Source: §V · p. 4
  • The detector reuses a world-model loss without extra networks; proposed next steps include testing other robot morphologies, graded forgetting, stronger baselines, and false-alarm calibration.The discussion names cheetah and quadruped bodies as broader morphology tests.Source: §V · p. 4

VI. CONCLUSION

Changepoint-Aware World Models detect abrupt dynamics shifts from internal prediction error and recover by forgetting stale replay while preserving the representation, outperforming passive retraining and fresh-model respawning across two shifts.

VI. CONCLUSION

  • Across two robot-relevant shifts, forgetting stale replay recovers faster than passive retraining and fresh-model respawning, with comparable asymptotic performance.Source: §VI · p. 4
  • The method detects abrupt dynamics shifts from a deep world model’s own prediction error and responds by forgetting stale replay while keeping the representation.Source: §VI · p. 4
  • The reported comparison spans two robot-relevant shifts and three seeds.Source: §VI · p. 4

Related papers

Loading 2609.18950v1…