PaperlayerView exact revision on arXiv

Source-linked AI summary

RIFAR: Reliability-Informed and Forgetting-Aware Replay for Continual Robot Learning with World-Action Models

Tu Lin, Zheng Lu, Zirui Song, Zixian Lin, Zebang Feng, Meijia Chen, Xiaoyu Sun, Wanru Li, Lijie Wang, Xiuying Chen

arXiv:2610.03079v3cs.AI

TL;DR

Continual robot learning must add skills without forgetting earlier ones, yet full demonstrations are costly and WAM-generated trajectories may be unreliable or insufficiently prioritized. RIFAR screens compactly reconstructed historical trajectories and uses action drift to guide replay. Across three LIBERO suites and real-world tasks, it outperforms REGEN; on LIBERO-Goal it reaches 90.97 AUC with 320 retained historical time steps per task.

  • Problem

    Continual learning must preserve earlier robot skills while adapting to new tasks, but full demonstrations are costly and WAM replay faces reliability and prioritization challenges.

    Source: §1 · p. 1

  • Method

    RIFAR reconstructs historical trajectories from compact contexts, screens action–visual consistency with a trainable inverse-dynamics model, then prioritizes reliable trajectories with greater action drift after initial adaptation.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    RIFAR consistently outperforms REGEN across three LIBERO suites and real-world tasks; on LIBERO-Goal, it achieves 90.97 AUC while retaining 320 historical time steps per task.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    RIFAR remains competitive with real-data replay while using compact historical contexts, including a LIBERO-Goal result slightly above ER(10) with fewer retained steps.

    Source: §5 · p. 8

    IDM-based screening cannot completely remove unreliable replay samples, which may add noise to IDM training and affect later reliability estimates.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Genuine embodied agency requires robots to turn continuous real-world experience into lasting, transferable skills. This demands continual learning that integrates new capabilities while preserving prior knowledge in evolving environments. Experience replay mitigates catastrophic forgetting, but storing complete demonstrations becomes costly as tasks accumulate. World-action models provide an alternative through generative replay. However, visually coherent rollouts may contain actions that cannot realize the predicted transitions, while new-task adaptation can disrupt learned behavior. To address these problems, we propose RIFAR (Reliability-Informed and Forgetting-Aware Replay), a reliability-informed and drift-aware generative replay framework. It reconstructs trajectories from compact historical demonstration prefixes and uses a trainable inverse-dynamics model to assess action-visual consistency. It first adapts to new tasks using current-task demonstrations and high-reliability replay, then evaluates how this adaptation changes action predictions on historical inputs and prioritizes reliable trajectories with greater action drift for further training. Across three LIBERO suites and real-world experiments, RIFAR outperforms the previous state of the art in WAM-based generative replay. On LIBERO-Goal, it achieves an AUC of 90.97 while retaining only 320 historical time steps per task, about 4.9% of those retained by 50-demonstration replay.

1 Introduction

Continual robot learning must add manipulation skills without losing earlier capabilities, but storing full demonstrations is costly and generated rollouts may be unreliable. RIFAR screens historical trajectories and uses action drift to prioritize replay; it outperforms REGEN across three LIBERO suites and real-world tasks with compact storage.

Motivation

  • Continual robot learning must acquire new manipulation skills while retaining earlier capabilities, because adapting a shared policy can disrupt previously learned behavior.Such disruption can cause catastrophic forgetting (Kirkpatrick et al., 2017).Source: §1 · p. 1

Background

  • Full demonstrations make storage increasingly costly as tasks accumulate, while generative replay reconstructs historical examples but WAM-based continual learning remains largely unexplored.REGEN is an initial approach, but preserving prior knowledge with generated trajectories remains challenging.Source: §1 · p. 1

Motivation

  • WAM replay faces two distinct challenges: generated actions may not realize predicted visual transitions, and reliable trajectories alone do not reveal which behaviors new-task adaptation most disrupted.RIFAR addresses these needs through action–visual reliability screening and drift-aware replay selection.Source: §1 · pp. 1–2

Approach

  • RIFAR reconstructs historical trajectories from compact prefixes, screens action–visual consistency with a trainable inverse-dynamics model, and prioritizes reliable trajectories with greater action drift after initial adaptation.Phase 1 combines current demonstrations with high-quality past-task trajectories; Phase 2 selects reliable trajectories with larger drift for further replay.Source: §1 · p. 2, Figure 1 · p. 2

Results

  • RIFAR improves AUC over REGEN by 23.17, 5.39, and 9.93 points across LIBERO-Goal, LIBERO-Object, and LIBERO-Spatial, respectively, and achieves 57.78 AUC in real-world tasks.On LIBERO-Goal, it reaches 90.97 AUC with 320 retained historical time steps per task, versus 90.70 AUC for ER(10) with approximately 1,300 steps.Source: §1 · p. 2

2 Related Work

Related work spans methods for retaining prior knowledge, continual learning in vision-language-action policies, and world-action models for generative replay. RIFAR builds on the latter while addressing reliability and replay-priority gaps identified for REGEN.

2.1 Continual Learning

  • Continual-learning approaches include Fisher-importance constraints in EWC, task-specific weights in PackNet, parameter-efficient LoRA, and replay of earlier examples.The cited work notes that parameter efficiency alone does not guarantee retention.Source: §2.1 · p. 3

2.2 Vision-Language-Action Models for Robot Manipulation

  • VLA continual-learning work includes adapter expansion and routing, goal-conditioned value modeling with asymmetric regularization, LoRA with on-policy reinforcement learning, and replay-based methods.PHASER allocates replay by manipulation phase and prioritizes samples using language–vision similarity and action divergence; Info-VLA uses replay-anchored contrastive learning.Source: §2.2 · p. 3

2.3 World Action Models for Continual Robot Learning

  • WAMs jointly predict actions and future observations, enabling recurrent trajectory generation for replay, while REGEN is an initial continual-learning approach.The cited discussion identifies missing action–visual consistency screening and adaptation-aware replay priorities as gaps motivating RIFAR.Source: §2.3 · p. 3

3 Method

RIFAR generates historical replay from compact task-compatible contexts, screens trajectories for action–visual consistency, and prioritizes reliable replay according to action drift after adaptation. Its two-phase training combines current-task demonstrations with screened replay, then reselects trajectories for further training.

3.1 Problem Setup

  • RIFAR trains a single policy across successive manipulation tasks, evaluating each updated policy on the new task and all previously learned tasks.Complete historical demonstrations are no longer available after a task becomes historical; compact contexts support replay generation.Source: §3.1 · pp. 3–4

3.2 Initial-State Anchored Historical Imagination

  • For each task, RIFAR retains 320 initial observation–proprioception pairs from selected demonstrations as anchors for generating long-horizon replay trajectories.It selects the first 20 demonstrations and retains their initial 16 time steps, then recursively feeds predicted future pairs into generation.Source: §3.2 · p. 4

3.3 Reliability Screening

  • An inverse-dynamics model assesses action–visual consistency, and RIFAR retains screened trajectories using smoothed agreement scores and the lowest-scoring 20% of chunks.The IDM predicts actions from initial and endpoint images; trajectories with persistent low agreement are also rejected.Source: §3.3 · pp. 4–5

3 Method

  • Training proceeds in two phases: 500 steps with current-task demonstrations and selected replay, followed by 300 steps with replay reselected by action drift.The two phases total 800 optimization steps per incremental task.Source: §3 · p. 3, §3.3 · p. 5, §3.4 · p. 5

3.4 Drift-Based Replay Prioritization

  • RIFAR compares action predictions on identical historical contexts before and after adaptation, then prioritizes reliable trajectories with greater drift for Phase 2 replay.The priority averages drift among contexts in the upper 20% tail, emphasizing localized policy changes; ten trajectories are selected per historical task.Source: §3.4 · p. 5

4 Experiments

RIFAR is evaluated on three LIBERO suites and real-world Franka tasks, with comparisons against REGEN and experience replay. Ablations examine anchor count, reliability screening, and drift-aware selection.

4.1 LIBERO Evaluation Setup

  • LIBERO evaluation uses Cosmos Policy across Goal, Object, and Spatial, with Tasks 0–5 trained jointly before Tasks 6–9 are introduced sequentially.RIFAR retains the first 16 time steps of selected demonstrations; both phases combine current-task demonstrations with replay.Source: §4.1 · p. 5

4.2 Main Results and Storage Efficiency

  • Across three LIBERO suites, RIFAR achieves higher AUC and lower NBT than REGEN; on LIBERO-Goal, AUC rises from 67.8 to 90.97 and NBT falls from 46.7 to 7.45.Source: Table 1 · p. 6, §4.2 · p. 6
  • On LIBERO-Goal, RIFAR’s 90.97 AUC slightly exceeds ER(10)’s 90.70 with approximately one-quarter as many retained time steps; ER(50) reaches 91.80 with approximately 6,500 steps per task.Source: Figure 3 · p. 6, §4.2 · p. 6

4.3 Ablation Studies

  • Increasing LIBERO-Goal anchors from 5 to 20 raises AUC from 80.5 to 90.97; K = 50 reaches 91.5, while K = 20 uses 40% as many anchors and stays within 0.53 points.Source: Figure 3 · p. 6, §4.3 · p. 6
  • Under matched 800-step training, drift-aware reselection reduces all three drift statistics on Task 3 across M6–M9, though maximum drift rises on some other tasks.Source: Figure 3 · p. 6, §4.3 · pp. 7–8
  • On LIBERO-Goal, adding drift-based selection to IDM screening raises AUC from 87.14 to 90.97 and lowers NBT from 10.61 to 7.45; either component improves on anchor-only replay.Source: §4.3 · p. 6, Table 2 · p. 7
  • IDM-screened trajectories achieve 76.9% execution success versus 44.0% for unfiltered candidates, a 32.9-point difference.The analysis links this screening signal to agreement between generated actions and visual transitions.Source: §4.3 · p. 7

4.4 Real-World Continual Learning

  • On Franka tasks, RIFAR reaches 57.78 AUC versus REGEN’s 50.39; ER(50) remains highest at 59.83, leaving RIFAR 2.05 points behind.RIFAR also improves FWT from 57.67 to 61.00 and lowers NBT from 26.77 to 8.84 versus REGEN.Source: Figure 5 · p. 8, Table 4 · p. 8, §4.4 · p. 8

5 Conclusion

RIFAR combines compact-context trajectory reconstruction, inverse-dynamics screening, and action-drift-guided replay to preserve earlier skills while learning new tasks. Across three LIBERO suites and real-world Franka experiments, it achieves higher AUC and lower NBT than REGEN while remaining competitive with real-data replay.

5 Conclusion

  • 90.97 AUC on LIBERO-Goal with 320 historical time steps per task, about 4.9% of ER(50)'s storage and above ER(10)'s 90.70 AUC.ER(10) retains approximately 1,300 steps per task.Source: §5 · p. 8
  • Across three LIBERO suites and real-world Franka experiments, RIFAR achieves higher AUC and lower NBT than REGEN while remaining competitive with real-data replay.Source: §5 · p. 8
  • RIFAR screens compact-context reconstructions for action–visual consistency, then uses action drift to prioritize earlier behaviors most affected by new-task learning.Source: §5 · p. 8

Limitations

RIFAR's reliability screening does not eliminate all unreliable replay, and generated trajectories can add noise to IDM training and subsequent reliability estimates. Its WAM also uses value-based termination, which may stop generation before task completion.

Limitations

  • Generated replay may be less reliable than demonstrations, adding noise to IDM updates and affecting later reliability estimates despite screening.The IDM is updated using newly collected demonstrations and WAM-generated replay data.Source: Limitations · p. 9
  • Value-based termination may stop autoregressive generation before a task is actually complete.Source: Limitations · p. 9
  • The authors propose stronger trajectory verification and reliability models, plus explicit task-success verification to improve termination reliability and trajectory completeness.Source: Limitations · p. 9

A.1 LIBERO Benchmark

  • LIBERO evaluation covers three suites of 10 language-conditioned tasks each, with Tasks 0–5 used for initial training and Tasks 6–9 introduced sequentially.The suites vary in goal-level manipulation, object-level generalization, and spatial reasoning.Source: §A.1 · p. 10

A.2 Real-World Experimental Details

Real-world continual learning experiments use a Franka robot to learn three manipulation tasks sequentially, with prior tasks reevaluated after each learning stage.

A.2 Real-World Experimental Details

  • Real-world continual learning is conducted on a Franka robot using the Cosmos Policy architecture and a 16-step action horizon.Source: §A.2 · p. 10
  • Cube pick-and-place, board wiping, and table cleaning are learned sequentially, with 50 demonstrations per task and 50 evaluation trials per task.Source: §A.2 · p. 10
  • Initial states are manually reset with small variations from training configurations, and previously seen tasks are evaluated after each stage using shared manual success criteria.Source: §A.2 · p. 10

B.1 Training Protocol

The LIBERO protocol begins with joint training on Tasks 0–5 and then introduces Tasks 6–9 sequentially. RIFAR uses a two-phase replay procedure, with an inverse-dynamics model trained and updated to assess replay reliability.

B.1 Training Protocol

  • Base policies are initialized from a pretrained Cosmos-Predict2-2B-Video2World checkpoint and fine-tuned in Cosmos-Policy; Tasks 0–5 are jointly trained before sequential learning on Tasks 6–9.The same optimization configuration is applied across all LIBERO suites.Source: §B.1 · p. 10, §B.2 · p. 10
  • Each continual-learning stage uses two optimization periods totaling 800 steps, with replay updated between periods using drift-aware selection.The first period is 500 steps and the second is 300 steps; data ratios change during the first period.Source: §B.1 · p. 10

B.3 IDM Training Procedure

  • The IDM predicts 16 intermediate 7-dimensional actions from image observations separated by 16 time steps, and is initially trained on real demonstrations from LIBERO Tasks 0–5.Source: §B.3 · p. 11
  • At each stage, the IDM update combines 50 current-task demonstrations with the top 10 replay trajectories from each previous task, selected by reliability score.For real-world experiments, the IDM starts on Task 1 and is updated as Tasks 2 and 3 arrive.Source: §B.3 · p. 11
  • Figure 6 organizes success rates by successive learning-stage checkpoints in rows and task groups S0–S4 in columns; masked cells are not-yet-introduced tasks.The lower-triangular matrices show acquisition and retention, averaged over three random seeds.Source: Figure 6 · p. 12

B.4 Complete RIFAR Algorithm

  • RIFAR reconstructs trajectories from compact prefixes, filters them for inverse-dynamics consistency, then selects high-drift reliable trajectories for a second replay phase.The final phase trains using the current-task data together with the selected high-drift replay.Source: §B.4 · p. 12

B.5 Evaluation Protocol

The protocol evaluates continual learning after sequential task additions, measuring both acquisition on new tasks and retention of earlier skills. It uses complementary aggregate metrics and per-task success rates to characterize performance.

B.5 Evaluation Protocol

  • Each stage evaluates all previously encountered tasks after new tasks are introduced sequentially, measuring acquisition and retention under a shared task order and evaluation setup.Training begins on Tasks 0–5; Tasks 6–9 are then introduced one at a time.Source: §B.5 · p. 12
  • AUC summarizes overall continual learning across task acquisition and knowledge retention, while FWT measures new-task acquisition and NBT measures degradation of earlier tasks.Higher FWT and AUC indicate better performance; lower NBT indicates less degradation.Source: §B.5 · p. 12

C.1 Task-wise Continual Learning Performance

The appendix details task-wise continual-learning evaluation, WAM rollout termination, and IDM-based replay reliability scoring. It specifies how rollout success is labeled and how generated trajectories are screened for replay.

C.1 Task-wise Continual Learning Performance

  • Task-wise success-rate matrices show performance after each continual-learning stage, with rows for stages and columns for previously encountered tasks.The 5 × 5 matrices cover the three LIBERO suites, with rates averaged over three random seeds; Figures 7 and 8 show per-task comparisons for LIBERO-Goal and LIBERO-Spatial.Source: §C.1 · p. 13, Figure 7 · p. 14, Figure 8 · p. 14

C.2 WAM Rollout and Termination Criteria

  • A WAM rollout is labeled successful if its value prediction reaches 1.0 followed by three consecutive predictions above 0.99, or stays above 0.99 for six consecutive steps.Rollouts that meet neither condition within the 200-step horizon time out and are labeled false before IDM filtering.Source: §C.2 · p. 13

C.3 IDM-based Trajectory Reliability Scoring

  • IDM reliability scoring uses prediction errors on 16-step, seven-dimensional action chunks and maps lower errors to higher scores.Chunk scores are smoothed with a three-chunk trailing average, and the trajectory soft score is the mean of the lowest 20% of smoothed values.Source: §C.3 · p. 14
  • A trajectory is reliable when its soft score is at least 0.65 and it has fewer than three consecutive chunks with IDM scores below 0.40.When fewer than 10 reliable trajectories are selected, trajectories are duplicated to reach 10; suspect trajectories are used only when none are reliable.Source: §C.3 · pp. 14–15

C.4 Drift Analysis across Continual Learning Stages

The appendix analyzes behavioral drift under drift-aware replay, the overlap between replay-selection stages, and computational resources. It reports generally lower aggregate drift with drift-aware selection and describes how replay-set overlap is interpreted.

C.4 Drift Analysis across Continual Learning Stages

  • Drift-aware replay generally yields lower Mean, Max, and Tail behavioral drift than replay without drift-aware selection, despite increases in some individual stage statistics.Tail is the average drift among the top 20% highest-drift samples.Source: §C.4 · p. 15, Table 10 · p. 16

C.5 Replay Selection Overlap Analysis

  • Replay-selection overlap counts shared trajectories between replay1 and replay2, indicating samples selected as valuable by both stages while other samples reflect subsequent filtering.Table 11 reports the number of shared trajectories for each entry.Source: §C.5 · p. 15, Table 11 · p. 16

C.6 Computational Resources

  • The experiments use 16 PRO 5000 GPUs, split between policy training and replay generation and evaluation; rollout generation, IDM updating, reliability screening, and drift scoring take less time than policy training.Generated replay is discarded after each adaptation stage.Source: §C.6 · p. 15
Loading 2610.03079v3…