PaperlayerView exact revision on arXiv

Source-linked AI summary

ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

Changyeon Kim, Kimin Lee, Yashraj Narang, Yijie Guo

arXiv:2610.18596v1cs.ROcs.AIcs.LG

TL;DR

Continual robot post-training must acquire new tasks while retaining earlier behaviors, but existing continual and off-policy methods leave gaps in transfer and learner persistence. ACCRUE carries lightweight actor–critic networks across tasks, uses current-task RL and prior-task actor distillation, and reports reduced real-robot training time with strong prior-task retention.

  • Problem

    Continual RL methods for VLAs offer limited forward transfer, while off-policy RFM methods improve single-task efficiency without addressing learner persistence across tasks.

    Source: §1 · p. 1

  • Method

    ACCRUE carries actor and critic networks across stages, adapts the critic on current-task transitions, and distills prior behavior from cached actor targets on retained episodes.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    31.0%–68.5% less robot RL interaction time for subsequent tasks, while maintaining approximately 97% average success on prior tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Simulation and real-robot experiments show faster subsequent-task learning with significantly reduced forgetting while retaining previously learned behaviors.

    Source: §6 · p. 12

    Experiments cover contact-rich assembly sequences of up to five stages; scaling to longer sequences under a fixed total memory budget and broader tasks remains future work.

    Source: §6 · p. 12

Abstract

from arXiv · show

Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0\% to 68.5\%.

1. Introduction

Continual robot deployment calls for learning new tasks from limited demonstrations while retaining earlier skills, but existing approaches either require fresh on-policy interaction or do not carry off-policy learners across tasks. ACCRUE inherits actor and critic networks and combines current-task learning with prior-behavior distillation; experiments report faster subsequent-task learning and strong retention.

1. Introduction

  • Practical continual post-training must learn new tasks with fewer interactions without sacrificing previously acquired capabilities.Robots may encounter new tasks and conditions without access to future tasks in advance.Source: §1 · p. 1
  • Existing continual RL methods mitigate forgetting but have limited forward transfer, while off-policy RFM methods improve single-task efficiency without learner persistence.Carrying a learner across tasks also risks forgetting, and small prior-task replay can leave inherited critic estimates unreliable.Source: §1 · p. 1
  • ACCRUE inherits lightweight actor and critic networks, adapts the critic only on current-task transitions, and distills the actor on retained prior-task episodes.A fixed external visual encoder maintains consistent features across base-policy updates.Source: §1 · p. 2
  • 31.0%–68.5% less robot RL interaction time on subsequent tasks, while maintaining approximately 97% average success on prior tasks.Across two multi-stage real-robot sequences.Source: §1 · p. 2

2. Related Work

Prior work separately studies RL post-training for RFMs and continual robot learning, leaving a gap in carrying off-policy learning across sequential tasks as the base model is updated. ACCRUE addresses this intersection by reusing actor–critic knowledge from a pretrained RFM while preserving prior behavior.

2.1. RL Post-Training for RFMs

  • RL post-training improves RFMs beyond demonstrations, but much prior work updates the model or action head with fresh policy rollouts.Other approaches use offline value learning, task-specific residual policies, or RL-generated data (13; 19; 35).Source: §2.1 · pp. 2–3
  • Lightweight off-policy methods refine frozen policies for single-task sample efficiency, whereas ACCRUE carries the learner across sequential tasks as the RFM is updated.Related approaches include action- or latent-space actor–critic methods (7; 33; 36; 40).Source: §2.1 · p. 3

2.2. Continual Learning for Robotics

  • Continual VLA methods use on-policy RL or demonstration-derived reinforcement fine-tuning, which respectively require fresh interaction or do not transfer environment-acquired value knowledge.Concurrent work CIDER instead combines current-task off-policy human-in-the-loop RL with prior-task actor distillation (17).Source: §2.2 · p. 3
  • ACCRUE builds on skills in a pretrained VLA or WAM and reuses earlier RL experience by retaining actor and critic across tasks.Figure 2 depicts the base policy, carried networks, and retained-memory distillation.Source: §2.2 · p. 3, Figure 2 · p. 3

3. Preliminaries

The paper formulates sequential post-training of a robot foundation model as learning across known task stages with current-task interactions and bounded prior-task replay. Its action-chunk RL formulation estimates value over full chunks and uses chunk-level backups.

3.1. Robot Foundation Models

  • A robot foundation model receives multi-view images, proprioception, and language, then proposes an H-step action chunk through an input–output interface.The framework is agnostic to the base policy’s internal architecture.Source: §3.1 · p. 3

3.2. Problem Setup

  • Tasks arrive sequentially with shared state and action spaces, and each stage supplies demonstrations while RL interactions come only from the current task.The learner also uses bounded prior-task online experience and is evaluated across tasks encountered so far.Source: §3.2 · p. 4

3.3. Action-Chunk Reinforcement Learning

  • Off-policy actor–critic learning operates on full action chunks, with an H-step discounted reward sum and chunk-level backup.Compared with single-action backups, chunk-level backups reduce the nominal credit-assignment path by a factor of H.Source: §3.3 · p. 4

4. Method

ACCRUE keeps a base policy fixed during each stage’s RL phase while carrying lightweight actor–critic networks across stages. It uses stable visual features and reference-action conditioning, and separates current-task RL from prior-task actor distillation.

4. Method

  • ACCRUE fine-tunes the base policy on demonstrations observed so far, then carries the actor and critic forward while retaining prior behavior through actor matching on retained samples.The critic is trained only on current-task transitions because limited replay may not preserve its earlier-task estimates.Source: §4 · p. 4, §4.3 · p. 6

4.1. Actor–Critic Inputs

  • A frozen external visual encoder and proprioception provide stable actor–critic inputs, while the instruction-conditioned base policy supplies reference action chunks.The actor and critic use trainable projectors, while replay stores encoder features so earlier transitions remain compatible across stages.Source: §4.1 · p. 5

4.2. Current-Task Off-Policy RL

  • At each stage, the inherited actor–critic pair learns from current-task replay, with the actor maximizing critic value while remaining close to the base policy’s reference chunk.The reference-action penalty supports local refinement rather than exhaustive search over the action space.Source: §4.2 · p. 5

4.3. Continual Operation

  • For later stages, ACCRUE initializes both networks from the previous checkpoint and retains the last N training episodes per completed task.The base policy is separately fine-tuned on accumulated demonstrations and held fixed during that stage’s RL.Source: §4.3 · pp. 5–6
  • A frozen preceding-stage actor supplies cached targets on retained inputs, and prior-task replay is used for actor distillation rather than critic or actor RL updates.The prior-task distillation loss is combined with the current-task actor loss, weighted by λ.Source: §4.3 · p. 6

5. Experiments

Experiments test retention, subsequent-task acceleration, and component effects across simulation and DROID hardware, using different base policies, RL algorithms, and interaction budgets.

5. Experiments

  • The study tests retention, new-stage acceleration, and component effects across FORGE and AutoMate simulations and NIST and AutoMate sequences on DROID.Simulation aggregates four FORGE orderings and four AutoMate asset sequences; hardware includes real 3D-printed AutoMate assets.Source: §5 · pp. 6–7, §5.2 · p. 8, §5.3 · p. 9

5.1. Experimental Setup

  • The experiments compare continual learners in a shared off-policy actor–critic architecture, including Sequential, ER, EWC, RETAIN, Single-task RL, and Isolated.Simulation stages use a shared environment-step budget; the main comparisons use the GR00T N1.7 backbone.Source: §5.1 · p. 7, §5.2 · p. 8

5.2. Simulation Experiments

  • Across FORGE and AutoMate, ACCRUE has the lowest NBT and final forgetting and the highest AUC and final accuracy among matched shared actor–critic methods, and exceeds Single-task RL in FWT.Results are aggregated over four FORGE task orderings and four AutoMate asset sequences, with four seeds.Source: §5.2 · p. 8, Table 1 · p. 9, Table 2 · p. 9
  • On AutoMate, ACCRUE approaches Isolated with higher FWT while maintaining one shared actor–critic pair; Simple Recipe has lower FWT and continual AUC on both benchmarks.Isolated stores a separate learner per task; Simple Recipe is an external reference.Source: §5.1 · p. 7, §5.2 · p. 9, Table 2 · p. 9

5.3. DROID Experiments

  • 31.0% to 68.5%: inheriting the actor and critic reduces robot time to the stopping threshold; earlier-task retention is at least 18 of 20 successes and 19.4 on average.The stopping threshold is 95% success; retention is measured on earlier tasks after later stages.Source: Figure 4 · p. 10, §5.3 · p. 10, Table 3 · p. 10

5.4. Ablation Studies and Analyses

  • Table 4 evaluates design choices on FORGE sequence S1 while holding the task sequence, stage budget, and shared hyperparameters fixed.The table marks the best and second-best values.Source: Table 4 · p. 11

Actor–critic inputs

ACCRUE’s ablations examine how visual features, actor guidance, critic updates, replay size, demonstration retention, and base-policy architecture affect continual learning. They support fixed visual features and current-task-only critic training, while showing that replay and demonstration retention involve trade-offs.

Actor–critic inputs

  • Replacing fixed DINOv3 features with stage-updated GR00T VLM features worsens retention and forward transfer, though the comparison does not isolate representation drift.The authors interpret the result as practical support for a fixed visual interface across base-policy updates.Source: Actor–critic inputs · p. 11
  • Removing the reference-deviation penalty substantially weakens forward transfer and task learning, while critic updates on prior-task replay increase forgetting and lower continual AUC and final success.Resetting the critic at each stage also modestly increases forgetting and reduces task-learning efficiency.Source: Actor–critic inputs · p. 11
  • 100–200 retained episodes per completed task, about 3% of the total replay buffer, provide the strongest joint retention–learning trade-off.Ten to twenty episodes are insufficient, while increasing the budget from 100 to 200 degrades performance.Source: Actor–critic inputs · p. 11, Table 5 · p. 11
  • Reducing prior-task demonstrations from 100% to 5% degrades base-policy and post-RL performance, yet ACCRUE maintains new-task learning efficiency while mitigating forgetting.The continual RL procedure and replay budget are held fixed in this comparison.Source: Actor–critic inputs · p. 12, Table 6 · p. 12
  • With Fast-WAM, ACCRUE maintains strong retention and final performance despite architectural differences and different pre-RL success rates; ER degrades under both policy families.The comparison is on FORGE sequence S1.Source: Actor–critic inputs · p. 12, Table 7 · p. 12

6. Conclusion

ACCRUE carries RL-acquired knowledge across deployment stages to learn new tasks efficiently while preserving earlier behavior. The evidence covers contact-rich assembly sequences of up to five stages, with longer sequences under a fixed total memory budget left for future work.

6. Conclusion

  • Across simulation and real robots, ACCRUE supports faster learning on new tasks while preserving earlier behavior with limited replay.The conclusion presents this as evidence that continual RL can support more sample-efficient RFM post-training as tasks arrive.Source: §6 · p. 12
  • ACCRUE inherits lightweight actor–critic networks guided by RFM reference actions and applies prior-task distillation.The approach uses limited replay to carry behavioral and value knowledge across tasks.Source: §6 · p. 12
  • The demonstrated scope is contact-rich assembly sequences of up to five stages with replay memory bounded per task; scaling under a fixed total memory budget remains future work.The authors also identify broader manipulation tasks and transfer across embodiments or action spaces as future directions.Source: §6 · p. 12

Related papers

Loading 2610.18596v1…