PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory Anchors for Continual Robot Learning

Maximilian Du, Zhanyi Sun, Chen Xu, Paarth Shah, Masha Itkina, Shuran Song

arXiv:2608.26545v1cs.RO

TL;DR

Continual robot policies must learn new tasks without losing existing behaviors, but replay buffers are sensitive to which past experiences they contain. The paper identifies conflict-specific Memory Anchors and enriches ER buffers with them, reducing high-conflict forgetting by 63% and improving real-robot task success by 1.7x over random ER.

  • Problem

    Continual robot learning must acquire new tasks without forgetting past behaviors when tasks share observations but require conflicting actions.

  • Method

    The paper identifies Memory Anchors through latent-space overlap, action disagreement, and retrieval of similar old-task data, then enriches ER buffers with them.

  • Results

    63% drop in catastrophic forgetting for the most conflicting task pairings follows from reserving 10% of an ER buffer for Memory Anchors, with 1.7x higher total task success on a real robot than random ER.

  • Takeaways & Limitations

    A small subset of replay experiences can be disproportionately important for retaining past performance, especially in high-conflict continual-learning settings.

  • Takeaways & Limitations

    The method requires access to the current policy’s latent space and past training data, is computationally intensive for larger datasets, and was tested on relatively short task sequences.

Abstract

from arXiv · show

Robot policies deployed in the wild should have the capability to continually learn new tasks without forgetting existing behaviors. A common approach to combat such catastrophic forgetting is to train on new task data with a replay buffer of previously learned task data. Although this buffer is commonly sampled randomly from all prior experiences, we show that a small set of these experiences contributes greatly in anchoring past performance. We call these experiences Memory Anchors. We identify Memory Anchors in regions where representations of new-task observations collapse onto those of old-task observations even though the tasks require conflicting actions, like when a familiar object must be manipulated in a new way. Rehearsing old data in this region plays a key role in preventing destructive overwriting of past task knowledge, serving as this critical Memory Anchor role. Excluding only 10% Memory Anchors before sampling the buffer leads to more than a 4.5x increase in catastrophic forgetting on the LIBERO benchmark suites. Conversely, enriching the replay buffer with Memory Anchors can decrease high-conflict task forgetting by 63% and enables successful continual learning of two task sequences on a real robot. Videos and additional visualizations can be found at https://robot-adaptation.github.io/MemoryAnchors

1 Introduction

The paper identifies Memory Anchors as a small subset of replay experiences that disproportionately preserve past task performance, especially when new and old tasks look similar but require conflicting actions. It proposes extracting and enriching these experiences in ER buffers, with benefits shown in simulation and on a real robot.

  • New and old tasks can share observations while requiring conflicting actions, creating catastrophic forgetting during continual learning.
  • Memory Anchors are a small set of old experiences that disproportionately anchor past task performance in conflicting regions.
  • The proposed extraction process finds latent-space overlap, isolates high action disagreement, and retrieves similar old-task data.
  • ANCHORER enriches ER buffers with Memory Anchors to reduce catastrophic forgetting, particularly for sequentially conflicting tasks.
  • 4.5x increase in catastrophic forgetting follows from reducing access to 10% of the best Memory Anchors on LIBERO.
  • 63% drop in catastrophic forgetting for highly conflicting pairings and 1.7x higher total task success on a real robot result from Memory Anchor enrichment.

2 Related Works

Continual learning methods address forgetting and loss of plasticity through regularization, architectural changes, or replay buffers. Prior replay work varies in sampling and buffer construction, motivating closer study of data selection in robotic continual learning.

  • Sequential learning introduces catastrophic forgetting and loss of plasticity, requiring balance between stability and forward transfer.
  • Continual learning approaches broadly use regularization, task-specific architectures, or replay buffers.
  • Experience Replay maintains a small buffer of past examples and samples them during new-task training.
  • Robotic continual learning adds challenges from continuous action spaces and diverse scenes and tasks.
  • Prior work explored random past-data subsets and improved buffer construction and sampling strategies.

3 Continual Learning under Experience Replay (ER)

The ER studies show that continual-learning outcomes depend on which memories are replayed and that forgetting concentrates in task pairs with overlapping representations but conflicting actions. These findings motivate identifying conflict-specific Memory Anchors.

  • Negative Backward Transfer measures the success drop of an earlier task as later tasks are learned.
  • RandER has 36% lower forgetting variance and 25% lower worst-case forgetting than FixedER, despite comparable average NBT.
  • Forgetting varies by up to 20% across DER samples even with fixed task ordering and the same starting checkpoint.
  • Continual-learning performance is especially sensitive to selected ER memories for worst-case forgetting tasks.
  • 3.2 Task Relationships in Catastrophic Forgetting: A Cream Cheese to Bowl task loses 24% performance when Bowl to Plate is learned, exceeding 5x the average NBT for LIBERO-Goal.
  • 3.2 Task Relationships in Catastrophic Forgetting: The worst two of ten tasks account for more than 50% of total negative backward transfer across LIBERO-Goal training orders.
  • 3.2 Task Relationships in Catastrophic Forgetting: Conflicting task pairs overlap in observation representations while requiring different or opposing action labels, enabling interference in shared model resources.

4 Memory Anchors

Memory Anchors are extracted from regions where new and old task representations overlap and actions disagree. The method then retrieves the most similar old-task data to regularize training around these conflicts.

  • Step 1: Find State Representation Overlap: The method first locates new-task observations within the latent manifold of past task data.
  • Memory Anchor candidates combine state-representation similarity with action disagreement relative to new-task data.
  • Step 2: Isolate Action Disagreements through Generative Policy: It then isolates overlapping new-task points with the largest disagreement between policy-predicted and ground-truth actions.
  • Step 2: Isolate Action Disagreements through Generative Policy: Noise-denoising action prediction helps identify samples requiring significant policy changes while reducing false positives from action multimodality.
  • Step 3: Extract Memory Anchors from Old Data: The method retrieves old-task data most similar to the disagreement set as Memory Anchors.

5 Impact of Memory Anchors on Forgetting

Memory Anchors are disproportionately important for retaining past tasks, especially when task observations overlap but required actions conflict. Removing them sharply increases forgetting, while enriching replay buffers with them improves retention and real-robot continual learning.

  • 5.2 Enriching Replay Buffers: ANCHORER reduced forgetting by 63% for the most conflicting task pairings by reserving replay-buffer capacity for Memory Anchors.The method enriched ER buffers with anchors extracted from new-task conflict regions.
  • 5.4 Memory Anchors on a Real Robot: On OpenJar, ANCHORER achieved 1.7x higher overall success than randomly sampled RandER while balancing plasticity and retention.The suite used visually similar jars requiring counterclockwise, clockwise, or direct-lift strategies.
  • 5.4 Memory Anchors on a Real Robot: On SweaterFold, ANCHORER achieved 1.7x higher final success rates than RandER on the same ER budget.The result extended to three sequential, language-conditioned bimanual folding behaviors.
  • 5.5 Baselines and Ablations: ANCHORER had the lowest average NBT among the compared buffer-selection methods, and action disagreement contributed to its advantage.Nearest-neighbor selection without action disagreement retrieved shared subtrajectories already rehearsed by new data.

6 Conclusion and Discussion

The paper introduces Memory Anchors as a small subset of past-task data used to preserve performance during sequential ER training. It validates the approach in simulation and on real robots, while noting access, computational, and sequence-length limitations.

  • 6 Conclusion and Discussion: Memory Anchors are a small subset of past-task data critical for maintaining performance during sequential ER training.The paper studies their importance for overall performance, high-conflict tasks, and small replay buffers.
  • 6 Conclusion and Discussion: The extraction process requires access to the current policy’s latent space and past training data, which may be unavailable for proprietary models.Computing representations for past datasets at every task is also computationally intensive, and the experiments use sequences of only 3–10 tasks.
  • 6 Conclusion and Discussion: The OpenJar suite uses three identically shaped jars requiring counterclockwise rotation, clockwise rotation, or direct lifting.The jars are distinguished by label text and icon.
  • 6 Conclusion and Discussion: The real-robot study collected demonstrations with a UMI setup and deployed the trained policy on an ARX robot arm.The setup used an iPhone camera for data collection.
  • 6 Conclusion and Discussion: OpenJar Memory Anchors concentrate near jar-lid contact, where new and old tasks produce conflicting actions.For clockwise opening, the old policy moves right toward the left gripper while the new task requires moving left toward the right gripper.
  • 6 Conclusion and Discussion: For the lift task, action disagreement occurs later during gripper closure and lifting, with retrieved anchors focusing on similar lifting and early reaching behavior.These locations correspond to task-specific conflict regions.

A.3 Additional Qualitative Results: OpenJar

OpenJar failures reveal distinct plasticity, forgetting, and execution-error patterns. ANCHORER reduces behaviorally incorrect outcomes relative to RandER, but precision remains a separate failure source.

  • Loss of Plasticity: RandER produced 15/20 incorrect Counterclockwise behaviors while learning Clockwise, whereas ANCHORER produced only 3/20.Rotating the Clockwise jar 15–20 degrees allowed the robot to finish, indicating competition near the neutral position.
  • Catastrophic Forgetting: RandER’s forgotten Counterclockwise behavior caused the robot to lift locked lids directly, while ANCHORER showed the same failure at lower frequencies.The incorrect lift strategy also affected the Clockwise jar.
  • Other Failures: A small proportion of failures for both methods were execution failures caused by insufficient precision.The main execution failure was unrotating the lid when the wrong gripper contacted it during the final grab.

A.4 Practical Lessons and Failure Analysis: OpenJar

The OpenJar study combines practical setup lessons with a three-step procedure for extracting Memory Anchors from latent observation overlap and action disagreement. At 5000 memories, ANCHORER retained prior behavior while learning the new task, whereas RandER exposed a stability–plasticity trade-off.

  • Practical Lessons: Specialist policies were tuned to nearly 100% success so continual-learning degradation reflected sequential training rather than intrinsically difficult tasks.
  • Practical Lessons: A pretrained VIT encoder was less sensitive to lighting than ResNet-50, while training the VIT from scratch caused full failures.
  • Practical Lessons: Additional lighting improved perception of gripper–lid contact, and final-grasp rotation bias prevented incidental contact from undoing progress.
  • Buffer Size Sensitivity: 5000 memories let ANCHORER learn Clockwise while retaining Counterclockwise, whereas RandER at 1000 memories learned Clockwise at Counterclockwise’s expense and could not learn Clockwise at 5000.
  • Memory Anchor Extraction: Memory Anchors are extracted by finding latent observation overlap, selecting high action disagreement, and retrieving nearby old-task samples.
  • Memory Anchor Extraction: The latent representation concatenates proprioception, language, and visual embeddings, with percentile distance statistics supporting overlap clustering.
  • Memory Anchor Extraction: For diffusion policies, action disagreement denoises lightly noised labeled actions and selects new samples exceeding the old-task baseline µ + 2σ.
  • Memory Anchor Extraction: Old-task anchors are ranked by median k-nearest-neighbor distance to the high-disagreement new-task set and selected in increasing score order.

B.3 LIBERO Experiment Details

The LIBERO experiments evaluate continual learning with standardized benchmark resets and permutations, focusing on NBT while also describing AUC, FWT, task conflict, and Memory Anchor concentration. These metrics distinguish retention, overall learning, plasticity, interaction-level forgetting, and anchor presence.

  • Evaluation Setup: LIBERO success rates use 50 benchmark reset configurations, while continual-learning results span three task permutations, producing 30 NBT data points across ten tasks.
  • Evaluation Setup: Reported LIBERO results use a 1000-transition ER buffer, while Anchor Addition uses an alternative 1% past-task allocation to expand capacity as tasks accumulate.
  • Metrics: Negative Backward Transfer measures task forgetting after later-task training; its reported average increases as forgetting worsens.
  • Metrics: AUC is the normalized area under the success curve from 0 to 1, combining initial learning and later retention into a holistic measure.
  • Metrics: FWT averages each task’s post-initial-training performance and measures policy plasticity to new information.
  • Task Conflict: Task conflict uses ∆i,j = ci,j−1 − ci,j to measure how training task j changes task i, complementing aggregate NBT with interaction-level analysis.
  • Anchor Concentration: Anchor Concentration is the proportion of top-10% Memory Anchors among the top-40% sampled buffer, with larger values indicating more anchor inclusion.

C.1 Additional Result: Memory Anchors on Smaller Buffer Sizes

Reducing buffer capacity makes Memory Anchor retrieval increasingly important. At larger capacities, uniform replay often contains anchors naturally, but under restriction ANCHORER preserves their representation while RandER forgets more.

  • Buffer Size Effects: The buffer-size sweep ranges from 0.5% to 100% and tests whether restricted replay makes Memory Anchor effects more visible.
  • Evaluation Context: The reported comparison concerns reduced Memory Anchor access and added-anchor performance under the LIBERO continual-learning experiments.
  • Buffer Size Effects: At higher buffer sizes, RandER and ANCHORER are indistinguishable because uniform sampling naturally includes Memory Anchors.
  • Buffer Size Effects: As buffer sizes become restricted, ANCHORER maintains Memory Anchor representation while RandER’s forgetting increases sharply.

C.2 Additional Result: Task-Task Relationships (§3.2)

Task–task interactions vary widely: a small subset produces most forgetting, while some interactions improve past behavior through later rebound. This analysis also bounds how Memory Anchors behave across homogeneous and heterogeneous task suites.

  • Task–Task Relationships: NBT aggregates interactions into one value, whereas ∆i,j exposes the diverse effects of training task j on task i and avoids relying solely on the original task performance reference.
  • Task–Task Relationships: Some interactions cause strong forgetting, others weak forgetting, and some improve past-task behavior after the conflicting task ends.
  • Task–Task Relationships: The average ∆i,j was 0.207 for the top five conflicts versus −0.0016 for the bottom 751, supporting concentration of forgetting in a small subset.
  • Additional Metrics: AUC also shows reduced performance variance, while FWT is less affected by replay sampling than retention-oriented NBT.
  • Scope and Interpretation: Memory Anchor extraction is most intuitive for homogeneous tasks with shared observations and differing critical decisions, but it still affects high-conflict interactions in heterogeneous suites.
  • Scope and Interpretation: In highly heterogeneous suites, overlap may select trajectory starts and action disagreement may include all overlapping new data when the policy fails to generalize.

C.5 Additional Result: All Metrics for Memory Anchor Removal (Section 5.1)

Reducing Memory Anchor availability worsens retention metrics, while enriching the replay buffer improves outcomes overall, with effects depending on task-suite homogeneity and buffer diversity.

  • NBT rises and AUC drops significantly as Memory Anchors become less available.
  • Forward transfer remains unaffected when Memory Anchors become less available.This supports the reported distinction between replay-buffer effects on past-policy stability and future-policy plasticity.
  • Enriching the replay buffer with ANCHORER produces overall higher AUC and lower NBT between tasks.
  • LIBERO-Goal shows measured improvement across all three metrics, whereas heterogeneous suites benefit more from diverse replay coverage.Adding Memory Anchors trades diversity for specificity; homogeneous tasks benefit broadly, while heterogeneous tasks benefit mainly on high-conflict interactions.
  • With 1000 memories per task, π0.5 achieves average NBT 0.02 on LIBERO-Goal.The result uses a buffer containing 20% of total past data.
Loading 2608.26545v1…