Source-linked AI summary
CIDER: Continual Interactive Distillation for Embodied Reinforcement Learning
Houlin Li, Minghui Xu, Guo Xu, Xuan Du, Xiaohan Yan, Chun Wang, Yuxiang Yan, Shukai Yang, Yongcheng Liu, Wei Shan, Maoqing Yao
TL;DR
Continual real-world robot RL must acquire new manipulation skills without losing earlier behaviors. CIDER freezes the accumulated policy as a teacher and interleaves current-task online learning with historical distillation, preserving six tasks while each new task is learned in approximately 10–20 minutes.
Problem
Continual real-world RL must accumulate new skills in one deployable policy while retaining previous behaviors, because sequential updates can cause catastrophic forgetting.
Method
CIDER freezes the previously consolidated policy as a teacher and interleaves current-task human-in-the-loop RL with distillation-based retention using a shared actor.
Results
CIDER preserves all six manipulation tasks, achieving a final average success of 100% over 100-trial evaluations.
Takeaways & Limitations
Interactive Distillation maintains continual skill accumulation while each new task reaches successful behavior within approximately 10–20 minutes of real-robot training.
Takeaways & Limitations
The evaluation focuses on short-horizon tasks within the available interaction budget; longer-horizon tasks with sparse rewards or delayed failures may require additional exploration, reward design, or supervision.
Abstract
from arXiv · showhide
Human-in-the-loop real-world reinforcement learning enables rapid acquisition of effective robotic manipulation policies for individual tasks, often within tens of minutes. Yet it remains unclear how to extend this paradigm to continual learning, where a single policy must acquire new skills without losing previously learned behaviors. Existing real-world continual learning methods do not explicitly constrain prior behaviors, leading to severe catastrophic forgetting. We introduce Continual Interactive Distillation for Embodied Reinforcement Learning (CIDER), a continual reinforcement learning framework that freezes the accumulated historical policy as a teacher before learning each new task and interleaves task learning with distillation-based retention. We further introduce gradient routing to separate the gradients used for acquiring new tasks from those used for preserving prior behaviors. We evaluate our method with a single shared actor on six real-world household and industrial manipulation tasks. Interactive Distillation maintains high measured success on previously learned tasks across our six-task real-robot sequence while acquiring each new task in 10 to 20 minutes, whereas every baseline forgets at least one previous task. Additional ablations reveal the key design choices that govern the tradeoff between stability and plasticity in real-world continual reinforcement learning.
1 Introduction
CIDER addresses continual real-world robot learning by combining current-task online RL with retention of previously learned behaviors. It targets catastrophic forgetting while preserving a single deployable policy that accumulates skills.
- Continual real-world RL requires one deployable policy to acquire new skills while continuing to execute previously learned behaviors.
- Sequential fine-tuning can overwrite previous action distributions, causing catastrophic forgetting and costly recovery through additional interaction, retraining, and evaluation.
- Naive sequential RL forgets previous skills, whereas Interactive Distillation interleaves current-task online RL with rehearsal using a frozen expert on prior-task states.
- CIDER formalizes continual real-world robot RL with a single shared actor evaluated across six short-horizon manipulation tasks.
- Across the task suite, CIDER substantially reduces measured forgetting compared with sequential fine-tuning, replay, regularization, offline updates, and post-hoc expert consolidation.
2 Related Work
Related work frames continual reinforcement learning as a stability–plasticity problem and reviews human-in-the-loop RL, forgetting mitigation, and policy distillation approaches. Existing methods constrain parameters, updates, structures, or action distributions in different ways.
- Human-in-the-loop RL uses operator corrections during execution to provide high-quality experience that can accelerate convergence and improve final manipulation policies.
- Continual RL requires balancing plasticity for acquiring new tasks with stability for retaining previous behaviors.
- Soft regularization methods constrain important parameters, while gradient projection reduces interference but can shrink the update subspace as tasks accumulate.
- Policy distillation transfers capabilities from expert policies to a student and has been explored for continual consolidation through task-specific teachers, aggregation, and reset-and-distill pipelines.
3 Preliminaries
The paper formulates continual real-world visual manipulation as sequential scenario learning with limited access to prior environments. Each task uses human-in-the-loop actor-critic RL on current data, while historical archives are reserved for behavioral distillation rather than value learning.
- A robot encounters a sequence of physical scenarios, each defined by observation space, shared continuous actions, scenario-specific transitions and rewards, and a discount factor.
- After completing a scenario, the learner cannot assume further physical access and may retain only bounded transition archives from previous scenarios.
- The continual objective is to produce one deployable policy that performs well on all scenarios seen so far.
- The single-task learner uses a stochastic actor, twin critics, and an active replay buffer containing autonomous transitions, demonstrations, and human corrections.
- RLPD applies SAC-style actor-critic updates to the mixed current-task replay buffer, using entropy-regularized actor learning and twin-critic targets.
- Historical transition archives are excluded from Bellman backups and contribute only their state components to behavioral distillation.
4 Interactive Distillation for Continual Real-World RL
Interactive Distillation retains prior behavior by freezing the consolidated actor as a teacher while the shared learner continues online RL on the current scenario. Training alternates current-task learning with historical distillation, using behavior-level constraints and routed gradients.
- At each stage, the learner maintains current-task data, historical archives, and a frozen teacher obtained from the previously consolidated policy.
- The stage output is one deployable policy and a new transition archive, with skills represented by a shared actor rather than task-specific heads.
- For the first scenario, CIDER trains with HIL-RL, stores an archive after success, and freezes the resulting actor as the first teacher.
- For later scenarios, training starts from the previous consolidated actor, freezes that actor as teacher, resets current replay, and collects new transitions only from the current scenario.
- Training alternates HIL-RL updates on current transitions with historical distillation updates on states extracted from archived transitions.
- Reverse KL constrains the current action distribution to match the frozen teacher on archived states, rather than directly constraining policy parameters.
- The actor receives current RL and distillation gradients, while the critic is trained only on current-task data; teacher outputs can be cached and archives sampled task-balanced.
- Interactive Distillation combines online adaptation with frozen-teacher behavioral supervision instead of old-transition Bellman backups, and consolidates during new real-world experience collection.
5 Experiments
Experiments evaluate CIDER on six sequential real-world manipulation tasks, testing continual retention, new-task acquisition, and design choices affecting stability and plasticity. CIDER preserves prior skills while learning subsequent tasks efficiently, outperforming the evaluated alternatives in measured retention.
- 5.1 Real-Robot Task Suite and Continual Learning Protocol: Across six household and industrial tasks, evaluation uses five independent sets of 20 real-robot trials per task after each sequential training stage.Tasks include socket insertion, toothbrush hanging, phone-box closing, connector plugging, safety-hook coupling, and chip-tray placement.
- 5.1 Real-Robot Task Suite and Continual Learning Protocol: All six independently trained single-task experts achieve 20 out of 20 successful trials, establishing that the task suite is solvable by the robot.These experts provide a task-performance reference and help distinguish forgetting from intrinsic task difficulty.
- 5.2 Main Results: Continual Skill Accumulation: Finetune, replay, EWC, sequential behavioral cloning, sequential offline RL, and DisCoRL each fail to maintain previously learned skills or suffer performance drops.The baselines test replay, parameter regularization, offline updates, and one-shot expert consolidation as retention mechanisms.
- 5.2 Main Results: Continual Skill Accumulation: CIDER is the only evaluated method that preserves all six manipulation tasks throughout sequential training, reaching 100% final average success over 100 trials.Every policy after stage k succeeds on all tasks seen up to that point.
- 5.2 Main Results: Continual Skill Accumulation: CIDER reaches successful behavior within approximately 10–20 minutes of real-robot training for each new task and requires no more training steps than independent single-task learning.It reaches the success criterion faster on some tasks, despite variation in objects, contact dynamics, and success conditions.
- 5.3 Ablations: Understanding CIDER: Default gradient routing blocks current-task actor gradients from the adapter while allowing historical distillation gradients to update it, completing the six-task sequence in around 15 minutes.Updating the adapter with both streams prevents convergence, whereas freezing it causes increasing distillation loss and slower current-task learning.
- 5.3 Ablations: Understanding CIDER: Continual initialization starts the fourth-task student from the consolidated policy, whereas from-scratch initialization must recover prior behaviors solely through historical distillation.The ablation compares these schemes under the same current-task training and historical distillation setup.
6 Limitations
CIDER is evaluated on short-horizon manipulation tasks within a limited real-robot interaction budget. Longer-horizon tasks with sparse rewards, multi-stage contacts, or delayed failures may require additional exploration, reward design, or supervision.
- The evaluation focuses on short-horizon manipulation tasks solvable within the available real-robot interaction budget.
- Longer-horizon tasks with sparse rewards, multi-stage contacts, or delayed failures may require additional exploration, reward design, or supervision before application.
A Task and Implementation Details
The study evaluates continual robot learning on six precise, contact-rich real-world manipulation tasks and provides shared experimental and algorithmic details. Interactive Distillation maintains a single deployable actor through staged initialization, current-task interaction, and historical-task distillation.
- Task Descriptions: Six real-robot manipulation tasks require precise pose alignment, contact-rich interaction, and recovery from small initial variations.
- Task Descriptions: Socket insertion, connector plugging, box closing, tray placement, toothbrush hanging, and safety-hook coupling test tight geometric and contact constraints.
- Experimental Parameters: The experimental setup includes common parameters for all tasks and varied parameters summarized separately across tasks.
- Interactive Distillation: Interactive Distillation initializes a single actor and critics, learns each task with HIL-RL, samples task-balanced historical batches, and returns one deployable actor.
C.1 Policy Architecture
CIDER uses a shared visuomotor actor-critic that processes visual observations without task identifiers entering the policy network. A visual adapter produces the shared representation used by both actor and critic heads.
- All experiments use the same visuomotor policy architecture and deployment interface.
- Two RGB viewpoints are encoded by ImageNet-pretrained ResNet-10 backbones, with zero placeholders used for unused views.
- The visual adapter applies eight learned spatial masks, projects features to 256 dimensions, and concatenates viewpoint vectors into z ∈ R^B×512.
- The shared representation feeds twin critic Q-values and an actor producing Gaussian action mean and log standard deviation through two-layer MLP heads.
- Deployment: The policy receives visual observations and proprioceptive states without an explicit task identifier, while deployment preprocessing selects scenario-specific camera and normalization settings.
- Deployment: The deployment task identifier selects perception preprocessing only and never conditions the policy network; the shared actor is unchanged across tasks.
D Baseline Implementation Details
Baselines share CIDER’s architecture and HIL-RL protocol but differ in data usage and optimization. They include fine-tuning variants, offline updates, expert distillation, and post-hoc consolidation comparisons.
- Common Setup: All baselines share CIDER’s policy architecture, HIL-RL interaction protocol, and generally identical RL hyperparameters.
- Online Baselines: Finetune resets historical replay and trains exclusively on the current-task buffer from the previous policy initialization.
- Online Baselines: Finetune with Replay mixes the full historical transition buffer into critic and actor updates without a capacity limit.
- Regularized Baselines: EWC adds Fisher-weighted parameter regularization using stored consolidated snapshots, with the penalty applied to all actor parameters.
- Offline Baselines: Sequential BC and Sequential Offline RL update policies offline without further real-environment interaction, using demonstrations or collected transitions.
- Reporting and Analysis: The appendix reports implementation details, standard deviations, success rates across continual stages, and post-hoc consolidation analysis.
- Distillation Baseline: DisCoRL independently trains six task experts and merges them into one student through offline reverse-KL policy distillation.
E Additional Ablations
The ablations identify design choices that balance retention of prior skills with adaptation to new tasks. Task-balanced sampling, an intermediate distillation ratio, and probabilistic distillation objectives provide the strongest overall behavior, while performance remains stable across task orders.
- Historical Task Sampling: Task-balanced sampling consistently outperforms dataset-proportional sampling in retention while achieving comparable current-task performance.It gives each previously learned task equal contribution to the distillation objective.
- Task Order Robustness: Different task orders produce only minor performance variations, with overall retention and final performance remaining stable across permutations.The experiment repeats continual learning with multiple random task permutations and fixed training settings.
- Distillation Loss: NLL, forward KL, and reverse KL achieve comparable performance, while MSE action matching consistently underperforms in retention.The ablation evaluates only the first two tasks because of time and cost constraints.
- Distillation Loss: Reverse KL is adopted as the default objective because it encourages the student to assign probability mass to teacher-favored actions.The similar performance of probabilistic objectives indicates robustness to the specific choice among NLL and KL-based losses.
F Scalability Analysis
CIDER’s storage and cached-teacher requirements grow linearly with the number of tasks, while task-balanced sampling keeps per-update distillation cost constant. The evaluation covers six tasks, leaving much larger-scale retention and representational capacity unresolved.
- Scaling Analysis: CIDER scales linearly with T in buffer storage and cached teacher outputs, while task-balanced sampling keeps per-update distillation cost constant.The primary scaling bottleneck is buffer storage rather than per-step gradient computation.
- Scaling Analysis: T = 6 tasks are evaluated, while whether a fixed-capacity shared actor can retain substantially more diverse skills remains an open question for T ≫ 6.Potential mitigations include increasing adapter capacity or combining CIDER with structural expansion methods.