PaperlayerView exact revision on arXiv

Source-linked AI summary

An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics

Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang, Richard Xue, Daniel Rakita

arXiv:2609.28807v1cs.RO

TL;DR

Robots may face changes that pretrained policies did not anticipate, and prior work had not established streaming deep RL as a framework for adapting robotic policies to such changes. This paper analyzes online adaptation of pretrained policies across locomotion and manipulation; its quadruped experiments report stronger performance than batch-based on-policy methods, while manipulation results expose stability and performance limits.

  • Problem

    Prior work had not established whether streaming deep RL could serve as a continual-learning framework for adapting robotic policies to unseen changes.

    Source: §I · p. 2

  • Method

    The paper analyzes pretrained RL policies adapting online to changes in the robot, environment, or goals, including the roles of optimizers and plasticity-loss mitigation.

    Source: §I · p. 2

  • Results

    The top four streaming variants outperform batch-based on-policy methods by 60% in average max success rate across adaptation scenarios, while manipulation results show performance decline over time and limited recovery in complex settings.

    Source: §I · p. 2

  • Takeaways & Limitations

    With careful optimizer and plasticity-mitigation choices, streaming deep RL can adapt pretrained policies to change and outperform common batch-based deep RL approaches such as PPO.

    Source: §VI · p. 7

    On manipulation tasks, streaming deep RL shows performance decline over time and limited recovery in more complex settings.

    Source: §I · p. 2

Abstract

from arXiv · show

Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes. In contrast, biological learning occurs moment-to-moment via a stream of experience, unlike the predominantly batch-based and offline nature of deep learning. Although recent works show the feasibility of stream-based deep reinforcement learning, where updates use only the latest experience, none have shown it to be a viable continual learning framework for adapting robotic policies to unseen changes. In this paper, we present the first analysis of streaming deep reinforcement learning for adaptive continual learning in robotics. In particular, we show that, following an initial pretraining phase, streaming deep RL can enable a robot to successfully adapt to unforeseen changes to itself, its environment, or goals. Our primary experiments within quadruped locomotion demonstrate that a deep neural network robotic policy with certain optimizers and plasticity loss mitigation techniques can successfully leverage domain task knowledge from its pretraining to quickly adapt online to diverse changes via stream learning, outperforming batch-based on-policy methods and improving task success rates by up to 90% over the pretrained policy. Furthermore, we perform additional evaluations on robotic manipulation tasks to determine if our previous observations extend to different robotic morphologies and scenarios. Our results show that the successes observed in quadruped locomotion can be partially realized in manipulation with stability and performance limitations. We conclude with a discussion on the limitations of our work and its implications for the future of continual robot learning.

I. INTRODUCTION

The introduction asks whether pretrained deep RL policies can adapt online to unseen robotic changes, a question not established by prior stream-learning work. It presents an analysis across quadruped locomotion and manipulation, finding strong locomotion adaptation but more limited manipulation results.

I. INTRODUCTION

  • Stream learning offers a way for robots to learn online from experience when pretrained models cannot anticipate every scenario.The introduction motivates this approach by noting that the learning world remains far larger and more complex than the agent.Source: §I · p. 1
  • A sufficiently plastic policy could let a quadruped with a damaged leg continue walking in an isolated setting far from assistance.The example contrasts adaptation from experience with waiting for repair.Source: §I · p. 1
  • Prior work had not established streaming deep RL as a continual-learning framework for adapting robotic policies to unseen changes.Recent progress in stream learning was driven by optimizer advances and network scaling and normalization techniques inspired by plasticity research.Source: §I · pp. 1–2
  • The paper analyzes pretrained policies adapting online to unforeseen changes in the robot, environment, or goals, focusing on optimizer and plasticity-loss mitigation choices.It also evaluates manipulation tasks to test whether observations from quadruped locomotion extend to other morphologies and scenarios.Source: §I · p. 2
  • Up to 90% improvement in success rate over the base policy on certain adaptation scenarios, while the top four streaming variants outperform batch-based on-policy methods by 60% in average max success rate.Manipulation results show performance decline over time and limited recovery in more complex settings.Source: §I · p. 2

II. RELATED WORKS

Related work traces the shift from incremental stream-based RL toward batch-based deep RL and describes advances that have revived streaming. It also reviews plasticity loss and mitigation strategies, positioning the paper's focus on adapting pretrained policies to new robotic scenarios.

A. Batch and Stream Reinforcement Learning

  • Deep RL shifted toward replay buffers, offline RL, and GPU-enabled parallel simulation for sample efficiency and to address instability in neural-network stream learning.Earlier incremental algorithms such as TD, SARSA, and Actor-Critic were largely limited to tabular settings.Source: A. Batch and Stream Reinforcement… · p. 2
  • ObGD bounds updates using eligibility traces, while normalization and scaling methods have improved streaming deep RL; prior work also proposed sigmoid-weighted activations and RG-estimated policy gradients.The cited examples include Elfwing et al. [6], Vasan et al. [24], and Elsayed et al. [7].Source: A. Batch and Stream Reinforcement… · p. 2
  • Prior work applied streaming deep RL to train robotic systems from scratch, but quadruped locomotion and object manipulation are challenging and inefficient to learn from scratch in a stream setting.This work instead bootstraps policies with pretraining and studies streaming RL for continual robotic adaptation.Source: A. Batch and Stream Reinforcement… · p. 2

B. Plasticity in Deep Learning

  • Plasticity is a network’s ability to keep adapting to new data, but standard deep networks can lose this capacity over time.Prior responses include reinitializing underused units, injecting parameters, adjusting optimizers, and adding normalization layers.Source: B. Plasticity in Deep Learning · p. 2

C. Continual Learning and Adaptation in Robotics

The paper distinguishes its focus on streaming adaptation of RL policies from other robotic adaptation approaches and limits its analysis to forward transfer. It frames robotic tasks as episodic MDPs whose policies are optimized for expected return.

C. Continual Learning and Adaptation in Robotics

  • Continual learning concerns adapting models to shifts in data distributions and objectives over time.Examples in robotics include online residual RL for frozen imitation models and dynamic-rank adjustment of pretrained diffusion policies.Source: C. Continual Learning and Adaptation… · p. 2
  • Unlike prior work using residual RL, torque compensation, batch online RL, or behavior-cloning adaptation, this work asks whether streaming deep RL itself can support continual robotic adaptation.The analysis focuses on forward transfer from prior knowledge to unseen conditions, not catastrophic forgetting or backward transfer.Source: C. Continual Learning and Adaptation… · pp. 2–3

A. Markov Decision Processes

  • The task formulation is an episodic MDP with continuous state and action spaces, transition probabilities, normalized dense rewards, and a discount factor.The policy’s parameters are chosen to maximize expected return over the episode horizon.Source: A. Markov Decision Processes · p. 3

B. Online and Streaming Reinforcement Learning

Streaming deep RL updates its policy and value parameters from the latest transition, trading potentially faster adaptation for instability risks. The analysis considers abrupt changes in transition dynamics or rewards and asks whether pretrained policies can adapt efficiently across them.

B. Online and Streaming Reinforcement Learning

  • In batch-based on-policy RL, agents collect a batch of transitions before updating actor-critic parameters with temporal-difference learning.The actor represents the policy, while the critic approximates the value function.Source: B. Online and Streaming Reinforcement… · p. 3
  • Streaming deep RL updates policy and value parameters using only the latest transition, which may enable faster adaptation but risks collapse from large noisy updates or poor learning from overly conservative ones.The method therefore requires controlling update size while maintaining network plasticity.Source: B. Online and Streaming Reinforcement… · p. 3

C. Area of Investigation

  • The adaptation setting allows transition dynamics or rewards to change at any time, with dynamics shifts linked to morphology or environment and reward shifts to goal distributions.The changes are modeled as abrupt transitions from pretrained conditions to an adaptation scenario, without gradual interpolation.Source: C. Area of Investigation · p. 3
  • The study asks whether pretrained parameters can be leveraged to produce a policy that performs strongly in the new MDP after a sharp change.The target is smooth and efficient adaptation across the change.Source: C. Area of Investigation · p. 3

D. Optimizers for Stream Learning

ObGD-family optimizers bound effective step sizes, with AdaptiveObGD adding a second-moment estimate; plasticity-loss mitigation evaluates layer normalization and continual backpropagation.

D. Optimizers for Stream Learning

  • ObGD and AdaptiveObGD use eligibility traces for multi-step credit assignment and bound the effective step size; AdaptiveObGD also uses a second-moment estimate.The actor and critic networks use separate optimizers.Source: D. Optimizers for Stream Learning · p. 3

E. Methods for Combating Plasticity Loss

  • Layer normalization normalizes intermediate representations across features for each sample, commonly before the layer activation.The paper evaluates it as a plasticity-loss mitigation technique.Source: E. Methods for Combating Plasticity Loss · pp. 3–4
  • Continual backpropagation reactivates least-used units using contribution utility, a running average based on outgoing-connection utility and gradient capacity for Tanh units.The paper substitutes gradient capacity for activation magnitude to favor units with lower saturation; a maturity threshold protects reset weights, while ρ controls the replacement fraction.Source: E. Methods for Combating Plasticity Loss · p. 4

IV. ANALYSIS ON QUADRUPED LOCOMOTION

The quadruped analysis tests streaming deep RL for continual adaptation and examines how optimizers and plasticity-loss mitigation affect performance. It evaluates policies during training and uses shared pretrained actor-critic architectures for comparison.

IV. ANALYSIS ON QUADRUPED LOCOMOTION

  • The quadruped experiments assess streaming deep RL's potential for continual robotic adaptation and the effects of optimizer and plasticity-mitigation choices.The analysis uses a quadruped locomotion domain.Source: §IV · p. 4

A. Experimental Setup

  • Policy success rate is measured over 50 episodes without parameter updates and evaluated every 10,000 online-training steps across five random seeds.Evaluation includes conditions with or without the adaptation scenario, depending on whether adaptation has begun.Source: A. Experimental Setup · p. 4
  • Comparative methods use the same pretrained weights from successful parallelized batch PPO training, except AdaptiveObGD + LN, which uses weights pretrained with layer normalization.Actor-critic networks use a four-layer, 256-unit-per-hidden-layer MLP with Tanh, consistent across experiments.Source: A. Experimental Setup · p. 4

B. Task Environment

The task is quadruped goal reaching, with a familiar-setting warm start before adaptation scenarios alter the robot, its environment, or its goal. Experiments compare streaming variants during a non-stationary adaptation phase.

B. Task Environment

  • AnymalC-Reachv1 requires an Anymal C quadruped to walk within 0.35 meters of a randomized goal using joint, velocity, goal-position, and success observations.Actions change joint positions, and the reward combines distance-to-goal reward with penalties for undesired contacts and high velocities.Source: B. Task Environment · p. 4

C. Adaptation Scenarios

  • The three adaptation scenarios simulate a stuck back-left leg, a slippery floor with friction coefficients changed from 0.3 to -1.8, and goals shifted from (0,0) to (0,3.0).The leg scenario zero-masks three action-space values.Source: C. Adaptation Scenarios · p. 4

D. Stream Adaptation Experiments

  • Five Stream-AC variants compare Adam, ObGD, AdaptiveObGD, AdaptiveObGD + CBP, and AdaptiveObGD + LN, with CBP used only during stream learning.Adam uses TD(0) Advantage Actor-Critic; ObGD and AdaptiveObGD use TD(λ).Source: D. Stream Adaptation Experiments · p. 4
  • After a 500,000-step unchanged warm start, adaptation lasts 1.5 million steps; success rates are sampled every 10,000 steps, with maximum and mean rates tracking recovery and sustained performance.The table caption identifies average success rates ± standard deviation across five seeds; MASD measures successive-evaluation differences as a stability metric.Source: D. Stream Adaptation Experiments · pp. 4–5, B. Task Environment · p. 4, Table 1 · p. 5
  • On goal shift, AdaptiveObGD + LayerNorm outperforms the other Stream-AC variants, while ObGD achieves higher performance than AdaptiveObGD at the cost of stability.The figure shows performance during quadruped goal-shift adaptation.Source: Figure 2 · p. 5

E. Quantitative Results

The optimizer and plasticity-mitigation results favor carefully designed streaming methods over Adam and batch PPO, while revealing trade-offs between performance and stability. Layer normalization and continual backpropagation offer distinct benefits across adaptation scenarios.

E. Quantitative Results

  • ObGD and AdaptiveObGD significantly outperformed Adam on all scenarios, while their relative performance and stability differed across tasks.AdaptiveObGD exceeded ObGD by 3.2% in max success rate on broken leg, but underperformed it by 15.2% on goal shift and 5.2% on slippery floor on average; ObGD had higher average MASD across scenarios.Source: E. Quantitative Results · p. 5, Table 2 · p. 5
  • Adding layer normalization to AdaptiveObGD gave the best performance on goal shift and slippery floor while retaining lower MASD on broken leg and goal shift.Its MASD was only slightly higher on slippery floor.Source: E. Quantitative Results · p. 5
  • AdaptiveObGD + CBP improved mean success rate over AdaptiveObGD on goal shift and slippery floor, despite no significant max-success benefit.It appeared to recover meaningfully first after the goal shift, a pattern the authors suggest may explain its higher mean success rate.Source: E. Quantitative Results · pp. 5–6

F. Comparison to Batch-Based Approaches

  • The top four Stream-AC variants significantly outperformed batch PPO on all adaptation scenarios in max and mean success rate.The comparison used batch PPO without a warm-start period and with a more aggressive learning rate.Source: F. Comparison to Batch-Based Approaches · p. 6

V. ANALYSIS ON OTHER ROBOTIC ENVIRONMENTS

The paper tests whether streaming deep RL adaptation extends beyond quadruped locomotion to manipulation tasks with different embodiments. It evaluates a Panda arm and a Unitree G1 humanoid in ManiSkill3.

V. ANALYSIS ON OTHER ROBOTIC ENVIRONMENTS

  • The manipulation analysis evaluates streaming deep RL adaptation across two tasks and distinct robot embodiments.The tasks use state observations, joint-position changes as actions, and the environments’ default dense rewards.Source: §V · p. 6, A. Task Environment · p. 6

A. Task Environment

  • Push Cube requires a Franka Emika Panda arm to push a randomized cube to a goal positioned a fixed distance in front of it.Success requires the cube to remain on the table and end within a specified radius of the goal; its dense reward covers push position, cube-to-goal distance, and keeping the cube on the table.Source: A. Task Environment · p. 6
  • Transport Box requires a Unitree G1 humanoid to move a randomized box from one table to another.Its four-stage dense reward encourages facing the box, grasping it stably, transporting it, and releasing it on the other table.Source: A. Task Environment · p. 6

B. Adaptation Scenarios

The manipulation experiments test streaming adaptation to abrupt goal or embodiment changes using four leading Stream-AC variants. Push Cube methods recover after a goal shift but later decline, while transport-box adaptation shows limited success.

B. Adaptation Scenarios

  • The scenarios impose distinct abrupt changes: a -0.15 y-axis goal offset in Push Cube and stuck or stiff shoulder, hand, and finger joints in Transport Box.Source: B. Adaptation Scenarios · p. 6

C. Stream Adaptation Experiments

  • The experiments compare ObGD, AdaptiveObGD, AdaptiveObGD + CBP, and AdaptiveObGD + LN over a warm-start followed by a 1.5-million-step adaptation phase.Max and mean success rates are evaluated every 10,000 steps during adaptation.Source: C. Stream Adaptation Experiments · p. 6

1) Push Cube:

  • AdaptiveObGD + LN achieved the highest Push Cube max success rate at 78.4%, while it and AdaptiveObGD had the highest mean success rate at approximately 23%.The authors report that methods recover to their respective peaks after the shift, then decline toward a stable success rate around 20%.Source: 1) Push Cube · pp. 6–7, Table 3 · p. 7, Figure 3 · p. 7
  • Transport Box adaptation showed limited success across variants; AdaptiveObGD + LN led with 23.3% max and 12.2% mean success rates.AdaptiveObGD + CBP was second strongest, narrowly ahead of AdaptiveObGD, but mitigation gains were marginal over the pretrained policy.Source: 1) Push Cube · p. 7, Table 3 · p. 7

2) Transport Box:

Streaming deep RL can adapt pretrained policies to changes and outperform batch-based methods, but recovery remains limited on complex tasks and real-world lifelong learning remains an open challenge.

2) Transport Box:

  • On more complex control tasks, noisy single-sample stream updates leave limited capacity for performance recovery and behavioral adaptation.Source: 2) Transport Box · p. 7

VI. DISCUSSION

  • With careful optimizer and plasticity-loss choices, streaming deep RL adapted pretrained policies to change and outperformed batch-based methods such as PPO.Source: §VI · p. 7
  • Streaming methods show strong forward transfer in quadruped locomotion, while manipulation shows signs of success but retains stability and adaptability ceilings.Source: §VI · p. 7
  • Managing the plasticity-stability tradeoff is critical; optimizer choices and network techniques such as continual backpropagation and layer normalization help preserve stability or plasticity.Source: §VI · p. 7
  • The experiments are an early simulation step toward real-world persistent autonomy, which requires advances in stability, safety, data efficiency, and system design.Source: §VI · p. 7
Loading 2609.28807v1…