PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

Jiaheng Hu, Jay Shim, Chen Tang, Yoonchang Sung, Bo Liu, Peter Stone, Roberto Martin-Martin

arXiv:2603.11653v3cs.LGcs.RO

TL;DR

Continual reinforcement learning for VLAs must balance adaptation to new tasks with retention of prior competence. The paper systematically evaluates CRL methods and finds that simple Seq. FT with LoRA achieves strong continual performance, with stability supported by pretrained models, parameter-efficient adaptation, and on-policy RL.

  • Problem

    VLA models are brittle in evolving or out-of-distribution settings, while conventional continual learning suggests Sequential Fine-Tuning causes catastrophic forgetting.

  • Method

    The paper conducts a systematic empirical study of CRL methods for large pretrained VLAs across diverse lifelong RL benchmarks, using LoRA and on-policy RL.

  • Results

    Across benchmarks, Seq. FT with LoRA achieves strong plasticity, minimal forgetting, enhanced zero-shot generalization, and frequently outperforms more sophisticated CRL methods.

  • Takeaways & Limitations

    Parameter-efficient Seq. FT is a simple, effective method for continual reinforcement learning with VLA models.

  • Takeaways & Limitations

    Continuous diffusion action heads often require more careful constraints, such as lower LoRA rank, and their sensitivity is left for future investigation.

Abstract

from arXiv · show

Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments. However, conventional wisdom from continual learning suggests that naive Sequential Fine-Tuning (Seq. FT) leads to catastrophic forgetting, necessitating complex CRL strategies. In this work, we take a step back and conduct a systematic study of CRL for large pretrained VLAs across diverse lifelong RL benchmarks. We find that, contrary to established belief, simple Seq. FT with low-rank adaptation (LoRA) is remarkably strong: it achieves high plasticity, exhibits little to no forgetting, and retains strong zero-shot generalization, frequently outperforming more sophisticated CRL methods. Through detailed analysis, we show that this robustness arises from a synergy between the large pretrained model, parameter-efficient adaptation, and on-policy RL. Together, these components reshape the stability-plasticity trade-off, making continual adaptation both stable and scalable. Our results position Sequential Fine-Tuning as a powerful method for continual RL with VLAs and provide new insights into lifelong learning in the large model era. Code is available at https://github.com/UT-Austin-RobIn/continual-vla-rl

1 Introduction

The paper studies whether large pretrained VLAs can continually adapt without the catastrophic forgetting expected from Sequential Fine-Tuning. It finds that Seq. FT with LoRA performs strongly because pretrained VLAs, parameter-efficient adaptation, and on-policy RL jointly support stability and plasticity.

  • VLA models generalize broadly but remain brittle in evolving or out-of-distribution environments, motivating continual adaptation through ongoing interaction.
  • Sequential Fine-Tuning adapts directly to arriving tasks but is traditionally associated with catastrophic forgetting on previously learned tasks.
  • Regularization, replay, and parameter isolation preserve prior performance but can reduce adaptation to new tasks, creating the stability–plasticity dilemma.
  • Large VLAs require parameter-efficient fine-tuning such as LoRA because full training is computationally costly, while their pretrained knowledge supports zero-shot performance.
  • Across diverse lifelong RL benchmarks, Seq. FT with LoRA achieves high plasticity, little to no forgetting, and strong zero-shot generalization, often surpassing sophisticated CRL methods and the multitask oracle.
  • The robustness of Seq. FT arises from the complementary interplay of pretrained VLAs, LoRA, and on-policy RL, with removing any component increasing forgetting.
  • The study establishes parameter-efficient Seq. FT as a simple, effective approach for continual reinforcement learning with VLA models.

2 Background & Related Work

The background introduces VLAs, RL post-training, continual reinforcement learning, and parameter-efficient adaptation. It emphasizes autoregressive and flow-based action policies, on-policy GRPO training, and LoRA's low-rank updates to frozen pretrained weights.

  • Vision-Language-Action Models: VLA models unify visual perception, language conditioning, and action generation, commonly using autoregressive action tokens or action chunking.
  • Reinforcement Learning Post-Training of VLA Models: RL post-training refines pretrained VLAs, whose generalization capabilities support exploration and learning from sparse rewards on challenging tasks.
  • Reinforcement Learning Post-Training of VLA Models: The study uses on-policy reinforcement learning with Group Relative Policy Optimization, a stable policy-gradient method for large-scale post-training.
  • Continual Reinforcement Learning: Continual reinforcement learning requires agents to adapt to non-stationary tasks or environments while retaining competence on previously encountered ones.
  • Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning updates only a small subset of parameters while keeping the pretrained backbone frozen, addressing the cost of full-parameter adaptation.
  • Parameter-Efficient Fine-Tuning: LoRA parameterizes weight updates as low-rank matrices while preserving the original pretrained weights.
  • Parameter-Efficient Fine-Tuning: For pretrained weights W0, LoRA trains matrices B and A with rank r much smaller than the matrix dimensions, then merges them as Wnew ← W0 + BA.
  • Parameter-Efficient Fine-Tuning: The paper adopts LoRA because it reduces trainable parameters while preserving the pretrained model's expressive capacity.

3 Problem Formulation

The paper formulates VLA post-training as language-conditioned finite-horizon MDPs and continual learning as sequential adaptation across tasks without previous-task data. It evaluates performance, forgetting, transfer, and retained zero-shot capabilities.

  • 3.1 Language-Conditioned MDP for VLA Post-Training: Each VLA post-training task is modeled as a finite-horizon, language-conditioned Markov Decision Process.
  • 3.1 Language-Conditioned MDP for VLA Post-Training: The MDP contains state and action spaces, transitions, horizon, initial-state distribution, language instruction ℓ, and sparse reward r.
  • 3.1 Language-Conditioned MDP for VLA Post-Training: The shared state space consists of camera images, while actions specify robot end-effector pose and gripper commands.
  • 3.2 Continual Reinforcement Learning in Language-Conditioned MDPs: In continual learning, the agent encounters T tasks sequentially in a fixed order beyond its control, with each task specified by a language instruction and sparse reward.
  • 3.2 Continual Reinforcement Learning in Language-Conditioned MDPs: The agent learns solely through environmental interaction and cannot access data or environments from previous tasks while learning the current task.
  • 3.3 Evaluation Metrics: Evaluation uses Average Success for overall performance, Negative Backward Transfer for forgetting, and Forward Transfer for transfer to new tasks.
  • 3.3 Evaluation Metrics: Task identity is observable because the VLA policy is explicitly conditioned on the natural-language instruction.
  • 3.3 Evaluation Metrics: The evaluation spans diverse tasks and benchmarks and introduces Zero-Shot Success to measure retention of pretrained VLA capabilities.

4 An Empirical Study of Continual RL for VLAs

Across three LIBERO benchmarks, the study finds that Sequential Fine-Tuning with LoRA remains competitive without explicit forgetting-prevention mechanisms. It achieves strong training performance, little forgetting, preserved zero-shot generalization, and robustness across controlled perturbations.

  • Experimental Setup: The study evaluates eight continual reinforcement learning algorithms across three LIBERO benchmarks using a consistent GRPO-and-LoRA training protocol.The comparison includes Sequential Fine-Tuning, multitask training, and representatives of regularization, replay, and parameter-isolation paradigms.
  • Performance Across CRL Methods: Sequential Fine-Tuning achieves training-task performance similar to replay and parameter-isolation methods while surpassing the other evaluated CRL methods.Its average success is generally only slightly below the multitask oracle.
  • Performance Across CRL Methods: Sequential Fine-Tuning preserves strong zero-shot generalization and often outperforms the multitask oracle on this evaluation axis.The results indicate that continual adaptation does not degrade, and often enhances, pretrained generalization capabilities.
  • Forgetting and Plasticity: Sequential Fine-Tuning exhibits little forgetting, with NBT consistently below 2% and sometimes negative, allowing optimization to focus on the current task without update constraints.This behavior removes the expected stability–plasticity trade-off in these experiments.
  • Robustness Under Perturbations: Across environmental, domain, architectural, and task-order perturbations, Seq. FT maintains strong performance, with AVG within 5% of the multitask oracle and NBT below 2%.The reported perturbation study examines changes to visual and state conditions, domain and model architecture, and continual task order.
  • Closing the Oracle Gap: The AVG gap to the multitask oracle can be closed by training Sequential Fine-Tuning for more episodes on lower-performing tasks.This result suggests the gap is not caused by Seq. FT becoming stuck at sub-optimal solutions, while multitask training may retain sample-efficiency synergies.

5 Analysis: What Makes Sequential Fine-Tuning So Effective?

The analysis attributes Sequential Fine-Tuning’s stability to the complementary effects of large pretrained VLAs, LoRA, and on-policy RL. Together, these components reduce interference while preserving plasticity and zero-shot generalization.

  • Ablation analysis: Removing the VLA, on-policy RL, or LoRA component causes significant losses in average performance and zero-shot generalization, with catastrophic forgetting.The authors characterize their combination as synergistic: no pair alone provides the same stable learning without forgetting.
  • Effect of On-Policy RL: On-policy policy-gradient updates reweight probability mass where the current policy already has support, limiting abrupt divergence from the pretrained policy.This implicit KL-drift constraint helps retain prior capabilities, unlike supervised updates that may shift mass toward low-probability actions.
  • Effect of Large Pretrained Models: Large pretrained models exhibit low interference because high-dimensional parameter spaces provide directions in which task updates minimally affect pretrained knowledge.The analysis connects this explanation to Fisher energy, which measures curvature along the current-task gradient and therefore potential interference.
  • Effect of Low-Rank Adaptation: LoRA restricts updates to a low-rank subspace, constraining per-layer update geometry and reducing uncontrolled structural modification.LoRA rank 512 achieves 0.6% NBT versus 40.9% for full fine-tuning, while its across-layer effective-rank standard deviation is 89.3 versus 465.0.
  • Zero-shot generalization: Sequential Fine-Tuning consistently preserves strong zero-shot generalization and often maintains a slight edge over multi-task oracle training.The reported gap is generally small on the evaluated benchmarks.

6 Conclusion

Sequential Fine-Tuning with LoRA achieves strong plasticity, minimal forgetting, enhanced zero-shot generalization, and often outperforms more sophisticated CRL methods. The combination of pretrained VLAs, parameter-efficient adaptation, and on-policy RL provides a simple, scalable recipe for continual post-training, while physical-robot deployment remains future work.

  • Sequential Fine-Tuning with LoRA achieves strong plasticity, minimal forgetting, enhanced zero-shot generalization, and frequently outperforms sophisticated CRL methods.
  • The observed stability emerges from the synergy between large pretrained models, LoRA-based adaptation, and on-policy reinforcement learning.
  • The findings provide a simple but scalable recipe for using reinforcement learning as continual post-training for large pretrained VLAs.
  • Applying these findings to physical robotic systems through sim-to-real transfer or real-world reinforcement learning is identified as future work.

Supplementary Materials

The supplementary material describes GRPO-based on-policy training for autoregressive and continuous-action VLAs, including stochastic flow dynamics for policy-gradient optimization. It also reports that Seq. FT remains robust as training sequences grow to 30 tasks.

  • Seq. FT remains robust after sequential training on 30 tasks, demonstrating resilience to extended training-task sequences.
  • GRPO samples trajectories from the previous policy and optimizes the policy using return-based advantages.
  • For autoregressive VLAs, GRPO treats action-token sequences as policy outputs and computes likelihood ratios over tokens.
  • For continuous flow or diffusion action heads, Flow-SDE adds controlled Gaussian noise to create a stochastic action policy.
  • The resulting stochastic trajectories support standard PPO or GRPO objectives through advantage-weighted likelihood optimization.

B Evaluation Metrics

The evaluation uses a success matrix to measure final performance, forgetting, transfer, and retention of zero-shot capabilities across sequentially arriving tasks. These metrics distinguish adaptation on trained tasks from performance on held-out tasks.

  • The success matrix records each task’s success rate after training through every sequential task stage.
  • Average Final Success measures the final policy’s average success rate across the entire training task sequence.
  • Negative Backward Transfer measures degradation on earlier tasks after subsequent learning, with lower values indicating better retention.
  • Forward Transfer measures whether learning earlier tasks improves performance on future tasks before those tasks are trained.
  • Held-out performance evaluates whether the final policy retains or enhances zero-shot success on tasks excluded from continual training.

C Evaluation Algorithms

The study compares sequential fine-tuning, oracle multitask training, and continual-learning methods based on regularization, replay, parameter isolation, and model merging. These algorithms represent different strategies for balancing adaptation with retention.

  • Sequential Fine-Tuning trains each arriving task without replay, parameter isolation, or task-specific regularization and is commonly treated as a lower-bound baseline.
  • Multi-Task Training jointly accesses all tasks and serves as an oracle upper-bound reference outside continual-learning assumptions.
  • Continual-learning methods are grouped into regularization, replay, and parameter-isolation paradigms that constrain updates or preserve prior information.
  • Elastic Weight Consolidation penalizes updates to parameters estimated as important for previous tasks using a Fisher-based quadratic constraint.
  • Expert Replay stores demonstrations for all tasks, whereas Dark Experience Replay stores prior interactions and logits, with both storage requirements growing linearly with task count.
  • Dynamic Weight Expansion assigns each task an isolated LoRA adapter, preventing gradient interference while increasing adapter weights linearly with task count.
  • SLCA decouples layerwise learning rates, while RETAIN merges task updates into the base model using a discount coefficient.
  • Together, the evaluated methods span established continual-learning paradigms and emerging large-model adaptation strategies.

D Experiment Setup

The experiments use multiple initially capable base models and standardized benchmark configurations to compare continual learning methods reproducibly across task qualities.

  • Base models: Base models receive supervised fine-tuning on small in-domain datasets to ensure non-zero initial success across multiple checkpoint qualities.This setup tests whether results depend on a single initial policy.
  • Environment standardization: ManiSkill evaluation fixes plate, background, and table colors while varying initial states across 40 object positions and 4 rotations.All methods use the same fixed task configurations to reduce evaluation variance.
  • Benchmark design: Four ManiSkill training tasks balance multiple seeds and baselines against a tractable total experimental budget.
  • Task selection: Training tasks avoid near-zero and near-saturated initial success, leaving both existing capabilities to refine and headroom for measurable improvement.

E Ablation Setup

The ablations compare reinforcement learning with supervised alternatives, smaller and full-model policies, and shared algorithm settings across the libero-spatial benchmark.

  • Ablation protocol: Libero-spatial ablations share the main environment setup, evaluation protocol, and hyperparameters unless otherwise specified.
  • Supervised baseline: Supervised fine-tuning replaces online RL with behavior cloning on 432 demonstration trajectories from the environment.It uses the same base model and input representation as the RL setup.
  • Model-size ablation: The Smaller Policy ablation replaces OpenVLA-OFT with an approximately 12M-parameter CNN, primed on 30 demonstrations before RL fine-tuning.Its RL setup matches Sequential Fine-Tuning.
  • Parameter-efficiency ablation: Without LoRA updates the full OpenVLA parameters directly during RL post-training while keeping other RL hyperparameters unchanged.
  • Hyperparameters: Shared post-training settings use GRPO with rank-32 LoRA adapters, while Sequential Fine-Tuning introduces no additional method-specific hyperparameters.

H.3.2 Held-Out Tasks

Held-out tasks include object-placement, appliance-interaction, and receptacle tasks, with evaluation also varying camera angles, lighting conditions, and robot initial position.

  • Object placement: Held-out object-placement tasks require placing alphabet soup, cream cheese, salad dressing, barbecue sauce, or ketchup in a basket.
  • Interaction and placement: Additional held-out tasks test pressing a coffee-machine button, closing a cabinet or microwave door, and placing objects on a plate.Placed objects include a ketchup bottle, fast-food cup, watering can, and toy bear.
  • Visual and state variation: Generalization is evaluated under changed camera angles, lighting conditions, and robot initial positions.
Loading 2603.11653v3…