PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning

Yuan Liu, Haoran Li, Shuai Tian, Yuxing Qin, Yuhui Chen, Yupeng Zheng, Yongzhen Huang, Dongbin Zhao

arXiv:2602.10503v2cs.RO

TL;DR

VLA adaptation with SFT requires substantial task-specific data and is prone to catastrophic forgetting. LifeLong-RFT uses chunk-level on-policy reinforcement fine-tuning with multi-dimensional process rewards, and reports strong multi-task results, including a 22% LIBERO continual-learning gain over SFT.

  • Problem

    SFT adaptation of VLAs requires substantial task-specific data, while continual learning must also retain previously acquired skills; existing VLA reinforcement fine-tuning rewards often depend on environment interaction or fallible reward models.

    Source: §I · p. 2

  • Method

    LifeLong-RFT combines chunking-level on-policy reinforcement fine-tuning with QACR, CTAR, and FCR, without requiring environment interaction or pre-trained reward models.

    Source: §I · p. 2

  • Results

    LifeLong-RFT shows strong multi-task performance across simulated and real-world tasks; on LIBERO continual learning it improves average success rate over SFT by 22% using only 20% of the training data.

    Source: §I · p. 2

  • Takeaways & Limitations

    The approach is described as enabling VLAs to learn new tasks with limited demonstrations while preserving original capabilities.

    Source: §I · p. 2

    The work focuses primarily on discrete action models, whose performance falls short of continuous action models.

    Source: §VI · p. 9

Abstract

from arXiv · show

Pretrained on large-scale and diverse datasets, VLA models demonstrate strong generalization and adaptability as general-purpose robotic policies. However, Supervised Fine-Tuning (SFT), which serves as the primary mechanism for adapting VLAs to downstream domains, requires substantial amounts of task-specific data and is prone to catastrophic forgetting. To address these limitations, we propose LifeLong-RFT, a simple yet effective Reinforcement Fine-Tuning (RFT) strategy for VLA models independent of online environmental feedback and pre-trained reward models. By integrating chunking-level on-policy reinforcement learning with the proposed multi-dimensional process reward mechanism, LifeLong-RFT quantifies the heterogeneous contributions of intermediate action chunks across three dimensions to facilitate policy optimization. Specifically, (1) the Quantized Action Consistency Reward (QACR) ensures accurate action prediction within the discrete action space; (2) the Continuous Trajectory Alignment Reward (CTAR) aligns decoded continuous action chunks with reference trajectories to ensure precise control; (3) the Format Compliance Reward (FCR) guarantees the structural validity of outputs. Comprehensive experiments across SimplerEnv, LIBERO, and real-world tasks demonstrate that LifeLong-RFT exhibits strong performance in multi-task learning. Furthermore, for continual learning on the LIBERO benchmark, our method achieves a 22% gain in average success rate over SFT, while effectively adapting to new tasks using only 20% of the training data. Overall, our method provides a promising post-training paradigm for VLAs. The project page is available at <https://yuan-liu-lifelong-rft.github.io>.

I. INTRODUCTION

SFT adaptation of VLAs is constrained by task-specific data needs and catastrophic forgetting, while existing reinforcement fine-tuning approaches depend on costly environment interaction or fallible reward models. LifeLong-RFT combines chunk-level on-policy learning with three-dimensional process rewards; experiments report strong multi-task performance and a 22% LIBERO continual-learning gain over SFT.

I. INTRODUCTION

  • SFT requires substantial task-specific data and can cause catastrophic forgetting, making continual VLA adaptation with limited data and retained skills difficult.Foundation-model representations reduce data requirements but do not eliminate forgetting; continual-learning techniques developed for specialized models struggle to scale to VLA task sets and parameterization.Source: §I · p. 2
  • LifeLong-RFT combines chunking-level on-policy reinforcement fine-tuning with QACR, CTAR, and FCR process rewards, without requiring environment interaction.The rewards assess discrete action-token consistency, continuous trajectory alignment, and output-format compliance.Source: §I · p. 2
  • LifeLong-RFT improves average success rate over SFT by 22% on LIBERO continual learning and adapts to new tasks using only 20% of the training data.Experiments across simulated and real-world tasks also demonstrate strong multi-task performance.Source: §I · p. 2

A. Vision-Language-Action Models

  • VLA models use discrete-token or continuous-action representations, and SFT-based post-training remains data-intensive and prone to forgetting.The models are pretrained on large-scale data before task-specific adaptation.Source: A. Vision-Language-Action Models · p. 3

B. Reinforcement Fine-tuning for VLA Models

  • VLA reinforcement fine-tuning methods use simulation rewards, real-world adaptation, or world models, but require environmental interaction and face deployment, cost, or reward-reliability challenges.Simulation methods use privileged states; real-world strategies incur human costs, while world-model prediction errors can increase reward-hacking susceptibility.Source: B. Reinforcement Fine-tuning for VLA… · p. 3

C. Continual Learning in Robotics

  • Continual-learning approaches for VLAs include model merging and continual imitation learning; LifeLong-RFT instead combines on-policy reinforcement learning with multi-dimensional process rewards.The stated aim is adaptation to new tasks while retaining previously learned knowledge.Source: C. Continual Learning in Robotics · p. 3

III. PROBLEM FORMULATION AND PRELIMINARIES

The preliminaries frame continual learning as acquiring skills across tasks despite limited access to historical data, while ordinary VLA post-training uses task-specific data. The method applies chunk-level on-policy reinforcement learning and group-relative policy optimization without environment interaction.

III. PROBLEM FORMULATION AND PRELIMINARIES

  • VLA models are pretrained on diverse data, then post-trained with task-specific data to adapt actions to downstream tasks and robot embodiments.SFT is described as the primary post-training approach, but it largely overlooks degradation of previously acquired capabilities.Source: §III · p. 3
  • Continual learning requires acquiring skills across a sequence of tasks while access to historical data is limited.Each task is associated with expert demonstrations, unlike single-stage adaptation that assumes concurrent access to all task data.Source: §III · p. 3
  • On-policy reinforcement learning updates a policy using its own generated outputs, and prior LLM findings suggest greater resistance to forgetting than SFT.It optimizes expected return over self-generated answers rather than relying on a fixed annotated dataset.Source: §III · p. 3

A. Chunking-Level On-Policy Reinforcement Learning

  • LifeLong-RFT evaluates sampled action chunks independently, removing the need to evaluate complete trajectories through environment interaction.The approach integrates chunk-level on-policy reinforcement learning with a multi-dimensional process reward.Source: A. Chunking-Level On-Policy… · pp. 3–4, §IV · p. 3, Figure 2 · p. 4
  • GRPO samples groups of action outputs for each observation and instruction, then computes relative advantages from their rewards.Group-wise comparisons avoid the explicit critic network used by PPO and reduce computational overhead.Source: A. Chunking-Level On-Policy… · p. 4
  • The policy objective uses expert demonstrations and clips the policy probability ratio to [1−ϵ, 1+ϵ] to stabilize optimization.A KL-divergence regularizer weighted by γ limits deviation from the reference policy; reward construction is central to the optimization.Source: A. Chunking-Level On-Policy… · p. 4

B. Multi-Dimensional Process Reward

The multi-dimensional process reward assesses action chunks across complementary dimensions that connect discrete token generation with continuous control. QACR checks action-token consistency after verifying that the output meets tokenizer format requirements.

B. Multi-Dimensional Process Reward

  • The process reward decomposes action-chunk assessment into complementary dimensions to connect discrete token generation with continuous robotic control.The section introduces the dimension-specific reward designs as a way to guide on-policy learning without environment interaction.Source: B. Multi-Dimensional Process Reward · p. 4

1) Quantized Action Consistency Reward:

  • QACR measures consistency between predicted and ground-truth action tokens for VLA outputs in the quantized action space.The reward uses position-wise token matches, with an indicator equal to 1 for a match and 0 otherwise.Source: 1) Quantized Action Consistency Reward · pp. 4–5
  • QACR first checks action-chunk size and action dimension against Fast+ specifications; invalid generations receive zero reward.Only validated generations proceed to consistency assessment, and the resulting QACR score lies in [0, 1].Source: 1) Quantized Action Consistency Reward · pp. 4–5

2) Continuous Trajectory Alignment Reward:

CTAR evaluates whether decoded continuous action chunks align spatially with reference trajectories, providing chunk-level feedback for manipulation. It combines pose and gripper rewards across the chunk horizon after rejecting invalid formats.

2) Continuous Trajectory Alignment Reward:

  • CTAR measures spatial alignment between decoded continuous action chunks and reference trajectories, providing dense feedback for dexterous manipulation.Invalid action-token formats receive zero reward; valid sequences are decoded into continuous action chunks.Source: 2) Continuous Trajectory Alignment… · p. 5
  • CTAR assesses pose and gripper components of decoded actions at each time step across the action chunk.The pose may represent end-effector pose or joint angles, while the gripper component indicates its open-close state.Source: 2) Continuous Trajectory Alignment… · p. 5
  • The pose reward decays exponentially with normalized L1 deviation from the ground-truth pose, with α controlling sensitivity.The gripper reward is binary, assigning 1 when predicted and ground-truth gripper states match and 0 otherwise.Source: 2) Continuous Trajectory Alignment… · p. 5
  • CTAR averages the weighted pose and gripper rewards over the chunk horizon H, with β setting their relative importance.The resulting reward quantifies discrepancies in both robot poses and gripper states.Source: 2) Continuous Trajectory Alignment… · p. 5

3) Format Compliance Reward:

The Format Compliance Reward (FCR) checks that generated action sequences have the structure needed for the other rewards to assess them. It is combined with QACR and CTAR in the multi-dimensional process reward.

3) Format Compliance Reward:

  • FCR addresses the structural validity on which QACR and CTAR depend: outputs must meet the specified action dimensions and chunk size.The reward guides the model toward structurally well-formed token sequences.Source: 3) Format Compliance Reward · p. 5
  • Fast+ validation gives a binary FCR: outputs matching the required shape receive 1, and failures receive 0.Valid outputs can be decoded into a continuous action chunk, supporting trajectory exploration.Source: 3) Format Compliance Reward · pp. 5–6
  • The combined process reward weights QACR against CTAR with ω and scales FCR with λ.ω ∈ [0, 1] controls the discrete-consistency versus continuous-alignment trade-off; λ scales format compliance.Source: 3) Format Compliance Reward · p. 6

V. EXPERIMENTS

Experiments evaluate LifeLong-RFT in simulation and on a Franka robot, including multi-task learning on SimplerEnv and LIBERO. In simulation, it improves average success rates over SFT on WidowX and Google Robot and reaches a 95.6% average success rate on LIBERO.

A. Implementation Details

  • Implementation uses NORA-Long with Fast+, full-parameter optimization, GRPO group size 8, and AdamW at a learning rate of 1 × 10^-6.CTAR uses α = 5 and β = 0.8; reward weights are ω = 0.7 and λ = 0.1, with experiments on 8 NVIDIA H20 GPUs.Source: A. Implementation Details · p. 6

1) Experimental Setup:

  • Training covers SimplerEnv with BridgeData V2 for WidowX and Fractal for Google Robot, and all 10 tasks in each LIBERO suite.The Franka real-world setup jointly trains on 40 demonstrations for each of the first three tasks and 50 for the last.Source: 1) Experimental Setup · p. 6
  • Evaluation repeats each SimplerEnv task for 24 trials, uses 500 trials per LIBERO suite, and runs 20 trials per real-world task.Average success rate is the metric across these evaluations.Source: 1) Experimental Setup · p. 6

2) Performance on Simulation:

  • 3.5% on WidowX and 4.4% on Google Robot are LifeLong-RFT’s average success-rate gains over SFT; its average success rate on LIBERO is 95.6%.The LIBERO result surpasses the competing continuous- and discrete-action models.Source: 2) Performance on Simulation · p. 6

3) Performance on Real-World:

The block reports real-world multi-task results alongside continual-learning experiments on LIBERO. It also describes sequential training with limited demonstrations and evaluates continual learning using task success rates.

3) Performance on Real-World:

  • 8.7% is LifeLong-RFT’s average success-rate improvement over SFT across four real-world tasks; the gain on Hang Chinese Knot is 15%.The tasks are Pick & Place (Banana and Bread), Pull Drawer, and Hang Chinese Knot.Source: 3) Performance on Real-World · p. 7, Figure 3 · p. 7

1) Experimental Setup:

  • LIBERO continual learning uses six base tasks with 50 demonstrations each, then four incremental tasks with 10 demonstrations each and 5 retained per prior task.The real-world sequence trains on four tasks, using 20 demonstrations for each new task and retaining 5 for each previous task.Source: 1) Experimental Setup · p. 7
  • Continual-learning evaluation uses FWT, NBT, and AUC derived from task success rates, with 50 LIBERO episodes and 20 real-world episodes.Higher FWT indicates better new-task adaptation, lower NBT less forgetting, and higher AUC better average success rates across tasks.Source: 1) Experimental Setup · p. 7

2) Performance on Simulation:

  • In evaluation on LIBERO against behavioral-cloning models and SFT-optimized large-scale VLAs, LifeLong-RFT consistently outperforms other methods across all task suites and gains 35.9 AUC over the SFT baseline on LIBERO-Goal.Source: 2) Performance on Simulation · p. 7

3) Performance on Real-World:

LifeLong-RFT improves continual-learning performance on real-world tasks and LIBERO, while its reward components and few-shot adaptation are evaluated on LIBERO. The conclusion identifies discrete-action models as a current scope boundary.

3) Performance on Real-World:

1) Effectiveness of Multi-Dimensional Process Reward:

  • Removing QACR drops average performance by 2.8% across four LIBERO suites, removing CTAR drops it by 90.9%, and removing FCR degrades LIBERO-Long by 4.4%.The authors report each reward component as effective; FCR is described as critical to structural output validity.Source: Table VI · p. 8, 1) Effectiveness of Multi-Dimensional… · p. 8

2) Efficiency of New Task Adaptation:

  • With 5 demonstrations, LifeLong-RFT reaches 100% on Pick Orange Juice and matches SFT trained on 50 demonstrations on Pick Bowl; with 10, it reaches 100% on Pick Bowl.With full demonstrations, it surpasses SFT by 30% on Put Wine Bottle and performs better on Put Alphabet Soup and Cream Cheese.Source: 2) Efficiency of New Task Adaptation · p. 8

VI. CONCLUSION AND FUTURE OUTLOOK

  • LifeLong-RFT combines chunking-level on-policy RL with three-dimensional process rewards, while its current focus on discrete action models leaves performance below continuous action models.The stated future direction is to extend the strategy to continuous action models.Source: §VI · pp. 8–9

APPENDIX

The appendix specifies simulation and real-world training settings, the continual-learning protocol, and detailed LIBERO and real-world outcomes.

1) Multi-Task Learning:

  • For SimplerEnv, WidowX uses batch size 512 for 30 epochs and Google Robot uses 1024 for 40; LIBERO-Long uses 256 for 35 epochs, while the other three LIBERO suites use 15 epochs.The remaining hyperparameters are described as consistent across the SimplerEnv platforms and unified across the three shorter LIBERO suites.Source: 1) Multi-Task Learning · p. 13, Table VII · p. 13, Table VIII · p. 13

2) Continual Learning:

  • The four Franka real-world tasks use 170 demonstrations, batch size 128, and 20 epochs; continual learning uses an initial base-task stage followed by lifelong learning with batch size 32 and 10 epochs.The real-world continual-learning experiment contains only the lifelong stage and learns four tasks sequentially.Source: 1) Multi-Task Learning · p. 13, 2) Continual Learning · p. 13, Table IX · p. 13

1) Detailed Continual Learning Results:

  • After LIBERO-Goal Task 8, performance improves on prior Tasks 2, 3, and 7, while LIBERO-Long remains weak on Task 7 at 36% and Task 9 at 34% with limited demonstrations.The authors describe the prior-task improvements as backward transfer and the low long-horizon results as a challenge for future work.Source: 1) Detailed Continual Learning Results · p. 13, Table XI · p. 14
  • On real-world continual learning, Pull Drawer reaches 100% success with 20 demonstrations, while Hang Chinese Knot reaches 60%.The paper identifies Hang Chinese Knot as a deformable-object task needing further improvement.Source: 1) Detailed Continual Learning Results · p. 14, Table XII · p. 14

2) Further Analysis of Continual Learning:

Additional analyses test extended LIBERO-Goal sequences, reward-weight sensitivity, training reward trends, and generalization to another discrete-action model.

2) Further Analysis of Continual Learning:

  • Across 10 LIBERO-Goal tasks, training uses 10 demonstrations per new task and replays 5 per previous task; Task 8 reaches 100% while prior knowledge is maintained.The authors report adaptability despite increasing task counts and limited samples.Source: 2) Further Analysis of Continual… · p. 14, Table XIII · p. 14

C. Analysis of Reward Combinations within the Multi-Dimensional Process Reward

D. Visualization of Training Process

E. Generalization across Discrete Action Models

  • On LIBERO-Spatial with RoboBrain-X0, LifeLong-RFT improves FWT by 12.2% and reduces NBT by 13.3%, with reported superiority across all four suites.The authors interpret these results as efficient new-skill acquisition with preservation of previously learned knowledge.Source: E. Generalization across Discrete… · p. 15, Table XIV · p. 15

F. Real-World Case Studies

The real-world case studies illustrate LifeLong-RFT executions across four manipulation tasks, each with a distinct handling challenge.

1) Pick Banana:

  • The banana example shows the model identifying and grasping a banana among distractor fruits, then placing it in a blue plate.The authors report robust completion despite clutter and distractors.Source: 1) Pick Banana · p. 15, Figure 7 · p. 16

2) Pick Bread:

  • The bread example centers on inserting bread into a narrow toaster slot, which the authors report LifeLong-RFT successfully completes.The paper characterizes this as fine-grained manipulation.Source: 2) Pick Bread · p. 15, Figure 8 · p. 16

3) Pull Drawer:

  • The drawer example requires grasping a handle and coordinating the end-effector with linear drawer motion to avoid jamming.The authors report robust manipulation in this constrained setting.Source: 3) Pull Drawer · p. 15, Figure 9 · p. 16

4) Hang Chinese Knot:

  • The Chinese Knot example requires lifting a deformable knot from a table and hanging it on a cabinet hook while adapting to deformation.The authors also note limitations in this task.Source: 4) Hang Chinese Knot · p. 15, Figure 10 · p. 16
Loading 2602.10503v2…