Source-linked AI summary
Improving Vision-Language-Action Model with Online Reinforcement Learning
Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, Jianyu Chen
TL;DR
Improving large VLA models through environment interaction is challenging because direct online RL can be unstable and computationally demanding. The paper proposes iRe-VLA, which alternates RL with supervised learning while freezing the VLM during RL and fine-tuning the full model on successful trajectories. Experiments in simulation and the real world report improved performance, generalization, and training stability, with a stated limitation under sparse-reward conditions.
Problem
Improving pretrained VLA models through physical interaction remains difficult because expert datasets are costly, distribution shift limits supervised learning, and online RL is unstable for long-horizon sparse-reward robot tasks.
Method
iRe-VLA alternates online reinforcement learning with supervised learning, freezing the VLM during RL and fine-tuning the entire model on successful trajectories.
Results
Across simulated and real-world manipulation tasks, iRe-VLA consistently improves VLA performance, stabilizes training, reduces computational demands, and improves generalization without catastrophic forgetting.
Takeaways & Limitations
Iterative RL and supervised learning can enhance existing VLA skills and generalization while preserving performance on previously learned tasks.
Takeaways & Limitations
Under sparse-reward online RL conditions, iRe-VLA can improve skills within seen types but cannot learn entirely new skills.
Abstract
from arXiv · showhide
Recent studies have successfully integrated large vision-language models (VLMs) into low-level robotic control by supervised fine-tuning (SFT) with expert robotic datasets, resulting in what we term vision-language-action (VLA) models. Although the VLA models are powerful, how to improve these large models during interaction with environments remains an open question. In this paper, we explore how to further improve these VLA models via Reinforcement Learning (RL), a commonly used fine-tuning technique for large models. However, we find that directly applying online RL to large VLA models presents significant challenges, including training instability that severely impacts the performance of large models, and computing burdens that exceed the capabilities of most local machines. To address these challenges, we propose iRe-VLA framework, which iterates between Reinforcement Learning and Supervised Learning to effectively improve VLA models, leveraging the exploratory benefits of RL while maintaining the stability of supervised learning. Experiments in two simulated benchmarks and a real-world manipulation suite validate the effectiveness of our method.
I. INTRODUCTION
VLA models extend pretrained vision-language models to low-level robotic control, but improving them through physical interaction remains difficult because expert data is costly and online RL is unstable. iRe-VLA addresses this by alternating online RL with supervised learning, freezing the VLM during exploration and fine-tuning the full model on successful trajectories.
- VLA models use pretrained vision-language models to produce low-level robotic control signals while retaining their common-sense and reasoning capabilities.
- Supervised fine-tuning is stable and scalable but depends on costly expert datasets and can suffer from distribution shift between demonstrations and physical environments.
- Online RL for embodied robots requires exploration over long-horizon, sparse-reward tasks and can be highly unstable for large neural networks.
- iRe-VLA alternates online RL and supervised learning to stabilize training and enhance VLA models.The RL stage freezes VLM parameters, while supervised learning fine-tunes the entire model on successful trajectories.
- Across simulated and real-world manipulation tasks, iRe-VLA improves task alignment, solves unseen tasks, and strengthens generalization while reducing computational demands.
II. RELATED WORKS
Prior work uses language and vision-language models for embodied reasoning, planning, and control, while RL fine-tuning has mainly aligned language models with human preferences. This paper instead applies RL directly to low-level VLA control in long-horizon, sparse-reward physical environments.
- Foundation models support embodied control by providing rewards or values, modeling dynamics, generating plans, or directly serving as policies.
- LLM-based planning methods generate textual steps or code, but these plans are not directly grounded in the physical world and require low-level skills.
- RLHF typically optimizes language models in offline-style settings using pre-collected human preference data and constrained policy updates.
- Unlike high-level planning approaches that assume skills such as pick or goto, this work uses RL to improve VLA-generated low-level control signals directly.
- The proposed setting involves physical environments with unknown dynamics, horizons of hundreds or thousands of steps, and sparse rewards.
IV. METHOD
The method begins with a VLA model trained on robotic demonstrations and applies online interactions to improve it while keeping computational costs affordable for robotic systems.
- iRe-VLA improves a demonstration-trained VLA model through online interaction under an affordability goal for robotic-system computation.
A. Model Architectures
The VLA architecture maps visual observations and language instructions to low-level actions using a pretrained VLM and lightweight action head. The design supports parameter-efficient adaptation while alternating VLM freezing and training across optimization stages.
- Model Architecture: The VLA model maps visual input and freeform language instructions to low-level robotic actions.
- Model Architecture: Its architecture combines a pretrained large VLM backbone with a lightweight action head.
- Iterative Optimization: During iterative fine-tuning, the VLM is frozen during exploration and trainable during supervised learning to balance stability with use of pretrained representations.
- Action Head: The action head converts the VLM’s hidden representation into action vectors through a token learner followed by an MLP.
- Parameter-Efficient Fine-Tuning: LoRA is used to fine-tune the VLM part because updating billions of parameters is computationally expensive and can overfit limited data.
B. Learning Pipeline
The iRe-VLA pipeline first obtains an SFT VLA model, then alternates online RL for exploration with supervised learning on expert and successful online data. Freezing the VLM during RL limits optimization to the action head, while later full-model supervision mitigates forgetting and broadens task coverage.
- Stage 0: Supervised Learning on Expert Dataset: Stage 0 initializes the VLA model through supervised fine-tuning on an expert robotic dataset.The resulting model serves as the starting policy for subsequent improvement.
- Stage 1: Online RL with Frozen VLM: During Stage 1, online RL adapts the action head while the VLM parameters remain frozen to reduce model collapse and accelerate learning.A critic head is introduced, and only action-head parameters are optimized.
- Stage 1: Online RL with Frozen VLM: Successful trajectories discovered during online RL are collected into the online dataset D_RL.The dataset is updated by adding newly discovered trajectories x_i.
- Stage 2: Supervised Learning on Both Expert and Online-collected Data: Stage 2 applies supervised learning to the full model using the union of expert data D_e and online data D_RL.This stage addresses the risk of forgetting previously learned tasks after RL.
- Iterate between Stage 1 and Stage 2: Alternating RL exploration with imitation of successful trajectories progressively expands task coverage while helping prevent catastrophic forgetting.The two stages are repeated as an iterative pipeline.
V. EXPERIMENTS
The experiments evaluate iRe-VLA across simulated and real-world settings to test why two-stage iterative RL is used, whether it stabilizes and improves VLA training, and whether it improves generalization.
- Experimental Questions: Experiments span the Metaworld and FrankaKitchen simulated benchmarks and real-world panda manipulation tasks.The evaluation is organized around the effectiveness and generalization of iRe-VLA.
- Experimental Questions: The study asks why two-stage iterative RL should replace standard RL for improving VLA models.This question targets the rationale for the proposed optimization process.
- Experimental Questions: The experiments test whether iRe-VLA stabilizes training and improves VLA performance on both expert and unseen tasks.The question explicitly covers training stability and performance across task familiarity.
- Experimental Questions: The experiments also test whether iRe-VLA produces better VLA generalization.Generalization is evaluated as a distinct experimental objective.
A. Experiment Setups
The evaluation uses one text-conditioned VLA model per domain and separates tasks into expert, RL-trained, and hold-out groups to assess improvement and generalization.
- Evaluation Domains and Task Groups: Experiments cover Metaworld, Franka Kitchen, and real-world panda manipulation, with one text-conditioned VLA model solving all tasks within each domain.Each domain contains expert, RL-trained, and hold-out task groups.
- Evaluation Domains and Task Groups: Expert tasks come from demonstration datasets, RL-trained tasks are enhanced through online RL, and hold-out tasks evaluate generalization.The three categories distinguish prior supervision, online improvement, and unseen-task evaluation.
- Dataset and Task Construction: Metaworld uses 25 expert tasks with 50 trajectories each, while Franka Kitchen uses an expert dataset containing 5 tasks.Novel tasks vary object shape, color, position, appearance, or position depending on the domain.
- Dataset and Task Construction: The real-world setting collects 2,000 trajectories through teleoperation and scripting.
B. Why do we adopt two-stage iterative optimization?
The paper motivates two-stage iterative optimization by observing instability from standard RL and computational difficulty in fully fine-tuning large VLA models. iRe-VLA addresses these issues by freezing the VLM during RL and distributing computation across stages.
- Stabilizing Training Process: Standard RL fine-tuning of the large VLA model can be unstable and cause performance drops, including on four of five sparse-reward Metaworld tasks.Similar instability has been reported for transformer-based RL policies.
- Managing the Model Training Burden: Fully fine-tuning a VLA model with billions of parameters exceeds most local machines’ computational capacity.Remote deployment can introduce parameter-transmission issues and reduce control frequency.
- Managing the Model Training Burden: The first RL stage reduces the training burden by freezing the upper-layer VLM and adapting only the lightweight action head.
C. Simulated Manipulation Experiments
Simulated experiments compare expert, RL-trained, and unseen tasks across MetaWorld and FrankaKitchen. iRe-VLA stabilizes online RL while improving performance on original, new, and unseen tasks, whereas permanently freezing the VLM reduces performance.
- Experiment setup: SFT policies provide effective starting points for new RL tasks, accelerating learning compared with learning from scratch.The experiments apply iRe-VLA sequentially to RL tasks and compare it with PPO and PPO-Replay.
- Training stability: Standard online RL can underperform SFT and degrade previously learned tasks because unstable RL updates affect pretrained representations.Table I reports this pattern across expert, RL-trained, and unseen task categories.
- Original tasks: iRe-VLA improves the FrankaKitchen left-door-open success rate from 0.43 to 0.83.The task was an expert task with limited demonstrations.
- RL-trained tasks: iRe-VLA successfully tackles new RL tasks in each domain without catastrophic forgetting.The method improves RL-trained tasks while retaining performance on previously learned tasks.
- Unseen tasks: After mastering four MetaWorld window-task types, the agent generalizes to windows with unseen colors and shapes.The experiments also report increased success rates on unseen tasks as the agent learns more task varieties.
- Ablations: Freezing the VLM in both stages causes performance drops, indicating that later supervised updates contribute beyond the action head alone.The authors attribute this partly to the action head’s limited expressiveness and report that online action data can enhance upper-layer VLM representations.
D. Real-world Manipulation Experiments
Real-world Panda experiments extend online RL from color-limited demonstrations to irregularly shaped objects and unseen objects. The iRe-VLA pipeline improves targeted picking success while keeping original-task performance stable under computational constraints.
- Experiment setup: The real-world VLA model is trained on 2,000 human-collected demonstrations spanning picking, placing, button-pressing, cable-routing, and drawer operations.The experiment follows the SERL real-world RL setup.
- Method: Real-world online RL uses SACfD and latent caching to improve sample efficiency and computational affordability.The implementation stores each VLM latent after processing an image once and trains in latent space.
- Computational constraint: Directly fine-tuning the entire VLA model exceeded the local machine’s computational capabilities, so standard real-world RL results were not reported.This constraint motivates the computationally lighter design choices used in the experiment.
- Results: The pipeline increases picking success on eggplants or carrots from 0.35 to 0.80 after demonstrations covered only four block colors.Each new real-world RL task requires around one hour of training.
- Results: Picking success on unseen objects improves from 0.37 to 0.61, while success on original tasks remains stable.The online RL tasks involve objects with irregular shapes, including eggplants and carrots.
VI. CONCLUSION AND LIMITATION
The paper concludes that iRe-VLA stabilizes online RL and reduces computational demands for improving VLA models across simulated and real-world manipulation tasks. It also identifies a scope limitation: sparse-reward online RL improves skills within seen types but does not learn entirely new skills.
- Conclusion: iRe-VLA stabilizes training and reduces computational demands while improving VLA models in simulated and real-world manipulation experiments.The conclusion reports effectiveness across both evaluation settings.
- Limitation: Under sparse-reward online RL, iRe-VLA can improve skills within seen types but cannot learn entirely new skills.This is stated as a potential limitation of the method.