Source-linked AI summary
VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, Ziwei Wang
TL;DR
Offline demonstrations provide limited state coverage for high-capacity VLAs, motivating online reinforcement learning for test-time improvement. VLA-RL formulates robotic manipulation as multi-modal multi-turn conversation, densifies rewards with a robotic process reward model, and adds scalable training techniques. It improves OpenVLA-7B by 4.5% on 40 LIBERO tasks, matches π0-FAST, and shows performance gains with increased test-time optimization.
Problem
Offline VLA training uses demonstrations with limited visited-state coverage, which can cause execution failure in out-of-distribution scenarios.
Method
VLA-RL applies online RL to pretrained auto-regressive VLAs, modeling manipulation trajectories as multi-modal multi-turn conversations and using a robotic process reward model for reward densification.
Results
4.5% improvement over OpenVLA-7B was achieved on 40 challenging LIBERO tasks, while matching the performance of π0-FAST.
Takeaways & Limitations
Performance consistently improves with increased test-time optimization, suggesting an emerging inference-scaling principle for robotic manipulation.
Takeaways & Limitations
Pseudo-reward extraction heuristics may not capture the nuances of more dexterous manipulation tasks, potentially making policy optimization inefficient.
Abstract
from arXiv · showhide
Recent high-capacity vision-language-action (VLA) models have demonstrated impressive performance on a range of robotic manipulation tasks by imitating human demonstrations. However, exploiting offline data with limited visited states will cause execution failure in out-of-distribution scenarios. Intuitively, an exploration-based method that improves on online collected data at test time could address this limitation. We present VLA-RL, an algorithmic and systematic framework that leverages online reinforcement learning (RL) to improve pretrained auto-regressive VLAs in downstream tasks. Within a unified perspective, we first introduce a trajectory-level RL formulation for auto-regressive VLA training, which models general robotic manipulation trajectory as multi-modal multi-turn conversation. To address the challenge of sparse rewards, we fine-tune a pretrained vision-language model as a robotic process reward model, which is trained on pseudo reward labels annotated on automatically extracted task segments. To scale up, we identify several implementation findings that improve the stability and efficiency including curriculum selection strategy, GPU-balanced vectorized environments, batch decoding, and critic warmup. VLA-RL enables OpenVLA-7B to surpass the strongest finetuned baseline by 4.5% on 40 challenging robotic manipulation tasks in LIBERO, and even matches the performance of advanced commercial models such as $π_0$-FAST. Notably, we observe that VLA-RL benefits from increased test-time optimization, indicating an early spark of inference scaling laws in robotics.
1 Introduction
Existing VLAs rely on offline imitation demonstrations, whose limited state coverage can cause failures in out-of-distribution scenarios. VLA-RL investigates online reinforcement learning to improve pretrained VLAs and reports gains across challenging manipulation tasks.
- Offline imitation learning can fail in out-of-distribution scenarios because demonstrations cover limited visited states.
- Reinforcement learning offers a route toward exploration-based test-time scaling using online-collected data with broader state coverage.
- Traditional robotics RL often requires extensive reward engineering and is commonly limited to low-dimensional states, small networks, or single tasks.
- VLA-RL applies scalable online RL to pretrained auto-regressive VLAs through a unified training framework.
- 4.5% improvement over the OpenVLA-7B imitation learning baseline was achieved on 40 challenging LIBERO robotic manipulation tasks.
2 Related Work
Prior work established general-purpose robotic foundation models and pretrained-model-assisted RL, but often retained offline-data requirements or simplified task settings. VLA-RL instead studies trajectory-level RL for large pretrained VLAs on general robotic manipulation tasks.
- Robotic Foundation Models: OpenVLA-7B uses a high-capacity vision-language model to generate robot actions as language tokens and generalize across tasks.
- Reinforcement Learning for Robotics Models: Prior pretrained-model RL studies often require complete offline datasets throughout fine-tuning.
- VLA-RL’s pipeline combines a transformer-based policy, a homogeneous value model, a frozen robotic process reward model, and vectorized environments.
- Reinforcement Learning for Robotics Models: Other robotics RL methods are frequently evaluated with simplified state representations, naive network structures, or narrow single-task paradigms.
- Reinforcement Learning for Robotics Models: VLA-RL explores trajectory-level RL from large robotics foundation models to support general tasks and environments.
3 VLA-RL
VLA-RL formulates auto-regressive VLA control as trajectory-level RL over multi-modal multi-turn conversations and combines policy optimization with value estimation and reward densification. Its system adds pseudo-label-based robotic process rewards and implementation techniques for scalable, stable training.
- Overview: The VLA-RL system trains a policy and value model while using a frozen robotic process reward model to densify sparse environmental rewards.
- General Robotic Manipulation as Multi-turn Conversation: The Markov decision process represents states as images paired with input token sequences and actions as generated output token sequences.
- General Robotic Manipulation as Multi-turn Conversation: The objective maximizes discounted trajectory rewards, and PPO provides stable policy optimization with clipped importance sampling and GAE advantages.
- Robotic Process Reward Model: The robotic process reward model predicts successful action-sequence likelihoods from trajectory states and actions using next-token prediction.
- Robotic Process Reward Model: Pseudo rewards are generated from automatically segmented successful trajectories, assigning positive labels to action sequences leading to keyframes.
- Robotic Process Reward Model: The final reward sums the environment’s sparse reward with the robotic process reward model’s prediction, accelerating learning while correlating with task success.
- System Implementation: Implementation improvements target learning efficiency and stability through adaptive curriculum selection, vectorized environments, batch decoding, and critic warmup.
4 Experiments
Experiments evaluate VLA-RL across LIBERO’s four task suites, examining benchmark performance, test-time scaling, training dynamics, efficiency, and design choices. VLA-RL improves OpenVLA-7B substantially, with reward densification, curriculum selection, sampling temperature, critic warmup, learning rate, and action coverage all affecting results.
- Major Experiments: VLA-RL improves OpenVLA-7B by 4.5% over SFT and 1.8% over DPO, matching π0-FAST after 48 GPU hours on LIBERO.The comparison covers LIBERO’s four task suites and uses average success rates and ranks over 500 evaluation episodes per method.
- Test-time Scaling: Success rates consistently improve across all four LIBERO task suites as test-time optimization increases.OpenVLA-7B is evaluated every 2500 training steps using average task success rates.
- Training Dynamics: Reward improvements correlate strongly with physical task success, while episode lengths decrease as VLA-RL learns more efficient action sequences.Reward plateaus correspond to curriculum transitions between task difficulties.
- Training Dynamics: GPU-balanced vectorized environments and vLLM acceleration reduce environment-evolution and rollout time, shifting the main efficiency bottleneck to training.Further training-efficiency improvements could therefore substantially reduce overall system time cost.
- Ablation Study: Reward densification raises success from 85.8% to 90.2%, while curriculum selection raises it from 88.0% to 90.2% over uniform random selection.The reward model supplies more frequent signals, and curriculum selection gradually increases task complexity based on agent performance.
- Ablation Study: Lowering sampling temperature from 1.5 to 1.0 reduces success from 90.2% to 85.8%, while critic warmup raises SR from 80.0% to 90.2%.The ablations associate reduced exploration with suboptimal policies and warmup with more accurate policy-gradient feedback.
5 Conclusions and Limitations
VLA-RL improves pretrained VLAs through online policy optimization and reports a 4.5% advantage over OpenVLA-7B on LIBERO. The method remains limited by pseudo-reward heuristics for more dexterous manipulation, including mature gripper closure during grasping.
- Limitations: Mature gripper closure during grasping is identified as a concrete manipulation challenge.
- Conclusions: 4.5% improvement over OpenVLA-7B is reported on LIBERO benchmarks through scalable online policy optimization.The framework formulates manipulation trajectories as multi-modal multi-turn conversations and uses a robotic process reward model for sparse rewards.
- Conclusions: Test-time optimization produces performance gains that the authors compare to emerging inference scaling laws in language models.The conclusion identifies this observation as an emerging principle for robotics.
- Limitations: Pseudo-reward extraction heuristics may not capture the nuances of more dexterous manipulation tasks, potentially making policy optimization inefficient.The authors identify this as a limitation and propose extending RL beyond auto-regressive VLAs and into real-world experience.