SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning

Haozhan Li; Yuxin Zuo; Jiale Yu; Yuhao Zhang; Zhaohui Yang; Kaiyan Zhang; Xuekai Zhu; Yuchen Zhang; Tianxing Chen; Ganqu Cui; Dehui Wang; Dingxiang Luo; Yuchen Fan; Youbang Sun; Jia Zeng; Jiangmiao Pang; Shanghang Zhang; Yu Wang; Yao Mu; Bowen Zhou; Ning Ding

Source-linked AI summary

SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning

Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, Dehui Wang, Dingxiang Luo, Yuchen Fan, Youbang Sun, Jia Zeng, Jiangmiao Pang, Shanghang Zhang, Yu Wang, Yao Mu, Bowen Zhou, Ning Ding

arXiv:2509.09674v1Sep 11, 2025cs.ROcs.AIcs.CLcs.LG

TL;DR

VLA training is limited by the scarcity and cost of human-operated trajectories for SFT and by weak generalization under distribution shift. SimpleVLA-RL extends veRL with VLA-specific online RL and scalable sampling infrastructure, achieving broad benchmark gains, improved data efficiency and generalization, and previously unseen policy patterns.

Problem
Scaling SFT for VLA models requires scarce and prohibitively expensive human-operated trajectories, while generalization remains limited under distribution shift.
Method
SimpleVLA-RL extends veRL with VLA-specific trajectory sampling and loss computation, parallel multi-environment rendering, and integrated training–inference–rendering for online RL.
Results
SimpleVLA-RL consistently improves performance by 10–15%, surpasses state-of-the-art baselines across LIBERO and RoboTwin, and improves data efficiency, generalization, and sim-to-real transfer.
Takeaways & Limitations
The results support RL as a way to reduce dependence on large-scale SFT data while enhancing VLA generalization and enabling effective real-world transfer.
Takeaways & Limitations
RL provides negligible gains when initial task competence is too low, indicating that effective online learning requires a minimal level of task ability.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have recently emerged as a powerful paradigm for robotic manipulation. Despite substantial progress enabled by large-scale pretraining and supervised fine-tuning (SFT), these models face two fundamental challenges: (i) the scarcity and high cost of large-scale human-operated robotic trajectories required for SFT scaling, and (ii) limited generalization to tasks involving distribution shift. Recent breakthroughs in Large Reasoning Models (LRMs) demonstrate that reinforcement learning (RL) can dramatically enhance step-by-step reasoning capabilities, raising a natural question: Can RL similarly improve the long-horizon step-by-step action planning of VLA? In this work, we introduce SimpleVLA-RL, an efficient RL framework tailored for VLA models. Building upon veRL, we introduce VLA-specific trajectory sampling, scalable parallelization, multi-environment rendering, and optimized loss computation. When applied to OpenVLA-OFT, SimpleVLA-RL achieves SoTA performance on LIBERO and even outperforms $π_0$ on RoboTwin 1.0\&2.0 with the exploration-enhancing strategies we introduce. SimpleVLA-RL not only reduces dependence on large-scale data and enables robust generalization, but also remarkably surpasses SFT in real-world tasks. Moreover, we identify a novel phenomenon ``pushcut'' during RL training, wherein the policy discovers previously unseen patterns beyond those seen in the previous training process. Github: https://github.com/PRIME-RL/SimpleVLA-RL

1. Introduction

SimpleVLA-RL applies online reinforcement learning to VLA models to address the scarcity and cost of human-operated trajectories and limited generalization in SFT-based training. Its framework enables scalable VLA-specific RL, improves benchmark performance and data efficiency, and reveals policies discovering previously unseen action patterns.

Motivation: Human-operated trajectories for scaling SFT are scarce and prohibitively expensive, while VLA generalization remains limited under distribution shift.These challenges constrain the scale and diversity of supervised training data and motivate alternative training approaches.
Motivation: RL’s success in Large Reasoning Models motivates testing whether it can strengthen VLA models’ step-by-step action generation.The paper frames this as a direct analogue between chain-of-thought reasoning and long-horizon action planning.
Approach: SimpleVLA-RL extends veRL with VLA-specific interactive trajectory sampling, loss computation, parallel multi-environment rendering, and integrated training–inference–rendering.These components support scalable and sample-efficient online RL for VLA models.
Findings: RL training reveals “pushcut,” where the policy discovers previously unseen patterns beyond those encountered in supervised data.This observation suggests that online RL can expand the policy’s learned solution patterns.
Results: 10–15% consistent performance improvements arise from exploration-enhancing strategies, while SimpleVLA-RL surpasses multiple state-of-the-art baselines on LIBERO and RoboTwin 1.0 and 2.0.The reported gains include benchmark-level improvements over strong VLA baselines.
Results: With one demonstration per task, RL raises LIBERO-Long success from 17.1% to 91.7% and outperforms SFT in spatial, object, and task generalization.The framework also transfers effectively from simulation to real-world deployment without real robot data.

2. Preliminaries

The preliminaries formalize reinforcement learning for LLMs and VLAs, contrasting token-based generation with interactive robotic control. They also introduce GRPO’s group-relative advantage computation and constrained policy updates.

2.1. RL Formulation for LLMs: LLM states combine the initial prompt with previously generated tokens, while actions select the next vocabulary token from a sampled policy distribution.The policy’s logits are converted to token probabilities using a temperature parameter.
2.1. RL Formulation for LLMs: LLM environments provide sequence-level rewards after completion, including binary rule-based rewards or continuous rewards from learned models or human feedback.Rollouts are sampled autoregressively without intermediate environmental feedback until termination.
2.2. RL Formulation for VLAs: VLA states combine visual observations, proprioception, and language instructions, while actions are robot control commands such as end-effector deltas or joint-angle targets.Actions may be produced through a diffusion action expert or a discrete action tokenizer.
2.2. RL Formulation for VLAs: VLA rollouts interact with the environment iteratively: the policy generates action chunks, the robot executes them, and updated states guide subsequent chunks.The resulting trajectory continues until task completion or the maximum episode length.
2.3. Group Relative Policy Optimization: GRPO removes the value function by normalizing advantages within groups of trajectories, then uses PPO-style clipping and KL regularization to constrain updates.The clipping parameter limits policy ratios, while KL regularization controls deviation from a reference policy.

3. SimpleVLA-RL

SimpleVLA-RL extends outcome-reward online RL to VLA manipulation by generating diverse, closed-loop trajectories and using exploration-oriented training modifications. Its framework combines interactive parallel rollouts, binary task-completion rewards, dynamic sampling, higher rollout temperature, and a wider clipping range.

Interactive VLA Rollout: SimpleVLA-RL generates multiple VLA trajectories through continuous environment interaction, updating visual observations and robot state after each action.Unlike text-only rollout, VLA sampling requires closed-loop feedback because actions alter the environment.
Interactive VLA Rollout: The rollout implementation replaces text generation with parallel environment initialization, iterative action sampling, and synchronous trajectory collection.The adopted pseudo-code uses an environment process pool and records state-action pairs across timesteps.
Outcome Reward Modeling: SimpleVLA-RL assigns each trajectory a binary reward: 1 for task success and 0 for failure.All tokens in a successful trajectory receive reward 1, while tokens in an unsuccessful trajectory receive reward 0.
Exploration Enhancements: Dynamic sampling excludes groups with uniformly successful or failed trajectories, retaining mixed-outcome groups for nonzero advantages and stable gradients.This addresses GRPO’s vanishing-gradient problem when group-relative rewards are identical.
Exploration Enhancements: The framework promotes exploration by widening the upper clipping threshold from 1.2 to 1.28 and increasing rollout temperature from 1.0 to 1.6.The higher temperature produces more diverse trajectories, while higher clipping permits larger probability increases for low-probability tokens.
Training Objective: The framework adopts GRPO-based online RL without KL regularization, removing the reference model and reducing memory use while potentially allowing broader exploration.The objective follows DAPO in removing the KL penalty from training.

4. Experiments

SimpleVLA-RL is evaluated on LIBERO, RoboTwin1.0, and RoboTwin2.0 using OpenVLA-OFT after SFT, with RL-trained policies compared against SFT-only and recent VLA baselines. Across these benchmarks, RL improves success rates, including long-horizon and dual-arm tasks.

Experimental Setup: SimpleVLA-RL is evaluated on LIBERO, RoboTwin1.0, and RoboTwin2.0, including real-world RoboTwin2.0 experiments.LIBERO covers five suites, while RoboTwin benchmarks target dual-arm manipulation with increasing task and domain diversity.
Evaluation: The study evaluates average Success Rate across 50 held-out test scenarios for each task.The benchmark suite includes long-horizon and extra-long-horizon task categories.
Experimental Setup: Experiments apply SFT followed by SimpleVLA-RL to OpenVLA-OFT, whereas baseline models use SFT alone.LIBERO uses 500 demonstrations per suite and 500 simulation scenarios; RoboTwin1.0 uses 50 demonstrations and 100 scenarios per task; RoboTwin2.0 uses 1,000 of each.
LIBERO: LIBERO average success rate rises from 91% with SFT-tuned OpenVLA-OFT to 99% with SimpleVLA-RL, achieving SoTA performance.On LIBERO-Long, success reaches 98.5%, compared with 86.5% for the baseline and 85.2% for π0.
RoboTwin1.0: RoboTwin1.0 performance increases from 39.8% to 70.4%, a 30.6% gain over the fine-tuned OpenVLA-OFT baseline.The result covers four dual-arm tasks.

5. Analysis

SimpleVLA-RL addresses data scarcity, generalization, and real-world transfer through online RL applied after SFT. It improves unseen-task performance and sim-to-real success while reducing reliance on demonstration trajectories.

Data: SimpleVLA-RL reduces demonstration-data requirements by applying RL after either one-trajectory or full-trajectory SFT.The scarce-data setting uses one demonstration per task, while the comparison uses 500 demonstrations per task.
Generalization: SimpleVLA-RL consistently improves unseen-task performance, whereas SFT often overfits despite exceeding 90% success on training tasks.SFT can exhibit catastrophic forgetting, while RL improves unseen tasks across spatial, object, and goal settings.
Generalization: On LIBERO-Object, RL gains reach 36.5% on Unseen Task 2 and 16.4% on Unseen Task 3.On LIBERO-Spatial, performance rises from 43.3% to 71.8% on Unseen Task 1, with additional gains of 7.1% and 13.3%.
Real-world transfer: 38.5% average real-world success follows SimpleVLA-RL, up from 17.5% for SFT and above RDT’s 23.5%.The models are trained entirely with simulation data and evaluated on unseen real-world backgrounds across four tasks.

6. Discussions

The discussion highlights pushcut as an RL-emergent shortcut and identifies initial task competence as a boundary condition for successful outcome-reward training. RL discovers novel strategies, but cannot improve a model that never produces successful trajectories.

6.1. “Pushcut”: Emergence of New Patterns through RL: “Pushcut” names RL-emergent pushing strategies that replace demonstration-based grasp–move–place routines.In RoboTwin2.0, the model pushes cans or objects directly into position instead of grasping them.
6.1. “Pushcut”: Emergence of New Patterns through RL: Outcome-level rewards permit equivalent successful strategies, allowing exploration beyond procedural constraints imposed by process supervision.Positive rewards reinforce effective behaviors while less efficient behaviors are eliminated.
6.2. Failure Modes of SimpleVLA-RL: RL fails completely when the base model has no initial task ability, remaining at 0% because sampling produces no successful trajectories.The limitation is specific to outcome-reward training without process rewards in this setting.
6.2. Failure Modes of SimpleVLA-RL: The 1000-trajectory SFT model improves average success from 28.2% to 50.4%, compared with 7.3% to 25.4% for the 100-trajectory model.Stronger initial capabilities provide more effective starting points for exploration.
6.2. Failure Modes of SimpleVLA-RL: RL effectiveness has a threshold: very low initial success rates yield only marginal improvements.For pick dual bottles, the 100-trajectory model rises from 1.2% to 4.3%, while the 1000-trajectory model rises from 29.7% to 68.3%.

7. Related Works

Related work spans RL for language-model reasoning, exploration enhancement, VLA architectures, and emerging RL methods for robotic manipulation. SimpleVLA-RL is positioned among efforts using preference optimization, alternating SFT and RL, dense rewards, and online RL.

RL for language models: RL for LLMs has induced complex reasoning behaviors such as self-verification and iterative optimization.
Exploration enhancement: DAPO’s Clip-Higher increases the upper PPO clipping bound to encourage low-likelihood tokens and exploration.The example sets 𝜖low = 0.2 and 𝜖high = 0.28.
VLA models: VLA models unify VLM or LLM backbones with action modules through end-to-end training for multimodal understanding and motor control.
VLA RL methods: Prior VLA RL methods use human preferences, alternating RL and SFT, dense rewards, or iterative SFT–RL frameworks.

8. Conclusion

SimpleVLA-RL extends online RL to VLA models with scalable sampling and training–inference–rendering, improving data efficiency, generalization, and sim-to-real transfer. The results support RL as a way to reduce SFT data demands and enhance VLA adaptability.

Conclusion: SimpleVLA-RL extends veRL with VLA-specific trajectory sampling and parallelized training–inference–rendering for scalable, sample-efficient online RL.
Conclusion: The framework delivers consistent improvements in data efficiency, generalization, and sim-to-real transfer across LIBERO and RoboTwin benchmarks.