PaperlayerView exact revision on arXiv

Source-linked AI summary

RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models

Hongzhi Zang, Mingjie Wei, Si Xu, Yongji Wu, Zhen Guo, Yuanqing Wang, Hao Lin, Peihong Wang, Liangzhi Shi, Yuqing Xie, Zhexuan Xu, Zhihao Liu, Kang Chen, Wenhao Tang, Quanlu Zhang, Weinan Zhang, Chao Yu, Yu Wang

arXiv:2510.06710v3cs.RO

TL;DR

RL post-training can address VLA performance degradation under deployment mismatch, but current efforts are fragmented and system design can limit scalable training. RLinf-VLA unifies models, algorithms, and simulators while optimizing resource allocation; its RL-trained models report strong performance across embodied benchmarks.

  • Problem

    VLA deployment can involve distribution mismatch, while fragmented implementations and insufficient embodied-system optimization complicate scalable RL training.

    Source: §I · p. 2

  • Method

    RLinf-VLA provides a unified interface for diverse simulators, VLA architectures, and RL algorithms, with flexible GPU allocation and hybrid fine-grained pipelining.

    Source: §I · p. 2

  • Results

    RL-trained models achieve strong performance across LIBERO, ManiSkill, and RoboTwin, including 98.11% success across 130 LIBERO tasks.

    Source: §I · p. 2, §V · p. 6

  • Takeaways & Limitations

    RLinf-VLA distills practices for PPO and GRPO and provides an open-source, maintained foundation for standardizing and scaling embodied-intelligence research.

    Source: §I · p. 2, §VI · p. 9

Abstract

from arXiv · show

Recent studies have demonstrated the potential of reinforcement learning (RL) to improve the task performance of vision-language-action (VLA) models through interaction. However, current efforts remain fragmented, lacking a unified platform for fair comparison across architectures and algorithms, as well as an efficient system design for scalable training. Therefore, we present RLinf-VLA, a unified and efficient framework for scalable RL training of VLA models. RLinf-VLA standardizes the integration of diverse VLA architectures, RL algorithms, and heterogeneous simulators through a unified interface, enabling extensibility and reproducibility. To improve efficiency, the framework adopts a flexible resource allocation architecture for rendering, inference, and training in RL pipelines. In particular, RLinf-VLA introduces a hybrid fine-grained pipeline allocation strategy that achieves a 1.61$\times$-1.88$\times$ training speedup on ManiSkill. Using this framework, RL-trained models achieve strong performance across embodied benchmarks, including 98.11% success on 130 LIBERO tasks, 97.66% success on 25 ManiSkill tasks, and 84.63% average success across 6 RoboTwin tasks. In addition, RLinf-VLA distills a set of effective practices for RL-based VLA training. We envision RLinf-VLA as a foundational framework for efficient, unified, and reproducible research in embodied intelligence.

I. INTRODUCTION

The introduction motivates RL post-training for VLA models and presents RLinf-VLA as a unified framework for scalable, efficient training. It addresses fragmentation and resource bottlenecks, with strong reported results across embodied benchmarks.

I. INTRODUCTION

  • RL post-training addresses performance degradation under deployment mismatch, and studies report stronger out-of-distribution generalization than SFT, particularly in semantic alignment and execution robustness [25].RL optimizes cumulative task rewards through trial-and-error interaction and supports exploration beyond narrow expert demonstrations.Source: §I · p. 2
  • Fragmented architectures, algorithms, and evaluation protocols hinder fair comparison, while simulator competition with inference and learning resources limits scalable VLA RL training.Online interaction can also create GPU idle time and pipeline bubbles without a system design tailored to the execution pattern.Source: §I · p. 2
  • RLinf-VLA integrates simulators, VLA architectures, and RL algorithms through a unified interface, with configurable execution modes and a hybrid GPU allocation mode.The listed examples include ManiSkill, LIBERO, and RoboTwin; OpenVLA, OpenVLA-OFT, and π0; and PPO and GRPO.Source: §I · p. 2
  • 98.11% success across 130 LIBERO tasks, 97.66% across 25 ManiSkill tasks, and 84.63% average across 6 RoboTwin tasks are reported for RL-trained models.LIBERO and ManiSkill results use a single model each; RoboTwin uses six task-specific models.Source: §I · p. 2

A. Reinforcement Learning for VLA

B. Simulators

  • Simulator interfaces vary, making a unified interface important for switching among environments used for scalable, safe, and controllable RL training.The cited platforms include ManiSkill for physics-rich manipulation, LIBERO for instruction-conditioned reasoning, and RoboTwin for bimanual tasks with domain randomization [43] [24] [9].Source: B. Simulators · p. 3

A. Reinforcement Learning Formulation

  • The framework formulates vision-based RL as a POMDP, where the agent receives observations rather than directly accessing the underlying state.The described observations include multiview images, language instructions, and proprioceptive robot states; actions may be continuous commands or action chunks.Source: A. Reinforcement Learning Formulation · p. 3

B. Action Formulation

VLA policies may produce action chunks rather than single token-level actions, and their RL pipeline couples generation, simulation, and training. RLinf-VLA offers flexible GPU allocation and fine-grained pipelining to reduce rollout inefficiencies.

B. Action Formulation

  • VLA actions form a hierarchy of chunks, atomic actions, and tokens, with tokens typically corresponding to robot-action dimensions such as end-effector pose or joint angles.A chunk may contain multiple atomic actions, supporting smooth control.Source: B. Action Formulation · p. 3

C. Pipeline of Reinforcement Learning for VLA models

A. GPU Allocation Strategies

  • Simulator resource demands vary by type: CPU-parallelized environments are typically CPU-bound, while GPU-parallelized environments contend for GPU memory and compute.RLinf-VLA supports collocated, disaggregated, and hybrid allocation modes.Source: A. GPU Allocation Strategies · p. 3

1) The Collocated Mode:

  • In collocated mode, components share a GPU set; the modified strategy offloads Simulator and Generation only at rollout boundaries, but their iterative waits waste resources and limit scalability.The original strategy kept only one component on GPUs at a time, making repeated offload-onload operations costly during embodied interaction.Source: 1) The Collocated Mode · pp. 3–4, Figure 4 · p. 3

2) The Disaggregated Mode:

  • Disaggregated allocation assigns GPUs to components in contiguous blocks, unlike the more flexible hybrid mode.The supplied figure identifies the disaggregated mode; the prose comparison describes contiguous-block allocation in existing designs.Source: 3) The Hybrid Mode with Fine-grained… · p. 4, Figure 5 · p. 4

3) The Hybrid Mode with Fine-grained Pipelining:

  • Hybrid allocation lets each component select GPUs flexibly, typically placing Generation and Simulator on separate partitions while Training can use all GPUs.Simulator waits for generated actions can still leave GPUs idle in this configuration.Source: 3) The Hybrid Mode with Fine-grained… · p. 4, Figure 6 · p. 4
  • Fine-grained pipelining partitions one GPU's simulator instance into sub-simulators, with k = 2 illustrating staggered observation generation and action processing.The sequence overlaps work across simulator instances and Generation.Source: 3) The Hybrid Mode with Fine-grained… · p. 4, Figure 7 · p. 4
  • Concurrent Simulator and Generation execution reduces idle time while preserving correctness, avoiding collocated offload overhead and disaggregated GPU bubbles.The cited passage describes the hybrid allocation schematic with fine-grained pipelining at k = 2.Source: 3) The Hybrid Mode with Fine-grained… · p. 4, Figure 7 · p. 4

4) The Auto-Placement Algorithm:

RLinf-VLA combines automatic rollout resource placement with a unified interface for integrating simulators. The placement algorithm profiles component runtimes and selects GPU allocation and pipeline depth under stated assumptions.

4) The Auto-Placement Algorithm:

  • The automatic placement algorithm combines runtime profiling with allocation optimization to choose a rollout configuration.It profiles generation and simulation execution times, then seeks the GPU allocation and pipeline stage count that minimize rollout time.Source: 4) The Auto-Placement Algorithm · pp. 4–5
  • The placement objective assumes training uses all N GPUs when active and restricts rollout GPU splitting to Simulator and Generation, with at most two pipeline stages.The algorithm considers N GPUs and nS simulation environments when choosing the allocation and stage count.Source: 4) The Auto-Placement Algorithm · pp. 4–5

1) Multiple Simulators Support:

  • The unified interface standardizes simulator interactions through Gym-style APIs, action chunking, flexible episode termination, and shared observation and action formats.It supports ManiSkill and RoboTwin GPU-parallelized environments as well as LIBERO CPU-parallelized wrappers.Source: 1) Multiple Simulators Support · p. 5, B. A Unified Interface · p. 5

2) Multiple Models Support:

The framework accommodates multiple VLA models and RL algorithms, with configurable design choices for action granularity and PPO. It also identifies stable off-policy VLA training as an open problem and outside this work’s primary focus.

2) Multiple Models Support:

  • RLinf-VLA integrates OpenVLA and OpenVLA-OFT, supporting single-step and multi-step action chunking, respectively, with LoRA for fine-tuning.The framework uses a unified data-processing pipeline and standardized interaction interface to integrate models.Source: 2) Multiple Models Support · p. 5

C. Design Choices for Algorithms

3) Multiple Algorithms Support:

  • Stable application of off-policy algorithms to VLA models remains an open problem, so off-policy methods are not the work’s primary focus.RLinf-VLA nevertheless supports DSRL, a SAC variant, for fine-tuning π-series models.Source: 3) Multiple Algorithms Support · p. 5

1) Multi-Granularity Calculation Support:

  • Advantage estimation can treat an action chunk as one macro-action with shared advantage or assign separate rewards and advantages to its atomic actions.Log-probability computation is also supported at multiple granularities.Source: 1) Multi-Granularity Calculation Support · p. 5

2) Design Choices for PPO:

  • PPO rollouts support fixed-length episodes with simultaneous resets or partial resets as each sub-environment terminates.Figure 8 illustrates the two reset modes; valid-action masking is described separately for GRPO.Source: 2) Design Choices for PPO · p. 5
  • PPO shares actor and critic parameters and adds a lightweight value head; action-level values consistently perform better than one value per chunk.Chunk-level estimation uses one scalar value, whereas action-level estimation produces a value for each atomic action in the chunk.Source: 2) Design Choices for PPO · pp. 5–6

3) Design Choices for GRPO:

RLinf-VLA combines GRPO design choices and flexible training support to evaluate VLA policies across simulated benchmarks and a preliminary real-world task. Experiments report benchmark gains, training-efficiency improvements, and practical guidance for RL-based VLA training.

3) Design Choices for GRPO:

  • GRPO design choices include masking post-completion actions, normalizing loss by valid trajectory length, and filtering groups with uniformly successful or failed trajectories.The filter requires mixed outcomes to compute non-zero advantages and is reported to accelerate convergence and improve performance.Source: 3) Design Choices for GRPO · p. 6, §V · p. 6

V. EXPERIMENT RESULTS

  • RLinf-VLA improves over base models by approximately 20–85% across three benchmarks; its unified 130-task LIBERO model averages 98.11%, and RoboTwin averages 84.63%.RoboTwin results consistently outperform SimpleVLA-RL across the six evaluated tasks.Source: §V · p. 6, A. Policy Performance · p. 7
  • Across GPU-parallelized and CPU-parallelized simulators, RLinf-VLA’s optimal configuration improves training throughput, achieving speedups of up to 1.88×.Source: §V · p. 6
  • In a preliminary real-world deployment study, RL post-training of policies improved real-world performance over SFT-only baselines in this setup, suggesting the potential practical value of simulation-based RL for embodied learning.Source: §V · p. 6
  • Evaluations on π-series models and off-policy methods are reported as demonstrating framework extensibility across diverse training settings.The cited passage refers to results in Section X-A without specifying their individual outcomes.Source: §V · p. 6

A. Policy Performance

  • On ManiSkill, RL improves success rates by 45%–70% over baseline, with PPO more stable and consistently stronger than GRPO; LIBERO success rises from about 73% to 98%.RoboTwin training remains stable and reaches high performance even from initial success rates as low as 3%.Source: A. Policy Performance · p. 7

B. System Efficiency

RLinf-VLA compares GPU-allocation modes across simulators and models, finding that the best configuration depends on the execution regime. Hybrid pipelining and system-level optimizations improve throughput, while flexible allocation supports efficiency across differing workloads.

B. System Efficiency

  • The optimal allocation varies by simulator and model: CPU-parallelized LIBERO can bottleneck hybrid pipelining, while hybrid mode outperforms collocated mode as GPU count increases on GPU-parallelized ManiSkill.OpenVLA-OFT action chunks shift the generation-to-execution ratio to about 1:15 in the described settings.Source: B. System Efficiency · pp. 7–8
  • For OpenVLA on ManiSkill, Hybrid (pipe=2) on 8 GPUs achieves up to 1.88× speedup over the disaggregated baseline.Across larger cluster sizes, fine-grained pipelining maintains a 1.61×–1.69× advantage over disaggregated mode.Source: B. System Efficiency · p. 8, Figure 10 · p. 8
  • For OpenVLA-OFT on LIBERO and RoboTwin, collocated RLinf-VLA achieves 1.34×–2.27× speedup over SimpleVLA-RL.Gains are attributed to vectorized rollout environments and training optimizations including adaptive communication and removal of redundant log-probability recomputation.Source: B. System Efficiency · p. 8

C. Ablation Study

Ablations identify training choices whose benefits depend on the algorithm, task, and model. A preliminary deployment experiment compares simulation-RL post-training with SFT, and the conclusion summarizes RLinf-VLA as a unified, efficient framework for VLA training.

1) Tips for PPO:

  • Action-level value estimation consistently yields higher success rates and lower value loss than chunk-level estimation for PPO with action chunks.The comparison is reported on OpenVLA-OFT ManiSkill and shows the same performance divergence on LIBERO-Goal.Source: 1) Tips for PPO · p. 8
  • Partial Reset consistently achieves higher success rates than Fixed Episode Length for a given number of training steps, across OpenVLA and OpenVLA-OFT.The comparison uses a “success once” objective and reports improved sample efficiency.Source: 1) Tips for PPO · pp. 8–9, Figure 12 · p. 8

2) Tips for GRPO:

  • On LIBERO-Goal, valid action masking improves performance, and combining it with trajectory-length normalization adds further improvement; neither technique shows clear benefit on ManiSkill.The mask excludes post-success actions; normalization aims to reduce bias when episode lengths vary.Source: 2) Tips for GRPO · p. 9, Figure 13 · p. 9, Figure 14 · p. 9
  • Success-rate filtering alleviates a training collapse around step 400 in OpenVLA ManiSkill, but its benefit is less pronounced in OpenVLA-OFT ManiSkill and LIBERO-Goal.The filter discards groups whose trajectories all have identical cumulative rewards.Source: 2) Tips for GRPO · p. 9

D. Real-World Experiment

  • In the preliminary drawer-closing experiment, both policies succeed in 8/10 real-world trials, but the RL-trained policy completes the task more efficiently.The SFT policy sometimes imitates suboptimal teleoperation behavior or predicts near-zero actions; the RL policy exhibits more direct, fluent pushing.Source: D. Real-World Experiment · p. 9

VI. CONCLUSION

  • RLinf-VLA integrates multiple simulators, algorithms, and VLA architectures with flexible execution modes and system-level optimizations for efficient training.The paper reports high simulated-task performance and offers actionable PPO and GRPO practices; it also describes the platform as open-source and maintained.Source: §VI · p. 9

Appendix for RLinf-VLA

The appendix defines return, value, action-value, and advantage quantities used in RL, then introduces PPO and its optimization approach. It describes PPO’s stability rationale and the role of generalized advantage estimation.

A. Definitions of RL

  • Return is the γ-discounted cumulative reward from timestep t; the value function is expected return from state s, and the action-value function is expected return after taking action a in s.The advantage function quantifies how much better action a is than the expected value of state s.Source: A. Definitions of RL · p. 13

B. Introduction of PPO

  • PPO constrains policy updates within a trust region using a clipped surrogate objective to support training stability and sample efficiency.The paper uses PPO as a foundation for its framework because of its robustness and simplicity.Source: B. Introduction of PPO · p. 13
  • Generalized Advantage Estimation estimates PPO advantages from rewards and value estimates; γ is the discount factor, λ controls the bias–variance trade-off, and T is the episode horizon.The PPO objective uses the rollout policy πθold and clipping parameter ϵ.Source: B. Introduction of PPO · p. 13

C. Introduction of GRPO

This section presents GRPO as a simpler alternative to PPO and describes interface and GPU-allocation options for supporting varied RL training setups.

C. Introduction of GRPO

  • GRPO removes PPO’s explicit value model and uses group-based trajectory comparisons, reducing model complexity and avoiding potential value-estimation inaccuracies.The section introduces a probability ratio and defines advantages using trajectory rewards relative to the group’s mean and standard deviation.Source: C. Introduction of GRPO · p. 14

A. The Unified Interface

  • Gym-style reset and step APIs support auto-reset, ignored terminations, and chunked actions with immediate or deferred resets.Utility functions include rollout visualization and fixed initial-state IDs for GRPO groups.Source: A. The Unified Interface · p. 14

B. GPU Allocation Modes

  • Users configure GPU allocation by assigning component GPU IDs, with independent offloading controls and a pipeline-stage setting for fine-grained pipelining.A pipeline-stage count above one splits each simulator instance into multiple stages; one disables pipelining.Source: B. GPU Allocation Modes · p. 14

C. Details for Design Choices

The section specifies supported action and probability granularities, trajectory-length loss normalization, evaluation conventions, and the base models used in the experiments.

1) Details for Multi-Granularity Support:

2) Details for Loss Normalization by Trajectory Length:

  • Under Valid Action Mask, each valid timestep is scaled by the trajectory’s valid-timestep count, preventing longer trajectories from dominating the gradient.The stated objective is balanced learning across trajectories of varying lengths.Source: 2) Details for Loss Normalization by… · p. 15

A. Details for Performance Experiments

  • Evaluation uses the “success_once” criterion, and curves are smoothed with a Gaussian filter of σ = 1 unless otherwise specified.An episode counts as successful if it reaches the success state at least once.Source: A. Details for Performance Experiments · p. 15

1) Base Models for Training:

  • The experiments use different base-model sources: RL4VLA’s OpenVLA checkpoint for ManiSkill, locally LoRA-fine-tuned OpenVLA-OFT for LIBERO, and SimpleVLA-RL’s checkpoint for RoboTwin.The LIBERO base is fine-tuned using demonstrations from all five task suites; RoboTwin reuses SimpleVLA-RL’s train and evaluation seeds.Source: 1) Base Models for Training · p. 15

2) Hyperparameter Settings for Training:

This section describes evaluation settings and baselines for measuring training throughput across simulators and GPU allocation modes.

2) Hyperparameter Settings for Training:

  • RL-trained models are evaluated with sampling enabled over three runs, with mean and standard deviation reported; supervised fine-tuned models use deterministic evaluation.Only the LIBERO setting reports standard deviation.Source: 3) Metric for Evaluation · p. 15

1) Hyperparameters for Efficiency Experiments:

3) Tasks for Efficiency:

  • Efficiency tests use a modified PutCarrotOnPlateInScene-v2 task on 8, 16, or 32 H100 GPUs, with 256 environments running 80 steps; LIBERO-Long and RoboTwin place_empty_cup scale environment counts and steps with GPU count.For the latter two tasks, environment counts are 64, 128, and 256 on 8-, 16-, and 32-GPU setups, respectively.Source: 3) Tasks for Efficiency · pp. 16–17

4) Baselines for Efficiency Experiments:

  • ManiSkill uses naive disaggregated allocation as its baseline, while LIBERO and RoboTwin compare against SimpleVLA-RL’s colocated mode and omit disaggregated results.For LIBERO and RoboTwin, the authors state hybrid one-stage allocation outperforms disaggregated allocation even accounting for offload–onload overhead.Source: 4) Baselines for Efficiency Experiments · p. 17

A. Extensibility of RLinf-VLA

The section describes RLinf-VLA’s model and algorithm extensibility and reports findings from ManiSkill evaluations and training ablations.

1) Model Extensibility:

  • The framework supports π-series flow-matching models and MLPs, with GRPO and PPO demonstrated for π-series models.The cited figures show results on π-series models and MLPs.Source: 1) Model Extensibility · p. 17

2) Algorithm Extensibility:

  • RLinf-VLA supports off-policy SAC in multiple settings, although stable SAC training for VLA models remains an open problem.The paper demonstrates DSRL on π0 and SAC on an MLP.Source: 2) Algorithm Extensibility · p. 17

B. Evaluation of OOD for ManiSkill

C. Ablation Studies

  • For chunk size 8, γchunk = 0.904 achieves performance comparable to γaction = 0.99, whereas γchunk = 0.99 performs substantially worse.The authors attribute the comparable result to properly aligning the discount factors for chunk-level training.Source: Figure 18 · p. 18, C. Ablation Studies · p. 18
  • The success-rate filter discards GRPO groups whose trajectories have identical cumulative rewards; it largely alleviates an OpenVLA ManiSkill collapse near step 400, but helps less in two other settings.The weaker effects are reported for OpenVLA-OFT ManiSkill and OpenVLA-OFT LIBERO-Goal.Source: Figure 19 · p. 18, C. Ablation Studies · p. 18
  • Larger rollout batches consistently yield higher success rates when evaluated by training epochs.The section notes this effect is especially relevant to on-policy PPO and GRPO.Source: C. Ablation Studies · p. 18, Figure 20 · p. 19
  • LoRA produces broadly similar training curves, but can require different hyperparameters: at learning rate 1 × 10−4, non-LoRA GRPO collapses while LoRA improves normally; at 1 × 10−5, non-LoRA improves stably.The authors conclude LoRA configurations may require separate hyperparameter tuning.Source: Figure 21 · p. 19, C. Ablation Studies · p. 19
Loading 2510.06710v3…