PaperlayerView exact revision on arXiv

Source-linked AI summary

$\pi_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models

Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Xiang Li, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, Chao Yu

arXiv:2510.25889v3cs.LG

TL;DR

Flow-based VLAs pose a challenge for online RL because flow matching makes executed-action likelihoods intractable. πRL introduces Flow-Noise and Flow-SDE formulations for fine-tuning these models, and experiments report gains over SFT across benchmarks, including out-of-distribution evaluations.

  • Problem

    Applying online RL to flow-based VLAs is challenging because flow matching makes executed-action log-likelihoods intractable, while deterministic ODE sampling also precludes exploration.

    Source: §4 · p. 3

  • Method

    πRL fine-tunes flow-based VLAs using Flow-Noise for exact likelihood estimation and Flow-SDE to couple denoising with policy-environment interaction.

    Source: §1 · p. 2

  • Results

    πRL consistently yields substantial gains over SFT across evaluated π0 and π0.5 benchmarks, with out-of-distribution evaluations indicating policy enhancement rather than narrow overfitting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework enables online RL fine-tuning of flow-based π0 and π0.5 models on the evaluated benchmarks.

    Source: §6 · p. 10

    Because online RL has low sample efficiency, the framework currently relies on sim-to-real deployment rather than real-world RL training.

    Source: §6 · p. 10

Abstract

from arXiv · show

Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying RL to large-scale flow-based VLAs (\eg, $\pi_0$, $\pi_{0.5}$) remains challenging due to intractable action log-likelihoods raised from flow matching. We address this challenge with $\pi_{\texttt{RL}}$, featuring two technical approaches: (1) \textbf{Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) \textbf{Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate $\pi_{\texttt{RL}}$ across various benchmarks, with experiments demonstrating that RL yields significant performance improvements in both in-distribution and out-of-distribution settings.

1. Introduction

SFT for VLAs requires costly expert demonstrations, while prior VLA-RL work largely targets autoregressive models and does not transfer directly to flow-based VLAs. πRL addresses the flow-matching likelihood challenge with two formulations and improves performance over SFT across benchmarks, including out-of-distribution evaluations.

1. Introduction

  • SFT requires laborious, costly curation of high-quality expert trajectories, and SFT-trained models tend to overfit to demonstrations and remain constrained by their quality.Source: §1 · p. 2
  • Prior VLA-RL advances largely target autoregressive VLAs, while flow-based VLAs use iterative flow-matching refinement and require a way to characterize executed-action likelihoods.The introduction identifies this likelihood problem as the fundamental obstacle to applying prior VLA-RL algorithms to flow-based models.Source: §1 · p. 2
  • πRL combines Flow-Noise, which models denoising as a discrete-time MDP for exact log-likelihood estimation, with Flow-SDE, which couples denoising and environment interaction through an ODE-to-SDE conversion.The two-layer MDP supports exploration, and PPO is used for further optimization.Source: §1 · p. 2
  • Across π0 and π0.5 benchmarks, πRL consistently improves over SFT, and out-of-distribution evaluations indicate policy enhancement rather than narrow target-environment overfitting.The introduction reports substantial gains across evaluated benchmarks but gives no numeric results here.Source: §1 · p. 2
  • The paper reports comprehensive ablation studies intended to provide empirical insights for future RL research on flow-based VLAs.Source: §1 · p. 2
  • The authors release code and models to support reproducibility and further research.Source: §1 · p. 2

2. Related Work

Related work covers VLA architectures, online RL fine-tuning for VLAs, and RL methods for flow models. It contrasts the broader VLA-RL literature with techniques that make flow-model likelihoods tractable or enable stochastic exploration.

2.1. Vision-Language-Action Models

  • VLA architectures such as Octo, RT, OpenVLA, π0, and GR00T integrate multimodal inputs for perception, reasoning, and control.Source: §2.1 · p. 2

2.2. Online RL Fine-tuning for VLA Models

  • Prior online VLA-RL work includes methods for long-horizon planning under data scarcity, comparisons of PPO, GRPO, and DPO with sparse rewards, and a framework for scalable RL training.The cited examples are SimpleVLA-RL, RL4VLA, and RLinf-VLA, respectively.Source: §2.2 · pp. 2–3

2.3. RL Fine-tuning for Flow Models

  • Flow-model RL methods include ODE-to-SDE conversion for stochastic exploration, learnable noise for tractable likelihoods, hybrid ODE-SDE rollouts for faster training, and advantage-weighted conditional-flow optimization.The passage associates these approaches with Flow-GRPO, ReinFlow, MixGRPO and TempFlow-GRPO, and FPO, respectively.Source: §2.3 · p. 3

3. Preliminary

The preliminaries formulate robot control as an MDP and describe how a flow-based VLA maps multimodal observations to action sequences. The model learns a conditional vector field and iteratively refines noise into actions.

3.1. Problem Formulation

  • The control problem is an MDP whose state is the robot observation, whose flow policy selects actions, and whose environment supplies transitions and rewards.The stated objective is to maximize expected discounted return over a finite horizon.Source: §3.1 · p. 3
  • The policy-gradient formulation uses sampled trajectories, with the advantage measuring an action value relative to the state value as a lower-variance policy-update signal.Source: §3.1 · p. 3

3.2. Flow-based Vision-Language-Action Model

  • A flow-based VLA maps RGB images, language tokens, and proprioception to a sequence of future actions by learning a conditional vector field from noise toward target actions.At inference, it samples Gaussian noise and iteratively refines it by integrating the learned vector field with forward Euler steps.Source: §3.2 · p. 3

4. Methodology

The methodology makes flow-based VLAs amenable to RL by addressing intractable likelihoods and deterministic sampling through two complementary formulations. Flow-Noise makes denoising transitions tractable for policy optimization, while Flow-SDE couples denoising with environment interaction.

4. Methodology

  • Flow-based VLA likelihood estimation and deterministic ODE sampling complicate RL, motivating Flow-Noise and Flow-SDE as routes to RL-compatible policies.The methodology targets exact likelihood estimation and stochastic exploration.Source: §4 · p. 3

4.1.1. STOCHASTICITY INJECTION

  • A learned noise network dynamically sets injected-noise magnitude; it is trained jointly with the velocity network and discarded after fine-tuning, leaving deterministic inference.Each denoising transition is Gaussian, with its mean from the forward Euler update and variance controlled by the noise network.Source: §4.1.1 · p. 4

4.1.2. LOG-LIKELIHOOD ESTIMATION

  • Flow-Noise models denoising as a discrete-time MDP, enabling policy optimization from the joint likelihood of the denoising sequence rather than the intractable final-action likelihood.The inference trajectory is discretized into K uniform steps, and its sequence likelihood is tractable.Source: §4.1 · p. 3, §4.1.2 · p. 4

4.2.1. STOCHASTICITY INJECTION

  • Flow-SDE converts ODE denoising to an equivalent SDE for stochastic exploration, preserving the generated actions' marginal probability density.Its discretized denoising transitions are isotropic Gaussians with specified means and variances.Source: §4.2.1 · pp. 4–5, §4.2 · p. 4

4.2.2. MDP FORMULATION

  • Flow-SDE embeds the denoising MDP inside the outer environment MDP, coupling action refinement with policy-environment interaction in a two-layer formulation.The formulation transforms action-likelihood estimation into Gaussian transition-likelihood estimation.Source: Figure 3 · p. 4, §4.2.2 · p. 5

4.2.3. HYBRID ODE-SDE SAMPLING

  • Hybrid ODE-SDE rollouts apply stochastic exploration at a randomly sampled denoising time and deterministic ODE updates at the remaining steps, shortening the MDP horizon.The authors describe this strategy as theoretically consistent with the two-layer framework.Source: §4.2.3 · p. 5

4.3.1. ALGORITHM

  • PPO optimizes the flow policy using chunk-level actions, GAE advantage estimates, and a clipped probability ratio that constrains updates for training stability.The action sequence is treated as one macro-step; the clipping interval is [1−ϵ, 1+ϵ].Source: §4.3.1 · pp. 5–6

4.3.2. CRITIC DESIGN

  • The shared actor-critic design adapts critic placement to each VLA: π0.5 uses VLM output, while π0 estimates value by averaging across the denoising trajectory.For π0, the action expert takes both the noisy action and state as inputs, complicating value prediction.Source: §4.3.2 · p. 6

5. Experimental Results

Experiments evaluate πRL on π0 and π0.5 across four robot-manipulation benchmarks. They report in-distribution gains over SFT, test out-of-distribution transfer, and examine critic, MDP, noise-injection, and hyperparameter choices.

5.2.1. IN-DISTRIBUTION RL TRAINING

  • Across all evaluated benchmarks, πRL yields substantial gains over SFT baselines: π0 achieves a maximum average improvement of +29.2%, while π0.5 demonstrates a +31.0% increase in average success rate.Source: §5.2.1 · p. 6

5.2.2. OUT-OF-DISTRIBUTION RL EVALUATION

  • ID gains transfer to environmental shifts in ManiSkill and CALVIN, but MetaWorld task-shift performance fluctuates without significant improvement.The authors interpret this pattern as localized mainly to action-level refinement rather than broader cross-task generalization.Source: Figure 5 · p. 7, §5.2.2 · p. 7

5.3.1. CRITIC DESIGN

  • The critic after the VLM performs slightly better than the critic after the action expert, while a four-layer MLP improves value approximation, performance, and training stability.The VLM-attached critic does not receive proprioceptive state; the action-expert critic must handle coupled state and noisy-action inputs.Source: §5.3.1 · pp. 7–8, Figure 6 · p. 8

5.3.2. FLOW POLICY MDP

  • Across MDP formulations, final success rates are consistent; the hybrid two-layer approach is 2× faster than the standard approach, while the one-layer formulation converges fastest.The one-layer MDP has no substantial wall-clock advantage over the standard two-layer model because it recalculates full denoising trajectories.Source: Figure 7 · p. 8, §5.3.2 · p. 8

5.3.3. STOCHASTICITY INJECTION

  • Fixed and learnable noise injection show similar initial training performance and both achieve efficient converged performance under the controlled comparison.The comparison sets fixed noise to 0.5 and learnable-noise log-variance bounds to 0.08 and 0.16, with the learnable-noise entropy bonus set to zero.Source: Figure 8 · p. 8, §5.3.3 · p. 8

5.4. Hyper-Parameters

  • Increasing noise leaves deterministic SFT evaluation unchanged but degrades its stochastic training performance; during RL, lower noise limits exploration degradation but can constrain refinement and destabilize training.At a = 0.2, training is unstable with a significantly higher clip fraction, attributed to larger gradient magnitudes.Source: Table 2 · p. 9, Figure 9 · p. 9, §5.4 · p. 10
  • At K = 1, training success plummets from ODE-to-SDE discretization error; larger K improves rollout performance but increases training complexity and computational overhead.K is the number of discretization steps for action generation, and increasing it extends the denoising sequence.Source: Figure 10 · p. 9, §5.4 · p. 10
  • Larger action chunks yield marginal performance gains but reduce interaction frequency and obscure reward credit assignment, diminishing explained variance and limiting subsequent RL refinement.The ablation compares chunk sizes of 5, 10, and 20.Source: Figure 11 · p. 9, §5.4 · p. 10

6. Conclusion

πRL enables online RL fine-tuning for π0 and π0.5 flow-based VLAs and achieves improvements over SFT on challenging benchmarks. The authors note that low online-RL sample efficiency currently makes the framework reliant on sim-to-real deployment.

6. Conclusion

  • πRL enables online RL fine-tuning of flow-based π0 and π0.5 models and achieves significant improvements over SFT baselines on challenging benchmarks.The conclusion identifies intractable flow-matching log-likelihoods as the challenge addressed by Flow-Noise and Flow-SDE.Source: §6 · p. 10
  • Because online RL has low sample efficiency, the framework currently relies on sim-to-real deployment; the authors aim to develop more efficient algorithms for real-world RL training.Source: §6 · p. 10

B.1. Benchmarks.

  • The evaluation spans LIBERO, ManiSkill, MetaWorld, and CALVIN, which probe task variation, environmental robustness, skill breadth, and long-horizon sequences.The benchmarks include 4,352 ManiSkill pick-and-place combinations and 50 MetaWorld manipulation primitives.Source: §B.1 · p. 13

B.2. Implementation Details.

  • During RL, the framework freezes VLM parameters and fine-tunes the 300M action expert, using a shared, co-located GPU allocation strategy.The implementation uses the RLinf codebase; experiments use eight NVIDIA H100 80GB GPUs.Source: §B.2 · p. 13

C.1. LIBERO Benchmark

The benchmark results report gains from RL fine-tuning on LIBERO, ManiSkill, CALVIN, and MetaWorld. The section also frames OOD evaluation as a test of whether ID improvements generalize to novel settings.

C.1. LIBERO Benchmark

  • Few-shot π0 Flow-SDE and Flow-Noise reach 96.1% and 97.6%, above full-dataset π0 SFT's 94.2%; single-trajectory π0.5 RL reaches 98.3% versus full-dataset SFT's 96.9%.Source: Table 3 · p. 14, §C.1 · p. 14

C.2. ManiSkill Benchmark

  • On ManiSkill, π0 success rises from 38.4% to 77.8%, while π0.5 rises from 40.1% to 90.9% with RL fine-tuning.Source: Table 4 · p. 14, §C.2 · p. 15

C.3. CALVIN Benchmark

  • On CALVIN, π0.5 Flow-SDE raises average completed subtasks from 3.838 to 4.717 and Len-5 success to 87.0%.The Flow-SDE versus SFT performance gap grows from 7.0% on Len-1 to 25.7% on Len-5 tasks.Source: Table 5 · p. 15, §C.3 · pp. 15–16

C.4. MetaWorld Benchmark

  • On MetaWorld, π0 and π0.5 achieve average success rates of 85.8% and 70.7%, respectively, surpassing the 68.2% SmolVLA baseline.Source: Table 6 · p. 15, §C.4 · p. 16

D. Out-of-Distribution RL Evaluation

D.1. ManiSkill

RL improves performance under visual and execution shifts, but gains do not reliably transfer to entirely new task objectives. Single-task and real-world evaluations extend the evidence, while training analyses identify stability trade-offs.

D.1. ManiSkill

  • For π0.5, Flow-SDE raises ManiSkill success by 126.7% in-distribution and 102.3% out-of-distribution.The OOD scenarios test vision, semantics, and execution variation; the proportional gains are comparable despite lower OOD improvements overall.Source: §D.1 · pp. 16–17

D.2. CALVIN

  • In CALVIN’s zero-shot Scene D evaluation, RL reaches 79.1% success versus 61.3% for the SFT baseline.Scene D differs in visual textures, lighting, and spatial layouts.Source: §D.2 · p. 17

D.3. MetaWorld

  • On MetaWorld’s novel task objectives, OOD performance oscillates, although RL retains skills learned during SFT.The authors report that RL gains do not transfer effectively to entirely novel objectives, while standard SFT can lose broader capabilities.Source: §D.3 · p. 17

D.4. Summary

  • RL improves robustness to visual and execution shifts, but transferring gains to entirely novel task objectives remains challenging.The reported distinction is between low-level environmental variations and high-level task generalization.Source: §D.4 · p. 18

E.1. SIMPLER

  • In SIMPLER, πRL raises π0’s average success rate from 67.2% to 86.7%.Three tasks—carrot, eggplant, and spoon—exceed 90% success.Source: Table 8 · p. 17, §E.1 · p. 18

E.2. Real2Sim2Real

  • In real-world zero-shot deployment, the RL-tuned policy succeeds on the pick-and-place task 40% of the time, while SFT fails to complete it.The policy is trained with RL after few-shot SFT using 20 expert trajectories.Source: Figure 12 · p. 17, §E.2 · p. 18

F.1. RL algorithms

  • RL training requires balancing rollout success against training stability because rollout-oriented parameters can constrain the performance ceiling.The authors conclude that careful parameter tuning is needed to balance rollout quality and stable convergence.Source: §F.1 · p. 19

F.2. VLM Fine-tuning Analysis

The analysis tests joint VLM and action-expert optimization with LoRA against a frozen VLM. On LIBERO, VLM fine-tuning shows no evident benefit, and joint optimization calls for a more conservative configuration.

F.2. VLM Fine-tuning Analysis

  • On LIBERO-Long with π0 and Flow-SDE, VLM LoRA-II follows a learning trajectory comparable to the frozen-VLM baseline.The authors conjecture that limited scene variability leaves pretrained VLM representations sufficiently robust, so fine-tuning brings no evident benefit on this benchmark.Source: §F.2 · p. 19
  • Jointly fine-tuning the VLM and action expert requires a more conservative optimization configuration for training stability.The tested LoRA configurations use rank 32; LoRA-II uses a 1e−6 learning rate and two updates per epoch.Source: §F.2 · p. 19

G. Insights from Large-Scale RL Training

The training insights cover hyperparameter choices, critic behavior, and temporal efficiency; the GR00T setup adapts the RL pipeline to a different foundation model and replaces expert-model dropout to support convergence.

G. Insights from Large-Scale RL Training

  • When train and evaluation performance diverge, reducing noise or increasing denoising steps can mitigate losses from stochastic execution; lower noise also calls for a smaller learning rate.The authors also report that more denoising steps can help when training improves steadily but evaluation oscillates, and that long-horizon tasks benefit from larger action chunks.Source: G. Insights from Large-Scale RL Training · p. 19
  • Cosine annealing prevents KL divergence from escalating during π0.5 LIBERO-Long training, stabilizing the process.The change addresses a steadily increasing KL divergence observed during training.Source: G. Insights from Large-Scale RL Training · p. 20
  • In ManiSkill, both π0 and π0.5 show an initial evaluation dip before improving as critic value estimates stabilize.The authors attribute the transient degradation to inaccurate critic signals during warm-up.Source: G. Insights from Large-Scale RL Training · p. 20
  • After RL training in ManiSkill, π0.5 episode lengths converge to the expert range, improving temporal efficiency.The authors attribute this to error correction and partial resets, which reward faster task completion through more resets and higher total reward within each update cycle.Source: G. Insights from Large-Scale RL Training · p. 20

H.1. Setup

  • GR00T N1.5 combines an Eagle 2.5 VLM with a Diffusion Transformer action-denoising head and supports multiple robot embodiments.The evaluated LIBERO suites are Spatial, Object, Goal, and Long.Source: §H.1 · p. 20
  • For GR00T RL, the setup fine-tunes the action expert while keeping the VLM fixed; the critic estimates values across the denoising trajectory.The critic network is integrated directly with the action head.Source: §H.1 · pp. 20–21
  • The GR00T pipeline replaces expert-model dropout with identity layers because dropout adds policy stochasticity that undermines training convergence.The authors describe this instability as compounded by per-step policy updates; Flow-SDE is the primary RL algorithm.Source: §H.1 · p. 21

H.2. Results

On the few-shot GR00T model, Flow-SDE improves LIBERO success over SFT, using default π0 configurations; task-specific tuning is left for future work.

H.2. Results

  • Flow-SDE raises few-shot GR00T LIBERO success from 52.5% with SFT to 89.9%.These results use default π0 configurations; the authors leave task-specific tuning for future work.Source: §H.2 · p. 21

I. Limitations and Future Work

The limitations identify open areas in noise injection, mixed ODE-SDE rollouts, and semantic generalization; the appendix records training hyperparameters across benchmarks.

I. Limitations and Future Work

  • ODE-to-SDE conversion causes a performance drop; coefficients-preserving sampling mitigates numerical error but yields limited RL improvement.The authors identify preserving the action distribution during conversion as a direction with potential.Source: §I · p. 21
  • Flow-SDE currently uses one randomly selected SDE denoising step, with all other steps remaining ODE steps.Richer mixed rollouts are proposed as future work that could improve training speed and performance.Source: §I · p. 22
  • ManiSkill OOD tests indicate that semantic generalization remains limited for both SFT and RL models.Future work proposes training on more diverse task distributions and varied linguistic instructions to foster cross-task adaptability.Source: §I · p. 22

J. Training Hyperparameters.

Loading 2510.25889v3…