Source-linked AI summary
VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
Hengtao Li, Pengxiang Ding, Runze Suo, Yihao Wang, Zirui Ge, Dongyuan Zang, Kexian Yu, Mingyang Sun, Hongyin Zhang, Donglin Wang, Weihua Su
TL;DR
VLA models rely heavily on imitation learning, which can compound errors under distribution shift, while RL alternatives require costly real-world interactions or face sim-to-real gaps. VLA-RFT uses a data-driven world model as a controllable simulator for action-conditioned visual rollouts and verified rewards, achieving efficient fine-tuning with improved performance, generalization, and robustness. Its practical scope remains bounded by expert-data quality, world-model capacity, and the lack of explicit world-model planning.
Problem
VLA models rely heavily on imitation learning, which is prone to compounding errors and weak robustness under distribution shift, while existing RL approaches face interaction, safety, or sim-to-real limitations.
Method
VLA-RFT trains a data-driven world model from robot interactions and uses it as a controllable simulator for action-conditioned visual rollouts and verified trajectory rewards.
Results
With as few as 400 fine-tuning steps, VLA-RFT outperforms strong supervised baselines, exceeds simulator-based RL efficiency, and sustains stable execution under perturbations.
Takeaways & Limitations
World-model-based reinforcement fine-tuning is presented as a practical post-training paradigm for improving VLA generalization and robustness with reduced real-world interaction costs.
Takeaways & Limitations
The verified reward remains tied to similarity with expert demonstrations, while world-model capacity and the absence of explicit world-model planning constrain the framework.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models enable embodied decision-making but rely heavily on imitation learning, leading to compounding errors and poor robustness under distribution shift. Reinforcement learning (RL) can mitigate these issues yet typically demands costly real-world interactions or suffers from sim-to-real gaps. We introduce VLA-RFT, a reinforcement fine-tuning framework that leverages a data-driven world model as a controllable simulator. Trained from real interaction data, the simulator predicts future visual observations conditioned on actions, allowing policy rollouts with dense, trajectory-level rewards derived from goal-achieving references. This design delivers an efficient and action-aligned learning signal, drastically lowering sample requirements. With fewer than 400 fine-tuning steps, VLA-RFT surpasses strong supervised baselines and achieves greater efficiency than simulator-based RL. Moreover, it exhibits strong robustness under perturbed conditions, sustaining stable task execution. Our results establish world-model-based RFT as a practical post-training paradigm to enhance the generalization and robustness of VLA models. For more details, please refer to https://vla-rft.github.io/.
1 INTRODUCTION
VLA-RFT addresses imitation-learning brittleness and the costs or limitations of existing reinforcement-learning pipelines with a data-driven world-model simulator. It reports efficient fine-tuning with improved generalization and robustness, motivating world-model-based post-training for VLAs.
- Imitation-trained VLA models accumulate errors under distribution shift, as deviations from expert demonstrations push policies toward unfamiliar states.
- Existing RL alternatives trade off interaction cost, safety, sim-to-real fidelity, or the ability to learn from action consequences.
- VLA-RFT uses a data-driven, controllable world simulator trained on robot interactions to predict future visual observations conditioned on action sequences.
- Policy actions are rolled out in the simulator to produce synthetic visual trajectories for reinforcement fine-tuning with trajectory-level rewards.
- With as few as 400 fine-tuning steps, VLA-RFT outperforms strong supervised baselines, is more efficient than simulator-based RL, and sustains execution under perturbations.
- The authors position world-model-based reinforcement fine-tuning as a practical post-training paradigm for scalable VLA training and deployment.
2 RELATED WORK
Related work situates VLAs at the intersection of pretrained vision-language models, imitation learning, reinforcement learning, and world models. Prior approaches improve capability but remain constrained by distribution shift, interaction demands, safety, or simulation fidelity.
- Vision-Language-Action Models: VLA models align visual and linguistic inputs with actions, commonly using imitation learning over large-scale datasets and supervised adaptation to task-specific action spaces.
- Vision-Language-Action Models: Lightweight adapters and post-training techniques have been explored to improve VLA training efficiency.
- Vision-Language-Action Models: Imitation learning remains vulnerable to error accumulation when distribution shifts move policies away from expert-data states.
- VLA with Reinforcement Learning: Simulation-based VLA reinforcement learning requires vast interactions and faces sim-to-real gaps, while real-world training is expensive and unsafe.
- VLA with Reinforcement Learning: Offline RL struggles with distribution shift and cannot learn from the consequences of its own actions without environmental interaction.
- World Models: World models learn environment dynamics for planning and control through explicit physics or latent predictive representations, with recent work extending them to multimodal and robotic settings.
3 METHOD
VLA-RFT uses offline pretraining to initialize a world model and VLA policy, then optimizes the policy through world-model rollouts and verified rewards. Its stochastic flow-based policy and GRPO update support action sampling, trajectory evaluation, and stable fine-tuning.
- Stage I: WM and Policy Pretraining: Stage I pretrains the world model on offline data and the VLA on expert demonstrations to provide stable initializations.The world model is trained with maximum likelihood, while the VLA learns stable action chunks through a flow-matching head.
- Stage II: VLA Optimization through WM Interaction: Stage II rolls out n action chunks from an initial frame and language instruction, with the world model generating trajectories and verified rewards.The resulting feedback is used to fine-tune the VLA with GRPO.
- World Model: The world model predicts future visual observations autoregressively, conditioning on the initial frame, executed actions, and previously generated frames.Its predictive distribution is trained through maximum likelihood on offline interaction data.
- SDE-Policy: The SDE policy combines a flow-matching action head with a Sigma Net to sample stochastic action chunks and compute rollout log-probabilities.Step-wise log-likelihoods are averaged across denoising steps, and the policy ratio is computed relative to the old policy.
- Training Pipeline: Algorithm 1 samples actions, generates trajectories with the world model, computes verified rewards and group-relative advantages, then updates the policy with GRPO.The pipeline repeats these operations across task instances, rollouts, and diffusion steps.
- Verified Reward: The verified reward is the negative weighted sum of per-frame reconstruction and perceptual similarity losses between generated and ground-truth frames.Trajectory alignment with offline ground-truth frames provides the reward signal.
4 EXPERIMENTS
Experiments evaluate VLA-RFT on LIBERO through world-model fidelity, standard-task performance, perturbation robustness, and reward-design ablations. Results show that limited RFT improves success rates and robustness, with trajectory-level world-model rewards providing the strongest gains.
- World Model Capabilities: The 138M-parameter autoregressive world model predicts action-conditioned visual dynamics on LIBERO with high frame fidelity.It achieves MSE 0.0039, PSNR 25.23 dB, SSIM 0.906, and LPIPS 0.059.
- Performance Improvements for VLA: 400 RFT iterations increase average LIBERO success rate from 86.6% to 91.3%, improving all four standard suites.Gains are Spatial +6.0 points, Object +6.4 points, Goal +2.6 points, and Long +3.0 points.
- Performance Improvements for VLA: VLA-RFT improves robustness across object-position, goal-position, robot-state, and combined perturbations, especially for goal and combined shifts.The largest gains exceed +6%, while RoboState improvements are smaller but consistent.
- Performance Improvements for VLA: VLA-RFT produces broader X- and Z-action coverage than the narrowly concentrated SFT-only policy, supporting adaptation under perturbations.The action-distribution comparison contrasts the RFT-trained policy with the SFT-only base policy.
- Key Factors for VLA-RFT: Trajectory comparison within the same world-model generative space is the strongest verified-reward design, reaching 91.1% average success rate.Action-only supervision yields +1.1 points, while reward type 3 achieves +4.5 points over the base policy and improves consistently across tasks.
5 CONCLUSION & LIMITATION
VLA-RFT uses a learned world model as a controllable simulator for efficient and safe policy optimization, improving VLA performance and generalization with minimal fine-tuning. Its main limitations concern dependence on expert data, world-model capacity, reward design, planning integration, and policy-architecture scope.
- Conclusion: VLA-RFT uses a learned world model as a controllable simulator to enable efficient and safe policy optimization.The framework bridges imitation and reinforcement learning while reducing real-world interaction costs.
- Limitations: The verified reward remains largely based on similarity between generated trajectories and expert demonstrations, constraining discovery beyond expert performance.Policy quality therefore remains dependent on the expert dataset.
- Limitations: World-model representational capacity limits out-of-distribution generalization, while explicit integration into planning remains absent.The authors identify larger, more diverse world models and planning integration as directions for improvement.
- Limitations: The verified reward mechanism could be improved with learned reward models that provide more task-relevant feedback.The current mechanism relies solely on expert similarity.
- Limitations: The current study focuses on flow-matching policies, leaving extension to broader policy architectures as an important direction.
A.1 MODEL ARCHITECTURE
The architecture combines a token-based autoregressive world model with a VLA policy whose action head is optimized during reinforcement fine-tuning. The world model predicts future visual observations from initial images and action sequences, while the policy uses a VLA-Adapter design.
- World Model: The world model encodes initial images and continuous actions as image and action tokens, then autoregressively predicts future state tokens.Decoders transform generated image tokens into predicted future images.
- World Model: The simulator generates future images I1, I2, . . . , IT from an initial image I0 and action sequence a0:T −1.
- VLA Policy: VLA-RFT selects VLA-Adapter as the base policy because competing flow-based implementations posed integration or computational-overhead challenges.
- Architecture Configuration: The world model is configured through architecture and pre-training hyperparameters summarized in Table 5, alongside VLA-Adapter settings in Table 6.
A.2 TRAINING DETAILS
Training first optimizes the world model and base VLA components, then performs reinforcement fine-tuning with a frozen world model and rollout-based GRPO updates. Perturbation experiments separately specify task, dimension, and retained-state settings.
- World Model Training: The world model is optimized on LIBERO datasets for 1.5 × 10^5 steps using AdamW, batch size 16, segment length 8, and learning rate 5 × 10^−5.
- VLA Policy Training: The base policy combines a VLM with a DiT-based flow-matching action head, using LoRA during VLM pre-training and joint action-head optimization.
- Reinforcement Fine-tuning: The world model is frozen during reinforcement fine-tuning.
- Reinforcement Fine-tuning: GRPO estimates advantages while reinforcement fine-tuning uses learning rate 1 × 10^−6, sigma learning rate 1 × 10^−5, auxiliary MSE coefficient 0.01, and entropy regularization 0.003.
- Reinforcement Fine-tuning: 400 training steps use batch size 16 and 16 rollouts per update to balance training stability and exploration.
- Perturbation Setup: Perturbation experiments distinguish Task 1 and Task 2, Dim 1 and Dim 2, and KP for keeping original states.
A.3 EXPERIMENT DETAILS
Experiments compare VLA-RFT with baseline VLA and VLA+RL methods across LIBERO suites and perturbed settings. The reported evaluations emphasize success rates, improvements, training steps, robustness, and visualization of disturbed scenarios.
- Perturbation Evaluation: Perturbation experiments evaluate different tasks and perturbation objects or robot states, with original states retained where indicated by KP.
- General Evaluation: VLA-RFT consistently achieves the highest scores among baseline policies in general LIBERO settings.Table 9 reports success rates across Spatial, Object, Goal, Long, and average metrics.
- VLA+RL Comparisons: VLA-RFT outperforms offline RL baselines and rivals online RL methods while requiring substantially fewer training steps.The comparison includes VLA-RL, ARFM, RWR, and ReinboT.
- VLA+RL Comparisons: Table 10 reports baseline success rate, fine-tuned success rate, improvement, and training steps for RL-method comparisons on Libero Average.
- Visualization: Figures 6 and 7 provide additional visualization results, including comparisons between original and disturbed scenarios.