Source-linked AI summary
World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy
Xiaokang Liu, Zechen Bai, Hai Ci, Kevin Yuchen Ma, Mike Zheng Shou
TL;DR
Real-world RL for VLA policies is costly, and action-conditioned video world models can misfollow subtle actions while lacking reliable native rewards. World-VLA-Loop combines SANS, a jointly trained video-and-reward world model, and iterative policy–simulator co-evolution; experiments report improved VLA performance with less reliance on physical interaction.
Problem
Real-world VLA reinforcement learning is expensive, while video world models have imprecise action following and lack native reliable rewards.
Method
World-VLA-Loop combines SANS success and near-success data, joint future-frame and binary-reward prediction from diffusion latents, and rollout-driven co-evolution of the world model and VLA policy.
Results
Experiments across simulation and physical deployment significantly improve VLA performance while reducing reliance on costly real-world interaction.
Takeaways & Limitations
The framework offers a scalable direction for world-model-based real-world RL with reduced dependence on costly physical interactions.
Takeaways & Limitations
Autoregressive video models suffer context-memory limits and quality drift in long-horizon tasks exceeding 300 frames.
Abstract
from arXiv · showhide
Reinforcement learning (RL) can refine Vision-Language-Action (VLA) policies beyond behavior cloning, but real-world RL remains expensive due to extensive rollouts, resets, supervision, and safety risks. Action-conditioned video world models offer an option to train in virtual environments, yet they exhibit imprecise action following, particularly on subtle near-success failures. Besides, they lack native reward signals for RL. Computing rewards based on inaccurate visual predictions remain unreliable. We introduce World-VLA-Loop, structured around two foundational designs and a higher-level co-evolving paradigm. We first curate SANS, dedicatedly mixing successful and near-success trajectories to improve action-outcome alignment. Then, we train a state-aware video world model that jointly predicts future frames and binary rewards from diffusion latents. It couples reward estimation to the generator rather than a separate module, and in turn, benefits visual prediction. Since VLA behavior shifts during RL, a fixed simulator can misalign with the updated policy, World-VLA-Loop therefore closes the loop by using the refined world model for iterative VLA post-training while feeding rollouts from each improved policy back to augment and fine-tune the world model. Across simulation and real-robot experiments, World-VLA-Loop substantially improves VLA performance while reducing reliance on costly physical interaction.
1 Introduction
Real-world RL for VLA policies is costly, while video world models struggle with precise action-conditioned failure simulation and lack native rewards. World-VLA-Loop addresses these gaps with SANS, joint video-reward prediction, and closed-loop co-evolution of the simulator and policy.
- Real-world VLA reinforcement learning requires costly physical rollouts, resets, supervision, and safety management.
- Action-conditioned video world models often misfollow minor actions, hallucinate success in near-success failures, and provide no native RL rewards.
- SANS combines successful and near-success trajectories to sharpen the world model’s boundary between successful and subtly failed outcomes.
- The state-aware world model jointly predicts future frames and binary rewards from diffusion latents, using rewards as intrinsic RL signals.
- World-VLA-Loop feeds rollouts from each refined policy back into world-model training, maintaining alignment as policy failure modes change.
- Experiments across simulation and physical deployment significantly improve VLA performance while reducing reliance on costly real-world interaction.
2 Related Work
Prior work spans imitation-learning-oriented VLA systems, controllable video world models, and emerging co-evolution strategies. World-VLA-Loop is situated among approaches that use action-conditioned video simulation while integrating native reward prediction and policy–world-model refinement.
- VLA research commonly uses foundation VLM priors for robotic manipulation, while reinforcement learning extends beyond imitation learning.
- Action-conditioned video world models support controllable simulation across gaming, autonomous driving, and robotics domains.
- The framework’s four-phase cycle curates SANS, trains a jointly supervised world model, performs GRPO policy optimization, and augments data with refined-policy rollouts.
- Related co-evolution methods identify imagination–outcome misalignment, whereas the described approach predicts rewards directly from world-model latents instead of relying on external reward models.
3 Method
The method builds a virtual RL environment from success and near-success data, an action-conditioned video model, and an integrated reward head. Iterative real-world rollouts then refine both the simulator and the VLA policy.
- SANS Dataset: SANS addresses success-dominated robotic datasets by adding near-success trajectories caused by minor end-effector inaccuracies.
- SANS Dataset: Near-success examples expose fine-grained spatial dynamics and better represent failure modes encountered during policy rollouts.
- SANS Dataset: The ManiSkill pretraining set contains 35k video-action pairs across 23 tasks, while LIBERO and real-world tasks use roughly 50 success and 50 failure trajectories per task.
- World Model: The world model conditions future video synthesis on observed frames and robot actions represented as 6-DoF end-effector poses with gripper state.
- World Model: A lightweight reward head maps denoised diffusion latents to scalar rewards and is jointly optimized with the original flow-matching loss.
- World Model: Joint reward-and-video training aligns reward prediction with generated visual outcomes and encourages clearer separation of successful and failed executions.
- Policy RL: The trained simulator generates multi-step observations and rewards for GRPO-based VLA post-training, replacing physics-based simulators during RL.
- Policy RL: After each RL update, success and near-success policy rollouts augment SANS for subsequent world-model fine-tuning and improved policy optimization.
4 Experiments
Experiments evaluate World-VLA-Loop as a visual simulator, reward-aware world model, and environment for VLA post-training across LIBERO and real-world tasks. The results show strong outcome alignment, improved policy success, and gains from iterative refinement and joint design choices.
- World Model Performance: 88.5% average visual alignment and 87.25% reward alignment establish the world model as a high-fidelity simulator for VLA rollouts.The reported evaluation covers both simulation and real-world scenarios.
- VLA Post-training: RL post-training improves OpenVLA-OFT success by 12.7% on LIBERO and by 23.4% and 13.3% in two real-world scenarios.Success rates are measured across benchmark and physical rollouts, with the real-world simulator curves distinguished from final physical results.
- Evaluation Setup: The world model is evaluated for visual fidelity, action-following precision, and its utility for improving VLA success during RL post-training.The evaluation spans simulated LIBERO tasks and a laboratory real-world setup.
- World Model Performance: More than 80% average alignment accuracy for both visual and reward metrics shows that the model distinguishes successful from failed trajectories across settings.Reward-based classifications are highly consistent with visual-alignment classifications.
- Iterative Refinement: After two closed-loop RL iterations, real-world success rises from 13.3% and 26.7% to 50.0% and 53.3% on two tasks.Each iteration feeds newly generated policy trajectories back into world-model training.
- Ablation Study: Removing reward supervision reduces visual alignment by approximately 30%, while excluding near-success trajectories also significantly decreases visual alignment.These ablations support joint reward-video training and inclusion of near-success failures in world-model data.
- Qualitative Results: Qualitative rollouts show that SANS augmentation helps the refined world model represent edge cases and yields more precise, robust grasping poses after reward-hacking behavior in the first iteration.The first RL iteration initially exploits spatial inaccuracies in the world model by moving in front of the cube.
5 Conclusion
World-VLA-Loop jointly optimizes a world model and VLA policy, using near-success data and integrated reward supervision for virtual policy refinement. Experiments report success-rate gains across simulation and real-world benchmarks, while current video models remain limited on long-horizon tasks exceeding 300 frames.
- World-VLA-Loop jointly optimizes the world model and VLA policy through a world-model-based reinforcement-learning framework.
- SANS near-success trajectories and integrated reward supervision support high-fidelity world-model training for policy refinement in a virtual environment.
- Experiments demonstrate significant success-rate gains across simulation and real-world benchmarks while reducing reliance on costly physical interactions.
- Current autoregressive video models suffer from limited context memory and quality drift on long-horizon tasks exceeding 300 frames.
A.1 World Model Training
The implementation transfers an action-conditioned Cosmos Predict 2 checkpoint and pretrains it on a ManiSkill SANS dataset. The learned model is then used with a VLA policy through a request-response interface for batch RL optimization.
- The world model is initialized from the Cosmos Predict 2 action-conditioned checkpoint and pretrained on a collected ManiSkill SANS dataset.This stage targets basic Franka-arm/action relationships and fundamental manipulation physics.
- The SimpleVLA-RL codebase replaces the physical simulator with the learned world model while preserving a consistent interface across environments.
- The VLA–world-model interaction uses a request-response architecture, with the world model operating as a backend server for batch GRPO optimization.The GRPO group size for advantage computation is 8.
A.3 Real-World Experimental Settings
Real-world evaluation uses a Franka arm and fixed third-person RealSense camera for randomized pick-and-place and pushing tasks. World-model generation and task adaptation are substantially cheaper than traditional physical-rollout training.
- Real-world experiments cover randomized pick-and-place cup and cube-pushing tasks using a Franka arm and fixed third-person RealSense D435 camera.The task prompts are “Pick up the cup and place it on the green plate” and “Push the cube into the dustpan.”
- A batched generation of 24 video frames takes approximately 6 seconds on a single NVIDIA H100 node.
- A full RL training session typically converges within 20–40 optimization steps, totaling roughly 25 to 35 hours.
- World-model fine-tuning on 100 trajectories typically converges within 4 hours on 8 H100 GPUs.This supports task adaptation with substantially lower physical-rollout burden.
B.1 Quality Analysis on Long-Horizon Tasks
Long-horizon quality analysis finds severe video degradation after 300 frames, affecting LIBERO-Long tasks more than shorter suites. An auxiliary VLM-reward experiment indicates realistic videos, while the integrated reward head is more computationally efficient.
- Severe video degradation typically occurs only after the 300-frame mark, covering most robotic manipulation tasks but limiting longer-horizon settings.
- LIBERO-Object, Spatial, and Goal tasks typically require around 220 frames, whereas LIBERO-Long tasks extend to 300–400 frames and become too blurry for RL accuracy.
- VLM rewards achieve reasonable performance, validating that generated videos are sufficiently realistic and accurate for external VLM interpretation.
- The integrated reward head achieves superior performance while being significantly more computationally efficient than VLM-based alternatives.
C.1 Additional World Model Generation Results
The world model generates action-conditioned rollouts that distinguish successful and failed executions, including nuanced failure cases. It also produces physically plausible trajectories for unseen action sequences.
- The model accurately follows erroneous action trajectories and distinguishes nuanced outcomes caused by incorrect execution.These generated failure rollouts demonstrate action adherence rather than hallucinated success.
- Successful action inputs produce trajectories that the world model clearly distinguishes from failure outcomes.Most illustrated successful trajectories preserve the success-versus-failure distinction.
- The model synthesizes physically plausible trajectories for action sequences unseen during fine-tuning.This capability leverages priors from ManiSkill pretraining and extends to novel robot-arm trajectories.