Source-linked AI summary
WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
Zhennan Jiang, Shangqing Zhou, Yutong Jiang, Zefang Huang, Mingjie Wei, Yuhui Chen, Tianxing Zhou, Zhen Guo, Hao Lin, Quanlu Zhang, Yu Wang, Haoran Li, Chao Yu, Dongbin Zhao
TL;DR
VLA reinforcement learning is limited by costly real-world interaction and unreliable closed-loop world-model rollouts, where hallucinated dynamics can corrupt policy optimization. WoVR regulates this interaction with a controllable action-conditioned world model, Keyframe-Initialized Rollouts, and policy–model co-evolution. It achieves the best reported LIBERO performance and consistent real-world gains, while remaining primarily empirically evaluated and underexplored for longer-horizon and mobile manipulation.
Problem
Real-robot RL requires massive interaction, while imperfect learned world models can hallucinate outcomes and provide unreliable optimization signals under policy-induced distribution shift.
Method
WoVR combines a rollout-stable action-controllable video world model, Keyframe-Initialized Rollouts, and policy–model co-evolution for hallucination-aware imagined RL.
Results
WoVR achieves the highest average LIBERO success rate of 95.9% under full-trajectory SFT and consistently improves performance across real-world robotic platforms.
Takeaways & Limitations
Hallucination-controlled world models can serve as practical simulators for VLA policy optimization in imagination.
Takeaways & Limitations
The analysis is primarily empirical, and WoVR remains underexplored for substantially longer-horizon tasks and mobile manipulation.
Abstract
from arXiv · showhide
Reinforcement learning (RL) promises to unlock capabilities beyond imitation learning for Vision--Language--Action (VLA) models, but its requirement for massive real-world interaction prevents direct deployment on physical robots. Recent work attempts to use learned world models as simulators for policy optimization, yet closed-loop imagined rollouts inevitably suffer from hallucination and long-horizon error accumulation. Such errors not only degrade visual fidelity, but also mislead policy optimization by providing unreliable learning signals. We propose WoVR, a reliable world-model-based RL framework for post-training VLA policies. Instead of assuming a faithful world model, WoVR explicitly regulates how RL interacts with imperfect imagined dynamics. It improves rollout stability through a controllable action-conditioned video world model, reshapes imagined interaction to reduce effective error depth via Keyframe-Initialized Rollouts, and maintains policy--simulator alignment through World Model-Policy co-evolution. Extensive experiments demonstrate that WoVR enables stable long-horizon imagined rollouts and effective policy optimization, achieving superior LIBERO performance and consistent real-world gains across multiple robotic platforms. These results show that world models can serve as practical simulators for RL when hallucination is explicitly controlled. Additional visualization results are available at https://wovr-corl.github.io.
1 Introduction
WoVR frames world-model-based RL for VLA policies as a reliability problem caused by hallucinated closed-loop dynamics, compounding prediction errors, and policy-induced distribution shift. It addresses this through controllable simulator design, reliable imagined interaction, and policy–model alignment, improving world-model quality and policy performance.
- Imitation learning limits VLA performance because policies remain constrained by demonstration quality and coverage.
- On-policy RL requires massive parallel real-world interaction, while off-policy methods can suffer from distribution shift and training instability.
- Closed-loop world models hallucinate visually plausible but physically incorrect outcomes as autoregressive errors compound and policy distributions shift.
- WoVR regulates hallucination through a rollout-stable action-controllable world model, Keyframe-Initialized Rollouts, and policy–model co-evolution.
- 23 FPS rollout efficiency accompanies strong perceptual and temporal consistency, while policy performance improves in simulation and across multiple real-world robotic platforms.
2 Related Work
Prior VLA-RL work reduces reliance on real interaction through on-policy, off-policy, and learned-world-model approaches, but does not fully address hallucinated dynamics in closed-loop optimization.
- On-policy VLA fine-tuning is impractical on real robots because it requires parallel rollouts, repeated resets, and tightly coupled policy–environment interaction.
- Learned world models have been used for reward construction, evaluation, and synthetic rollouts, but these approaches stop short of closed-loop RL simulation.
- Prophet, World-Env, and WMPO replace real interaction with imagined rollouts but largely treat the world model as a drop-in simulator.
- Without explicit hallucination control, accumulated prediction errors can be exploited by the policy and mislead reinforcement-learning optimization.
3 Methods
WoVR constructs a controllable, rollout-stable video world model, reshapes imagined interaction with KIR, and regulates policy–model alignment through PACE. These mechanisms target error accumulation, hallucinated success, and distribution shift during imagined RL.
- 3 Methods: WoVR controls hallucination at simulator, interaction, and alignment levels through a stable action-conditioned model, KIR, and PACE.
- 3.1 Stabilized Action-Conditioned World Model: Dual-channel action injection provides frame-level control, while first-frame anchoring and noisy context augmentation stabilize chunk-wise autoregressive generation.
- 3.1 Stabilized Action-Conditioned World Model: The simulator predicts video chunks from anchored visual context and policy actions, producing long-horizon imagined trajectories.
- 3.2 Hallucination-Aware Policy Optimization in Imagination: KIR initializes some rollouts near task-critical states, shortening effective prediction depth and reducing errors before decisive contacts or corrections.
- 3.2 Hallucination-Aware Policy Optimization in Imagination: GRPO uses imagined rollouts, while trajectory-length normalization increases the contribution of short, task-critical segments.
- 3.3 Policy–Model Alignment: PACE periodically refines the world model with rollouts from the evolving policy to reduce policy–simulator mismatch.
4 Experiments
WoVR is evaluated as a stable, efficient world-model simulator for imagined RL, its downstream LIBERO policy gains, and transfer to two real-world robotic platforms. Across these settings, the reported results favor WoVR over representative baselines, including under long-horizon rollouts and multiple VLA backbones.
- Experimental design: The experiments test world-model stability, policy improvement, and real-world transfer using closed-loop video generation, LIBERO suites, and two robotic platforms.World-model evaluation includes perceptual, distributional, temporal, and throughput metrics; policy evaluation covers four LIBERO suites and real-world manipulation.
- World-model evaluation: WoVR outperforms all compared world-model baselines across reported metrics, with improvements becoming more pronounced at longer rollout horizons.The comparison covers EVAC, Cosmos-Predict2, and OpenSora under the same chunk-wise autoregressive generation protocol.
- Policy optimization: 69.5%: WoVR raises average LIBERO success under one-trajectory SFT, exceeding GRPO's 4.1% improvement and WMPO's 50.9% average success rate.WMPO records 0% improvement on LIBERO-Long, while WoVR achieves the best performance across all suites.
- Real-world transfer: +28.9 points on Franka and +13.4 points on AgileX Piper: WoVR improves real-world success over the base policy across both robotic platforms.The real-world evaluations use three tasks per platform and 30 independent trials per task.
- Real-world transfer: +22.3 points: WoVR improves the average real-world success rate of a π0.5 policy on AgileX Piper across all evaluated tasks.The reported gain includes a +10.0-point improvement on the challenging Fold Towel task.
5 Conclusion
WoVR identifies hallucination in closed-loop imagined interaction as a central reliability bottleneck for world-model-based RL in VLA policy optimization. It addresses this through simulator stability, Keyframe-Initialized Rollouts, and policy–simulator alignment, with experiments supporting hallucination-controlled world models as practical RL simulators.
- Conclusion: WoVR treats hallucination as a reliability bottleneck and regulates simulator stability, imagined interaction, and policy–simulator alignment.Its mechanisms are an action-controllable video world model, KIR, and PACE.
- Conclusion: Experiments on LIBERO and real-world manipulation tasks show that WoVR enables effective policy optimization.
- Conclusion: Hallucination-controlled world models can serve as practical RL simulators within a closed-loop process that refines simulators and VLA policies.
6 Limitations
WoVR’s evaluation scope remains limited in theory and deployment setting. The paper does not formally characterize hallucination propagation or provide regret guarantees, and longer-horizon and mobile-manipulation settings remain underexplored.
- The analysis of hallucination remains primarily empirical, without a formal characterization of propagation into policy optimization or a regret bound against the real-environment optimum.
- Real-world validation focuses on tabletop manipulation, leaving substantially longer-horizon tasks and mobile manipulation underexplored.These settings may amplify error accumulation through longer horizons, changing viewpoints, larger state spaces, and more diverse contact dynamics.
- Scaling WoVR to broader scenarios may require more diverse embodied-data pretraining and more sophisticated mechanisms for suppressing compounding errors.
C Evaluation Metrics
The paper evaluates generated videos using perceptual, distributional, motion-aligned, and throughput metrics. Together, these metrics assess visual similarity, temporal or motion consistency, and generation efficiency.
- LPIPS measures frame-level perceptual similarity using deep visual features.
- FID evaluates distributional similarity between generated and real frames using image-level feature statistics.
- FVD evaluates distributional similarity between generated and real videos using spatiotemporal feature statistics.
- FloLPIPS measures perceptual similarity after motion alignment along estimated optical-flow trajectories.
- FPS measures the number of frames generated per second and therefore quantifies world-model generation efficiency.
D.1 Ablation on World Model Mechanisms
The ablation study tests how reference frames, memory context, and noisy context augmentation affect long-horizon generation. Results show that the full context strategy improves fidelity, temporal consistency, and rollout stability, especially as horizons lengthen.
- Multi-frame context with a fixed reference anchor improves LPIPS, FID, FVD, and FloLPIPS over single-frame context across rollout horizons.
- Removing the fixed reference frame causes clear degradation, particularly at longer rollout horizons.Qualitatively, models without the reference frame show spatial drift and object disappearance.
- Disabling noisy context augmentation causes noticeable performance drops, with a larger gap as rollout length increases.The reported effect is moderate for short rollouts and more pronounced for longer ones.
- The full strategy combines a fixed reference frame, multi-frame memory, and noisy context augmentation to stabilize closed-loop autoregressive generation.The study uses 1,500 VLA rollout trajectories for training and 24 held-out trajectories for evaluation on LIBERO-Spatial.
D.2 Ablation on PACE and KIR
The ablations show that WoVR’s reliability gains come from both policy-aligned simulator refinement through PACE and keyframe-initialized rollouts through KIR. Together, these mechanisms improve imagined training signals and long-horizon policy optimization.
- Conclusion: PACE and KIR provide complementary reliability benefits: one refines the simulator toward the evolved policy distribution, while the other reduces effective rollout depth.The ablation separates policy-aligned data collection from keyframe-initialized imagined interaction.
- Results: 82.5% average success rate makes full WoVR the best-performing variant across LIBERO-Spatial and LIBERO-Object.The average is computed over the two reported suites.
- PACE: Increasing base-policy trajectories from 1,500 to 2,500 raises the two-suite average only from 75.8% to 77.5%.The +1.7-point gain indicates limited benefit from adding data without changing the policy distribution.
- PACE: After PACE updates WMBase to WMEvo, imagined training becomes more consistent with real-environment evaluation.Before the transition, imagined success can improve while real evaluation lags; post-transition alignment supports more faithful training signals.
- KIR: Removing KIR lowers average success from 82.5% to 79.7%, with Spatial falling from 84.2% to 81.6% and Object from 80.8% to 77.8%.KIR initializes imagined rollouts near task-critical states, shortening effective prediction depth and reducing early error accumulation.
D.3 Ablation on Reward Modeling
Dense rewards accelerate early learning, but sparse and dense reward models reach similar final performance. The dense model’s substantially higher rollout cost motivates using the lightweight sparse model in the main experiments.
- Final performance: Both reward designs converge to similar final performance.The comparison uses a ResNet-based binary-success model and a Qwen3-VL-based fine-grained progress model.
- Learning dynamics: Dense rewards improve sample efficiency early in training, producing faster initial performance gains than sparse rewards.All other components and evaluation metrics are held fixed in the comparison.
- Efficiency: Dense-reward rollout is approximately 3× slower than sparse-reward rollout.The computational cost comes from the Qwen3-VL 2B vision-language backbone.
- Design choice: WoVR adopts the sparse reward model in all main experiments because binary rewards are common in LIBERO and real-world robotic setups.The choice combines comparable final performance with greater time efficiency.
E Qualitative Failure Mode Analysis
WoVR changes recurrent real-world failure patterns across Franka Panda and AgileX Piper tasks, improving grasping, release, placement, and contact-rich execution. Deployment uses delta end-effector actions, Rotation 6D, and platform-specific handling of state-estimation noise.
- Franka Panda: On Franka Panda, WoVR improves banana transfer and produces more direct motions compared with repeated grasp attempts or failed placement by the base policy.The platform evaluation covers Pick Banana, Pick Bread, and Open Drawer.
- AgileX Piper: On AgileX Piper, WoVR reduces unstable placement and release errors arising under noisier state estimation and less stable low-level control.The platform evaluation covers Pick Cube, Pick Tomato, and Fold Towel.
- Failure-mode changes: WoVR reduces recurrent action-level failures across two robotic platforms, including unstable grasping, gripper-timing errors, inaccurate placement, and incomplete interactions.The qualitative changes are consistent with the reported quantitative real-world improvements.
- Action space: The policy outputs relative delta end-effector displacements, which are converted to absolute poses and then executable joint commands through inverse kinematics.The same delta EEF action space feeds both the VLA policy and world model.
- Rotation representation: Rotation 6D replaces direct delta-RPY regression to avoid angle discontinuities, singularities, and quaternion double-cover ambiguity.Predicted 6D vectors are projected to valid rotation matrices through orthogonalization.
- Platform-specific control: AgileX Piper uses an accumulated action formulation because noisy, biased state feedback causes drift under the standard update rule.The accumulated formulation reduces sensitivity to noisy feedback and improves long-horizon execution stability, while external-contact discrepancies are compensated by visual feedback.
F.4 Task Setting
WoVR scales supervised and world-model data according to task difficulty. Harder tasks receive more demonstrations and rollout data when they involve complex dynamics, longer horizons, or more diverse interactions.
- Task-specific scaling: Harder tasks receive more VLA SFT demonstrations so the supervised policy reaches sufficient initial performance before reinforcement learning.World-model training follows the same task-difficulty principle.
- Task-specific scaling: World-model datasets are enlarged for tasks with complex dynamics, longer horizons, or more diverse interactions to support reliable modeling.The allocation is explicitly tied to task difficulty rather than a uniform data budget.
- Real-world collection: Real-world data collection omits PACE for efficiency and instead combines rollouts from multiple VLA SFT checkpoints saved at different training steps.Checkpoint diversity broadens trajectory and action-space coverage by inducing distinct behavior distributions.