PaperlayerView exact revision on arXiv

Source-linked AI summary

WoVR+: World Model as Reliable Simulator for Versatile VLA Policy Optimization with RL

Shangqing Zhou, Zhennan Jiang, Ning Ma, Xin Fu, Yupeng Zheng, Shuai Tian, Huangrui Li, Jiahao Liu, Chao Yu, Zhongpu Xia, Haoran Li, Dongbin Zhao

arXiv:2610.17922v1cs.RO

TL;DR

World-model hallucinations can mislead VLA policy optimization, while richer interfaces and long-horizon tasks add consistency, exploration, and rollout-error challenges. WoVR+ redesigns the learned environment and imagined interaction for these settings. It improves success over SFT across RoboTwin and real-robot tasks, with gains also reported for long-horizon phone packaging.

  • Problem

    World-model hallucinations can produce spurious imagined successes, and extending WM-RL to multi-view VLA interfaces and long-horizon tasks raises consistency, exploration, and compounding-error challenges.

    Source: §I · p. 2

  • Method

    WoVR+ combines a geometry-grounded multi-view world model with proprioceptive feedback, chunk-wise exploration, and stage-local imagined training.

    Source: §I · p. 2

  • Results

    WoVR+ improves average success over SFT by 21.09% on 2 RoboTwin tasks and 16.7% across 6 real-robot tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    WoVR+ supports policy optimization across multi-view VLA interfaces and long-horizon manipulation, with reported phone-packaging success increasing from 30% to 75%.

    Source: §I · p. 2, §VI · p. 15

    Under trajectory-level binary supervision, failed rollouts receive the same return regardless of their intermediate progress.

    Source: 1) Limitations of Full-Start WM-RL · p. 7

Abstract

from arXiv · show

Action-conditioned world models can reduce costly real-robot interaction in reinforcement learning for Vision-Language-Action models (VLA-RL), but hallucinations can mislead policy optimization. Extending them to richer VLA interfaces and long-horizon tasks raises challenges in spatiotemporal and cross-modal consistency, exploration-reliability balance in RL, and compounding rollout errors. We present WoVR+, extending WoVR's hallucination-aware WM-RL framework by redesigning the learned environment and imagined interaction protocol: A VLA-interface-aligned world model provides multi-view prediction and proprioceptive updates while mitigating hallucinations. Chunk-wise noise preserves flow-based policy exploration while limiting prediction-degrading action jitter. For long-horizon tasks, stage-local imagined training combines stage-entry resets, stage-specific rewards, local horizons, and controllable sampling to shorten generated prefixes and balance training across stages. WoVR+ improves success over SFT by 21.09% and 16.7% on RoboTwin and 6 real-robot tasks. Long-horizon phone-packaging success rises from 30% to 75%. Videos are available at https://youmo445.github.io/WoVR_Plus.github.io/.

I. INTRODUCTION

WoVR+ extends hallucination-aware world-model RL to richer VLA interfaces and long-horizon manipulation, where imperfect imagined dynamics, costly robot interaction, and rollout errors challenge reliable policy optimization. It combines interface-aligned prediction, coherent exploration, and stage-local training; experiments report gains over SFT in simulation and on real robots.

I. INTRODUCTION

  • VLA policy training through SFT is constrained by demonstration quality and coverage, while robot interaction for on-policy RL is costly and world-model rollouts can produce misleading successes.Hallucinated trajectories can incentivize exploiting model errors instead of true task progress.Source: §I · p. 1

III. WOVR FRAMEWORK AND EXTENSIONS

  • WoVR+ retains WoVR’s reliability-oriented WM-RL framework while redesigning the learned environment and imagined interaction protocol for multi-view VLA policies and long-horizon manipulation.Its design uses geometry-grounded multi-view prediction and proprioception, chunk-wise Flow-SDE exploration, and stage-local training with resets, local horizons, and controllable sampling.Source: §I · p. 2, §III · p. 3

I. INTRODUCTION

  • 21.09% on 2 RoboTwin tasks and 16.7% across 6 real-robot tasks are WoVR+’s average success improvements over SFT; 4-stage phone-packaging success rises from 30% to 75%.The evaluation also reports improvements in multi-view video-generation fidelity and effective policy optimization through WM-RL.Source: §I · p. 2, Figure 1 · p. 2

A. Multi-View World Models

  • Multi-view world models increasingly support closed-loop VLA interaction, but policy optimization requires consistency across observations, actions, camera motion, and proprioceptive transitions.Prior work includes multi-view prediction and, in some systems, proprioception and success-signal prediction.Source: A. Multi-View World Models · pp. 2–3

B. World-Model-Based VLA Reinforcement Learning

  • Existing world-model RL methods address reliability issues incompletely, leaving joint design of a VLA-aligned environment, effective exploration, and reliable imagined optimization insufficiently studied.WoVR frames world-model RL as a reliability problem and uses simulator control, reliable interaction, and policy–model alignment.Source: §I · p. 1, B. World-Model-Based VLA Reinforcement… · p. 3

A. WM-RL framework in WoVR

WoVR controls hallucination through simulator design, reliable imagined interaction, and policy–model alignment. WoVR+ extends that framework with a VLA-aligned interface, temporally coherent action exploration, and stage-local rollouts for long-horizon tasks.

A. WM-RL framework in WoVR

1) Simulator-level control:

  • The learned simulator is trained on recorded robot transitions using action-conditioned video prediction, then frozen to serve as the policy’s imagined interaction environment.The training context combines an initial-frame anchor with recent observations, and the prediction target is the recorded future observations.Source: 1) Simulator-level control · p. 3, 2) Interaction-Level Reshaping · p. 3

2) Interaction-Level Reshaping:

  • WoVR reshapes imagined interaction with success-masked rollouts and keyframe-initialized contexts that shorten generated prefixes and reduce effective prediction depth.Trajectory-length normalization gives shorter rollouts greater per-timestep weight.Source: 2) Interaction-Level Reshaping · p. 3

3) Alignment-Level Regulation:

  • WoVR uses low-frequency real-world recalibration to address policy–model mismatch as policy actions drift from the world model’s training distribution.Additional policy trajectories fine-tune the world model, which is then frozen during renewed imagined optimization.Source: 3) Alignment-Level Regulation · pp. 3–4

B. Extensions and Scope of the Present Work

  • WoVR+ extends the framework to multi-view observations and proprioceptive updates, coherent chunk-wise exploration, and stage-local rollouts for long-horizon, multi-stage manipulation.Its extensions include absolute joint-and-gripper commands, geometry-grounded action and camera conditioning, stage-entry resets, stage-specific rewards, local horizons, and controllable stage sampling.Source: B. Extensions and Scope of the Present… · p. 4

IV. METHODS

WoVR+ couples a VLA-aligned multi-view world model with geometry-aware action conditioning and latent-aligned control injection. Its design conditions video prediction on robot actions and camera geometry while retaining the pretrained backbone architecture.

A. A VLA-Aligned Multi-View World Model for Reliable Local Dynamics

  • The world model combines multi-view history, explicit action and camera geometry, and command-based proprioceptive updates for reliable local dynamics.Its inputs use absolute-joint action chunks and camera intrinsics and extrinsics.Source: A. A VLA-Aligned Multi-View World… · pp. 4–5

1) Geometry-Grounded Action and Camera Conditioning:

  • Absolute-joint targets are converted through forward kinematics into view-specific EEF maps encoding position, orientation, depth, and gripper opening.Semi-transparent compositing retains overlapping cues for bimanual interaction.Source: 1) Geometry-Grounded Action and Camera… · p. 5, Figure 5 · p. 5
  • Per-view ray origins and directions encode camera geometry alongside action maps, so conditions describe both commanded manipulation and its camera configuration.Camera poses are updated at each frame using forward kinematics and fixed hand–eye calibration.Source: 1) Geometry-Grounded Action and Camera… · p. 5

2) Latent-Aligned Control Injection:

  • Action-map latents and temporally grouped ray coordinates are aligned with video latents, assembled per view, and augmented with view embeddings.The first-frame latent slot uses repeated rays, while subsequent slots group four frames of ray origins and directions.Source: 2) Latent-Aligned Control Injection · p. 5, Figure 6 · p. 6
  • A parallel DiT control stream learns action and camera effects while retaining the pretrained backbone architecture; the DiT branches train while the VAE and text encoder remain frozen.Control features are injected into selected backbone blocks through learnable projections.Source: 2) Latent-Aligned Control Injection · pp. 5–6

3) Closed Loop Context and Proprioception Update:

The interaction protocol combines anchored sparse history and command-based proprioception with training strategies for recursive prediction. Flow-SDE exploration uses chunk-wise noise to maintain temporally coherent deviations from the ODE policy.

3) Closed Loop Context and Proprioception Update:

  • Anchored sparse history pairs a persistent first-frame scene anchor with fixed-stride recent observations within a five-frame context.With stride four, clipping early indices to zero supplies valid context near initialization.Source: 3) Closed Loop Context and… · p. 6
  • Under the absolute-position interface, shifting commanded joint and gripper targets by one step approximates proprioception without a learned proprioception model.The command-following approximation is evaluated with a learned transition model in Appendix A.Source: 3) Closed Loop Context and… · p. 6

4) Robust Training and Efficient Generation:

  • To reduce the gap between training and imagined contexts, training blurs selected EEF-near regions and samples synchronized temporal strides from 1 to 4.The anchor and other views remain unchanged; stride sampling exposes the model to different motion magnitudes and prediction intervals.Source: 4) Robust Training and Efficient… · p. 6
  • The action-conditioned video predictor uses a flow-matching objective, with five integration steps for the standard model and a DMD2 one-step generator for higher-throughput imagined interaction.The predictor is conditioned on sparse history and geometric control.Source: 4) Robust Training and Efficient… · p. 6

B. WM-RL with Temporally Coherent Exploration

  • Flow-SDE sampling provides tractable Gaussian transition scores for policy optimization, while chunk-wise noise shares one innovation across the horizon instead of independently perturbing each token.Training uses one randomly selected SDE step per action chunk and ODE updates at the remaining steps; chunk-wise noise yields temporally coherent deviations.Source: 1) From Flow-ODE to Flow-SDE · pp. 6–7, Figure 7 · p. 7

2) Chunk-Wise Exploration Noise:

Chunk-wise Flow-SDE noise preserves each token’s innovation scale while making neighboring action perturbations coherent. Full-start WM-RL, meanwhile, gives later stages fewer training opportunities and relies on sparse terminal rewards over increasingly error-prone imagined prefixes.

2) Chunk-Wise Exploration Noise:

  • Independent innovations perturb neighboring absolute-joint targets in different directions, creating high-frequency action variations; simply lowering noise weakens exploration and can destabilize optimization.The destabilization is associated with substantially larger gradient magnitudes.Source: 2) Chunk-Wise Exploration Noise · p. 7
  • Chunk-wise noise preserves each token’s Gaussian innovation marginal while directing exploration toward temporally coherent perturbations instead of independent deviations.The initial noise, transition mean, and schedule remain unchanged; noise sharing is limited to positions within the sampled action chunk.Source: 2) Chunk-Wise Exploration Noise · p. 7
  • Correlated action-chunk transitions make the retained per-token Gaussian transition log-scores inexact as joint likelihood ratios for the surrogate policy ratios.Grouped imagined returns still provide GRPO advantages.Source: 2) Chunk-Wise Exploration Noise · p. 7

1) Limitations of Full-Start WM-RL:

  • Full-start WM-RL has three coupled limitations: insufficient later-stage state coverage, sparse trajectory-level binary rewards, and compounding errors in long autoregressive rollouts.With terminal-only rewards, all valid steps share the same outcome-based advantage, so failed rollouts receive the same return despite differing intermediate progress.Source: 1) Limitations of Full-Start WM-RL · p. 7
  • Later-stage training opportunities decay multiplicatively because fewer full-start rollouts reach each stage, and those that do depend on longer generated prefixes.The increasingly long prefixes make imagined states more susceptible to accumulated world-model errors.Source: 1) Limitations of Full-Start WM-RL · p. 7

2) KIR and Its Limitations:

WoVR+ extends keyframe initialization into stage-organized policy optimization, combining data-supported stage-entry resets with stage-specific rewards and local horizons. Balanced stage starts give the shared policy direct training opportunities across stages, while deployment still runs the complete task without resets.

2) KIR and Its Limitations:

3) Stage-Local Imagined Policy Optimization:

  • The protocol constructs data-supported entry distributions for every stage, with stage-specific binary completion rewards, local horizons, and controllable stage-sampling probabilities.Stage boundaries follow manipulation-skill transitions and observable subgoal completion; each stage can be optimized without requiring full-task success.Source: 3) Stage-Local Imagined Policy… · p. 8
  • Balanced weights give each stage N/M direct rollout starts in expectation, branching a GRPO group from each reset state for that stage’s local horizon.The resets avoid replaying preceding stages in generated prefixes.Source: 3) Stage-Local Imagined Policy… · p. 8
  • Stage-balanced resets provide direct training opportunities across all stages and eliminate cross-stage model-generated prefixes; stage rewards optimize each skill segment independently.All stages update one shared VLA policy, which executes the full task end-to-end without resets at deployment.Source: 3) Stage-Local Imagined Policy… · p. 8

V. EXPERIMENTS

The experiments assess world-model prediction, policy-level utility, exploration, policy optimization, and long-horizon learning across simulation and real robots. WoVR+ maintains stronger multi-view fidelity over time than the compared models, while all methods degrade during autoregressive prediction.

V. EXPERIMENTS

  • Across the experiments, RoboTwin supports controlled world-model and WM-RL studies, while real robots are used for policy consistency, multi-task WM-RL, and long-horizon evaluation.The evaluation is organized around predictive capability, policy-level utility, exploration, policy optimization, and long-horizon learning.Source: §V · p. 8

1) Video Prediction Setup:

2) Multi-View Prediction Fidelity and Efficiency:

  • WoVR+ achieves the best overall fidelity across head and wrist views among the compared models, while the three-view WoVR adaptation improves localization but retains cross-view layout inconsistencies.The comparison uses held-out RoboTwin trajectories and includes Ctrl-World and EVAC as representative ACWM baselines.Source: 2) Multi-View Prediction Fidelity and… · p. 9, 1) Video Prediction Setup · p. 9

3) Autoregressive Prediction Fidelity:

  • All methods degrade over time, but WoVR+ maintains the best overall autoregressive fidelity across views and horizons.Ctrl-World and EVAC show structural distortions, WoVR has less realistic surface appearance, and WoVR+ better preserves geometry and visual realism.Source: 3) Autoregressive Prediction Fidelity · p. 9

4) Prediction under Policy-Induced Actions:

The supplied findings span policy-induced prediction, world-model ablations, policy-level consistency, and synthetic-data supervision. They report stronger prediction under policy-induced action shift, correlated learned and real success rates, and useful but incomplete synthetic supervision.

4) Prediction under Policy-Induced Actions:

  • WoVR+ outperforms the three-view baselines on prediction under policy-induced action distribution shift, indicating stronger local generalization on executable but underrepresented exploratory actions.The evaluation conditions each model on the same history and Flow-SDE actions, then compares predictions with ground-truth futures.Source: 4) Prediction under Policy-Induced… · p. 9, Table II · p. 10

5) Ablation Study of Model Structure and Training methods:

  • Removing camera raymaps gives the weakest performance, direct latent ray encoding beats image-based encoding, and view embeddings further improve prediction quality.The findings indicate complementary contributions from continuous camera geometry and explicit view identity beyond EEF projection alone.Source: 5) Ablation Study of Model Structure… · p. 9, Table III · p. 10
  • All three world-model training strategies improve generation quality over the raw configuration; combining them gives the strongest distribution-level and wrist-view performance.Temporal-stride sampling alone remains slightly better on several frame-aligned head-view metrics; anchored sparse history and geometry-aware local corruption also improve robustness in their stated ways.Source: 5) Ablation Study of Model Structure… · pp. 9–10, Table IV · p. 11

1) Policy-Level Consistency:

  • The learned-environment and real-robot success rates are strongly correlated across six task–policy pairs (Pearson r = 0.97, Spearman ρ = 0.94).Each policy-task pair had 120 rollouts in each environment, with ACWM rollouts initialized from corresponding real-robot initial images.Source: 1) Policy-Level Consistency · p. 10, Figure 11 · p. 11

2) Synthetic-Data Utility:

  • WM-BC improves success from 71.00% to 91.00% under fixed resets and from 72.66% to 76.56% under randomized resets.Success-filtered real and synthetic trajectories formed an 881-trajectory dataset; the smaller randomized-reset gain indicates useful but incomplete synthetic supervision.Source: 2) Synthetic-Data Utility · pp. 10–11

C. Policy-Level Utility of the Learned Environment

  • Filtered synthetic demonstrations yield a smaller success gain under randomized than fixed resets, motivating direct WM-RL with both successful and failed rollouts.The synthetic trajectories are generated from matched reset states and retained after success filtering.Source: 2) Synthetic-Data Utility · pp. 10–11

1) Effect on WM-RL Optimization:

Chunk-wise exploration supports more stable WM-RL, and WoVR+’s learned environment strengthens downstream policy optimization. The approach also improves real-robot success across six tasks.

1) Effect on WM-RL Optimization:

  • Chunk-wise exploration produces more stable policy improvement and higher final success rates than I.I.D. exploration across both tasks and evaluation settings.The comparison varies only the temporal correlation of Flow-SDE innovation.Source: 1) Effect on WM-RL Optimization · p. 11, Table V · p. 11

2) Temporal and Task-Space Analysis:

1) RoboTwin Policy Optimization:

  • WoVR+ achieves the strongest overall performance among ACWM environments on RoboTwin and outperforms budget-matched Direct RL.It approaches converged Direct RL, and multi-view policies consistently outperform their single-view counterparts.Source: 1) RoboTwin Policy Optimization · p. 12, Table VI · p. 12

2) Real-Robot WM-RL:

  • Real-robot WM-RL raises average success from 39.2% to 55.8% across six tasks.The shared policy is evaluated in 20 matched trials per task.Source: 2) Real-Robot WM-RL · p. 13

E. Long-Horizon Policy Learning

Long-horizon WM-RL faces imbalanced access to later stages, sparse trajectory-level rewards, and compounding errors from long imagined prefixes. On four-stage Phone Packaging, uniform stage-reset training achieves the highest end-to-end success.

E. Long-Horizon Policy Learning

2) Overall Policy Performance:

  • Uniform stage-reset sampling achieves 75% end-to-end success, versus 30% for SFT, 50% for KIR, and 50% or 55% for full-start WM-RL.The full-start variants use trajectory-level or stage-wise rewards, respectively.Source: 2) Overall Policy Performance · p. 13, Figure 17 · p. 14
  • Uniform stage sampling yields a more balanced completion profile, while arrival-matched sampling retains intermediate-stage success without converting it into final task success.Full-start stage-wise rewards improve progress through Stages 2 and 3 but leave a large Stage 4 gap.Source: 2) Overall Policy Performance · p. 13

3) Effect of Reward Granularity:

  • With full-start imagination, stage-wise rewards raise end-to-end success from 50% to 55%.Informative GRPO groups rise from 51% with trajectory-level rewards to approximately 94% with stage-wise rewards, while visitation and prefix length remain unchanged.Source: 3) Effect of Reward Granularity · p. 14

4) Effect of Autoregressive Prefix Length:

Stage-entry initialization reduces the long imagined prefixes that distort later-stage predictions. In long-horizon training, uniform stage allocation improves final-stage completion and training throughput; the framework still relies on predefined stages and data-supported entry states.

4) Effect of Autoregressive Prefix Length:

5) Effect of Stage-Training Allocation:

  • Uniform sampling raises Stage 4 completion from 4/18 (22.2%) to 15/16 (93.8%) compared with arrival-matched sampling.It sacrifices only small amounts of Stage 2–3 completion and produces a more balanced profile.Source: 5) Effect of Stage-Training Allocation · p. 15

6) Optimization Efficiency of Stage-Local Training:

7) Comparison with KIR used in WoVR:

  • Uniform stage-local training achieves 75% end-to-end success versus KIR’s 50%, showing that prefix shortening alone does not explain the improvement.The method combines stage-specific rewards, stage-entry initialization, and balanced training opportunities while retaining a shared policy.Source: 7) Comparison with KIR used in WoVR · p. 15

VI. CONCLUSION

  • WoVR+ combines a geometry-grounded multi-view world model, chunk-wise exploration, and stage-local imagined training, but relies on predefined stages and data-supported stage-entry states.The conclusion identifies automatic stage discovery and adaptive reset-state construction as future directions.Source: §VI · p. 15

APPENDIX A VALIDATION OF COMMAND-BASED PROPRIOCEPTION

Command Shift uses temporally shifted absolute joint targets for proprioceptive updates, relying on the low-level controller to track those targets. It performs best under cross-task shift in the reported long-horizon evaluation.

APPENDIX A VALIDATION OF COMMAND-BASED PROPRIOCEPTION

  • Command Shift updates proprioception by temporally shifting absolute joint-and-gripper commands instead of learning a separate state-transition model.Its approximation relies on the low-level controller tracking absolute targets sufficiently closely.Source: Appendix A · pp. 15–16
  • Under cross-task shift, Command Shift performs best and remains more stable than EEF MLP and ∆EEF MLP.Joint MLP has the lowest task-internal error at 200 steps.Source: Appendix A · p. 16, Figure 21 · p. 16

APPENDIX B REWARD MODEL USED IN WM-RL

  • Reward models are task-specific ResNet classifiers using synchronized head and wrist images, without actions, proprioception, or observation history.A detected success latches the reward, and the models remain frozen during WM-RL.Source: Appendix B · p. 16

APPENDIX C REAL-ROBOT PLATFORM AND DATA INTERFACE

  • Real-robot observations combine synchronized head, left-wrist, and right-wrist RGB images with a 14-D proprioceptive state.Streams are synchronized and resampled to the 10-Hz control timeline.Source: Appendix C · p. 16

APPENDIX D TASK DEFINITIONS AND SUCCESS CRITERIA

  • Phone Packaging is decomposed into four stages, each with a local completion criterion used for reward routing and stage-wise analysis.The six multi-task evaluation tasks include both single-arm and bimanual skills.Source: Appendix D · p. 17, Table VII · p. 17, Table VIII · p. 17

APPENDIX E TRAINING CONFIGURATIONS

  • The appendices report the three-view world-model architecture and training settings alongside shared policy, WM-RL, and task-specific rollout configurations.These configurations are summarized in Tables IX and X.Source: Appendix E · p. 17, Table IX · p. 17, Table X · p. 17

Related papers

Loading 2610.17922v1…