PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldGym: World Model as An Environment for Policy Evaluation

Julian Quevedo, Ansh Kumar Sharma, Yixiang Sun, Varad Suryavanshi, Percy Liang, Sherry Yang

arXiv:2506.00613v3cs.ROcs.AI

TL;DR

Robot policy evaluation is costly and difficult because real-world testing is expensive and handcrafted simulators struggle with complex dynamics. WorldGym uses an action-conditioned video world model for Monte Carlo rollouts from real initial frames, with VLM-based rewards. Its policy values correlate with real-world success and preserve rankings across model variants, while realistic object interaction remains challenging.

  • Problem

    Robot policy evaluation needs a practical alternative to costly real-world testing and manually engineered simulators that struggle with complex dynamics.

    Source: §1 · p. 1

  • Method

    WorldGym learns an action-conditioned video world model, rolls policies out from real initial frames, and uses a VLM to assign rewards.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    WorldGym policy values highly correlate with real-world success rates and preserve relative rankings across policy versions, sizes, and training checkpoints.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    WorldGym enables safe, reproducible policy sanity checks and OOD evaluation before deployment using only initial frames.

    Source: §1 · p. 2, §4.2 · p. 7, §6 · p. 9

    Realistic object interactions remain challenging, although robot motions are faithfully emulated.

    Source: §6 · p. 9

Abstract

from arXiv · show

Evaluating robot control policies is difficult: real-world testing is costly, and handcrafted simulators require manual effort to improve in realism and generality. We propose a world-model-based policy evaluation environment (WorldGym), an autoregressive, action-conditioned video generation model which serves as a proxy to real world environments. Policies are evaluated via Monte Carlo rollouts in the world model, with a vision-language model providing rewards. We evaluate a set of VLA-based real-robot policies in the world model using only initial frames from real robots, and show that policy success rates within the world model highly correlate with real-world success rates. Moreoever, we show that WorldGym is able to preserve relative policy rankings across different policy versions, sizes, and training checkpoints. Due to requiring only a single start frame as input, the world model further enables efficient evaluation of robot policies' generalization ability on novel tasks and environments. We find that modern VLA-based robot policies still struggle to distinguish object shapes and can become distracted by adversarial facades of objects. While generating highly realistic object interaction remains challenging, WorldGym faithfully emulates robot motions and offers a practical starting point for safe and reproducible policy evaluation before deployment.

1 INTRODUCTION

Robot policy evaluation is costly and difficult because real-world testing risks damage, while handcrafted simulators struggle with complex dynamics and the sim-to-real gap. WorldGym addresses this by using one video world model across tasks and environments to evaluate policies before deployment.

1 INTRODUCTION

  • Real-world robot evaluation is costly and potentially damaging, motivating reliable pre-deployment testing.Handcrafted simulators also require manually encoding complex dynamics and interactions, limiting realism and generality.Source: §1 · p. 1
  • Handcrafted simulators struggle with high-dimensional dynamics and complex interactions, contributing to the sim-to-real gap.Source: §1 · p. 1
  • Because all tasks share one physical world, a single world model can in principle evaluate many policies across tasks and environments.Source: §1 · p. 2
  • WorldGym learns an action-conditioned video world model, rolls policies out from real initial frames, and uses a VLM to determine task success.It aligns diffusion horizons with policy action chunks for efficient inference.Source: §1 · p. 2
  • WorldGym’s policy values highly correlate with real-world success rates and preserve relative rankings across policy variants, while enabling OOD evaluation from a single initial frame.The paper also reports remaining difficulty with object-shape recognition and adversarial object facades.Source: §1 · p. 2

2 PROBLEM FORMULATION

The paper formulates robot policy evaluation as offline evaluation in image-based, partially observable, multi-task environments. It learns one action-conditioned world model from offline data and estimates policy values through stochastic rollouts with learned rewards.

2 PROBLEM FORMULATION

  • The setting is a finite-horizon, multi-task POMDP with states, actions, observations, goals, rewards, transitions, emissions, and horizon length.Policies act toward language-defined goals while observations are emitted from underlying states.Source: §2 · p. 2
  • Rewards are sparse: only the terminal reward can be 0 or 1, while all earlier rewards are zero.Source: §2 · p. 3
  • Offline policy evaluation estimates a policy’s expected future reward from previously logged interaction data.Source: §2 · p. 3
  • The method learns a single world model from image observations and action sequences, then estimates policy values with Monte Carlo rollouts.A learned reward function evaluates generated observation sequences.Source: §2 · p. 3
  • A shared world model can exploit diverse tasks and environments while matching the image-based observations available on real robots.Source: §2 · p. 3

3 BUILDING AND EVALUATING THE WORLD MODEL

WorldGym combines an autoregressive action-conditioned video model, iterative policy rollouts, and VLM-based rewards. Validation shows that it follows real robot dynamics and arbitrary end-effector controls across morphologies, while flexible horizon matching supports efficient policy evaluation.

3.1.1 WORLD MODEL TRAINING

  • The world model is a latent Diffusion Transformer trained autoregressively on frame sequences paired with robot actions.Action dropout and classifier-free guidance improve adherence to controls, while causal temporal attention conditions generation on prior frames.Source: §3.1.1 · p. 3

3.1.2 ROLLING OUT A POLICY IN THE WORLD MODEL

  • WorldGym iterates between policy action chunks and generated observations, returning each latest predicted frame as the policy’s next input.Source: §3.1.2 · p. 3

3.2.1 AGREEMENT WITH VALIDATION SPLIT

  • Generated rollouts generally follow real-robot videos across Bridge, RT-1, VIOLA, and Berkeley UR5 morphologies.The qualitative comparison uses matching action sequences and initial observations.Source: Figure 2 · p. 4, §3.2.1 · p. 4

3.1.2 ROLLING OUT A POLICY IN THE WORLD MODEL

  • Matching diffusion horizon length to each policy’s action-chunk size enables efficient parallel generation from one world-model checkpoint.This avoids wasted compute for shorter chunks and lost parallelism for longer chunks.Source: §3.1.2 · p. 4

3.1.3 VLM AS REWARD

  • GPT-4o evaluates generated rollout frames against language instructions and can provide manually specified partial-credit signals when policies do not fully succeed.Source: §3.1.3 · p. 4

3.2.2 END-EFFECTOR CONTROL SWEEPS

  • WorldGym evaluates VLA success rates in simulation and the real world, providing the basis for comparing policy performance across environments.Source: Figure 4 · p. 5
  • Generated videos closely follow gripper opening, closing, and directional controls even for control sequences absent from training data.Source: §3.2.2 · p. 5, Figure 3 · p. 5

4 EVALUATING POLICIES IN WORLDGYM

WorldGym evaluates robot policies through simulated video rollouts and reproduces real-world performance patterns, including policy rankings across models and training stages. It also enables OOD testing, revealing both generalization strengths and persistent perceptual failures.

4.1 CORRELATION BETWEEN REAL-WORLD AND SIMULATED POLICY PERFORMANCE

  • WorldGym policy success rates strongly correlate with real-world outcomes, with Pearson r = 0.78 and mean-rate differences averaging only 3.3%.RT-1-X reaches 18.5% versus 15.5%, Octo 20.0% versus 23.82%, and OpenVLA 70.6% versus 67.4% in the real world versus WorldGym.Source: §4.1 · p. 6

4.2 POLICY RANKING WITHIN A WORLD MODEL

  • WorldGym preserves the known ranking of RT-1-X, Octo, and OpenVLA across the 17-task Bridge evaluation suite.The same relative ordering appears in WorldGym and the reported real-world results.Source: §4.2 · p. 7
  • Larger and more recent policy versions outperform smaller or earlier counterparts, while later training checkpoints generally receive higher WorldGym success rates.These rankings agree with reported real-world comparisons and validation-loss trends across Octo, OpenVLA, UniPi, and DexVLA evaluations.Source: §4.2 · p. 7
  • WorldGym completes reported rollouts in under an hour on one GPU using only initial images, avoiding manual real-robot evaluation setup.The authors contrast this with real-world evaluation that can take days.Source: §4.2 · p. 7

4.3 OUT-OF-DISTRIBUTION INPUTS

  • OOD tests show OpenVLA succeeds on color classification but struggles with shape-based distinctions and can select a carrot image on a laptop instead of the real carrot.OpenVLA picks the correct colored paper in all trials, while advanced classification is near chance and the laptop distraction causes errors in 15% of trials.Source: §4.3 · pp. 7–8, Figure 8 · p. 7, Figure 10 · p. 8
  • Adding unrelated distractors degrades all tested VLAs, while modified language tasks show OpenVLA generalizes best among the evaluated policies.The OOD studies use edited initial frames and changed task instructions to evaluate unseen inputs without new real-world setups.Source: §4.3 · pp. 7–8, Figure 11 · p. 8, Figure 12 · p. 9

5 RELATED WORK

WorldGym extends prior video-generation and policy-evaluation research by using a shared action-conditioned world model to evaluate robot policies across tasks, environments, and robot morphologies. Unlike handcrafted simulators, it supports OOD image and language evaluation without comparable manual engineering.

ACTION-CONDITIONED VIDEO GENERATION

  • WorldGym uses action-conditioned video generation for quantitative robot-policy evaluation, extending prior work on simulated interactions, plans, and games.Its architecture draws on Diffusion Forcing and Diffusion Transformers while supporting variable horizons for different action chunk sizes.Source: §5 · p. 9

POLICY EVALUATION

  • WorldGym performs model-based policy evaluation with one image-based world model shared across policies and tasks, rather than task-specific dynamics or value models.The shared model is motivated by the common physical world underlying many tasks and policies.Source: §5 · p. 9
  • Compared with SIMPLER and narrower world-model evaluation work, WorldGym targets diverse environments and morphologies while enabling OOD language and image testing.The comparison identifies OOD evaluation as difficult for software simulators without significant hand engineering.Source: §5 · p. 9

6 CONCLUSION

WorldGym is presented as a world-model environment that realistically emulates robot interactions and correlates with real-world policy outcomes. Its flexible OOD evaluation supports safer, more reproducible pre-deployment testing, although some interactions remain unrealistic.

6 CONCLUSION

  • WorldGym combines realistic robot-interaction emulation, strong simulated-to-real correlations, and flexible OOD evaluation for robot policies.It supports OOD language instructions and altered initial frames.Source: §6 · p. 9
  • Because not all interactions are fully realistic, WorldGym is positioned as an important step toward—not a replacement for—safe and reproducible evaluation before deployment.Source: §6 · p. 9

A ADDITIONAL DETAILS OF THE AUTOREGRESSIVE DIFFUSION TRANSFORMER

WorldGym trains an action-conditioned video model on diverse robot data and rolls out policies in sliding windows aligned to their action chunks. A VLM then scores generated videos using binary or partial-credit success rubrics.

A ADDITIONAL DETAILS OF THE AUTOREGRESSIVE DIFFUSION TRANSFORMER

  • The world model trains on nine unified robot datasets with 256×256 frames, seven-dimensional actions, and a 20-frame context window.Actions combine six-dimensional end-effector position with binary gripper state; longer rollouts use a sliding window.Source: A ADDITIONAL DETAILS OF THE… · p. 16
  • At inference, prediction blocks use horizon h equal to each policy’s action-chunk size, enabling efficient sliding-window rollouts.The algorithm predicts each block in parallel, concatenates generated observations with history, and continues until rollout completion.Source: A ADDITIONAL DETAILS OF THE… · p. 16

B.1 PROMPT FOR VLM AS REWARD

  • The VLM reward prompt asks GPT-4o to judge task completion from rollout frames and instructions, returning an exact score of 0, 0.5, or 1 when partial credit applies.Without partial credit, the rubric uses only failure and success scores of 0 or 1.Source: §B.1 · p. 17
  • Partial-credit rubrics distinguish little progress, partial completion, and full completion when tasks are not completed end-to-end.The prompt explicitly treats full completion as success even when partial-credit criteria also apply.Source: §B.1 · p. 17

B.2 VALIDATING VLM SUCCESS PREDICTIONS

The authors validate GPT-4o as a success evaluator by comparing its judgments on RT-1 rollout videos with ground-truth task labels. GPT-4o achieves high true-positive and true-negative rates, especially avoiding false-positive success judgments.

B.2 VALIDATING VLM SUCCESS PREDICTIONS

  • GPT-4o achieves high true-positive and true-negative rates on RT-1 videos labeled with ground-truth task success.The evaluation averages performance and standard error across four runs.Source: Table 3 · p. 17, §B.2 · p. 17
  • Low false-positive rates make GPT-4o useful for avoiding overestimation of policy value during evaluation.A false positive occurs when a failed rollout is judged successful.Source: Table 3 · p. 17, §B.2 · p. 18

C ARCHITECTURE AND TRAINING DETAILS OF VIDEO BASED POLICY

The video-based policy combines language-conditioned video prediction with inverse dynamics to generate actions from task instructions. Its validation visualization predicts nine future frames from an instruction encoding and an initial frame.

C ARCHITECTURE AND TRAINING DETAILS OF VIDEO BASED POLICY

  • The policy follows UniPi by combining a language-conditioned video prediction model with an inverse dynamics model.The video model uses frozen UMT5-xxl instruction embeddings with adaptive layer normalization, while inverse dynamics predicts actions from ten-frame clips.Source: C ARCHITECTURE AND TRAINING DETAILS OF… · p. 19

C.1 VALIDATION VISUALIZATION OF LANGUAGE CONDITIONED VIDEO GENERATION MODEL

  • The language-conditioned video model takes a UMT5-xxl instruction encoding and an initial frame, then predicts the next nine frames.Figure 14 shows validation visualization on BridgeV2.Source: Figure 14 · p. 19

D ARCHITECTURE AND TRAINING DETAILS OF DIFFUSION POLICY

The diffusion policy adapts a pretrained Qwen2-VL-2B backbone to BridgeV2 using LoRA and a diffusion control head. Additional real-robot evaluations show that the world model generally follows varied robot actions and gripper controls.

D ARCHITECTURE AND TRAINING DETAILS OF DIFFUSION POLICY

  • DexVLA adaptation trains LoRA matrices and the policy head on BridgeV2 while preserving the pretrained Qwen2-VL-2B backbone.Actions are rescaled to (−1, 1), and the control head uses DDPM training with DDIM sampling.Source: D ARCHITECTURE AND TRAINING DETAILS OF… · p. 20

E.1 ADDITIONAL RESULTS ON REAL-ROBOT VIDEOS

  • The world model generally generates videos resembling original real-robot videos when conditioned on the same actions.This qualitative evaluation covers actions from different robots.Source: Figure 15 · p. 21
  • Across left-right, forward-backward, and gripper open-close controls, generated videos generally follow the simulated end-effector actions.The sweep evaluates separate gripper-control dimensions on Bridge.Source: Figure 16 · p. 21

E.2 ADDITIONAL RESULTS ON GOOGLE ROBOT

On Google Robot, OpenVLA outperforms RT-1-X and Octo, though by a smaller margin than on Bridge; detailed evaluations also show robustness and dataset-size effects.

E.2 ADDITIONAL RESULTS ON GOOGLE ROBOT

  • OpenVLA outperforms RT-1-X and Octo on Google Robot, but by a smaller margin than on the Bridge dataset.A partial score of 0.5 is assigned when the robot attempts to reach the target location.Source: Table 4 · p. 22, §E.2 · p. 22

E.4 DETAILED RESULTS ON OOD IMAGE EVALUATION TASKS

  • OpenVLA appears more robust across object generalization, distractions, and classification settings in detailed Bridge OOD image evaluations.Source: Table 6 · p. 24
  • All policies suffer significant performance drops when unrelated distractors are present, although OpenVLA is more robust than RT-1-X and Octo.Source: Table 7 · p. 24

F.1 DATASET SIZE ANALYSIS

  • Larger training datasets improve MSE, LPIPS, and SSIM when generated videos are compared with ground-truth validation videos.Source: §F.1 · p. 25, Table 8 · p. 25

F.2 PARALLELISM EFFICIENCY ANALYSIS

WorldGym improves rollout efficiency by matching its parallel denoising horizon to each policy’s action-chunk size.

F.2 PARALLELISM EFFICIENCY ANALYSIS

  • Increasing the horizon from 1 to 4 frames achieves a 2.8× speedup for generating rollouts.Matching the horizon to policy chunk sizes supports policies such as single-action OpenVLA and four-action Octo.Source: §F.2 · p. 25, Table 9 · p. 25
Loading 2506.00613v3…