Source-linked AI summary
WorldEval: World Model as Real-World Robot Policies Evaluator
Yaxuan Li, Yichen Zhu, Junjie Wen, Chaomin Shen, Yi Xu
TL;DR
Real-world evaluation of generalist robot policies is difficult to scale safely and reproducibly. The paper introduces Policy2Vec and WorldEval to generate action-following videos and automate policy evaluation, reporting strong correlation with real-world performance and safety-detection capabilities.
Problem
Real-world robot-policy evaluation is time-consuming, difficult to scale across tasks and conditions, and risky for hazardous behaviors.
Method
Policy2Vec uses latent policy-network representations to condition a video model, while WorldEval automates policy evaluation with generated videos and success classification.
Results
WorldEval strongly correlates with real-world policy performance and can rank policies while identifying collapsed or hazardous actions.
Takeaways & Limitations
World models can provide scalable, reproducible, and risk-free proxy evaluations for real-world robot policies.
Takeaways & Limitations
FID is useful mainly for simple tasks, while its effectiveness decreases on intricate tasks such as Table Busing.
Abstract
from arXiv · showhide
The field of robotics has made significant strides toward developing generalist robot manipulation policies. However, evaluating these policies in real-world scenarios remains time-consuming and challenging, particularly as the number of tasks scales and environmental conditions change. In this work, we demonstrate that world models can serve as a scalable, reproducible, and reliable proxy for real-world robot policy evaluation. A key challenge is generating accurate policy videos from world models that faithfully reflect the robot actions. We observe that directly inputting robot actions or using high-dimensional encoding methods often fails to generate action-following videos. To address this, we propose Policy2Vec, a simple yet effective approach to turn a video generation model into a world simulator that follows latent action to generate the robot video. We then introduce WorldEval, an automated pipeline designed to evaluate real-world robot policies entirely online. WorldEval effectively ranks various robot policies and individual checkpoints within a single policy, and functions as a safety detector to prevent dangerous actions by newly developed robot models. Through comprehensive paired evaluations of manipulation policies in real-world environments, we demonstrate a strong correlation between policy performance in WorldEval and real-world scenarios. Furthermore, our method significantly outperforms popular methods such as real-to-sim approach.
1 Midea Group 2 East China Normal University worldeval.github.io
WorldEval is presented as an adaptive approach for evaluating real-world policies across diverse tasks, with success rates that correlate strongly between simulation and real robots.
- WorldEval evaluates real-world policies across diverse tasks.
- Its world-simulator success rates strongly correlate with success rates observed using real robots.
- The paper lists equal contribution and corresponding-author designations among its authors.
1 Introduction
The paper addresses the difficulty of scaling, comparing, and safely conducting real-world evaluation of generalist robot policies. It proposes Policy2Vec and WorldEval, which use policy representations and generated videos to evaluate and rank policies through a world simulator.
- Real-world evaluation becomes impractical as robot policies cover many tasks, checkpoints, and changing environments.
- World models are proposed as scalable, reproducible complements to traditional real-world assessments.
- The paper asks whether a world simulator can automatically evaluate and rank manipulation policies trained on real-world data.
- Direct action inputs and explicit high-dimensional action encoders face ambiguity in manipulation actions and difficulty representing unseen objects and environments.
- Policy2Vec uses latent representations from the policy network itself as action representations for generated policy videos.
- WorldEval experiments report strong correlation between simulated policy performance and real-world performance across multiple public manipulation policies.
2 Related Work
Related work spans real-world, simulation-based, and automated robot-policy evaluation, action representation, and robotic world models. WorldEval uses the policy network itself as an action representation within a video-based evaluation pipeline.
- Evaluation for robot policies: Existing robot-policy evaluation methods include real-world testing, simulation-based evaluation, and automated laboratory evaluation.
- Robot action representation: Robot action representations include discretization, function approximation, latent spaces, and language-token sequences.
- World model for robotics: World models predict future states and outcomes from learned action representations and have been applied to robot navigation and manipulation.
- Imitation learning for robotic control: Imitation-learning and vision-language-action methods use large-scale datasets and powerful models to learn robotic tasks.
- WorldEval pipeline: WorldEval's architecture extracts Policy2Vec embeddings from robot policies and injects them into a pretrained video-generation model for evaluation.
3 WorldEval: World Simulator for Evaluation of Robot Manipulation
WorldEval evaluates policies by comparing relative performance in generated videos with real-world behavior rather than reproducing every real-world action exactly. Its pipeline uses latent policy actions to condition video generation and an automated success classifier to score trials.
- WorldEval targets consistent relative performance rankings between generated videos and real-world evaluations, not exact behavioral replication.
- Policy2Vec: Policy2Vec addresses the challenge of building a world simulator that accepts robot actions and generates corresponding realistic videos.
- Latent action embedding: Latent action embeddings extracted before a policy's action decoder encode intended robot behavior.
- Latent action embedding: The latent embeddings, language instruction, and initial frame condition a fine-tuned video-generation model to produce future robot frames.
- Integrating latent action: Policy latent representations are projected, scaled by learnable weight α, added to language embeddings, and fed into the DiT module.
- Automated policy verifier: Gemini-2.0 classifies generated-video trial success as a practical automated verifier approximating the ground-truth success function.
4 Experiments
WorldEval is evaluated across real-world robotic tasks and policies as a proxy for policy performance, with experiments testing correlation, simulation alternatives, action embeddings, generalization, and safety detection.
- Experimental setup: Experiments evaluate WorldEval across multiple real-world robotic manipulation policies and tasks, using paired real-world and generated-video assessments.The study includes over 1,000 real-world trials and evaluates four open-source policies across experimental tasks.
- Comparison with real-to-sim evaluation: WorldEval achieves the lowest MMRV and highest Pearson r, outperforming real-to-sim despite matched tasks and objects and minimized visual discrepancies.The comparison uses RoboTwin with an embodiment identical to the real robot and applies SIMPLER-inspired techniques for visual alignment.
- Action embedding ablation: Policy2Vec outperforms VQVAE and one-hot action encodings across reported metrics, reducing MMRV by 0.1 and 2.24 relative to those methods.Table 2 reports Pearson correlation, MMRV, FID, and eventual rank across five tasks.
- Policy-video faithfulness and generalization: Generated videos closely align with real robot actions for both successful and failed π0 tasks, including the novel Collect Toy scenario.Collect Toy uses an unseen instruction, object, and task configuration, yet the generated video mirrors the real robot’s behavior.
- Safety detection: WorldEval identifies hazardous collapsed actions by producing meaningless mosaic videos when a policy generates unsafe behavior.A DexVLA example involves the robot unexpectedly raising both arms, while the corresponding generated video collapses into a mosaic pattern.
- Metric limitations: FID is useful for preliminary evaluation in low-complexity tasks but becomes less effective for intricate tasks such as Table Busing.For complex tasks, the Pearson correlation coefficient r decreases significantly, indicating domain-specific limitations.
5 Conclusion
WorldEval combines Policy2Vec with automated real-to-video evaluation to assess robot policies without physical deployment. It correlates with real-world performance, ranks policies and checkpoints, detects unsafe behavior, and generalizes to novel environments and objects.
- WorldEval uses Policy2Vec to generate action-following policy videos from latent representations of manipulation policies.The approach avoids explicit action encoding or high-dimensional action inputs.
- WorldEval provides scalable, reproducible, and risk-free evaluation of real-world robot policies.
- WorldEval correlates strongly with real-world performance across diverse manipulation policies.
- WorldEval ranks policies, identifies checkpoint improvements, and detects unsafe behaviors without physical deployment.
- WorldEval outperforms conventional real-to-sim approaches and generalizes to novel environments and objects.
A Limitations
WorldEval remains constrained by its reliance on action outputs and pretrained generative models, while generated videos can contain artifacts, especially for low-performing policies.
- WorldEval currently uses only policy action outputs as control signals, excluding language instructions and intermediate reasoning.Adding these modalities may better bridge simulated evaluation and real-world performance.
- Generated videos can show object deformation, visual hallucinations, unnatural dynamics, overexposure, and robotic-arm ghosting.These artifacts are especially pronounced when evaluating low-performing policies.
- WorldEval’s reliance on pretrained generative models may limit adaptability in significantly novel tasks and environments.The paper identifies architectural refinements and additional fine-tuning as possible directions for improving robustness.
B More Experiments
Additional experiments examine checkpoint scaling, distribution shifts, and unseen environments. WorldEval maintains strong video-reconstruction and real-world correspondence across several novel settings, while FID is most suitable for simpler tasks.
- Checkpoint analysis: WorldEval task success rates improve as policy training progresses, while FID decreases across checkpoints.The paper links these changes to differences in latent action distributions and reports strong FID correlation with real-world success on simple tasks.
- Distribution shifts: WorldEval maintains robust video reconstruction and strong real-to-video versus real-world correlation for unseen objects, instructions, and tasks.Collect Toy combines an unseen object, unseen instruction, and new task configuration.
- Unseen environments: MMRV is 0.047 across office, living-room, and kitchen environments versus 0.044 in the laboratory, with Pearson r of 0.927.The tests cover placing a cup, striking a block, and handing over a block in unseen backgrounds.
C More on Experimental Setup
The experimental setup trains a video-generation model on real robot trajectories and evaluates policies across multiple manipulation tasks. Comparisons include Robotwin-based real-to-sim evaluation and WorldEval-generated policy videos.
- Training setup: The video-generation model is trained on 1,400 real-world robot trajectories for policy-video synthesis.Training uses WAN2.1 14B with 81-frame, 480 × 480 samples and 50 diffusion steps per generated video.
- Video comparison: WorldEval-generated policy videos are compared with real-world policy videos for successful and failed tasks.
- Real-to-sim comparison: Robotwin evaluation uses an embodiment identical to the real robot and covers StrikeBlock, HandoverBlock, and PlaceCup.Policies trained on real-world data are evaluated in the RoboTwin simulation environment.
- Data collection: WorldEval evaluates policy networks trained on data collected at different frequencies.
- Task design: The experiments include Bussing Table, Collect Toy, Place Cup, Handover Block, and Strike Block.These tasks span sorting and disposal, novel toy collection, cup placement, block handover, and block striking.
D More Visualization
This section provides additional visualizations of real-world images and generated videos, covering both successful and failed cases. It also examines whether data collection frequency affects generated-video quality.
- Figures 8 and 9 present success and failure cases, respectively, for real-world images and generated videos.
- The data-frequency analysis compares datasets collected at 10Hz and 50Hz using separate policy networks trained on equal numbers of trajectories.A single WorldEval model is used for evaluation.
- The section investigates whether collection-frequency differences affect generated-video quality and reports that WorldEval effectively captures the resulting differences.