PaperlayerView exact revision on arXiv

Source-linked AI summary

Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test

Chun-Kai Fan, Xiaowei Chi, Xiaozhu Ju, Hao Li, Yong Bao, Yu-Kai Wang, Lizhang Chen, Zhiyuan Jiang, Kuangzhi Ge, Ying Li, Weishi Mi, Qingpo Wuwu, Peidong Jia, Yulin Luo, Kevin Zhang, Zhiyuan Qin, Yong Dai, Sirui Han, Yike Guo, Shanghang Zhang, Jian Tang

arXiv:2601.04137v1cs.ROcs.AIcs.CV

TL;DR

Embodied world models need to preserve human-perceived fidelity while serving as robust priors for real-world agents, but current evaluations do not fully assess those needs. WoW-World-Eval tests five capabilities using curated robot-manipulation data and a multi-metric protocol. The results show strong perceptual performance can coexist with weaknesses in planning and real-world execution.

  • Problem

    Existing evaluations leave open whether video models maintain human-perceived fidelity and robustly support real-world embodied agents, while conventional video metrics can misalign with human judgments.

    Source: §1 · pp. 1–2

  • Method

    WoW-World-Eval benchmarks five embodied abilities on robot-manipulation data with 22 metrics, alongside human-preference and inverse-dynamics tests.

    Source: §1 · pp. 2–3

  • Results

    Models show perceptual fidelity but struggle with long-horizon reasoning and real-world execution, often collapsing under the IDM Turing Test.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The evaluation exposes a substantial gap between visual generation and embodied action in the tested models.

    Source: §5 · p. 10

    Planning scores correlate only moderately with human ratings (r = 0.43, ρ = 0.51), reflecting difficulty in automatically evaluating long-horizon reasoning and current models’ limited planning ability.

    Source: §10.2 · p. 24

Abstract

from arXiv · show

As world models gain momentum in Embodied AI, an increasing number of works explore using video foundation models as predictive world models for downstream embodied tasks like 3D prediction or interactive generation. However, before exploring these downstream tasks, video foundation models still have two critical questions unanswered: (1) whether their generative generalization is sufficient to maintain perceptual fidelity in the eyes of human observers, and (2) whether they are robust enough to serve as a universal prior for real-world embodied agents. To provide a standardized framework for answering these questions, we introduce the Embodied Turing Test benchmark: WoW-World-Eval (Wow,wo,val). Building upon 609 robot manipulation data, Wow-wo-val examines five core abilities, including perception, planning, prediction, generalization, and execution. We propose a comprehensive evaluation protocol with 22 metrics to assess the models' generation ability, which achieves a high Pearson Correlation between the overall score and human preference (>0.93) and establishes a reliable foundation for the Human Turing Test. On Wow-wo-val, models achieve only 17.27 on long-horizon planning and at best 68.02 on physical consistency, indicating limited spatiotemporal consistency and physical reasoning. For the Inverse Dynamic Model Turing Test, we first use an IDM to evaluate the video foundation models' execution accuracy in the real world. However, most models collapse to $\approx$ 0% success, while WoW maintains a 40.74% success rate. These findings point to a noticeable gap between the generated videos and the real world, highlighting the urgency and necessity of benchmarking World Model in Embodied AI.

1. Introduction

Embodied world models must combine perceptual fidelity with robust, actionable predictions, yet existing evaluations often miss the physical plausibility and planning demands of robotics. WoW-World-Eval benchmarks these capabilities and pairs human preference assessment with a real-world execution test.

1. Introduction

  • Existing video benchmarks often emphasize visual fidelity or coarse task success, while embodied models also require physical plausibility, planning, and actionability.Conventional video-quality scores can misalign with human judgments in embodied settings.Source: §1 · pp. 1–2
  • WoW-World-Eval evaluates embodied world models across five abilities and five metric dimensions using about 609 annotated robot-manipulation samples and 22 metrics.The abilities are perception, planning, prediction, execution, and generalization; metric dimensions include video quality, instruction understanding, planning reasoning, physical law, and execution accuracy.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • A two-alternative forced-choice human test measures whether generated videos fool evaluators, and its overall score aligns with human preference at Pearson correlation above 0.93.The protocol is presented as a proxy for a Human Turing Test.Source: §1 · p. 3
  • The Inverse Dynamics Model Turing Test checks whether generated videos produce plausible actions executable by a real-world-trained model.The test uses an IDM trained only on real-world execution sequences.Source: §1 · p. 3

2. Related work

Prior video-generation and physical-reasoning benchmarks cover selected evaluation needs, while holistic world-model benchmarks leave important robotics capabilities unmeasured. WoW-World-Eval adds a robotics-centric benchmark design that explicitly covers planning reasoning and execution accuracy.

2. Related work

  • Existing benchmark families target general video quality, physical reasoning, or holistic world-model evaluation, rather than one shared evaluation scope.Table 1 marks whether benchmarks explicitly include core dimensions and metric categories.Source: §2 · p. 3, Table 1 · p. 3
  • Video-generation benchmarks assess visual fidelity, temporal coherence, compositionality, controllability, anticipation, or intrinsic faithfulness.Examples include TC-Bench [29], T2V-CompBench [78], and VBench-2.0 [97].Source: §2 · p. 3
  • Physical-reasoning benchmarks evaluate physical plausibility through diagnostics, semantic and physical assessments, question answering, or rule-based and learned evaluators.The cited examples are Physics-IQ [61], PhyGenBench [59], PhysBench [18], and Videophy [4].Source: §2 · p. 3
  • WorldModelBench [46] and EWMBench [95] cover dimensions such as perception, prediction, and generalization, but omit planning and execution.WoW-World-Eval adds robotics-domain metrics for planning reasoning and execution accuracy.Source: §2 · p. 3, Table 1 · p. 3

3. WoW-World-Eval: A Multi-faceted Benchmark for Embodied World Models

WoW-World-Eval frames embodied world-model assessment as conditional video generation from an initial image and instruction. It combines a five-capability framework, curated robot data, targeted metrics, and a common scoring procedure.

3. WoW-World-Eval: A Multi-faceted Benchmark for Embodied World Models

  • The benchmark evaluates conditional video generation from an initial image and instruction, probing instruction understanding, planning, perception, and physically grounded dynamics.Its design treats five capability dimensions as fundamental and orthogonal.Source: §3 · p. 4, §3.1 · p. 4

3.1. Core Evaluation Dimensions

  • Perception, planning, prediction, execution, and generalization are assessed through distinct tasks, including object and spatial recognition, long-horizon plans, physical futures, robot actions, and OOD inputs.Planning evaluation includes 25 samples; execution uses nine in-house tasks compatible with real-robot experiments.Source: §3.1 · pp. 4–5

3.2. Data Curation Pipeline

  • The 609-sample dataset combines open-source robotics data, in-house trajectories, and AI-generated OOD samples, with GPT-4o filtering followed by human verification and annotation.Each entry includes an instruction, initial image, ground-truth video, and annotated keypoints; prediction and perception account for 50.57% and 40.89% of samples, respectively.Source: §3.2 · p. 5

3.3. Multi-faceted Evaluation Metrics

  • The metrics cover visual fidelity, instruction alignment, physical consistency and causal reasoning, and planning; the planning score combines DAG node correctness with task completion.Instruction scoring uses GPT-4o, while physical evaluation includes regional and trajectory consistency measures and fine-tuned physical-common-sense scoring.Source: §3.3 · pp. 5–6

3.4. Overall Benchmark Score

  • Overall scoring maps raw metrics through absolute pre-scaling and monotone transformations, then averages available metrics within groups and combines groups using normalized weights.The protocol specifies higher-is-better and lower-is-better pre-scaling and permits an unweighted mean across available groups.Source: §3.4 · p. 6

4. Experiments

The experiments compare video-generation models across embodied capabilities, test prompt enrichment, and relate benchmark scores to human judgments and real-world executability. Results show balanced benchmark performance and human alignment, alongside persistent planning and execution gaps.

4.1. Quantitative Evaluation Results

  • Hailuo leads overall at 52.55, WoW-cosmos2 leads open source at 50.74, while Hailuo's Planning DAG remains only 17.27.Planning Reasoning scores are markedly lower and more compressed than other dimensions.Source: §4.1 · pp. 7–8
  • WoW-cosmos2 scores 70.36 in Instruction Understanding, narrowly surpassing Hailuo at 70.11, while WoW-cosmos1 leads Seq. Match at 63.33.Hailuo leads Caption Score (88.78) and Exec. Quality (70.48); WoW-cosmos2 is described as more procedurally consistent across multi-step instructions.Source: §4.1 · p. 8
  • Kling and Hailuo lead Physical Law overall at 68.02 and 66.72, while WoW-cosmos2 leads open source at 66.18.WoW-cosmos2 has the strongest Robot Trajectory score, Hailuo the strongest Object Trajectory score, and camera motion is broadly stable across models.Source: §4.1 · p. 8

4.2. Dense prompts quantitative results

  • Dense prompts improve dimensions broadly: WoW-cosmos1 gains +7 in Physical Law, while Planning Reasoning's DAG metric changes minimally.In Video Quality, the best embodied world model scores 60.55 versus the closed-source baseline's 56.09.Source: §4.2 · p. 8, Table 4 · p. 8

4.3. Qualitative Evaluation and Human Evaluation

  • Qualitative outputs show visual artifacts, unintended camera motion, task failures, and physics violations; WoW-cosmos2 videos are described as more likely to be indistinguishable from reality.The reported qualitative comparison is consistent with the quantitative failure patterns.Source: §4.3 · pp. 8–9
  • Among 15 experts rating over 1,200 videos, benchmark Overall Score correlates with human judgment (Pearson r = 0.93, Spearman ρ = 0.91).Figure 3b reports the metric–preference correlation.Source: Figure 3 · p. 9, §4.3 · p. 9

4.4. Turing Test

  • Higher benchmark scores track higher deceive-human ratios (r = 0.679); Video Quality (r = 0.874) and Physical Law (r = 0.753) have the highest dimension correlations.The authors conclude that both visual quality and physical plausibility matter for generated videos to appear real to human observers.Source: §4.4 · p. 9, Figure 4 · p. 9
  • Real-world IDM success remains low for high scorers (Kling 9.88%, Hailuo 2.47%), while WoW-wan reaches 40.74% and WoW-cosmos2 18.52%.The evaluation covers nine manipulation tasks; the table reports execution accuracy on real-world tasks.Source: Table 5 · p. 10, §4.4 · p. 10

5. Conclusion

The paper presents WoW-World-Eval as a dual-perspective benchmark for perceptual realism and physical executability. Experiments identify a gap between video fidelity and embodied reasoning or action.

5. Conclusion

  • WoW-World-Eval combines fine-grained human preference assessment with an Inverse Dynamics Model test to evaluate perceptual realism and physical executability.The benchmark uses 609 real-world robot manipulation data.Source: §5 · p. 10
  • Some models achieve high perceptual fidelity but struggle with long-horizon reasoning and real-world execution, often collapsing under the IDM test.Source: §5 · p. 10
  • The authors identify a substantial gap between visual generation and embodied action and call for more physically grounded, generalizable world models for robotics.Source: §5 · p. 10

7. More Related Works

Related work spans video-generation architectures and predictive models for embodied agents. The review contrasts latent-state and pixel-space approaches with generalist simulators built for broad generalization.

7. More Related Works

  • Autoregressive video models predict discrete visual-token sequences, aligning with world-model state transitions but limiting fine visual detail through quantization.VideoGPT [89] uses VQ-VAE [82], while Genie [8] applies the paradigm to interactive environments and predictive models.Source: §7 · p. 16
  • Diffusion Transformers model unified spacetime patches; recent systems are presented as candidates for high-fidelity simulation of dynamics and occlusion.Examples include Wan [83], Cosmos, and Sora.Source: §7 · p. 16
  • Embodied world models predict future states conditioned on actions; prior approaches trade compact latent-state efficiency against pixel-space visual detail and long-horizon prediction stability.Dreamer [36] supports efficient RL planning, whereas RoboNet [20] predicts frames from interaction logs but can produce blurry long-horizon predictions.Source: §7 · p. 16
  • Generalist simulators such as UniSim [93] and GAIA-1 [38] use internet-scale data to learn high-level physics for zero-shot generalization.Source: §7 · p. 16

8. Models Detail

The evaluated set includes commercial, open-source, and embodied world models, with versions and generation settings specified for comparison. WoW is distinguished by its robotics-focused training and physically grounded prediction target.

8. Models Detail

  • The paper specifies model versions, parameter counts, durations, and resolutions to facilitate comparison and reproduction.Source: §8 · p. 16
  • Klingv2.1 is evaluated with 5-second 720p generation, while Hailuo-02 is evaluated at 5 seconds and 768p.Kling is described as a latent-space DiT with 3D VAE compression; Hailuo is a commercial text-to-video and image-to-video model.Source: §8 · p. 16
  • CogVideoX-I2V-5B [94] generates 5-second 1360x768 videos; Cosmos-Predict1 and Predict2 [1] generate 5-second 720p videos.The passage gives Cosmos-Predict1 and Predict2 parameter counts as 7B and 2B, respectively.Source: §8 · p. 16
  • Wan2.1-I2V-14B [83] is evaluated at 5 seconds and 832x480 resolution; the model family uses 3D VAE compression and DiT backbones.Source: §8 · pp. 16–17
  • WoW [17] is a 14B-parameter embodied model for robotic manipulation, conditioning future video rollouts on visual observations and task instructions.Its inductive biases include DINOv2 token distillation, and its target capabilities include planning, contact dynamics, and multi-step execution.Source: §8 · p. 17

9. Detailed Metrics

This section presents complementary metrics for visual fidelity, instruction semantics, regional and trajectory consistency, and physical and causal reasoning, alongside a physical-reasoning evaluator and an overall benchmark scoring scheme. The scheme normalizes metrics and calibrates their mappings against human ratings.

9.1. Visual Fidelity.

  • Visual fidelity is assessed with FVD, SSIM, PSNR, DINO, and DreamSim, spanning reconstruction accuracy, perceptual similarity, and distributional realism.DreamSim combines CLIP, OpenCLIP, and DINO embeddings and is fine-tuned on human perceptual judgments.Source: §9.1 · p. 17

9.2. Instruction Semantic Correctness.

  • Caption Score, Sequence Match Score, and Execution Quality Score each show strong agreement with human preferences across supervised and fully OOD evaluation settings.Caption Score averages VLM judgments over five caption components; Sequence Match Score assesses action–object selection and order; Execution Quality Score rates action completion on a five-level rubric.Source: §9.2 · pp. 17–18

9.3. Mask-guided Regional Consistency.

  • Mask-guided regional consistency tracks object, gripper, and background features across frames, recording an all-zero mask when a region is occluded or blurred.GroundingSAM-2 tracks point-prompted masks, DINOv3-Large extracts patch features, and RLE stores masks; the protocol reports separate temporal consistency scores for the three regions.Source: §9.3 · pp. 18–19

9.4. Trajectory-level Consistency.

  • Trajectory consistency compares tracked end-effector and object paths using L2Norm Error, DTW, and Fréchet Distance; smaller camera ATE/RPE indicate more reliable corrected trajectories.The evaluation subtracts estimated camera motion to reduce viewpoint drift or jitter in generated videos, while larger camera errors add evaluation noise and indicate a need for improved generation or stabilization.Source: §9.4 · p. 19

9.5. Physical and Causal Reasoning.

  • Held-out video-understanding accuracy rose from 60.83% to 71.51% after Stage 1 fine-tuning of Qwen-2.5-VL 7B.Stage 1 used approximately 50,000 multiple-choice VQA samples from six video-understanding benchmarks, with group size G = 8 and rule-based rewards for matching ground-truth answers.Source: Stage 1: Foundational Video… · p. 21
  • The final evaluator uses a two-stage fine-tuned model and a zero-shot prompt focused on physical plausibility across six dimensions.The model combines physical common-sense understanding from Stage 1 with structured scoring ability from Stage 2.Source: Final Inference for Evaluation · p. 22

9.6. Overall Benchmark Score.

  • Metrics are normalized to 0–100; FVD and PSNR are bounded at 2000 and 50, respectively, before monotonic transformation and rescaling.Metrics already in the 0–100 range are retained as-is.Source: §9.6 · p. 22
  • Each metric’s single-parameter monotone mapping is selected on development data to maximize Pearson correlation with human ratings, with Spearman correlation as a tie-breaker, then frozen.Selection uses a grid of parameter values and Fisher-z-averaged Pearson correlation across K-fold cross-validation.Source: §9.6 · p. 22

10. Extended Experiment Analysis

The appendix specifies equally weighted human ratings and finds they track human preferences strongly, especially for physical law, while planning remains harder to score; ground-truth replay also validates GC-IDM.

10.1. WoW-World-Eval Human Evaluation Rule

  • Four independent 1–5 ratings cover video quality, instruction understanding, physical law, and planning; their equal-weight sum ranges from 4 to 20.Video quality concerns perceptual integrity, instruction understanding judges the final outcome, physical law assesses physical plausibility, and planning assesses action-sequence coherence.Source: §10.1 · pp. 22–24

10.3. Ground-truth video replay of IDM

  • GC-IDM achieves 90% overall replay accuracy across nine manipulation tasks and outperforms ResNet-based inverse dynamics and AVDC on the four displayed tasks.The authors use this ground-truth replay accuracy to support GC-IDM as a physical-plausibility evaluator; they attribute its failures on generated videos to deficiencies in video realism.Source: Table 7 · p. 25, §10.3 · p. 25

11. Case Results Visualization

The section presents additional generated results from different models for comparison.

11. Case Results Visualization

  • Figures 8–10 visualize additional generated results from different models for comparison.Source: §11 · p. 25

12. Prompts

This section documents prompts for selecting and classifying manipulation data, evaluating generated videos across execution, instruction following, video quality, planning, and physical plausibility, and producing standardized dense episode descriptions. The prompts emphasize evidence grounded in visible content and avoid unsupported scene details.

12. Prompts

  • The GPT data-selection prompt classifies samples using a task instruction and first-frame image, requiring visible evidence and excluding dimensions when evidence is insufficient or ambiguous.It requests JSON with dimension assignments, confidence scores, and a rewritten instruction for Perception dimensions.Source: §12 · p. 25
  • Perception includes object-centric attributes such as color, number, shape, size, and type, with paths specifying the selected attribute and rewrites expressed in attribute-based terms.Perception instructions are rewritten using attributes, functions, spatial relations, or affordances; Prediction and Planning retain the original instruction.Source: §12 · pp. 25–26

Instruction Semantic Correctness: Extract Video Semantic Caption

  • The semantic-caption prompt extracts five aspects—initial state, processing state, final state, action, and object—from the instruction and available ground-truth video.Descriptions must be objective and video-grounded; unclear or invisible aspects are marked “Unknown,” without assumptions or imagined information.Source: Instruction Semantic Correctness:… · pp. 29–30

Instruction Semantic Correctness: Compare Video Semantic Caption

  • Generated and ground-truth captions are compared across five dimensions: initial state, processing state, final state, action, and manipulated object.The comparison focuses strictly on caption semantics and avoids inferring unstated actions or states.Source: Instruction Semantic Correctness:… · p. 30
  • Caption consistency is scored as 1 for fully consistent, 0.5 for partially consistent, and 0 for inconsistent descriptions.The overall score may average the five dimensions while also considering whether task intent and outcome remain aligned.Source: Instruction Semantic Correctness:… · p. 30

Instruction Semantic Correctness: Evaluate Video Action Execution

  • The action-execution prompt compares intended and observed object-action sequences, scoring the fraction of instruction pairs executed in the same relative order.Missing or out-of-order pairs reduce the sequence-match score, which ranges from 0.00 to 1.00.Source: Instruction Semantic Correctness:… · p. 31
  • The action-execution evaluation also includes a separate numerical assessment of execution quality for each object-action pair.The supplied prompt excerpt introduces this second part but does not provide its scoring criteria.Source: Instruction Semantic Correctness:… · p. 31
  • Execution quality uses a conservative 1–5 scale based strictly on visible evidence, from no attempted action or movement to both target-state achievement and successful completion.The rubric distinguishes partial movement or action attempts from successful completion.Source: Instruction Semantic Correctness:… · p. 31
  • The execution prompt requests an instruction-sequence match score in the 0.00-1.00 range and execution-quality scores from 1 to 5 for object-action pairs.Source: Instruction Semantic Correctness:… · p. 31

Physical Common Sense: Stage 2: Scoring Alignment Fine-tuning Prompt

  • The four-dimension fine-tuning rubric rates video quality and instruction following from 1 to 5, spanning severe defects or unrelated content to clear video and prompt fidelity.Video-quality criteria include stability and artifacts; instruction-following criteria judge whether core elements and details match the prompt.Source: Physical Common Sense: Stage 2:… · pp. 31–32
  • The same rubric scores physical consistency and planning logic from 1 to 5, assessing interaction anomalies and whether actions form an efficient, successful plan.Its criteria range from chaotic interactions or actions to physically consistent behavior and goal achievement without redundant movements.Source: Physical Common Sense: Stage 2:… · p. 32

Physical Common Sense: 6-Dimension Evaluation Prompt

  • The six-category evaluation judges physical plausibility rather than whether a video is AI-generated, assigning each category a score from 1–5, or null when no relevant video content applies.Source: Physical Common Sense: 6-Dimension… · p. 33
  • The dense-prompts extension asks for standardized robot-episode captions of no more than 150 words to clarify ambiguous actions and unify phrasing across datasets.The prompt calls for a richer, more precise narrative using technical terminology where appropriate.Source: Physical Common Sense: 6-Dimension… · p. 34
  • The caption structure requests the episode goal, environment and ambient conditions, robot characteristics, and camera perspective and motion.Robot details include appearance and gripper type when relevant or inferable.Source: Physical Common Sense: 6-Dimension… · p. 34
  • The prompt fragment includes camera-motion guidance, mentioning following the action, smooth tracking, and handheld jitter when applicable.Source: Physical Common Sense: 6-Dimension… · p. 34
  • The action breakdown asks the model to state the robot’s immediate sub-goal and describe granular actions using standardized terminology.It also asks the recaption to identify manipulated objects and changes to their state or location.Source: Physical Common Sense: 6-Dimension… · pp. 34–35
  • The prompt asks for the robot’s and manipulated objects’ states after the action sequence is complete.Source: Physical Common Sense: 6-Dimension… · p. 35
  • The prompt limits recaptions to approximately 100–150 words of factual, observable details, excluding invented or out-of-scene elements.It also calls for consistent action and object terminology and avoids subjective stylistic filler.Source: Physical Common Sense: 6-Dimension… · p. 35
  • The apple example describes a single-arm robot on a countertop, viewed from a static third-person angle, with the goal of depositing the apple.The example specifies a brightly lit kitchen and a bowl as the target.Source: Physical Common Sense: 6-Dimension… · p. 35
  • The example sequence moves an already-grasped apple above a bowl, lowers the arm, releases the apple, and may retract the arm.The apple ends inside the bowl, with the gripper open and clear of it.Source: Physical Common Sense: 6-Dimension… · p. 35
Loading 2601.04137v1…