RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

1 Introduction

• Large-scale video generation has produced temporally coherent, visually plausible futures [16] [49].These predictions are a potential source of scalable supervision for robot learning [9] [23] [45].Source: §1 · p. 1
• Embodied manipulation requires generated interactions to respect physical constraints and support executable control.Robotics-oriented video world models aim to better capture robot–environment interactions and embodiment-aware dynamics [1] [2] [9] [12] [21] [46].Source: §1 · p. 1
• Existing benchmarks assess visual fidelity, semantic consistency, and temporal coherence [15] [59], while newer efforts extend evaluation toward physical plausibility.These benchmarks provide diagnostics of whether generated videos are physically plausible.Source: §1 · p. 1

• RoboWM-Bench converts robotic videos through inverse dynamics and human demonstrations through pose tracking and retargeting into actions validated in high-fidelity real-to-sim environments.It covers both human-hand and robotic scenarios, diverse object dynamics and task horizons, and single-arm and bimanual interactions; it combines step-level executability checks with task-level success rates.Source: RoboWM Bench · p. 2
• Visual realism does not always translate into executable behavior: success tends to fall with complexity, especially in long-horizon tasks and deformable-object manipulation.Fine-tuning on manipulation-specific data improves executability, but physically consistent interactions remain challenging.Source: RoboWM Bench · p. 2
• Inaccurate contact prediction, unrealistic object deformation, and geometric distortions can produce dynamically infeasible actions.These recurring execution difficulties help diagnose why predicted interactions fail under embodiment constraints.Source: RoboWM Bench · p. 2
• Action-extraction accuracy and real-to-sim execution consistency analyses support the reliability of RoboWM-Bench’s evaluation findings.The benchmark addresses the cost and limited scalability of broad, reproducible physical robot validation [14].Source: RoboWM Bench · p. 2

2.1 World Models for Robotics

• Large-scale video models such as Sora, Veo, Wan [49], and Seedance [16] show visual realism and temporal coherence, motivating robotics-oriented world models and evaluation of physical consistency and control feasibility.Prior work includes video-based robot planning [3] [9], data generation [23] [46], embodied interaction prediction [12] [21] [57], and compositional video planning [62].Source: §2.1 · p. 3

2.2 Learning Robotic Actions from Video

• Video-to-action research spans visual representation pretraining, latent-action and domain-adaptation methods, inverse dynamics, human-to-robot transfer, and VLA or world-model approaches.This work uses video-to-action interfaces as evaluation tools for testing whether generated manipulation videos support successful embodied execution, not as a policy-learning objective.Source: §2.2 · p. 3

2.3 Evaluation and Benchmarks of World Models

• Existing video-world-model benchmarks assess perceptual quality and physical reasoning, but measure action-level task completion under embodied constraints less directly.VBench, EvalCrafter, and T2VEval cover visual fidelity, temporal consistency, text–video alignment, and motion coherence; PAI-Bench [61] and VBench-2.0 [60] extend evaluation toward physical reasoning and embodied capabilities.Source: §2.3 · p. 3
• RoboWM-Bench provides standardized task suites and unified execution-based evaluation across simulation-native and real-to-sim scenarios to assess physical executability and action-level consistency.LVP [9] evaluates video-conditioned planning on real-world manipulation tasks, while Wow-wo-val [14] introduces embodied protocols; dedicated hardware can limit scalable real-robot comparisons.Source: §2.3 · p. 3

3.1 Benchmark Overview

• RoboWM-Bench converts generated human-hand or robotic manipulation videos into action sequences, executes them in simulation, and evaluates whether they complete tasks.The benchmark covers simulation-native scenarios and real-to-sim reconstructions, with tasks spanning object properties, interaction regimes, and temporal horizons.Source: §3.1 · pp. 3–4

3.2 High-Fidelity Real-to-Sim Framework

• The real-to-sim framework combines open-source simulation environments with reconstructed real-world scenes to support controlled, reproducible, and scalable evaluation.It reduces dependence on specific physical setups while preserving physical dynamics.Source: §3.2 · p. 4

3.3 Embodied Video-to-Action Execution

• Separate video-to-action pathways support human-hand videos and robotic videos, which reflect complementary evaluation settings for current world models and downstream robot learning.The benchmark executes the resulting action sequences in corresponding simulation environments.Source: §3.3 · p. 4

3.3.1 Human-Centric Retargeting

• For human-hand videos, the benchmark estimates 3D hand poses and retargets them to robot end-effector position, orientation, and gripper-opening actions.Contact-relevant fingertip geometry defines pose and gripper targets, followed by trajectory smoothing and temporal denoising.Source: §3.3.1 · p. 5

3.3.2 Robot-Centric Execution

• For robotic videos, an inverse dynamics model predicts joint-space action chunks from consecutive frames, using simulation pretraining followed by adaptation on real trajectories.Simulation trajectories provide action-labeled supervision, while real-world trajectories adapt the model to real observations.Source: §3.3.2 · p. 5

3.4 Manipulation Task Suite with Diverse Complexity

• The task suite spans rigid, articulated, and deformable interactions, as well as long-horizon compositional tasks and bimanual coordination.Examples include pickup, drawer opening, towel folding, hamburger assembly, and object handover.Source: §3.4 · p. 5

3.5 Evaluation of Embodied Executability

• Executability is assessed at key action nodes and at task level, with success requiring every node check to pass and the final objective to be achieved.Key nodes represent meaningful interaction stages, such as grasping or lifting; the hierarchical protocol supports failure diagnosis as well as overall completion measurement.Source: §3.5 · pp. 5–6

4.1.1 Tasks and Environments

• The task suite includes robotic and human-hand manipulation in randomized scenes, with fixed viewpoints, standardized protocols, and 10 shared initial configurations per task.Robotic tasks use Franka arms in simulation-native or real-to-sim scenes; human-hand tasks use real scenes with visible hands, following LVP [9], because simulated human dynamics would be unfaithful.Source: §4.1.1 · p. 6

4.1.2 Baselines

• Baselines span Veo3.1, Wan2.6 [49], Wan2.2 [49], Cosmos-Predict2.5 [2], interaction-focused LVP [9], and Cosmos-Finetune trained on 50 real-world trajectories per task.Source: §4.1.2 · p. 6

4.2 Embodied Executability of Video World Models

• Execution success is higher for human-hand than robotic videos, falls with task complexity, and improves for robotic behaviors after manipulation-specific fine-tuning.The fine-tuned Cosmos-Predict2.5 variant uses 50 trajectories per task; fine-tuning reduces robot-geometry artifacts, though object localization and grasp stability remain failure factors.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Perceptual Plausibility vs. Embodied Executability

• PAI-Bench scores are near-saturated across models, whereas RoboWM-Bench distinguishes physical executability, revealing plausible-looking actions that remain infeasible.The results indicate that perceptual domain scores and embodied execution provide complementary evaluations.Source: §4.3 · p. 7, Figure 4 · p. 8

4.2 Embodied Executability of Video World Models

• Wan2.6 is the strongest overall model, yet plausible-looking videos can fail execution through unstable grasps, incorrect drawer motion, or distorted robot geometry.Examples include fingers lifting an object without a stable grasp and an apparent end-effector reach that conflicts with the robot’s true forward kinematics.Source: §4.2 · p. 7, Figure 3 · p. 8

4.4.2 Video-to-Action Replay Success

• Human-hand retargeting achieves near-perfect replay, with residual cylindrical-contact failures; IDMSim+Real exceeds IDMReal, while shallow-grasp boundary contacts remain difficult.IDMSim+Real combines large-scale simulation pretraining with fine-tuning on 50 real-world trajectories per task, providing motion priors for inverse-dynamics prediction.Source: Table 2 · p. 9, §4.4.2 · p. 9

4.4.1 Real-to-Sim Execution Consistency

• Replaying 10 successful and 10 failed real-world trajectories per task in reconstructed simulations consistently reproduced their outcomes, supporting task-relevant reconstruction fidelity.Using identical robotic action sequences keeps this test focused on reconstruction fidelity; video-to-action recovery is assessed separately.Source: §4.4.1 · pp. 8–9, Figure 5 · p. 9

5 Conclusion

• RoboWM-Bench converts predicted human-hand and robotic manipulation videos into executable actions for standardized simulation and real-to-sim evaluation.It covers diverse object dynamics, task horizons, and coordination requirements, using step-level and task-level execution metrics in a scalable, reproducible protocol.Source: §5 · p. 9
• Embodied execution reveals challenges in complex manipulation, especially precise contact, deformable-object reasoning, and long-horizon consistency.Experiments evaluate state-of-the-art video world models.Source: §5 · p. 10
• RoboWM-Bench provides an execution-grounded diagnostic framework for advancing physically consistent world models for robotic manipulation.Its execution-based assessment complements perceptual video evaluation.Source: §5 · pp. 9–10

A Limitations

• Intermediate video-to-action interfaces can introduce errors, especially in contact-sensitive or near-boundary interactions where small pose or grasp deviations change execution outcomes.The protocol validates its real-to-sim environments and action-extraction modules, but still depends on these interfaces.Source: A Limitations · p. 14
• For human-hand videos, executability is assessed through retargeted robot actions rather than direct simulation of human-hand dynamics, which current physics engines struggle to model faithfully.Source: A Limitations · p. 14
• The task suite does not exhaustively cover manipulation settings, and step-level evaluation depends on task-specific key nodes and manually specified checkers.Broader scene layouts and object categories could extend coverage; LLM- or VLM-assisted generation of success predicates and key-node checkers could aid scaling, with execution retained as final validation.Source: A Limitations · p. 14

B Additional Results for Purely Simulated Robotic Tasks

• For each task, models predict behaviors from simulation observations and task descriptions, then convert predictions into executable actions and run them in simulation.The setup follows the main paper’s protocol.Source: B Additional Results for Purely… · p. 14
• In purely simulated tasks, most models’ task- and step-level execution success rates decrease as complexity rises, particularly for long-horizon reasoning or precise contact interactions.The overall trends are consistent with those in the real-to-sim evaluation.Source: B Additional Results for Purely… · p. 14, Table 3 · p. 14

C Additional Results on Human Tasks

• Bimanual human-hand tasks are generally more challenging because they require coordinated interactions between both hands to maintain physically feasible manipulation.Detailed execution success rates are reported in Table 4.Source: C Additional Results on Human Tasks · p. 14
• The additional human-hand manipulation tasks involve bimanual coordination.Source: C Additional Results on Human Tasks · p. 14
• Detailed execution success rates for the human-hand tasks are reported in Table 4.Source: C Additional Results on Human Tasks · p. 14

D Comparison with PAI-Bench Quality Scores

• Approximately 0.78 quality scores cluster across tasks and models while RoboWM-Bench execution accuracy varies substantially, showing visual plausibility does not necessarily imply physical correctness.Figure 6 compares PAI-Bench average quality scores with RoboWM-Bench execution accuracy for human-hand and robotic tasks.Source: D Comparison with PAI-Bench Quality… · pp. 14–15, Figure 6 · p. 15

E More Visualizations

• Figure 7 adds qualitative examples showing predicted manipulation videos converted into robot actions and executed in simulation.These examples complement those in Figure 3 of the main paper.Source: E More Visualizations · p. 17, Figure 7 · p. 17

F Discussion of Depth in Human-Hand Tracking

• Because world-model videos lack ground-truth depth beyond the first frame, tested estimated absolute and aligned relative depth remained imperfect and did not improve tracking or retargeting, so the final pipeline omits depth.The relative-depth strategy aligns subsequent-frame estimates with ground-truth depth from the first frame.Source: F Discussion of Depth in Human-Hand… · p. 18, Figure 8 · p. 18

G Compute Resources

• Human-hand pose estimation and retargeting took approximately 112 seconds and 5.5 GB of GPU memory; robotic IDM inference took approximately 1.5 seconds and 4 GB.The human-hand video was 1280×720 at 15 FPS for up to 25 seconds, processed on one RTX 4090; the robotic video was 1280×720 at 30 FPS for 5 seconds, processed on one A800.Source: G Compute Resources · p. 18

H Broader impacts

• RoboWM-Bench is intended as a diagnostic evaluation protocol, with simulation or real-to-sim validation, not as sufficient evidence for real-world deployment.The paper identifies risks from overinterpreting benchmark scores or using video-to-action pipelines without safeguards; it states that no personally identifiable or sensitive data are released.Source: H Broader impacts · pp. 18–19

I Prompt Details for World Model Video Generation

• The supplement introduces details on generating instruction prompts for world-model video generation.It covers prompt generation for both human-hand and robotic manipulation tasks.Source: I Prompt Details for World Model Video… · p. 19

I.1 Human-Hand Tasks

• For human-hand tasks, Qwen3-VL-Flash generates concise prompts from initial images and task instructions; constraints on hand visibility, camera stability, unnecessary motion, and stationary objects improve video quality.The generated descriptions are summarized in Tables 9 and 10.Source: §I.1 · p. 19

I.2 Robotic Tasks

• For robotic tasks, Qwen3-VL-Flash generates concise prompts from initial images and task instructions; constraints on gripper structure, arm-joint motion, and camera perspective improve video quality.The generated descriptions are summarized in Tables 11 and 12.Source: §I.2 · p. 21

J Implementation Details of PAI-Bench Domain Score

• Each PAI-Bench suite has one scenario-dependent task-completion question and four questions identical across tasks.The benchmark provides separate targeted VQA suites for human-hand and robotic tasks.Source: J Implementation Details of PAI-Bench… · p. 23

J.1 VQA Pairs for Human-Hand Tasks

• The human-hand suite asks whether the human completes the described task and makes physical contact with its target object.Both questions use yes, no, or unclear options, with yes as the stated ground truth.Source: §J.1 · pp. 23–24
• Human-hand questions check anatomical plausibility, object structural integrity, and physically plausible object motion without sudden displacement.They flag impossible hand bends, broken fingers, extra joints, implausible object deformations, and sudden spatial displacements.Source: §J.1 · p. 24

J.2 VQA Pairs for Robotic Tasks

• The robotic suite asks whether the robot completes the described task and whether its gripper or hand contacts the target object.Both questions use yes, no, or unclear options, with yes as the stated ground truth.Source: §J.2 · p. 24
• Robotic-system plausibility questions check arm and gripper integrity, rigid-link and gripper deformation, and realistic joint rotations throughout the video.The question asks whether the system maintains a physically plausible configuration.Source: §J.2 · p. 24
• Robotic-task questions also check that relevant objects retain structural integrity and move plausibly without sudden spatial displacement.The suite flags physically implausible object deformation and motion.Source: §J.2 · p. 24