PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction

Tianyi Wang, Jiazhou Chen, Yiming Xu, Xiangyu Li, Tianyi Zeng, Chih-Hsien Chou, Ning Lu, Liang Peng, Junfeng Jiao, Christian Claudel

arXiv:2608.28718v1cs.ROcs.AIcs.CVcs.ETeess.SY

TL;DR

RoboPhys-3D addresses the lack of a unified 3D-grounded evaluation for embodied world models by comparing generated and ground-truth rollouts through shared reconstruction and execution-aware metrics. Across representative models, Cosmos 3 achieves the highest task-aligned score, while that score strongly agrees with human judgments.

  • Problem

    Existing EWM benchmarks lack a unified 3D-grounded protocol for testing scene-state agreement and action executability.

  • Method

    RoboPhys-3D evaluates 50 manipulation tasks using shared reconstruction, 50 metrics across 18 sub-dimensions and four levels, AFS, and task-aligned RoboPhyscore.

  • Results

    Cosmos 3 achieves the highest RoboPhyscore at 0.6330, or 92.7% of ground truth, while RoboPhyscore agrees strongly with human evaluation.

  • Takeaways & Limitations

    State- and execution-grounded metrics expose failures that perceptual and VLM-based judgments can miss, supporting grounded evaluation of EWM capability.

Abstract

from arXiv · show

Video world models increasingly serve as data engines, action planners, and simulators for embodied AI, but conventional embodied world model (EWM) benchmarks lack a unified 3D-grounded protocol for establishing whether generated rollouts preserve the underlying 3D scene state or translate into executable actions. We introduce RoboPhys-3D, a 3D-grounded EWM benchmark built on RoboTwin 2.0, covering 50 manipulation tasks across four regimes, with 5,000 episodes and 25,000 multi-view ground-truth videos. A defining feature of RoboPhys-3D is that generated and ground-truth videos are processed through the same 3D reconstruction pipeline, enabling reconstruction-induced error to be distinguished from generation-induced error. The RoboPhys-3D benchmark organizes 50 complementary metrics into 18 sub-dimensions across four levels: pixel-level fidelity, 3D geometry consistency, state-level understanding, and task-level completeness. We further introduce Average Full Score, a hierarchical score averaging all 50 metrics for comprehensive evaluation, and RoboPhyscore, a compact task-aligned score averaging the metrics most strongly correlated with task success. Among the four representative video world models, Cosmos 3 achieves the highest RoboPhyscore (0.6330, 92.7% of ground truth), while state- and execution-grounded metrics reveal substantial failures that perceptual and vision-language model-based judgments fail to capture. RoboPhyscore further exhibits strong agreement with human evaluation (Pearson r = 0.9761 and Spearman \r{ho} = 0.8962), demonstrating the importance of grounded, execution-aware evaluation for EWM capability.

1. Introduction

Existing EWM benchmarks lack a unified 3D-grounded protocol that tests agreement with ground-truth scene state and action executability. RoboPhys-3D addresses this gap with shared 3D reconstruction and complementary evaluation scores.

  • EWMs support embodied data synthesis, action planning, and simulation, but existing benchmarks do not provide a unified 3D-grounded evaluation protocol.
  • Conventional video benchmarks emphasize perceptual quality and plausibility, while prior results show high scores can coexist with physically impossible predictions and poor task success.
  • RoboPhys-3D evaluates 50 manipulation tasks across four regimes using shared reconstruction of generated and matched ground-truth videos, separating generation-induced from reconstruction-induced error.
  • Average Full Score provides comprehensive diagnostics, while RoboPhyscore offers a compact task-relevant assessment based on metrics strongly correlated with task success.
  • The benchmark introduces 50 metrics across 18 sub-dimensions and four levels, covering visual, geometric, physical, semantic, and execution behavior.

2. Related Work

Related work spans 3D reconstruction, video generation, and embodied-world-model evaluation. RoboPhys-3D builds on these areas by using reconstruction as a probe of geometric coherence within manipulation settings.

  • Classical SfM and MVS recover 3D structure, while NeRF, 3DGS, and feed-forward methods improve novel-view synthesis or estimate geometry without per-scene optimization.
  • Reconstruction is conventionally evaluated through rendering fidelity, geometric fidelity, depth error, and camera-pose accuracy.
  • Recent EWM benchmarks evaluate embodied settings, motivating a framework that also measures scene-level geometric agreement with manipulation environments.
  • Video generation has progressed from diffusion models to scalable Transformers, enabling longer, higher-resolution, temporally coherent, and action-conditioned outputs.
  • General video benchmarks assess visual quality and physical commonsense, whereas GWM benchmarks emphasize camera trajectories and viewpoint synthesis.

3. RoboPhys-3D Dataset

RoboPhys-3D constructs a multi-view manipulation dataset and applies a common generation, reconstruction, normalization, and execution pipeline. Its design varies manipulator count and target-location requirements across 50 tasks.

  • The four-stage pipeline collects multi-view videos, generates view-aware prompts and annotations, reconstructs videos in 3D, and conditions candidate models on an initial frame and prompt.
  • RoboPhys-3D contains 145,000 final-evaluation videos, including 5,000 ground-truth, 20,000 reconstructed, and 120,000 generated videos.
  • Each model output is normalized, lifted into a shared 3D representation, rendered from the main perspective, and decoded by an IDM into executable actions.
  • The dataset retains 50 RoboTwin 2.0 tasks with 100 episodes each, five synchronized views per episode, 5,000 episodes, and 25,000 ground-truth videos.
  • Tasks span single- versus dual-arm manipulation and with- versus without-target regimes, varying coordination complexity and goal-directed placement requirements.
  • View-aware prompts explicitly describe visually observable details such as the acting gripper, approach direction, and manipulation trajectory.
  • Four reconstruction methods—VGGT, VGGT-Ω, 4DGS, and 4C4D—are applied identically to every video to support comparable scoring.

4. RoboPhys-3D Evaluation

RoboPhys-3D evaluates rollouts across four hierarchical levels, from pixels to executable task completion. AFS balances the taxonomy, while RoboPhyscore selects metrics with consistently strong associations to task success.

  • The framework has four levels: pixel-level fidelity, 3D geometric consistency, state-level understanding, and task-level completeness.
  • Pixel-level fidelity: Pixel-level fidelity covers 20 metrics across image quality, aesthetic quality, distribution similarity, motion quality, and content consistency.
  • 3D geometric consistency: 3D geometric consistency uses reconstruction-based geometry, depth consistency, camera trajectory geometry, and roundtrip consistency across eight metrics.
  • State-level understanding: State-level understanding measures object localization, trajectory accuracy, interaction quality, and related semantic and physical behaviors across 16 metrics.
  • Task-level completeness: Task-level completeness evaluates instruction following, collision safety, and task success through six metrics, including replayed-action and policy outcomes.
  • Average Full Score: AFS averages metrics within sub-dimensions, sub-dimensions within levels, and the four levels equally, preventing metric counts from determining level weight.
  • RoboPhyscore: RoboPhyscore retains eight non-task-success metrics whose minimum Pearson correlation with Action Planner, Data Engine, and VLM-2 exceeds 0.70.

5. Experiments

Experiments compare four video world models across multiple reconstruction methods and RoboPhys-3D evaluation levels. Cosmos performs best overall, while reconstruction quality, state understanding, task execution, prompt specificity, and inverse dynamics materially affect measured capability.

  • Overall results: Cosmos achieves the highest RoboPhyscore at 0.6330, followed by Wan at 0.5359, RoboDreamer at 0.3492, and CogVideoX at 0.3363.Cosmos ranks first under every reconstruction method.
  • Overall results: Reconstruction methods retain 78.2%–96.1% of ground-truth scores, with VGGT-Ω highest at 96.1% and 4DGS lowest at 78.2%.Reconstruction-induced error therefore contributes materially to benchmark scores.
  • Per-dimension analysis: State-level metrics expose large gaps despite plausible outputs: Cosmos retains 61.2% of ground truth for object localization and 73.2% for trajectory accuracy.Wan retains only 33.2% and 63.3% on the same metrics, respectively.
  • Per-dimension analysis: Execution-grounded task success remains limited even when aggregate task scores are high: Cosmos retains 85.9% of ground truth, while CogVideoX reaches 0.9650 collision safety but only 16.5% task success.The results show that safety or perceptual plausibility alone does not establish task completion.
  • Per-task analysis: Performance generally declines with coordination complexity, and dual-arm target-directed manipulation is the clearest challenge; Cosmos retains 91.0% of ground truth in the hardest regime versus Wan’s 73.6%.Task 4 is the lowest-scoring regime for every evaluated model.
  • Ablation study: View-aware prompts improve RoboPhyscore, raising Wan from 0.4962 to 0.5782 and Cosmos by 4.3% over its instruction baseline.The ablation fixes the visual substrate to ground-truth video.
  • Ablation study: DreamGen is selected for execution-grounded evaluation because it achieves MAE = 0.0112, Action Planner success of 0.9825, and Data Engine success of 0.5875.It outperforms J-IDM and MIDM on all three reported measures.
  • Human alignment: RoboPhyscore agrees strongly with human judgments, with Pearson r = 0.9761 and Spearman ρ = 0.8962.The human evaluation covers whether videos are coherent, physically plausible, and correctly execute the intended task.

6. Conclusion

RoboPhys-3D evaluates embodied world models through shared 3D reconstruction, multi-level metrics, and task-aligned scoring. Across representative models, Cosmos 3 performs best, while grounded metrics expose gaps between perceptual plausibility and executable manipulation.

  • RoboPhys-3D compares generated rollouts with ground-truth manipulation trajectories within a shared reconstructed 3D scene across 50 tasks and four regimes.
  • 50 metrics span 18 sub-dimensions and four levels, while AFS offers diagnostic coverage and RoboPhyscore provides a compact task-aligned indicator.
  • 0.6330 RoboPhyscore makes Cosmos 3 the strongest evaluated model, reaching 92.7% of ground truth under every reconstruction method and task regime.
  • Pearson r = 0.9761 and Spearman ρ = 0.8962 show strong agreement between RoboPhyscore and human judgment.
  • Grounded metrics reveal weaker object localization, trajectory accuracy, and task success despite strong perceptual, VLM-based, or collision-safety scores.
  • Reconstruction quality changes RoboPhyscore by up to 21.8%, motivating separation of reconstruction-induced and generation-induced error.

A.1. Dataset Statistics and Task Taxonomy

The appendix defines RoboPhys-3D’s dataset organization, evaluation assets, and metric suite. It covers task regimes, episode-level data handling, prompt variants, reconstruction and world-model substrates, and pixel-level visual criteria.

  • Dataset Statistics and Task Taxonomy: RoboPhys-3D organizes 50 manipulation tasks into four regimes based on single- versus dual-arm operation and the presence of a designated target.
  • Dataset Statistics and Task Taxonomy: Episodes use a 70%/15%/15% per-task train/validation/test split, with synchronized views, textual conditions, and generated data kept within one episode split.
  • Evaluation Assets: Evaluation compares video world models and inverse dynamic models across raw videos and reconstructions from VGGT, VGGT-Ω, 4DGS, and 4C4D, with ground truth as the reference ceiling.
  • Prompting: The benchmark ablates original, Physion-Eval, and view-aware scene annotations, including prompts designed to describe robotic manipulation from an egocentric viewpoint.
  • Metric Taxonomy: Pixel-level fidelity covers image quality, aesthetic quality, distribution similarity, motion quality, and content consistency.
  • Metric Taxonomy: The visual suite combines pixel fidelity, learned aesthetic predictors, distribution distances, optical-flow motion measures, temporal smoothness and flickering, and subject or background consistency.

B.4.3. Task Success

Task-success evaluation connects generated videos to executable behavior and policy utility, while normalization converts heterogeneous metrics into a common higher-is-better scale. The section also documents benchmark-specific bounds and direction adjustments.

  • Task Success: Task success uses Action Planner replay, Data Engine policy improvement, and VLM-2 Score judgments of execution success.
  • Task Success: Action Planner replays decoded actions in RoboTwin 2.0 and measures success against each task’s ground-truth criterion.
  • Task Success: Data Engine measures policy-performance gains from synthetic videos and recovered actions, whereas VLM-2 Score judges success from the generated video alone.
  • Score Normalization: Each raw metric xm is mapped to a normalized score sm in [0, 1], with higher values indicating better performance.
  • Custom Normalization: Normalization combines native scores, clipped affine mappings, absolute anchors, and task-specific transforms according to metric provenance and direction.
  • Custom Normalization: PSNR and FVD use absolute anchors UPSNR = 50 and UFVD = 2000, while percentile bounds support metrics lacking usable upstream normalization ranges.

B.6. Metrics Classification

RoboPhys-3D classifies metrics by their grounding signals and visualizes sub-dimension scores across models and reconstruction methods. Cosmos shows the strongest overall balance, but level-specific strengths do not guarantee task-level accuracy.

  • Metrics Classification: Reference-based metrics comprise 13 video-grounded, 12 VLM-grounded, and 5 simulator-grounded metrics, each using external information beyond the evaluated video.
  • Metrics Classification: Sub-dimension scores are divided by theoretical maxima so radar axes lie in [0, 1] and remain comparable across native ranges.
  • Video World Models: 2.7797 AFS places Cosmos first among evaluated world models, ahead of Wan at 2.5714, CogVideoX at 2.2517, and RoboDreamer at 2.1246.
  • Video World Models: Cosmos retains 93.7% of ground truth in pixel-level fidelity and 92.0% in state-level understanding, while exceeding ground truth in 3D geometry consistency and task-level completeness.
  • Video World Models: CogVideoX achieves the strongest 3D geometry consistency, yet its state-level and task-level scores fall to 72.4% and 69.9% of ground truth.
  • Video World Models: Wan retains stronger perceptual and semantic behavior but shows pronounced degradation in object localization at 33.2% and trajectory accuracy at 63.3%.

B.7.2. Reconstruction Methods

Feed-forward reconstruction methods achieve the strongest overall benchmark performance, while reconstruction sensitivity varies substantially by evaluation dimension.

  • VGGT-Ω achieves the highest AFS at 2.5598, retaining 97.0% of the reference score.
  • VGGT, 4C4D, and 4DGS retain 95.8%, 92.8%, and 90.4% of the reference AFS, respectively.
  • RoboPhyscore decomposes into eight task-aligned source metrics normalized to [0, 1].

B.8.1. Video World Models

Cosmos provides the strongest and most balanced task-aligned profile among the generated models, but reference-grounded state and motion metrics still expose substantial deficits.

  • Cosmos achieves a RoboPhyscore of 0.6330, the strongest balanced task-aligned profile among the generated models.
  • Average recall reaches only 0.4773 and nDTW 0.4297 for Cosmos, corresponding to 67.3% and 72.4% of their references.
  • Cosmos’s strong semantic and physical plausibility does not guarantee accurate state recovery or motion evolution.
  • The benchmark’s detailed results are presented for video world models and reconstruction methods, including RoboPhyscore profiles.

B.9. Full Metric-Level Correlation Analysis

RoboPhys-3D reveals strong redundancy within metric families but weaker or negative relationships across perceptual, geometric, state, semantic, and task dimensions.

  • The matrix shows that related measurements capture common quality aspects, but no single metric family characterizes EWM capability.
  • PSNR correlates with SSIM at r = 0.913 and LPIPS at r = 0.940, while SSIM and LPIPS reach r = 0.965.
  • Average precision and average recall (OL) are nearly redundant at r = 0.992, while HSD and trajectory dynamics correlate at r = 0.967.
  • Action planner and data engine are nearly perfectly correlated at r = 0.999, whereas their correlations with VLM-2 are lower at r = 0.869 and r = 0.876.
  • Roundtrip spatial consistency correlates negatively with interaction score, physical commonsense, and normalized score at r = −0.862, −0.890, and −0.936.

B.10. Threshold Sensitivity of RoboPhyscore

Threshold sensitivity exposes a trade-off between compact task-aligned metric sets and multidimensional coverage, leading the authors to adopt 0.7 as the default threshold.

  • Increasing the correlation threshold reduces the selected metric set from 19 metrics at 0.5 to none at 0.9.
  • Human evaluation reaches its highest Pearson correlation at 0.7, with r = 0.9761, R2 = 0.9528, and MAE = 0.0317.
  • Threshold 0.8 yields the highest human Spearman correlation, ρ = 0.9692, but narrows RoboPhyscore to a single metric.
  • At threshold 0.7, RoboPhyscore retains eight complementary metrics spanning perceptual, state, semantic, physical, and task-level properties.
  • The authors adopt 0.7 because it balances task alignment with metric diversity rather than maximizing a single correlation.

C. Supplementary Details about Qualitative Results

Qualitative analyses examine reconstruction, textual conditioning, and inverse dynamic model choices, while human-study analysis compares unified scores against independent judgments. RoboPhyscore shows the strongest linear agreement with human evaluation, whereas AFS best preserves ranking and calibration.

  • Qualitative comparisons: The qualitative comparisons span reconstruction methods, textual conditioning, and inverse dynamic models across the Move Pillbottle Pad, Grab Roller, and Put Object Cabinet tasks.The Grab Roller example reports embodiment morphing and unstable grasping under some conditions, while the Put Object Cabinet comparison reports recovered phase structure with differences concentrated in contact-rich precision.
  • Human-study protocol: Human evaluation uses randomized, blinded presentation and asks participants to judge overall coherence, physical plausibility, and successful execution of the intended interaction.Samples are stratified across world models, reconstruction methods, task categories, and RoboPhyscore ranges.
  • Human-study protocol: Human reference scores are averaged across independent annotators and compared with unified scores using Pearson correlation, Spearman correlation, R2, and MAE.Analyses are performed at the sample level rather than only on model-level averages.
  • Human alignment: RoboPhyscore achieves the highest Pearson correlation with human evaluation at r = 0.9761, while AFS achieves the highest rank correlation at ρ = 0.9715.RoboPhyscore also reaches R2 = 0.9528; AFS reaches R2 = 0.9499.
  • Human alignment: Both proposed scores agree more strongly with human evaluation than WorldModelBench, EWMBench, and WorldArena.WorldArena reaches r = 0.9263, compared with 0.8090 for WorldModelBench and 0.6417 for EWMBench.
  • Human alignment: AFS has the lowest mean absolute error, 0.0284, and is described as the best-calibrated score on the common scale.RoboPhyscore is the marginally stronger linear predictor, while AFS preserves configuration ordering most faithfully.

D.1.2. Task Alignment

Task-alignment results show that aggregate and component metrics distinguish world models and prompt effects, while reconstruction quality materially shapes geometry-sensitive scores. Perceptual gains do not necessarily indicate faithful physical or task execution.

  • World-model comparison: 3.6128/6: Cosmos achieves the highest aggregate IQ-1, followed by Wan, RoboDreamer, and CogVideoX.The scores correspond to 83.7%, 81.4%, 74.2%, and 71.4% of the RoboTwin reference score, respectively.
  • World-model comparison: 0.3307, 0.8577, 0.7740, and 0.9135: Cosmos leads mean PSNR, SSIM, LPIPS, and FVD among generated models.Wan leads MUSIQ and CogVideoX leads Inception Score, showing that perceptual quality or diversity alone does not ensure reference fidelity.
  • Prompt specification: 7.0%: Wan gains in IQ-1 under the proposed current prompt, whereas Cosmos improves only 0.8%.Wan rises from 3.4498 to 3.6911, while Cosmos rises from 3.6961 to 3.7247; prompt sensitivity is therefore model-dependent.
  • Prompt specification: 31.0%: Wan’s distribution similarity improves from 0.3120 to 0.4086 with richer prompting, while Cosmos gains approximately 2.4%.For Wan, most of the improvement comes from replacing the generic instruction with a richer description.
  • Prompt specification: 3.2053: Wan reaches its highest motion quality with the current prompt, while Cosmos improves modestly by 0.6%.The prompt improves Wan’s dynamic degree, flow, smoothness, and temporal stability, but the intermediate prompt temporarily lowers aggregate motion quality.
  • Reconstruction-based geometry: 29.8%: CogVideoX’s reconstruction-based geometry score exceeds the RoboTwin reference, driven largely by reprojection photometric consistency rather than physical geometry.The result demonstrates that reconstruction self-consistency and geometric correctness are related but distinct.
  • Prompt specification: 4.1%: Wan’s aggregate reconstruction-based geometry score decreases with richer prompting, despite a 17.9% increase in Fidelity.The aggregate decline reflects a trade-off because Stability decreases as Fidelity improves.
  • Reconstruction quality: 74.7%: 4DGS retains this fraction of the RoboTwin ground-truth depth-consistency score, compared with 97.9% for VGGT.Depth-related evaluation is more sensitive to reconstruction choice than aggregate reconstruction-based geometry.

E.4.3. Task Success

Task success depends on both reconstruction quality and prompt specification. Moderate reconstruction artifacts can be tolerated, but severe degradation harms executability, while improved prompts substantially raise success, especially for Wan.

  • The task-success evaluation compares different world models and different prompts in separate tabulated comparisons.
  • Moderate reconstruction artifacts can be tolerated at the task level, but sufficiently severe degradation causes an abrupt loss of downstream executability.
  • 0.4200 overall task-success gain (28.7%) is achieved for Wan when moving from the instruction-only to the proposed current prompt.Wan’s task-success score rises from 1.4650 to 1.8850 across the compared prompts.
  • 66.0% and 66.3% increases in Action Planner and Data Engine scores, respectively, accompany the improved prompt specification for Wan.Action Planner rises from 0.3525 to 0.5850, while Data Engine rises from 0.2075 to 0.3450.
Loading 2608.28718v1…