PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Video Generation Model for the Embodied World

Yufan Deng, Zilin Pan, Hongyu Zhang, Xiaojie Li, Ruoqing Hu, Yufei Ding, Yiming Zou, Yan Zeng, Daquan Zhou

arXiv:2601.15282v1cs.CVcs.AIcs.RO

TL;DR

Robot-oriented video generation lacks task-specific evaluation and sufficiently diverse training data for physical interactions. This paper introduces RBench and RoVid-X, showing that current models still struggle with physically realistic robot behaviors while providing a large annotated dataset for training and evaluation.

  • Problem

    Robot video generation lacks standardized task-specific benchmarks and large, diverse datasets needed to assess and support physical interaction capabilities.

    Source: §2.2 · p. 3, §3 · p. 5

  • Method

    The paper introduces RBench for multi-task, multi-embodiment evaluation and constructs RoVid-X with approximately 4 million annotated robotic videos.

    Source: §1 · pp. 2–3

  • Results

    Evaluation of 25 representative models reveals significant room for improvement in generating physically realistic robot behaviors.

    Source: §1 · pp. 2–3, §6 · p. 12

  • Takeaways & Limitations

    RBench and RoVid-X provide a foundation for systematic assessment and training of embodied video models.

    Source: §1 · p. 3

    The evaluated models still have significant room for improvement in their overall performance.

    Source: §5.2.2 · p. 11

Abstract

from arXiv · show

Video generation models have significantly advanced embodied intelligence, unlocking new possibilities for generating diverse robot data that capture perception, reasoning, and action in the physical world. However, synthesizing high-quality videos that accurately reflect real-world robotic interactions remains challenging, and the lack of a standardized benchmark limits fair comparisons and progress. To address this gap, we introduce a comprehensive robotics benchmark, RBench, designed to evaluate robot-oriented video generation across five task domains and four distinct embodiments. It assesses both task-level correctness and visual fidelity through reproducible sub-metrics, including structural consistency, physical plausibility, and action completeness. Evaluation of 25 representative models highlights significant deficiencies in generating physically realistic robot behaviors. Furthermore, the benchmark achieves a Spearman correlation coefficient of 0.96 with human evaluations, validating its effectiveness. While RBench provides the necessary lens to identify these deficiencies, achieving physical realism requires moving beyond evaluation to address the critical shortage of high-quality training data. Driven by these insights, we introduce a refined four-stage data pipeline, resulting in RoVid-X, the largest open-source robotic dataset for video generation with 4 million annotated video clips, covering thousands of tasks and enriched with comprehensive physical property annotations. Collectively, this synergistic ecosystem of evaluation and data establishes a robust foundation for rigorous assessment and scalable training of video models, accelerating the evolution of embodied AI toward general intelligence.

1 Introduction

Video models are increasingly important for embodied perception, reasoning, and action, but robotic video generation lacks evaluations that capture task correctness and physical feasibility. This paper addresses the gap with RBench and RoVid-X, combining systematic assessment with large-scale training data.

Background

  • Video models increasingly support robot learning, action prediction, and controllable trajectory simulation, expanding their role in embodied perception-reasoning-action loops.These applications help address the shortage of large-scale human teleoperation data and support more generalizable intelligent systems in the physical world.Source: §1 · pp. 1–2

Motivation

  • Existing evaluations often overlook task completion, action-goal alignment, and physical feasibility because they emphasize perceptual quality and lack task-specific physics criteria.This can produce overly optimistic scores for videos with unnatural movements or incomplete tasks.Source: §1 · p. 2

Benchmark

  • RBench evaluates robotic video generation across five task categories and four robot types using 650 image–text pairs and reproducible metrics for task completion and visual quality.Its sub-metrics include structural consistency, physical plausibility, and execution completeness.Source: §1 · pp. 2–3

Model evaluation

  • Evaluations of 25 representative video models reveal substantial limitations in generating physically realistic robotic videos and a persistent gap with embodied-task requirements.The evaluation spans open-source, commercial, and robotics-specific models.Source: §1 · pp. 2–3

Dataset

  • Approximately 4 million curated robotic videos make RoVid-X the largest dataset specifically designed for embodied video generation models.A four-stage pipeline integrates over 20 open-source datasets and multi-source platforms, then filters quality, segments and captions tasks, and annotates physical properties.Source: §1 · p. 3

2 Related Work

Related work applies video generation to robot trajectory synthesis, action extraction, policy learning, and future-state prediction, but remains constrained by limited robot datasets and evaluation gaps. RBench addresses these gaps with robotic-video metrics combining physical realism and task-oriented action completeness.

2.1 Video World Modeling for Robotics

  • Video generation supports robot trajectory synthesis [7] [8] [47], action extraction via inverse dynamics or latent action models [4] [22] [89] [112], and policy learning [37] [50] [60] [63] [113].These approaches aim to reduce reliance on costly teleoperation data and improve training efficiency and task performance.Source: §2.1 · p. 3

2.2 Datasets for Robot Learning

  • Robot-learning datasets span real-world robot, human-video, and synthetic-robot data, but costly collection, limited morphology and environment diversity, and inconsistent formats hinder scalable cross-dataset training [8].The proposed data focus is broader robot morphologies and entities with unified physical attributes.Source: §2.2 · pp. 3–4

2.3 Benchmarks for Video Generation

  • Existing video benchmarks assess visual semantics, temporal dynamics, or physical plausibility [6] [13] [62] [67] [72] [73] [85] [93] [108], but RBench unifies physical realism with task-oriented action completeness.It targets robotic task correctness and spatial constraints across multiple embodiments, including failures such as structural distortion, floating components, and omitted key actions.Source: §2.3 · pp. 4–5, Figure 2 · p. 4

3 RBench

RBench evaluates robotic video generation in realistic physical scenarios through diverse task and embodiment coverage, combining task-validity and visual-quality metrics to expose characteristic failure modes.

3 RBench

  • RBench targets robotic physical scenarios, revealing failures such as robot-shape distortion, object-attribute drift, non-contact attachment, and related errors.It addresses the limitations of general-scene benchmarks [42] [46] and physical-reasoning benchmarks [36] [72].Source: §3 · p. 5

3.1 Benchmark Construction

  • The benchmark contains 650 evaluation cases spanning five tasks and four embodiments, with 250 task-oriented and 400 embodiment-specific image-text pairs.Tasks include Common Manipulation, Long-horizon Planning, Multi-entity Collaboration, Spatial Relationship, and Visual Reasoning; embodiments include dual-arm, humanoid, single-arm, and quadruped robots.Source: §3.1 · p. 5
  • Human-verified reference images, redesigned prompts, filtered samples, and recorded metadata reduce train-test overlap while supporting analysis across objects, viewpoints, environments, and skills.Selected evaluation videos are excluded from the subsequent training database, and metadata records manipulated objects, embodiment types, and camera viewpoints.Source: §3.1 · p. 5

3.2 Automatic Metrics

  • Automatic metrics extend MLLM-based evaluation with manually designed indicators that assess both robotic task validity and visual realism using Qwen3-VL [3] and GPT-5.The framework separates task-completion evaluation from visual-quality evaluation, addressing criteria absent from perceptual protocols such as VBench [46].Source: §3.2 · p. 6

3.2.1 Task Completion

  • Task-completion metrics assess physical-semantic plausibility and task adherence, checking grounding, causal entity changes, contact and grasping, required actions, ordering, semantic fidelity, and responsiveness.Temporal frame grids are evaluated with MLLM-based VQA protocols and checklists targeting floating or penetration, spontaneous emergence, missing actions, incorrect order, and semantic drift.Source: §3.2.1 · p. 6

3.2.2 Visual Quality

  • Visual-quality metrics measure subject motion, robot and object stability, and temporal smoothness, targeting insufficient activity, morphology or attribute drift, deformation, jitter, blur, and aliasing.Motion amplitude discounts camera-induced movement; stability uses contrastive MLLM VQA, while smoothness uses Q-Align frame-quality stability with motion-adaptive anomaly thresholds.Source: §3.2.2 · pp. 6–7

4 RoVid-X

RoVid-X is an open-source robotic video dataset built through a refined four-stage pipeline from licensed internet and open-source embodied-video sources. It contains 4 million clips spanning diverse robots, tasks, actions, and interaction objects, with annotations designed to support physically realistic robot-video generation.

4 RoVid-X

  • The four-stage pipeline combines licensed video collection, quality filtering, task segmentation and captioning, and physical-property annotation.Sources include public-domain or non-copyrighted internet videos and open-source embodied video datasets.Source: §4 · p. 7, §4.1 · p. 7

4.1 Dataset Construction

  • The pipeline collects over 20 open-source embodied datasets and approximately 3 million raw clips, then uses GPT-5 to identify robotic tasks and remove irrelevant content.The collected videos cover varied robot types, actions, and task scenarios.Source: §4.1 · p. 7
  • Quality control removes non-robot scenes and scores clips for clarity, dynamic effects, aesthetics, and OCR before retaining high-quality data.Each clip receives a quality score based on these criteria.Source: §4.1 · pp. 7–8
  • Task segmentation labels dynamic actions with timestamps and standardized captions specifying the action subject, manipulated object, and operation details.The process excludes static or irrelevant actions and uses a video understanding model [34] with a prompt template and MLLM [34] descriptions.Source: §4.1 · p. 8
  • Physical-property annotation enhances resolution with FlashVSR [114], tracks subjects using AllTracker [43], and generates relative depth maps with Video Depth Anything [16].These annotations provide reference data for spatial relationships, action consistency, and training and evaluation of robot video generation models.Source: §4.1 · p. 8

4.2 Dataset Analysis

  • 4 million robotic video clips make RoVid-X a large-scale open-source dataset designed specifically for training video generation models under physical interaction and real-world dynamics.The dataset aims to bridge conventional video generation with embodied robot learning by providing high-quality data for training and evaluation.Source: §4.2 · p. 8

5 Experiment

RBench evaluates 25 video generation models across tasks and embodiments, revealing major gaps in physical intelligence despite progress in visual quality. Human-study agreement and RoVid-X finetuning results support the benchmark’s validity and dataset effectiveness.

5.1 Evaluation Setups

  • The evaluation covers 25 state-of-the-art models spanning closed-source, open-source, and robotics-specific systems across multiple tasks and embodiments.Open-source models use official default configurations, closed-source models use developer-recommended APIs, and each sample averages three generated videos.Source: §5.1 · p. 9

5.2.1 Quantitative Results

  • Wan 2.6 ranks first at 0.607, rising from Wan 2.1’s rank 14 and 0.399, while Seedance 1.5 Pro rises from rank 6 to rank 2.These gains indicate that iterative scaling and optimization improve physical reasoning, distinct motion patterns, and control logic, not only visual quality.Source: §5.2.1 · p. 10
  • Closed-source models occupy the benchmark’s top seven positions, while Sora v2 Pro ranks 17th with an average score of 0.362.The results expose a domain gap between consumer-oriented visual smoothness and the physical fidelity and motion control required for embodied tasks.Source: §5.2.1 · pp. 10–11
  • Robotics-specific training improves performance, but fine-tuned robot-entity models rank near the bottom, showing that physical data cannot fully replace broad world knowledge.Cosmos 2.5 outperforms significantly larger open-source video models, whereas Vidar and UnifoLM struggle, revealing a specialization trade-off.Source: §5.2.1 · pp. 10–11
  • The benchmark identifies cognitive and manipulation bottlenecks: Wan 2.6 scores 0.531 on Visual Reasoning, and models perform better on locomotion than fine-grained manipulation.Precise contact dynamics for object interaction remain more difficult than generating coarse legged-locomotion patterns.Source: §5.2.1 · p. 11

5.2.2 Qualitative Results

  • Qualitative cases show uneven capabilities: models succeed on selected visual, planning, and spatial tasks but also omit actions, misidentify objects, or introduce physically implausible interventions.Seedance 1.0 and Hailuo identify some objects correctly, Wan 2.5 completes a long-horizon sequence, and LongCat-Video adds an unrealistic human arm.Source: Figure 5 · p. 10, §5.2.2 · p. 11

5.3 Human Preference Study

  • RBench scores align strongly with human judgments, achieving Spearman ρ = 0.96 with two-sided p < 10^-3 across a ten-model subset.Thirty participants compared paired outputs, and the close ranking agreement supports the automated metrics as a reliable evaluation standard.Source: Table 3 · p. 11, §5.3 · p. 12

5.4 Validation of RoVid-X

  • RoVid-X finetuning improves performance across five task domains and four embodiments using Wan2.1 14B and Wan2.2 5B initializations.The validation randomly sampled 200k dataset instances and used MSE loss exclusively, supporting both the dataset and its collection pipeline.Source: Table 4 · p. 11, §5.4 · p. 12

6 Conclusion · Appendix

The paper introduces RBench and RoVid-X as complementary foundations for evaluating and training robot-oriented video generation models. It identifies current physical-realism limitations and outlines future work toward actionable policies, grounded evaluation, and physically capable generation.

6 Conclusion

  • RBench combines task-level accuracy and visual fidelity, including structural consistency and physical plausibility, to address gaps in robot-video evaluation.Its evaluation of 25 models reveals that current video generators still struggle to produce physically realistic robot behaviors.Source: §6 · p. 12
  • RBench scores strongly correlate with human evaluations, supporting the benchmark’s effectiveness for assessing robot-oriented video generation.Source: §6 · p. 12
  • RoVid-X provides a large-scale, diverse dataset for video generation tasks, complementing RBench as a foundation for advancing robotics video models.Source: §6 · p. 12
  • The findings expose shortcomings in current video foundation models and motivate improved methods for exploring embodied intelligence through video world models.Source: §6 · p. 12
  • Future work will use Inverse Dynamics Models to recover executable actions from generated videos for closed-loop control in simulation and on real-world hardware.Source: §6 · p. 12
  • The authors also plan automated, physically grounded metrics that assess the kinematic and dynamic feasibility of generated robot behaviors.Source: §6 · p. 12
  • Further training advances will target video models capable of generating robot videos with high-fidelity actions, supporting video-driven embodied intelligence.Source: §6 · p. 12

A Evaluation Set Details … A.1.5 Visual Reasoning

RBench’s task-oriented evaluation set spans five core dimensions, using annotated image-text pairs to test robotic manipulation, planning, collaboration, spatial reasoning, and visual-semantic reasoning. Its scenarios range from everyday object handling to complex multi-stage actions and fine-grained visual grounding.

A.1 Task-Oriented Evaluation Set

  • RBench evaluates five task dimensions using 50 initial-frame images per task and human-created, verified prompts covering diverse robotic scenarios.The corpus targets everyday manipulation, complex planning, multi-entity interaction, spatial reasoning, and visual-semantic reasoning.Source: §A.1 · pp. 20–21

A.1.1 Common Manipulation

  • Common Manipulation tests single-arm, dual-arm, and humanoid robots performing grasping, placing, pushing, rotating, and pressing with plausible, coherent goal-directed behavior.The task focuses on everyday object handling and natural physical execution.Source: §A.1.1 · p. 21

A.1.2 Long-Horizon Planning

  • Long-Horizon Planning evaluates multi-stage action decomposition, physically reasonable transitions, continuous spatial chaining, and coordinated complex motion across time.Examples include refrigerator manipulation sequences, stair-and-plank navigation, and quadruped flips followed by balancing.Source: §A.1.2 · p. 21

A.1.3 Multi-Entity Collaboration

  • Multi-Entity Collaboration pairs a primary robot with a human, animal, or robot to assess natural, temporally coherent, task-consistent handovers, assistance, following, and guidance.Evaluation spans semantic, temporal, and physical collaboration levels across four primary robot embodiments.Source: §A.1.3 · p. 21

A.1.4 Spatial Relationship

  • Spatial Relationship tests whether robots maintain specified above/below, left/right, and front/behind relations through consistent positions, orientations, layouts, and motion trajectories.The task probes spatial understanding and geometric reasoning among humanoid, single-arm, and quadruped robots interacting with objects.Source: §A.1.4 · p. 21

A.1.5 Visual Reasoning

  • Visual Reasoning covers color, numerical ordering, attribute-category matching, geometric properties, text semantics, and visual features in complex robot-action scenes.The task emphasizes fine-grained visual grounding, logical consistency, and alignment between actions and high-level visual-semantic requirements.Source: §A.1.5 · p. 22

A.2 Embodiment-Specific Evaluation Set · B Automatic Metrics Details · B.1 Physical-Semantic Plausibility

RBench evaluates robot video generation across four embodiment-specific subsets and five complementary automatic metrics. Its Physical-Semantic Plausibility metric uses key-frame grids to detect physical, semantic, and causal violations that standard visual metrics may miss.

A.2 Embodiment-Specific Evaluation Set

  • RBench evaluates dual-arm, humanoid, single-arm, and quadruped robots using 100 verified initial-frame images per embodiment.The subsets are drawn from public datasets, with human-created prompts checked for accuracy and linguistic diversity.Source: §A.2 · p. 22
  • The subsets vary robot models, actions, objects, environments, and viewpoints to test bimanual coordination, tool use, precise manipulation, and terrain adaptation.They include both first-person and third-person perspectives, with quadruped evaluation emphasizing motion continuity.Source: §A.2 · p. 22
  • Cross-embodiment evaluation exposes model biases: humanoid scenarios often achieve higher task completion and visual quality, whereas fine-grained single-arm manipulation remains difficult.RBench therefore identifies where current image-to-video models excel or fail across robot embodiments.Source: §A.2 · p. 22
  • Embodiment-aware benchmarking supports fair comparisons and encourages models to generalize beyond human-centric motion priors toward robot-specific, physically grounded behavior.The stated goal is to move from visually appealing but brittle outputs toward robust, controllable, deployment-ready robotic generation.Source: §A.2 · p. 22

B Automatic Metrics Details

  • RBench combines five fine-grained metrics—Physical-Semantic Plausibility, Task-Adherence Consistency, Motion Amplitude, Robot-Subject Stability, and Motion Smoothness—with MLLM and computational evaluation.The MLLM protocol analyzes key-frame grid images, while low-level indicators capture pixel-level motion statistics and temporal dynamics.Source: B Automatic Metrics Details · p. 23

B.1 Physical-Semantic Plausibility

  • Physical-Semantic Plausibility detects floating or unsupported entities, interpenetration, sudden appearance or disappearance, duplication, and incorrect non-contact grasping.Examples include a spoon or manipulator suspended in air, a hand penetrating a box, and objects spuriously duplicating.Source: §B.1 · p. 23, Figure 6 · p. 23, Figure 7 · p. 24
  • The VQA-style evaluator assesses both local physical violations and whether the overall action sequence follows reasonable causal progression and basic physical laws.These anomalies are treated as severe credibility failures in generated robotic videos.Source: §B.1 · pp. 23, 25, Figure 8 · p. 24

B.2 Task-Adherence Consistency … B.6 Score Aggregation

The benchmark evaluates robotic video generation through task adherence, robot–subject stability, motion amplitude, and motion smoothness, then aggregates five normalized signals into Task Completion and Visual Quality scores. These metrics target semantic correctness, physical plausibility, temporal coherence, and visual consistency across diverse robotic behaviors.

B.2 Task-Adherence Consistency

  • Task-Adherence Consistency uses task-specific VQA criteria to assess responsiveness, key-action completeness, execution plausibility, and logical consistency across five task families.It evaluates completion and action effectiveness for manipulation, collaboration, spatial reasoning, visual reasoning, and long-horizon planning, preventing high-level correctness from masking missing or physically invalid actions.Source: §B.2 · pp. 25–26
  • For visual reasoning and long-horizon tasks, the metric verifies trigger–feedback–outcome chains and ordered event completion while penalizing discontinuous or physically implausible execution.Question-chain verification treats missing or incorrect events as unfulfilled steps, and event-list scoring ensures partially correct ordering is not over-rewarded without valid motion.Source: §B.2 · p. 26

B.3 Robot-Subject Stability

  • Robot-Subject Stability separately measures whether robot morphology and target-object identity remain visually and semantically consistent throughout the generated video.Examples include robots changing embodiment, sleeves changing form, and cups changing color and shape; the metric captures such structural drift and appearance changes.Source: §B.3 · pp. 26, 28

B.4 Motion Amplitude

  • Motion Amplitude measures meaningful robot articulation by tracking robot keypoints with GroundingDINO, SAM2, and CoTracker while compensating for camera and background motion.Its soft-zero strategy retains a small residual when robot motion does not exceed background motion, improving robustness to tracking noise and partial occlusion.Source: §B.4 · pp. 28–30

B.5 Motion Smoothness

  • Motion Smoothness detects temporal discontinuities by measuring adjacent-frame Q-Align quality fluctuations against a motion-amplitude-adaptive threshold.Abnormal frames and adjacent frames are flagged to capture abrupt artifacts, and the Motion Smoothness Score is the proportion of normal frames; higher values indicate smoother, more coherent motion.Source: §B.5 · pp. 30–31

B.6 Score Aggregation

  • Score aggregation normalizes five metrics—PSS, TAC, RSS, MS, and MA—and applies penalties for insufficient motion amplitude and unstable robot or object appearance.The motion-amplitude penalty uses t = 0.1, tlow = 0.05, and δ = 0.1, while stability grades B–E map to penalties 0.2, 0.4, 0.6, and 0.8; grade A has zero penalty.Source: §B.6 · pp. 31–32
  • Final model scores average Task Completion and Visual Quality across samples for quantitative benchmark comparison and ranking.Task Completion combines Physical-Semantic Plausibility and Task-Adherence Consistency, while Visual Quality combines Robot-Subject Stability and Motion Smoothness with motion-amplitude and stability penalties.Source: §B.6 · p. 32

C Model Descriptions and Implementation Setups … C.3 Robotics-specific Models

The benchmark evaluates commercial, open-source, and robotics-specific video generation models under their official or default inference configurations. These setups span diverse architectures, modalities, resolutions, frame rates, and video durations.

C.1 Commercial Models

  • Commercial models include Wan 2.6, Hailuo, Veo 3, Kling 2.6 pro, and Seedance 1.0, evaluated through official APIs across 5–6-second outputs at 720p–1080p and 24–30 fps.Wan 2.6 uses 5-second 720P at 30 fps; Hailuo uses 6 seconds at 1364 × 768 and 24 fps; Veo 3 uses up to 720p, 8 seconds, and 24 fps; Kling 2.6 pro uses 1920 × 1080 at 24 fps; Seedance 1.0 uses 1280 × 720 at 24 fps.Source: §C.1 · p. 32
  • Additional commercial baselines comprise Wan 2.5, Seedance 1.5, Sora v1, and Sora v2 Pro, using official models or APIs with default parameters.The reported configurations range from 4–5-second videos at 1280 × 720 and 30 fps to 720P at 24 fps.Source: §C.1 · pp. 32–33

C.2 Open-source Models

  • Cited baselines include open-source Wan2.2_A14B, LongCat-Video, Wan2.1, and SkyReels-V2; official default-setting runs use Wan2.1 at 832 × 480 for 5 seconds, HunyuanVideo at 1248 × 704 for 5 seconds, and SkyReels-V2 at 960 × 544 for approximately 4 seconds.Source: §C.2 · p. 33
  • FramePack and CogVideoX receive explicit implementation details, generating 5-second 832 × 480 videos and 6-second 720 × 480 videos at 16 and 8 fps, respectively.FramePack compresses input frames by importance for fixed-context long-video synthesis, whereas CogVideoX uses expert-adaptive LayerNorm and multi-resolution frame packing.Source: §C.2 · p. 34

C.3 Robotics-specific Models

  • Robotics-specific models include Cosmos 2.5, DreamGen (GR), DreamGen (DROID), UnifoLM-WMA-0, and Vidar, evaluated with robotics-oriented generation or world-modeling configurations.Cosmos 2.5 uses NVIDIA’s official codebase; DreamGen models generate trajectory videos; UnifoLM-WMA-0 combines world modeling with action policy learning; Vidar targets robotic manipulation.Source: §C.3 · p. 34
  • Vidar generates 4-second multi-view interactive videos at 704 × 480 and 15 fps with automatically annotated action trajectories.Its architecture combines a video diffusion prior with a Masked Inverse Dynamics Model (MIDM).Source: §C.3 · p. 34

D Human Preference Study Details · E Prompt Template · F Additional Qualitative Comparisons

The supplementary sections validate RBench against human judgments, detail the structured Visual Reasoning prompt, and extend qualitative comparisons across RBench’s five task categories. Together, they specify reproducible evaluation procedures and illustrate model behavior across diverse embodied scenarios.

D Human Preference Study Details

  • Thirty participants compared paired videos for each prompt, with votes converted into model scores using a 5/3/1 win-tie-loss scheme.The questionnaire is provided in Figure 14.Source: D Human Preference Study Details · p. 34, Figure 14 · p. 35
  • The human–benchmark agreement analysis uses LOO linear calibration to correct systematic scale mismatch before computing Bland–Altman bias and 95% limits of agreement.The calibration is fit using all models except the model being evaluated, avoiding overfitting from the held-out item.Source: D Human Preference Study Details · pp. 34–35
  • Bias = 0.002 and LoA = [−0.108, 0.112] indicate close absolute agreement between calibrated benchmark scores and human judgments, alongside a strong model-level correlation with RBench (ρ = 0.96).Source: D Human Preference Study Details · pp. 34, 36

E Prompt Template

  • The Visual Reasoning prompt supplies viewpoint, content, manipulator, and object context, then uses a two-part question-chain constructor and video assessment prompt.The question chain converts instructions into binary, stepwise checks of causal and temporal action dependencies.Source: E Prompt Template · p. 36
  • The assessment combines the reasoning chain with a chronological 3×2 frame grid and instructs the MLLM to score completion, stability, consistency, and physical plausibility in structured JSON.This protocol is designed to make evaluation reproducible and interpretable across models.Source: E Prompt Template · pp. 36–37

F Additional Qualitative Comparisons

  • Supplementary qualitative comparisons show two representative cases per RBench task category using videos from ten state-of-the-art I2V models.The five categories are Common Manipulation, Long-Horizon Planning, Multi-Entity Collaboration, Spatial Relationship, and Visual Reasoning, visualized in Figures 19–23.Source: F Additional Qualitative Comparisons · p. 37

G Comprehensive Quantitative Results

This section reports comprehensive evaluations across five robotic tasks and four embodiments using GPT- and Qwen-based evaluators. It defines the reported metrics and provides complete task-level and embodiment-level results in accompanying tables.

Evaluation scope

  • The evaluation covers five tasks and four robot embodiments, with per-model scores reported using both GPT- and Qwen-based evaluators.Tasks are Common Manipulation, Long-Horizon Planning, Multi-Entity Collaboration, Spatial Relationship, and Visual Reasoning; embodiments are Dual Arm, Humanoid, Single Arm, and Quadruped.Source: G Comprehensive Quantitative Results · pp. 37, 39

Task-level metrics

  • Task-level metrics include action execution, task completion, object and robot consistency, physical semantics, event completion, entity consistency, coordination, spatial relations, manipulation feasibility, and visual reasoning.The abbreviations are AES, TCS, OCS, RCS, PSS, ECR, ECS, ACS, SRS, MFS, and VRS.Source: G Comprehensive Quantitative Results · p. 37

Robot-level metrics

  • Robot-level metrics assess physical semantics, task-adherence consistency, robot–subject stability, motion smoothness, and motion amplitude.These metrics are abbreviated PSS, TAC, RSS, MSS, and MAS.Source: G Comprehensive Quantitative Results · p. 38

Task results

Embodiment results

Related papers

Loading 2601.15282v1…