Source-linked AI summary
Rethinking Video Generation Model for the Embodied World
Yufan Deng, Zilin Pan, Hongyu Zhang, Xiaojie Li, Ruoqing Hu, Yufei Ding, Yiming Zou, Yan Zeng, Daquan Zhou
TL;DR
Robotic video generation lacks standardized evaluation and sufficiently diverse training data for physically grounded embodied intelligence. The paper introduces RBench to assess task and visual fidelity and RoVid-X to support training; evaluations expose substantial physical-realism deficiencies in current models.
Problem
Robotic video generation lacks systematic evaluation and large-scale, diverse training data covering varied tasks, environments, objects, and robot types.
Method
The paper introduces RBench, a benchmark with task-level and visual-fidelity metrics, and constructs RoVid-X as a large-scale robotic video dataset.
Results
Evaluations of 25 representative models reveal significant room for improvement in generating physically realistic robot behaviors.
Takeaways & Limitations
RBench provides a benchmark for rigorous robotic video assessment, while RoVid-X supplies training data for embodied video models.
Takeaways & Limitations
Current video generation models still require significant improvement in overall performance and physical realism for robotic behaviors.
Abstract
from arXiv · showhide
Video generation models have significantly advanced embodied intelligence, unlocking new possibilities for generating diverse robot data that capture perception, reasoning, and action in the physical world. However, synthesizing high-quality videos that accurately reflect real-world robotic interactions remains challenging, and the lack of a standardized benchmark limits fair comparisons and progress. To address this gap, we introduce a comprehensive robotics benchmark, RBench, designed to evaluate robot-oriented video generation across five task domains and four distinct embodiments. It assesses both task-level correctness and visual fidelity through reproducible sub-metrics, including structural consistency, physical plausibility, and action completeness. Evaluation of 25 representative models highlights significant deficiencies in generating physically realistic robot behaviors. Furthermore, the benchmark achieves a Spearman correlation coefficient of 0.96 with human evaluations, validating its effectiveness. While RBench provides the necessary lens to identify these deficiencies, achieving physical realism requires moving beyond evaluation to address the critical shortage of high-quality training data. Driven by these insights, we introduce a refined four-stage data pipeline, resulting in RoVid-X, the largest open-source robotic dataset for video generation with 4 million annotated video clips, covering thousands of tasks and enriched with comprehensive physical property annotations. Collectively, this synergistic ecosystem of evaluation and data establishes a robust foundation for rigorous assessment and scalable training of video models, accelerating the evolution of embodied AI toward general intelligence.
1 Introduction
Robotic video generation lacks systematic evaluation and sufficiently diverse training data. The paper addresses both gaps with RBench and RoVid-X, while evaluations reveal substantial limitations in current models.
- Video models support embodied perception, reasoning, and action, but robotic video generation still lacks systematic evaluation.
- RBench evaluates robotic video generation across task completion and visual quality using structural consistency, physical plausibility, and execution completeness.It contains 650 image–text pairs spanning five task categories and four robot types.
- 25 representative video models reveal a persistent gap between current foundation models and physically realistic embodied robotic behaviors.
- Robotic training data remains constrained in scale, diversity, environments, objects, tasks, and robot morphologies, limiting video-model generalization.
- RoVid-X contains approximately 4 million curated robotic videos with standardized task descriptions and physical property annotations.
2 Related Work
Video generation offers scalable robot-training data, but existing datasets remain costly, limited in diversity, and difficult to combine, while evaluation lacks robotics-specific coverage. This work motivates RBench as a benchmark for robotic physical scenarios and task-oriented action completeness.
- Video generation models can synthesize robot trajectories and support embodied-intelligence training with richer perception, reasoning, and action data.
- Large-scale robot-learning datasets remain scarce, with real-world collection constrained by high costs and limited scale.
- Existing robot datasets emphasize similar robot types and restricted environments, while inconsistent collection and storage hinder cross-dataset co-training.
- Video-generation evaluation spans visual-semantic fidelity, temporal dynamics, and physical plausibility, but existing benchmarks do not comprehensively target robotic tasks.
- RBench introduces reproducible metrics intended to unify physical realism with task-oriented action completeness in robotic video generation.
3 RBench
RBench evaluates robotic video generation across diverse tasks and embodiments using task-specific criteria that extend beyond conventional perceptual metrics. Its dimensions cover physical-semantic plausibility, task adherence, robot-subject stability, motion amplitude, and motion smoothness.
- RBench targets robotic physical scenarios, addressing the task-specific gaps of general video-generation benchmarks and perceptual protocols.
- Benchmark scope: Five task categories—Common Manipulation, Long-horizon Planning, Multi-entity Collaboration, Spatial Relationship, and Visual Reasoning—contain 250 image-text pairs.
- Benchmark scope: Four embodiment categories—Dual-arm, Humanoid, Single-arm, and Quadruped robots—contain 400 image-text pairs in total.
- Benchmark scope: The evaluation set uses manually verified keyframes, redesigned prompts, and videos excluded from the subsequent training database to reduce overlap.
- Evaluation dimensions: Physical-Semantic Plausibility uses MLLM-based VQA over temporal frame grids to identify grounding, penetration, and related physical or semantic violations.
- Evaluation dimensions: Task-Adherence Consistency checks prompt responsiveness, required key actions, ordering, and semantic alignment through temporal-grid VQA.
- Evaluation dimensions: Motion Amplitude discounts camera movement to penalize videos with smooth appearance but insufficient robotic-subject activity.
- Evaluation dimensions: Robot-Subject Stability evaluates morphology and target-object attributes over time, including manipulators, topology, joints, object identity, color, and position.
4 RoVid-X
RoVid-X is constructed through a four-stage pipeline that collects, filters, segments, captions, and physically annotates robotic videos. The resulting open-source dataset contains 4 million clips spanning diverse robot actions, tasks, and types for video-generation training and evaluation.
- Data collection: RoVid-X is built from licensed internet robotic videos and more than 20 open-source embodied video datasets covering varied robot types and task scenarios.GPT-5 filters collected content to remove low-quality or irrelevant clips.
- Video quality filtering: The pipeline applies scene segmentation and multidimensional quality scoring to retain clips meeting clarity, dynamic-effect, aesthetic, and OCR standards.Each retained clip receives a quality score based on these criteria.
- Task segmentation and captioning: Task segmentation uses a video-understanding model and prompt template to timestamp robot actions and generate standardized subtitles describing each task segment.Static scenes and irrelevant actions are excluded, while action intervals are labeled with start and end times.
- Physical property annotation: Physical-property annotation enhances resolution with FlashVSR and applies unified optical-flow tracking with AllTracker to record robot actions consistently across scenes.
- Dataset scale and coverage: 4 million robotic video clips make RoVid-X an open-source dataset designed for training video-generation models on physical interaction, spatial relationships, and real-world dynamics.Its data covers diverse robotic actions, tasks, robot types, and scenarios.
5 Experiment
RBench evaluates 25 video generation models across robotic tasks and reveals a substantial gap in physically realistic behavior. Results also show strong alignment with human judgments and continued room for improvement.
- Quantitative Results: 25 representative models were evaluated across model types using RBench.The study included closed-source, open-source, and robotics-specific models under standardized evaluation procedures.
- Quantitative Results: Top commercial models are beginning to function as World Simulators by modeling interaction-rich physical scenarios beyond visual fidelity.Wan 2.6 and Seedance 1.5 Pro are identified as examples of this shift toward Physical AI.
- Quantitative Results: Wan improved from Rank 14, 0.399 in Wan 2.1 to Rank 1, 0.607 in Wan 2.6, indicating gains in physical reasoning alongside model iteration.Seedance also rose from Rank 6 to Rank 2 between versions 1.0 and 1.5 Pro.
- Quantitative Results: Sora v2 Pro ranked 17th with Avg 0.362, showing that creative video-generation proficiency does not necessarily transfer to embodied tasks.The passage attributes this domain gap to optimization for visual smoothness and cinematic transitions rather than physical fidelity and motion control.
- Quantitative Results: Closed-source models occupied the top seven positions, while robotics-specific training helped Cosmos 2.5 outperform larger open-source video models.Models fine-tuned on specific robot entities, including Vidar and UnifoLM, ranked near the bottom.
- Quantitative Results: Fine-grained manipulation and visual reasoning were persistent bottlenecks compared with execution-oriented and coarse locomotion tasks.Wan 2.6’s Visual Reasoning score was 0.531, while manipulation required more difficult contact dynamics than legged locomotion.
- Human Preference Study: ρ = 0.96 (two-sided p < 10^-3) measured the Spearman correlation between RBench scores and human preference scores.The study used 30 participants and a ten-model subset; remaining discrepancies motivate further metric refinement.
6 Conclusion
The paper introduces RBench and RoVid-X to address gaps in evaluating and training robot-oriented video generation models. Together, they provide a foundation for assessing physical realism and advancing robotic video models.
- Conclusion: RBench evaluates robot-oriented video generation using task-level accuracy, visual fidelity, structural consistency, and physical plausibility.Its evaluation of 25 models reveals that current systems need substantial improvement in physically realistic robot behavior.
- Conclusion: RoVid-X provides a large-scale and diverse resource for robotic video generation tasks.The paper positions the benchmark and dataset together as a foundation for rigorous assessment and scalable training.
- Future Work: Future work will connect generated videos to executable robot policies through inverse dynamics and closed-loop simulation and hardware experiments.The authors also plan more automated metrics grounded in kinematic and dynamic feasibility.
A.1 Task-Oriented Evaluation Set
The task-oriented evaluation set measures robotic video generation across five task dimensions using curated image–text prompts. It spans manipulation, planning, collaboration, spatial relations, and visual-semantic reasoning.
- Task Dimensions: RBench organizes evaluation into Common Manipulation, Long-Horizon Planning, Multi-Entity Collaboration, Spatial Relationship, and Visual Reasoning.These dimensions target multi-dimensional task execution capabilities in robotic scenarios.
- Task Dimensions: Each task uses 50 initial images with human-created and verified prompts covering diverse robotic scenarios.The resulting image–text pairs span everyday manipulation, complex planning, multi-entity interaction, spatial reasoning, and visual-semantic reasoning.
- Common Manipulation: Common Manipulation tests physically plausible and temporally coherent grasping, placing, pushing, rotating, and pressing behaviors.Scenes include single-arm, dual-arm, and humanoid robots handling everyday objects.
- Long-Horizon Planning: Long-Horizon Planning evaluates action decomposition, stage transitions, cross-time reasoning, and continuous body coordination.Examples include sequential object manipulation, multi-step spatial planning, and complex quadruped motion.
- Multi-Entity Collaboration: Multi-Entity Collaboration assesses interactions between a primary robot entity and a human, animal, or second robot.The task includes behaviors such as object handover and assistance.
- Spatial Relationship: Spatial Relationship evaluates relative positions, orientations, and motion trajectories among robots and objects.Relations include above/below, left/right, and front/behind.
- Visual Reasoning: Visual Reasoning covers color, ordering, attribute, geometric, textual, semantic, and visual-feature understanding.The task requires fine-grained visual grounding, logical consistency, and alignment between robot actions and visual-semantic requirements.
A.2 Embodiment-Specific Evaluation Set
The embodiment-specific evaluation set compares robot video generation across heterogeneous morphologies and uses metrics designed to expose physical and semantic failures. It also highlights stronger performance on humanoids than fine-grained manipulation.
- Embodiment Coverage: RBench covers four representative robot categories to analyze morphology-specific modeling and generalization challenges.The design accounts for differences in degrees of freedom, control modes, and task objectives.
- Embodiment Coverage: Embodiment subsets vary robot models, actions, objects, environments, and camera perspectives.Dual-arm robots emphasize coordination, humanoids tool use and posture, single-arm robots precision, and quadrupeds terrain adaptation and continuity.
- Embodiment Findings: Models often perform better on humanoid scenarios than on fine-grained single-arm manipulation.The passage relates this bias to extensive pretraining on human activity data.
- Evaluation Metrics: Five fine-grained metrics jointly assess physical-semantic plausibility, task adherence, motion amplitude, robot-subject stability, and motion smoothness.An MLLM-based VQA protocol operates on key-frame grids and is supplemented by pixel-level motion and temporal-dynamics indicators.
- Physical-Semantic Plausibility: Physical-Semantic Plausibility detects floating entities, interpenetration, sudden changes, and incorrect grasping through key-frame analysis.The evaluator also judges whether action sequences and causal progression follow basic physical laws and common sense.
- Physical-Semantic Plausibility: These violations reduce generated-video credibility because they expose failures in physical laws and semantic causality.Examples include unsupported manipulators, hands penetrating boxes, duplicated objects, and non-contact attachment.
B.2 Task-Adherence Consistency
Task-Adherence Consistency evaluates whether generated robotic videos follow the prompt’s objectives, action sequence, spatial relations, and physically plausible execution. The section also introduces complementary stability and motion measures for detecting embodiment drift and smooth-but-inactive behavior.
- Task-Adherence Consistency: Figure 9 illustrates failures where a static gripper ignores a grasp instruction and omitted faucet actions break the causal chain leading to water flow.These examples represent non-responsiveness and incomplete key actions.
- Motivation: RBench uses explicit task-adherence evaluation because conventional low-level perception metrics miss ignored objectives, omitted actions, and inconsistent execution.The protocol uses key-frame grids and MLLM-based assessment to evaluate responsiveness, action completeness, and execution consistency.
- Task-Adherence Consistency: Task-specific criteria cover common manipulation, long-horizon planning, multi-entity collaboration, spatial relationships, and visual reasoning.These criteria assess completion, ordered events, role coordination, spatial accuracy, manipulation feasibility, and trigger-feedback-outcome logic.
- Task-Adherence Consistency: Action Effectiveness penalizes discontinuous, incomplete, or physically implausible motions even when high-level reasoning or event ordering is correct.The measure considers natural gripper closure, contact locations, trajectories, body coordination, stable landing, and transitions between stages.
- Visual Stability: Robot-Subject Stability separately measures whether the robot and target subject preserve visual and semantic consistency throughout generation.Examples include robot embodiment changes, sleeve-to-sweater transformation, cup identity changes, extra manipulators, and unrealistic material deformation.
- Motion Amplitude: Motion Amplitude measures meaningful robot articulation after compensating for camera movement, addressing videos that appear smooth while the robot remains nearly static.GroundingDINO, segmentation masks, and CoTracker isolate subject motion from background drift.
- Motion Smoothness: Motion Smoothness detects temporal discontinuities and motion artifacts, while adjacent frames around abnormal indices are also flagged.The metric targets artifacts ranging from low-level aliasing to high-level jitter and blur.
C Model Descriptions and Implementation Setups
The implementation set spans open-source, commercial, world-modeling, and robot-learning video generators. Models are evaluated using official APIs, weights, scripts, and default configurations, with generation settings varying by model.
- Implementation Setups: Open-source and commercial video models are run with official default configurations or developer-recommended APIs for consistency.The set includes Wan, Hailuo, Veo, Kling, Seedance, Sora, LTX, FramePack, CogVideoX, HunyuanVideo, and SkyReels-V2.
- Implementation Setups: Generation settings vary substantially across models, including video duration, spatial resolution, and frame rate.Examples range from 4-second 960 × 544 outputs to 6-second 1364 × 768 outputs and differing frame rates such as 8, 16, 24, and 30 fps.
- Model Descriptions: Wan2.2_A14B is evaluated with its default model at 1280 × 720 resolution and 16 fps for 5-second videos.The model supports both text and image inputs.
- Model Descriptions: LongCat-Video is evaluated with default configurations at 1280 × 704 and 15 fps using coarse-to-fine generation.It unifies text-to-video, image-to-video, and video continuation within a Diffusion Transformer architecture.
- Model Descriptions: The evaluation also includes Cosmos world-simulation models and robot-learning systems such as DreamGen and UnifoLM-WMA-0.These systems generate videos under model-specific resolutions and frame rates for physical-AI or robotic-learning settings.
D Human Preference Study Details
The human preference study compares generated videos pairwise and converts votes into model-level scores, which are then compared with RBench. Spearman correlation and Bland–Altman analysis indicate strong ranking and close calibrated agreement between benchmark and human judgments.
- Human Preference Study: Thirty participants compare paired outputs for the same prompt and video instance, choosing whether A, B, or neither is better.Votes are aggregated per model using scores of 5 for wins, 1 for losses, and 3 for ties.
- Agreement with Human Judgments: ρ = 0.96 measures the strong correlation between model-level human scores and RBench benchmark scores.The comparison evaluates whether the benchmark reproduces the human-based model ranking.
- Agreement with Human Judgments: Bland–Altman analysis complements rank correlation by assessing absolute agreement between human and automatic scores.Because score scales may differ, the analysis applies leave-one-out linear calibration before computing agreement statistics.
- Agreement with Human Judgments: Bias = 0.002 and LoA = [−0.108, 0.112] indicate close agreement between calibrated benchmark scores and human judgments.The solid line represents bias, while dashed lines represent the 95% limits of agreement.
- Visual Reasoning Evaluation: Visual Reasoning uses a question-chain constructor and a video assessment prompt to verify causal and temporal dependencies in robot actions.The assessment combines binary reasoning questions, contextual information, and chronologically ordered frames with structured scoring and JSON output.
G Comprehensive Quantitative Results
The comprehensive evaluation reports task-level and robot-level results across five tasks, four embodiments, and GPT- and Qwen-based evaluators. Supplementary figures visualize representative cases, while the tables organize results by task and embodiment.
- Evaluation Scope: Five task domains and four robot embodiments are evaluated with detailed scores from both GPT-based and Qwen-based evaluators.The tasks are Common Manipulation, Long-Horizon Planning, Multi-Entity Collaboration, Spatial Relationship, and Visual Reasoning; embodiments are Dual Arm, Humanoid, Single Arm, and Quadruped.
- Metrics: Task-level metrics include AES, TCS, OCS, RCS, PSS, ECR, and ECS, while robot-level metrics include PSS, TAC, RSS, MSS, and MAS.The reported abbreviations distinguish task evaluation from per-robot embodiment evaluation.
- Results Organization: GPT-based results are organized across five task tables and four embodiment tables, with corresponding Qwen-based tables provided separately.The tables cover task-specific and embodiment-specific evaluations for both evaluator families.
- Qualitative Results: Figures 19–23 visualize representative generated videos for Common Manipulation, Long-Horizon Planning, Multi-Entity Collaboration, Spatial Relationship, and Visual Reasoning.Each task category includes two representative cases from ten state-of-the-art image-to-video models.