EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models

1 Introduction

• Conventional video benchmarks emphasize visual fidelity, language alignment, and human preference, but do not capture embodied requirements such as coherent motion and plausible action execution.For manipulation, backgrounds, object configurations, and embodiment structure should remain static as instructed poses and interactions evolve.Source: §1 · p. 2
• EWMBench evaluates embodied generation across scene consistency, motion correctness, and semantic alignment using systematic tools, including video-based MLLM prompts.The benchmark asks models to generate future frames from an initial segment and manipulation instruction.Source: §1 · p. 2
• The dataset uses 30 samples across ten tasks with action-ordering constraints, static initial frames, and varied trajectories for tasks with multiple valid solutions.The samples are selected from Agibot-World, and voxelized scoring encourages trajectory variation.Source: §1 · p. 2
• The paper presents EWMBENCH as the first world-generation benchmark tailored to embodied tasks, alongside a diverse dataset, open-source evaluation metrics, and analysis of existing models.The framework targets robotic manipulation and evaluates physically and kinematically constrained trajectories.Source: §1 · p. 2

2.1 Video Generative Models

• Video-generation research includes diffusion models, robot-learning applications addressing data scarcity, and models predicting future states from textual and visual inputs.Some robotics work uses video models to predict future frames from text and visual inputs.Source: §2.1 · p. 3

2.2 Evaluation of Video Generative Models

• Video evaluation has expanded from appearance-focused FID, IS, and FVD to multi-capability benchmarks, while specialized tools assess physical laws and compositionality.Existing frameworks still emphasize visual quality, aesthetics, and motion smoothness.Source: §2.2 · p. 3
• Earlier benchmarks generally overlook embodied factors such as action plausibility, object interactions, and manipulation, motivating EWMBENCH’s scene, motion, and semantic evaluation pipeline.The benchmark initializes models with scene images, instructions, and optional action trajectories, then evaluates generated frames.Source: §2.2 · p. 3, Figure 2 · p. 4

3.1 EWMBENCH Overview

• The benchmark initializes generation with scene images and instructions, optionally supplies an action trajectory, and evaluates outputs through multimodal analysis and visual models.The trajectory is a sequence of 6D poses, and raw generated frames are normalized before evaluation.Source: §3.1 · p. 4

3.2 Dataset Construction

• Ten household and industrial tasks are selected for clear goals and sequential dependencies, with voxel-grid trajectory encoding and greedy selection promoting motion diversity.Each task is decomposed into 4–10 atomic sub-actions paired one-to-one with video segments and step-level captions.Source: §3.2 · p. 4, Figure 3 · p. 5

3.3 Evaluation Metrics

• Scene Consistency uses fine-tuned DINOv2 patch embeddings to measure layout, object permanence, and viewpoint coherence across consecutive and initial frames.Higher similarity scores indicate more stable scene structure and coherent viewpoints.Source: §3.3 · p. 5
• Trajectory evaluation compares generated end-effector paths with ground truth using HSD for spatial deviation, NDTW for spatial-temporal alignment, and DYN for motion dynamics.Models provide three candidate trajectories per task, and the candidate with the best Hausdorff distance is selected.Source: §3.3 · p. 5
• Semantic evaluation measures instruction alignment through video captions and ground-truth annotations, and estimates task-space diversity from CLIP video-feature similarity.The diversity score is 1 − similarity.Source: §3.3 · p. 5

3.4 MLLM Prompt Suite Design

• The MLLM prompt suite assesses global caption alignment with task instructions, step-by-step descriptions against ground-truth steps, and logical errors such as hallucinated manipulations.The prompt suite is designed to be compact yet representative.Source: §3.4 · pp. 5–6

4 Experiments

• The evaluation covers seven open-source, commercial, and domain-adapted models across ten tasks, generating three videos per episode and selecting the best prediction.Source: §4 · p. 6

4.1 Evaluation Results

• Domain-adapted models, including EnerVerse and LTX_FT, consistently outperform commercial and open-source models across evaluation dimensions.Source: §4.1 · p. 6, Table 2 · p. 7
• Scene consistency can remain high for visually plausible static videos, whereas trajectory consistency reflects motion aligned with ground truth and task success.Source: §4.1 · p. 6, Figure 5 · p. 7

4.2 Human Evaluation

• EWMBENCH’s rankings align more closely with human judgments than VBench’s rankings in an evaluation of predictions from four representative models.Source: §4.2 · p. 7, Figure 6 · p. 8

4.3 Complementarity of Trajectory Metrics

• Sequence reversal selectively lowers nDTW; outliers substantially lower HSD and DYN; frame repetition raises nDTW but lowers DYN, showing complementary metric sensitivities.Source: §4.3 · p. 7, Figure 6 · p. 8

4.4 Further Analysis and Discussions

• Domain-adapted models perform best particularly in semantic and dynamic dimensions, yet occasionally exhibit empty grasping that exposes limits in fine-grained action grounding.Source: §4.4 · pp. 7–8
• Kling leads general models; Hailuo is reasonably capable zero-shot but cartoon-like, while COSMOS and LTX show robotic-context or continuity failures and OpenSora jitters or stalls.Source: §4.4 · p. 8
• EWMBENCH’s fine-tuned DINOv2-based Scene Consistency metric captures layout structure and viewpoint changes more effectively than VBench’s subject-level metrics.Source: §4.4 · p. 8

• EWMBENCH pairs a motion-diverse, multi-scene dataset with multidimensional, human-aligned metrics for measuring progress in embodied world model development.Source: §5 · p. 8
• The current method evaluates robotic-arm end-effector trajectories; future work will incorporate the state and configuration of the entire arm.Source: §5 · p. 8
• Evaluation currently uses fixed-viewpoint scenes, with flexible viewpoints such as dynamic camera setups left for future research.Source: §5 · p. 8
• The benchmark currently covers manipulation, with navigation and mobile manipulation identified as future extensions.Source: §5 · p. 8

Pre-Processing

• Reference images are resized to 640 × 480, and prompts use aligned step-level captions for the current sub-action.Source: §A.1 · p. 12
• A prompt constraint is appended to preserve the robot’s first-person view and keep it unchanged across frames.Source: §A.1 · p. 12

Video Normalization

• Generated videos are resized to 640 × 480 and resampled to 30 FPS before scene-consistency and semantic-alignment evaluation.Source: §A.1 · p. 12

Trajectory Extraction

• The pipeline detects both end-effectors with fine-tuned YOLO-World and tracks them with BoT-SORT, then evaluates the hand with the largest convex-hull spatial extent.Source: §A.1 · p. 12

YOLO-World Training Details

• On Agibot-World frames, with Freezer Restocking and Factory Packing held out for validation to construct a hard-sample set, the model is trained for 100 epochs and achieves recall 0.91667 and precision 1.0.Source: §A.1 · p. 12

Task Selection

• The ten selected tasks span household and industrial contexts and challenge spatial reasoning, tool use, and action ordering.Source: §A.2.1 · p. 12
• The task set includes household activities such as retrieving toast, pouring water, and storing objects, alongside industrial factory packing.Source: §A.2.1 · p. 12

Scene Diversity

• Tasks vary in object properties, spatial layouts, and interaction complexity; Figure 7 shows task scenes, trajectory overlays, and object-property distributions.Source: §A.2.1 · p. 12

Action Motion Sampling Strategy

• The evaluation set uniformly samples 100 videos per task and decomposes each into primitive actions aligned with video segments, sub-action labels, and textual descriptions.Source: §A.2.2 · p. 12
• Figure 7’s first two rows show task scenarios with action-motion trajectories, while its third row groups examples by object properties.Source: Figure 7 · p. 13
• Left and right end-effector positions are represented as voxel grids, and pairwise trajectory similarity is computed with 3D IoU.V^L and V^R denote the voxel grids for the left and right end-effectors, respectively.Source: §A.2.2 · p. 13
• A greedy strategy starts with the trajectory having the lowest average IoU to all others, then iteratively selects the lowest-average-IoU trajectory relative to the selected set.The procedure selects ten representative trajectories per task; the resulting set includes common and atypical motion patterns.Source: §A.2.2 · p. 13

A.3.1 Scene Consistency: DINOv2 vs. VBench Analysis

• Fine-tuned DINOv2 focuses on task-relevant agents and tools, while VBench’s features miss foreground-background distinctions and its scores remain high despite viewpoint or layout shifts.The appendix attributes this contrast to DINOv2’s adaptation to embodied data and reports decreased cosine similarity where VBench remains insensitive.Source: §A.3.1 · pp. 13–14, Figure 8 · p. 14, Figure 9 · p. 14

Symmetric Hausdorff Distance Consistency(HSD Consistency)

• HSD measures the maximum spatial deviation between generated and ground-truth trajectories, with its reciprocal used to make higher scores indicate greater consistency.It evaluates spatial alignment of the generated path with the true trajectory.Source: Symmetric Hausdorff Distance… · p. 14

Normalized Dynamic Time Warping Distance Consistency(NDTW Consistency)

• NDTW captures trajectory shape and temporal alignment, including action timing and order, and its reciprocal is used as the similarity score.Unlike HSD’s focus on spatial deviation, NDTW assesses how generated actions align with the timing and sequence of true actions.Source: Normalized Dynamic Time Warping… · pp. 14–15

Dynamic Consistency (DYN)

• DYN compares velocity and acceleration distributions with Wasserstein distance, using amplitude normalization to improve robustness across motion amplitudes.Soft matching avoids requiring strict temporal alignment; the normalization corrections are intended to prevent low-amplitude trajectories from numerically amplifying the assessment.Source: Dynamic Consistency (DYN) · p. 15

A.4 Visual Examples of Model Generation Results

• Domain-adapted models can depict the correct scene and goal yet still produce an empty grasp without object interaction.Figure 10 illustrates a robot hand moving toward the correct region but failing to close its gripper on the object.Source: §A.4 · p. 15, Figure 10 · p. 15
• LTX exhibits abrupt scene transitions and object discontinuity, while COSMOS fails to maintain a consistent viewpoint despite explicit viewpoint-control instructions.The examples describe these as temporal-coherence and spatial-constraint failures, respectively.Source: §A.4 · pp. 15–16, Figure 12 · p. 16, Figure 11 · p. 16
• OpenSora can depict the intended manipulation context while its robotic arm jitters and fails to move smoothly.The example illustrates motion-control difficulty despite partial task-semantic understanding.Source: §A.4 · p. 16, Figure 14 · p. 17
• COSMOS and LTX often generate human hands instead of robotic arms, indicating difficulty adapting their representations to robotic manipulation.The appendix links this pattern to a bias toward human-hand representations in their training data.Source: §A.4 · p. 15, Figure 13 · p. 16