Source-linked AI summary
WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Models
Yu Shang, Zhuohang Li, Yiding Ma, Weikang Su, Xin Jin, Ziyou Wang, Lei Jin, Xin Zhang, Yinzhou Tang, Haisheng Su, Chen Gao, Wei Wu, Xihui Liu, Dhruv Shah, Zhaoxiang Zhang, Zhibo Chen, Jun Zhu, Yonghong Tian, Tat-Seng Chua, Wenwu Zhu, Yong Li
TL;DR
Embodied world-model evaluation has focused largely on perceptual video quality, leaving downstream functional utility insufficiently assessed. WorldArena unifies perceptual, embodied-task, and human evaluation, introduces EWMScore, and finds a consistent gap between visual quality and embodied task performance across 14 models.
Problem
Existing embodied world-model benchmarks emphasize video-level quality and insufficiently assess functional roles in downstream embodied tasks.
Method
WorldArena evaluates 16 video metrics across six dimensions, three embodied task roles, and human judgments, while EWMScore aggregates multidimensional video metrics.
Results
Across 14 representative models, WorldArena reveals a consistent gap between perceptual quality and embodied task performance, while EWMScore correlates well with human judgments.
Takeaways & Limitations
Strong visual generation alone is insufficient for reliable embodied decision-making, motivating evaluation that jointly tracks perceptual and functional capability.
Abstract
from arXiv · showhide
While world models have emerged as a cornerstone of embodied intelligence by enabling agents to reason about environmental dynamics through action-conditioned prediction, their evaluation remains fragmented. Current evaluation of embodied world models has largely focused on perceptual fidelity (e.g., video generation quality), overlooking the functional utility of these models in downstream decision-making tasks. In this work, we introduce WorldArena, a unified benchmark designed to systematically evaluate embodied world models across both perceptual and functional dimensions. WorldArena assesses models through three dimensions: video perception quality, measured with 16 metrics across six sub-dimensions; embodied task functionality, which evaluates world models as data engines, policy evaluators, and action planners integrating with subjective human evaluation. Furthermore, we propose EWMScore, a holistic metric integrating multi-dimensional performance into a single interpretable index. Through extensive experiments on 14 representative models, we reveal a significant perception-functionality gap, showing that high visual quality does not necessarily translate into strong embodied task capability. WorldArena benchmark with the public leaderboard is released at https://world-arena.ai, providing a framework for tracking progress toward truly functional world models in embodied AI.
1. Introduction
WorldArena addresses fragmented embodied world-model evaluation by combining perceptual quality, embodied task utility, and human assessment. Across 14 representative models, it also introduces EWMScore and analyzes the gap between visual fidelity and functional capability.
- Motivation: Existing evaluations mainly measure video quality, leaving world models’ broader embodied roles and practical task utility insufficiently assessed.These roles include environment proxies, synthetic data engines, and policy-evaluation tools.
- Benchmark: WorldArena unifies perceptual and functional evaluation for embodied world models.The benchmark is designed as a comprehensive evaluation framework rather than a video-quality-only protocol.
- Benchmark: Its perceptual component uses 16 numerical metrics across six dimensions, while functional evaluation covers data synthesis, policy evaluation, and action planning.Human evaluation adds qualitative judgments about physical plausibility and instruction adherence.
- Experimental scope: The benchmark evaluates 14 representative world models using bimanual robotic manipulation scenarios from RobotTwin 2.0, which contains 50 diverse scenarios.The evaluated models include general video-generation models and specialized embodied world models.
- EWMScore: EWMScore is proposed as a unified objective metric, and human studies show that it aligns highly with subjective judgment as a reliable, interpretable index.This metric is intended to summarize multidimensional generative performance.
- Implications: WorldArena’s systematic analysis provides multidimensional evidence about model strengths and limitations for future embodied-world-model research.The benchmark is positioned as a framework for evaluating both perceptual and functional capabilities.
2. Related Works
Prior world-model benchmarks largely emphasize perceptual video-generation quality, while embodied benchmarks and robot-specialized models receive less comprehensive functional evaluation. WorldArena is framed against this division of general-purpose and embodied benchmark settings.
- World model scope: Embodied world models predict future observations of physical scenes involving robot locomotion and manipulation.They include video-generation-based, 3D-reconstruction-based, and latent-space models.
- Benchmark categories: Existing world-model benchmarks are broadly divided into general-purpose and embodied benchmarks.This categorization organizes the comparison underlying WorldArena’s related-work discussion.
- WorldArena comparison: WorldArena compares existing benchmarks across three key evaluation dimensions to position its unified perceptual and functional scope.The comparison is presented in Table 1.
- General-purpose benchmarks: General-purpose benchmarks primarily assess perceptual and generative video qualities, but generally omit functional roles in decision-making and interaction.They commonly evaluate visual fidelity, motion realism, content consistency, and sometimes physical or geometric consistency.
3. The WorldArena Benchmark
WorldArena evaluates embodied world models through open-loop video quality, closed-loop embodied tasks, and human judgments, then aggregates normalized video metrics into EWMScore. Its video evaluation spans six perceptual dimensions with specialized metrics for appearance, motion, consistency, physics, geometry, and controllability.
- Framework overview: The benchmark combines six-dimensional video evaluation, three embodied downstream tasks, human annotations, and the interpretable EWMScore index.Video quality uses 16 metrics, while embodied-task evaluation measures closed-loop performance.
- Video quality evaluation: Video evaluation covers visual quality, motion quality, content consistency, physics adherence, 3D accuracy, and controllability.These dimensions target perceptual reliability, dynamic coherence, scene stability, physical realism, spatial structure, and instruction responsiveness.
- Visual quality: Visual quality uses Image Quality, Aesthetic Quality, and JEPA Similarity to assess clarity, visual appeal, and similarity to ground-truth videos.The metrics rely on MUSIQ, a LAION aesthetic predictor, and V-JEPA feature distributions.
- Motion quality: Motion quality measures intensity and temporal continuity through Dynamic Degree, Flow Score, and Motion Smoothness.Optical flow captures movement strength, while frame interpolation evaluates smoothness against real frames.
- Content consistency: Content consistency evaluates subject, background, and photometric stability across frames using feature similarity and optical-flow-based alignment.Higher AEPE indicates poorer alignment for the photometric-consistency calculation.
- Functional perceptual dimensions: Physics adherence, 3D accuracy, and controllability assess physical interactions, trajectory correctness, spatial geometry, perspective plausibility, and instruction response.Controllability includes instruction following, semantic alignment, and action-conditioned video diversity.
- EWMScore: EWMScore linearly normalizes the 16 video metrics to [0,100] and averages them into an objective automated score of overall generative quality.The metric aggregates multidimensional video-quality results into a single composite index.
4. Experiments
Experiments evaluate 14 representative world models across video quality, embodied data synthesis, policy evaluation, action planning, and human assessment. Results show a persistent gap between perceptual quality and functional embodied performance.
- Experimental Setup: 14 representative world models are evaluated, including general-purpose video models and embodied-specific models, with available models post-trained on the evaluation dataset.The dataset uses RoboTwin 2.0, covering 50 robotic scenarios and 2,500 videos.
- Video Quality Evaluation: General video models excel mainly in visual quality, whereas embodied models perform better on structure- and interaction-related metrics.CtrlWorld and TesserAct score highly on subject consistency, background stability, and trajectory accuracy, while commercial models achieve the highest visual and aesthetic scores.
- Embodied Data Engine: Most generated data improves downstream policy performance but remains below real-data training; only RoboMaster and WoW surpass real-data training on Task 2.The data-engine evaluation trains policies from 25 synthetic trajectories per world model on two manipulation tasks.
- Embodied Policy Evaluator: CtrlWorld correlates strongly with the RoboTwin simulator for policy evaluation, while Cosmos-Predict 2.5 correlates more weakly and both models overestimate success rates.Five policies with varying performance levels are evaluated through action-conditioned observation rollouts.
- Embodied Action Planner: World models achieve non-trivial action-planning success rates but remain substantially below π0.5, especially for reliable long-horizon closed-loop execution.The comparison executes predicted action sequences in the RoboTwin simulator.
- Inter-metric Analysis: EWMScore correlates strongly with human judgments (Pearson r = 0.825), moderately with data synthesis (r = 0.600), and weakly with action planning (r = 0.360).The cross-dimensional analysis indicates that perceptual realism does not translate proportionally into downstream embodied-task performance.
5. Conclusion and Future Work
WorldArena unifies perceptual and functional evaluation of embodied world models through video metrics, embodied tasks, and human assessments. Across 14 models, it finds that visual generation quality alone is insufficient for reliable embodied decision-making, while EWMScore aligns well with human judgments.
- Conclusion: WorldArena integrates multidimensional video quality metrics, embodied task evaluations, and human assessments into one benchmark.The framework evaluates embodied world models from both perceptual and functional perspectives.
- Conclusion: Evaluation of 14 representative models reveals consistent gaps between perceptual quality and embodied task performance.The paper identifies strong visual generation alone as insufficient for reliable embodied decision-making.
- Future Work: EWMScore captures overall generative capability and correlates well with human judgments.The benchmark is intended to support continued expansion toward perceptually strong and functionally reliable embodied world models.
A. Additional Details on Metrics
The metric details combine no-reference image-quality assessment with per-frame aesthetic scoring for generated videos. These measures extend evaluation beyond technical sharpness to include visual appeal and perceptual consistency.
- Image Quality: MUSIQ evaluates per-frame technical image quality without reference images, capturing distortions such as overexposure, sensor noise, and compression artifacts.The resulting Simg score reflects visual purity and clarity, with higher values indicating better quality.
- Image Quality: For video sequence V = {I1, I2, . . . , IT}, Simg aggregates frame-level predictions from the pretrained MUSIQ quality function.The metric evaluates image quality at the digital-frame level across the sequence.
- Aesthetic Quality: The LAION Aesthetic Predictor maps each frame into an aesthetic feature space and derives an average score for the sequence.This captures harmonious lighting, color composition, perceptual coherence, and artistic consistency beyond pixel-level sharpness.
A.3. JEPA Similarity
JEPA Similarity evaluates generated videos by comparing their high-level spatiotemporal feature distributions with reference demonstrations. It avoids FVD’s Gaussian assumption and is designed to align more closely with human perception.
- Metric definition: JEPA Similarity measures maximum mean discrepancy between generated and reference video feature distributions.The features are extracted with a pretrained V-JEPA encoder.
- Metric definition: Unlike FVD, JEPA Similarity does not assume Gaussian feature distributions.It uses a second-order polynomial kernel to estimate the distribution distance.
- Interpretation: Higher JEPA Similarity values indicate closer alignment with reference demonstrations.The metric compares generated-video features against reference expert demonstrations.
- Advantages: JEPA Similarity is sensitive to high-level spatiotemporal breakdowns and offers improved sample efficiency over conventional metrics.The authors also report substantially stronger correlation with human subjective assessments for complex embodied-operation logic.
A.4. Dynamic Degree
Dynamic Degree measures motion intensity in generated videos using optical flow, emphasizing the most active pixels. The score is mapped adaptively so higher values represent more pronounced dynamic responses.
- Computation: Dynamic Degree uses RAFT optical flow and focuses on pixels in the top 5% of motion magnitudes.This targets salient movements such as robotic-arm grasping.
- Computation: The score applies a smooth numerical mapping with a resolution-adaptive threshold based on the video dimensions.The threshold is τ = 6 256 × min(H, W), while α controls mapping steepness.
- Interpretation: Dynamic Degree values closer to 1 indicate more pronounced dynamic responses in the video.The metric is intended to capture motion in salient regions rather than overall pixel activity alone.
- Purpose: Optical-flow motion scoring helps identify static-frame degeneration, minimal drift, and unnatural global distortions.The metric therefore assesses both insufficient motion and implausible scene dynamics.
A.6. Motion Smoothness
Motion Smoothness evaluates temporal coherence by predicting intermediate frames from odd-indexed inputs and comparing them with the real intermediate frames. Its weighting accounts for motion magnitude rather than treating static backgrounds as equally informative.
- Method: Motion Smoothness uses VFI-Mamba to reconstruct intermediate frames from odd-indexed video frames.The predicted intermediates are compared with ground-truth frames to assess temporal plausibility.
- Scoring: The score combines interpolation fidelity with the logarithm of motion magnitude between neighboring frames.The displayed formulation multiplies SSIM by ln(1 + diff(Iprev, Inext)).
- Scoring: Motion weighting prevents static backgrounds from inflating the smoothness score.It also accounts for the greater difficulty of interpolation during large motion.
A.7. Subject Consistency
Subject Consistency measures whether objects remain consistent across video frames using DINO feature similarities. WorldArena adjusts this consistency score with dynamic degree so near-static videos cannot obtain artificially high scores from frame similarity alone.
- Subject Consistency: Subject Consistency compares DINO features from the first, current, and previous frames using cosine similarity.Higher similarity indicates better object consistency across frames.
- Subject Consistency: Static-video shortcuts can produce artificially high frame-consistency scores despite weak dynamic generation.This motivates incorporating dynamic degree into the subject-consistency calculation.
- Subject Consistency: The dynamic-degree adjustment penalizes subject-consistency scores when video motion falls below a predefined threshold.The mechanism prevents static or near-static videos from receiving high scores solely from frame-level similarity.
- Background Consistency: Background Consistency uses CLIP features to measure scene stability relative to the first and previous frames.CLIP emphasizes global scene semantics and helps detect uncontrolled background variation.
- Photometric Consistency: Photometric Consistency evaluates pixel-level stability by warping frames with forward and backward optical flow.It uses average end-point error as the reconstruction-error measure.
- Photometric Consistency: Dynamic-degree weighting ensures full photometric-consistency credit is retained only when motion satisfies Sdyn ≥γ.Static sequences are proportionally penalized, encouraging fidelity during actual motion.
B. The Prompt of VLM-based Policy Success Judgement in Policy Evaluator Task
The policy evaluator uses a VLM-based judge to compare policy-generated rollouts with ground-truth trajectories, prioritizing arm selection, task completion, and action intent. Success requires the correct arm when specified, a similar final state, and instruction-consistent motion, while rendering differences are tolerated within defined limits.
- The VLM judge compares policy-generated video rollouts against ground-truth reference trajectories to determine task success.The input contains uniformly sampled ground-truth and policy frames.
- Evaluation prioritizes correct arm selection, similarity between final task states, and consistency between the motion process and instruction semantics.An explicitly required left or right arm must be used; final-frame comparison determines whether the task is basically completed.
- Visual hallucinations, minor trajectory differences, and video-length differences are treated as tolerable rendering variations.
- A successful judgment requires the correct arm, a final state similar to ground truth, and the correct action intent.The output is a binary answer: 1 for success and 0 for failure.
- Failure includes using the wrong arm, producing a substantially different final state, moving in the wrong direction, or operating the wrong object.
C. Case Comparison of Each Metric in EWMScore
The metric examples contrast good and bad generated videos across visual quality, motion quality, content consistency, physics adherence, 3D accuracy, and controllability. Good cases preserve structure, coherent motion, stable content, physical interaction, spatial accuracy, and instruction-aligned behavior, whereas bad cases exhibit distortions, artifacts, or incorrect actions.
- Visual Quality: Good visual-quality examples preserve sharp structural details, realistic lighting, clear reflections, and close style and morphology alignment.Bad examples show motion blur, noise, geometric distortion, and artifacts.
- Motion Quality: Good motion-quality examples show complete manipulation sequences and fluid pixel-level movement, while bad examples remain static or barely move.
- Content Consistency: Good content-consistency examples maintain the object’s identity, background, and camera perspective throughout interaction.Bad examples deform the object or destabilize the background and viewpoint.
- Physics Adherence: Good physics-adherence examples show physical gripper contact and ground-truth-aligned trajectories, unlike examples with unsupported lifting or anomalous deviations.
- 3D Accuracy: Good 3D-accuracy examples align generated depth with ground truth and preserve realistic perspective and lighting.Bad examples suffer geometric distortion, spatial-integrity collapse, ghosting, or blur.
- Controllability: Good controllability examples follow instructions and preserve semantic meaning, whereas bad examples move the wrong object or transform content into unrelated forms.