Source-linked AI summary
Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models
Meiqi Wu, Zhixin Cai, Fufangchen Zhao, Xiaokun Feng, Rujing Dang, Bingze Song, Ruitian Tian, Jiashu Zhu, Jiachen Lei, Hao Dou, Jing Tang, Lei Sun, Jiahong Wu, Xiangxiang Chu, Zeming Liu, Kaiqi Huang
TL;DR
Existing benchmarks measure visual quality and text–video alignment but do not adequately evaluate interactive response. Omni-WorldBench introduces a hierarchical, agent-based benchmark and evaluates 18 models, finding strong conventional video quality alongside limitations in action-conditioned evolution and causal interaction consistency.
Problem
Existing evaluation methods emphasize visual fidelity and text–video alignment, but do not adequately capture consistent, plausible responses under varying interaction conditions.
Method
Omni-WorldBench combines Omni-WorldSuite, a hierarchical prompt suite spanning interaction levels and scenario types, with Omni-Metric, an agent-based protocol measuring action impacts on final outcomes and intermediate state transitions.
Results
Evaluation of 18 representative world models reveals strong conventional video quality but clear limitations in action-conditioned world evolution, causal interaction consistency, and joint camera-object control.
Takeaways & Limitations
The results support evaluating world models beyond passive video quality toward agent-centric interactive generation.
Takeaways & Limitations
Omni-WorldBench cannot fully capture open-world interactive environments, especially long-horizon and highly dynamic settings, and human-aligned evaluation results remain planned.
Abstract
from arXiv · showhide
Video--based world models have emerged along two dominant paradigms: video generation and 3D reconstruction. However, existing evaluation benchmarks either focus narrowly on visual fidelity and text--video alignment for generative models, or rely on static 3D reconstruction metrics that fundamentally neglect temporal dynamics. We argue that the future of world modeling lies in 4D generation, which jointly models spatial structure and temporal evolution. In this paradigm, the core capability is interactive response: the ability to faithfully reflect how interaction actions drive state transitions across space and time. Yet no existing benchmark systematically evaluates this critical dimension. To address this gap, we propose Omni--WorldBench, a comprehensive benchmark specifically designed to evaluate the interactive response capabilities of world models in 4D settings. Omni--WorldBench comprises two key components: Omni--WorldSuite, a systematic prompt suite spanning diverse interaction levels and scene types; and Omni--Metrics, an agent-based evaluation framework that quantifies world modeling capabilities by measuring the causal impact of interaction actions on both final outcomes and intermediate state evolution trajectories. We conduct extensive evaluations of 18 representative world models across multiple paradigms. Our analysis reveals critical limitations of current world models in interactive response, providing actionable insights for future research. Omni-WorldBench will be publicly released to foster progress in interactive 4D world modeling.
1 Introduction
Existing benchmarks mainly measure visual fidelity and text–video alignment, leaving interaction-conditioned world-state evolution insufficiently evaluated. Omni-WorldBench addresses this gap with a hierarchical prompt suite and an agent-based protocol for interactive response.
- Conventional metrics such as FID, FVD, and VBench measure visual fidelity or text–video alignment but miss consistent responses under varying interaction conditions.
- Omni-WorldBench organizes prompts across three interaction levels, from object-local effects to localized object interactions and global environmental changes.The suite also covers real-world physical settings and application domains such as autonomous driving, embodied robotics, and gaming.
- Omni-Metrics evaluates dynamic, controllable, and interaction-aware generation rather than focusing predominantly on static visual fidelity.
- The benchmark evaluates 18 representative world models and reports their performance boundaries and limitations in interactive response.Human-alignment studies further demonstrate that Omni-Metric aligns well with human preferences.
2 Related Works
Video-based world models have expanded across autonomous driving, robotics, and gaming, but their evaluation remains dominated by generic video metrics and benchmarks with limited interaction coverage. Omni-WorldBench responds by evaluating multiple interaction levels and scenario types.
- Video-based world models support distinct application goals, including long-horizon traffic evolution, vehicle decision-making, object-centric dynamics, and manipulation control.
- Existing evaluations rely heavily on FID, IS, FVD, and video-generation tools such as VBench, whose scores can diverge from human perceptual judgments.
- Prior coverage is largely restricted to camera motion, whereas contemporary world models increasingly address broader interaction types.
- Omni-WorldBench systematically covers multiple interaction complexities and aims to characterize the interactive expressiveness of world models.
3 Omni-WorldSuite
Omni-WorldSuite combines diverse scene domains with a three-level hierarchy of interaction effects, using grounded prompts, initial frames, and optional camera trajectories. Its 1,068 prompts span physical, causal, spatial, camera, and task-oriented capabilities.
- Scene coverage: Omni-WorldSuite spans general daily-life scenes and task-oriented environments, covering physical laws, commonsense, causality, camera motion, loop closure, and spatial constraints.
- Interaction hierarchy: Level 1 confines effects to the acting object, Level 2 models one object affecting another, and Level 3 captures broader changes involving multiple objects.
- Prompt construction: Dataset-grounded prompts use source-video first frames and camera trajectories, VLM captions, and manual verification to improve spatial and attribute consistency.
- Prompt construction: Concept-driven prompts sample scene, entity, action, and interaction-level prototypes through a generate–verify–refine pipeline.
- Benchmark comparison: Compared with prior benchmarks, Omni-WorldBench supports text, image, and trajectory inputs while covering general and task-oriented scenes and explicitly treating interaction type as an evaluation dimension.
- Statistics: 1,068 evaluation prompts cover six annotation dimensions, with Level 2 containing the most prompts, followed by Level 3 and Level 1.The dimensions are Physics Principles, Commonsense, Causality, Camera Motion, Loop-Closure Consistency, and Spatial Constraints.
4 Omni-Metric
Omni-Metric evaluates world models across video quality, camera-object controllability, and interaction effect fidelity. It combines structured video representations with metrics for visual coherence, controllability, temporal stability, and causal consistency.
- Evaluation dimensions: Omni-Metric evaluates Generated Video Quality, Camera-Object Controllability, and Interaction Effect Fidelity as complementary dimensions.Together, these dimensions assess perceptual quality, scene and object coherence, and interaction-related dynamics.
- Evaluation pipeline: The evaluation pipeline generates a video from an evaluation prompt, then extracts entity trajectories, optical flow, and relative camera motion before computing metrics.Entity masks are obtained with GroundingDINO and SAM, optical flow with RAFT, and camera motion is approximated from flow variations.
- Camera-Object Controllability: Camera-Object Controllability measures camera motion, object consistency, and scene transitions to assess whether static scene elements remain coherent.Its metrics cover camera trajectory discrepancies, prompt-specified object grounding, and detected scene boundaries.
- Camera-Object Controllability: Transition Detection uses content-based scene boundaries, assigning full continuity credit when N=1 and zero when N>1.PySceneDetect flags boundaries using frame-to-frame HSV dissimilarity, a threshold τ, and a minimum scene-length constraint L.
- Interaction Effect Fidelity: Interaction Effect Fidelity evaluates long-term consistency, causal event ordering, physical plausibility, and object-level responses through dedicated interaction metrics.InterStab-L measures revisit-pair consistency, while InterCov checks affected entities’ semantic responses and unaffected entities’ temporal stability.
- Interaction Effect Fidelity: InterCov verifies that interaction-affected entities respond semantically consistently while unaffected entities maintain temporal stability.This provides a high-level semantic complement to low-level flow-based coverage and targets adherence to causal structure.
5 Experiments
Omni-WorldBench evaluates 18 representative world models across generation paradigms using 15 metrics covering video quality, interaction fidelity, and controllability. Results show trade-offs across these dimensions, with strong visual quality but persistent weaknesses in interaction consistency and joint control.
- Evaluation Setup: 18 representative world models across T2V, IT2V, and camera-controlled generation are evaluated with Omni-WorldBench.The models span diffusion-based, autoregressive, and hybrid paradigms.
- Evaluation Setup: 15 metrics assess generated video quality, interaction effect fidelity, and camera and object controllability.The protocol uses Omni-WorldSuite as its custom test set and Omni-Metric as the evaluation framework.
- Overall Performance: IT2V models show the highest overall performance potential, with Wan2.2 achieving the highest overall AgenticScore at 75.92%.Cosmos follows at 75.42%, while HunyuanVideo leads pure T2V models at 73.96%.
- Interaction Effect Fidelity: Wan2.2 achieves the highest IT2V interaction-effect-fidelity average at 67.34%, while WonderWorld ranges from 84.96% on InterStab-L to 24.89% on InterStab-N.The contrast indicates a substantial trade-off for some camera-controlled models when handling complex interaction logic and camera scheduling.
- Generated Video Quality: Most models exceed 95.00% on Temporal Flickering and Motion Smoothness, but Dynamic Degree varies substantially across models.ViewCrafter and WonderWorld achieve 100.00% Dynamic Degree, shifting key differences toward content alignment and dynamic responsiveness.
- Controllability: Camera-aware methods lead explicit control, with WonderWorld scoring 96.12% on Camera Control and Cosmos and Wan2.2 reaching 94.90% and 94.01% on Object Control.HunyuanWorld obtains the best average controllability score in its group at 79.67%.
- Overall Findings: Current models remain limited in action-conditioned world evolution, causal interaction consistency, and joint camera-object control despite strong conventional video quality.These results motivate evaluation beyond passive visual quality toward agent-centric interactive generation.
- Qualitative Analysis: Qualitative comparisons show Wan2.2 producing a complete, anatomically reasonable throwing motion while preserving structural integrity and scene coherence.For camera-controlled generation, HunyuanWorld remains relatively stable, whereas ViewCrafter introduces a spurious building that degrades visual consistency.
6 Conclusion
Omni-WorldBench is introduced as an interaction-centric benchmark for evaluating action-driven scene evolution, intermediate state transitions, and causal consistency. Evaluation of 18 models reveals substantial gaps between visual realism and true interactivity, while the benchmark remains limited for open-world, long-horizon settings.
- Contribution: Omni-WorldBench evaluates interactive response through hierarchical prompts and agent-based metrics measuring action effects on outcomes and intermediate state transitions.The framework also assesses non-intervention consistency, spatiotemporal causal coherence, and visual quality through an overall AgenticScore.
- Findings: Evaluation across 18 video generation and world models reveals substantial gaps between visual realism and true interactivity.The benchmark emphasizes action-driven scene evolution rather than visual quality or motion realism alone.
- Limitations: Omni-WorldSuite cannot fully capture open-world interactive environments, especially long-horizon and highly dynamic settings.The authors also plan future human-aligned evaluation results to complement and validate interaction-quality assessment.