Source-linked AI summary
WorldOlympiad: Can Your World Model Survive a Triathlon?
Yuke Zhao, Wangbo Zhao, Weijie Wang, Zeyu Zhang, Dakai An, Akide Liu, Yinghao Yu, Jiasheng Tang, Fan Wang, Wei Wang, Bohan Zhuang
TL;DR
Video-based world-model benchmarks provide limited evidence about physical rules, 3D structure, and controllable long-horizon interaction. WorldOlympiad evaluates these dimensions across gaming, robotics, and real-world scenarios, finding current models remain far from reliable world simulators.
Problem
Existing evaluations provide limited evidence that video-based world models preserve state continuity, physical and geometric constraints, user responsiveness, and plausible long-horizon dynamics.
Method
WorldOlympiad evaluates gaming, robotics, and real-world videos through complementary metrics for physical faithfulness, geometric consistency, and chunk-level interactive generation.
Results
Experiments show current world-model pipelines remain far from reliable simulators, failing on physical rules, 3D structure, or long-horizon state preservation.
Takeaways & Limitations
Perceptually plausible video generation does not establish controllable world modeling, motivating structured evaluation across physical, geometric, and interaction capabilities.
Takeaways & Limitations
The benchmark does not yet isolate memory mechanisms from confounding differences in model scale, training data, and architecture.
Abstract
from arXiv · showhide
We introduce WorldOlympiad, a benchmark for diagnosing video-based world models across physical faithfulness, geometric consistency, and interaction fidelity. While existing benchmarks often focus on visual quality, semantic alignment, or short-term temporal coherence, they provide limited insight into whether generated videos obey physical rules, preserve coherent 3D structure, and sustain controllable interactions over long horizons. To address this gap, WorldOlympiad decomposes world-model evaluation into three complementary dimensions. The physical track uses object segmentation and MLLM-as-judge to assess whether generated videos follow interpretable rules in mechanics, thermal phenomena, and material properties. The geometry track reconstructs generated videos with Gaussian splatting and evaluates structural consistency, cross-view coherence, and camera-trajectory alignment. The interaction track assesses whether generated rollouts follow complex action prompts and maintain smooth, coherent transitions across consecutive video chunks. WorldOlympiad further covers three major downstream scenarios, including gaming, robotics, and general real-world videos, capturing diverse challenges from interactive control and embodied manipulation to open-domain motion and camera dynamics. Together, these tracks and scenarios form a scalable and interpretable evaluation suite that exposes failure modes beyond generic video quality. Experiments on state-of-the-art models reveal substantial gaps in physical reasoning, 3D consistency, and long-horizon interaction, underscoring the need for more structured evaluation protocols for generative world models.
1 Introduction
WorldOlympiad addresses limitations in existing video-generation benchmarks by evaluating video-based world models across long-horizon physical, geometric, and interactive capabilities. It provides a unified protocol spanning gaming, robotics, and real-world scenarios, built from 1,000 long videos and applied to 8 generation pipelines.
- Motivation: Video-based world models must predict future visual states from observations and controls while preserving more than visual fidelity.The benchmark motivation emphasizes requirements for game simulation, robotic policy development, and real-world scene generation.
- Limitations of Existing Benchmarks: Existing video benchmarks mainly measure perceptual quality or text–video alignment and do not fully capture long-horizon controllable interaction.VBench, VBench-2.0, CLIP-based metrics, and VBench++ remain insufficient for core world-modeling properties.
- Limitations of Existing Benchmarks: Current world-model benchmarks often target a single domain, limiting unified comparison across gaming, robotics, and real-world scenarios.The paper frames the unresolved question around reliable simulation of world dynamics across multiple domains and long horizons.
- WorldOlympiad: WorldOlympiad evaluates long-horizon consistency, physical-law adherence, 3D geometric consistency, and faithful responses to control signals across diverse domains.Its multi-dimensional judge metrics assess physical laws, 3D geometry, and chunk-by-chunk interactive generation.
- WorldOlympiad: WorldOlympiad comprises 1,000 high-quality long videos and benchmarks 8 long-video generation pipelines across gaming, robotics, and real-world applications.The dataset covers interactive game control, embodied robotic manipulation, and open-domain motion and camera dynamics.
2 Related Work
Prior work shows that video world models can exhibit physical consistency and support applications such as gaming and robotics, but existing generators and benchmarks remain limited in duration, domain coverage, and interactive-functionality assessment.
- Video Generation Models: Diffusion-based video generators exhibit emergent physical consistency, including object permanence, 3D coherence, and plausible motion dynamics.These properties are attributed to large-scale training.
- Video Generation Models: Early diffusion-based video generators typically produce short clips of 5–10 seconds, limiting their use as persistent world-model simulators.The passage identifies short-clip optimization as a limitation for persistent simulation.
- World-Model Applications: World models have been applied to interactive game generation and robotics simulation, with systems demonstrating controllable character actions and environment dynamics.Real-time interaction remains a central capability requirement regardless of the target application.
- Existing Benchmarks: General video benchmarks such as VBench and VBench 2.0 evaluate visual quality, motion authenticity, semantic consistency, and physical plausibility.World-model benchmarks additionally target dimensions including physical law adherence, simulation fidelity, and functionality.
- Existing Benchmarks: Existing benchmarks lack unified coverage of gaming, robotics, and general scene generation, while interactive functionality remains notably underassessed.Robotics-focused benchmarks extend evaluation to controllability, action conditioning, and closed-loop interaction, but do not resolve the broader coverage gap.
3 WorldOlympiad
WorldOlympiad is a multi-domain benchmark for evaluating world-model behavior across robotics, gaming, and real-world videos. It standardizes long-video annotations and evaluates physical faithfulness, geometric consistency, and interaction-related semantic adherence using structured preprocessing and model-based judgments.
- Benchmark composition: WorldOlympiad contains 400 robotics, 400 gaming, and 200 real-world videos targeting manipulation, interactive control, long-context state evolution, open-domain motion, and camera dynamics.The robotics, gaming, and real-world subsets emphasize complementary world-modeling requirements.
- Benchmark composition: The benchmark draws robotics videos from RoboCOIN, gaming videos from GameGen-X, and filtered long, high-motion real-world videos from LVD-2M.RoboCOIN supports bimanual manipulation and physically grounded interactions; GameGen-X supports interactive game behavior; LVD-2M supports everyday dynamics and camera movement.
- Annotation pipeline: A three-stage chunk-caption-refine pipeline uses Gemini-3-Pro-Preview to produce accurate, comprehensive annotations for long videos.Videos are chunked into at most six contiguous, gap-free intervals; each chunk receives an action field and caption, followed by full-video refinement for continuity and camera-label validation.
- Evaluation tracks: Physical faithfulness spans mechanics, thermodynamics, and material properties, combining MLLM entity identification, SAM3 masks and trajectories, and relevance-aware metric evaluation.The rule-based metrics cover phenomena such as gravity, buoyancy, compression, impact, phase transitions, color mixing, solubility, hardness, and combustibility.
- Evaluation tracks: Geometric consistency uses reconstruction, diagnostic meta-view, and camera-trajectory agreement signals derived from Gaussian-splat reconstruction and calibrated MLLM judgments.Depth Anything 3 estimates the Gaussian scene and camera parameters, while judgments assess layout, 3D structure, cross-view geometry, and prompt-consistent organization.
- Evaluation tracks: Interaction evaluation includes a CLIP-based semantic-adherence score computed from uniformly sampled frames within each video chunk.The supplied passage identifies this as the first component of the interaction metric.
4 Experiment
Experiments evaluate eight video world-model pipelines across gaming, robotics, and general real-world videos using WorldOlympiad’s physical, geometric, interaction, and composite scores. Results show strong physical performance but persistent geometric and long-horizon interaction weaknesses, while automatic rankings closely match human preferences.
- Evaluation setup: Eight video-generation pipelines are evaluated across gaming, robotics, and general real-world video scenarios.The evaluated pipelines span gaming-centric, robotics-centric, and general long-video world-model families.
- Overall results: LingBot-World achieves the highest overall score (0.683), alongside strong physical faithfulness (0.942).Cosmos-Predict-2.5 also generalizes beyond embodied manipulation and achieves strong physical fidelity across diverse downstream settings.
- Physical faithfulness: Physical scores are high for several pipelines, but thermodynamics and material-level questions remain more fragile than many mechanics questions.LingBot-World, Cosmos-Predict-2.5, Rolling Forcing, LongLive, and Yume-1.5 score 0.942, 0.906, 0.873, 0.863, and 0.863, respectively, on physical faithfulness.
- Geometric consistency: Geometric consistency remains unresolved: Hunyuan-WorldPlay leads at 0.424, while most models remain between 0.25 and 0.40.The passage links this weakness partly to interaction designs that rely primarily on camera or viewpoint control.
- Fine-grained diagnostics: WorldOlympiad’s diagnostic breakdowns and qualitative cases identify failures in physical rules, 3D reconstruction, semantic grounding, and long-range interaction drift.The benchmark decomposes aggregate scores into domain, physical, reconstruction, and interaction submetrics to explain model failures.
- Human alignment: Automatic and human rankings are highly consistent, with a Spearman correlation coefficient of ρ = 0.95 across eight annotated models.Human comparisons cover perceived quality, physical plausibility, temporal coherence, and interaction fidelity.
5 Conclusion · A WorldOlympiad Judge Prompt Templates · B Detailed Results
WorldOlympiad evaluates video world models through physical faithfulness, geometric consistency, and interaction fidelity, using unified diagnostics across gaming, robotics, and general-world pipelines. Its prompt templates operationalize dynamic-object extraction, physical judging, interaction assessment, and 3D reconstruction, while detailed results report domain-wise and model-level diagnostics.
- 5 Conclusion: WorldOlympiad measures physical faithfulness, geometric consistency, and interaction fidelity beyond surface-level visual quality.It combines rule-based physical judging, 3D reconstruction diagnostics, and chunk-level plus long-range interaction evaluation.
- 5 Conclusion: Experiments across gaming-centric, robotics-centric, and general world-model pipelines diagnose whether generated videos behave as reliable world simulations.
- 5 Conclusion: Future work will use controlled evaluation to isolate how memory mechanisms affect long-horizon consistency and interactive controllability.Different model scales, training data, and architectures currently confound attribution of performance gains to memory itself.
- A WorldOlympiad Judge Prompt Templates: The judge prompt templates cover dynamic-object extraction, physical consistency, interaction quality, and 3D reconstruction quality.Dynamic-object extraction identifies primary moving or deforming objects for SAM-based masking, visualization, and background completion, returning at most three objects.
- A.2 Physical Judge Prompts: The physical pipeline first determines which rules are judgeable in the reference, then compares generated candidates against the reference for compliance.Judged rules span mechanics, thermodynamics, and material behavior, with requirements for plausible physics, identity consistency, temporal order, and visible evidence.
- A.3 Interaction Judge Prompts: The interaction pipeline evaluates generated videos at chunk, adjacent-transition, and stitched-full-video levels.It checks action and caption adherence, smooth transitions, stable identities, coherent scene and camera behavior, and global semantic alignment using scores from 0 to 5.
- A.4 3D Judge Prompts: The 3D pipeline rewrites prompts for static-scene reconstruction after masking and inpainting dynamic foreground actors, then scores Gaussian-Splat renders and meta-view images.Camera-trajectory score Straj is computed from DA3 camera-motion similarity.
- B Detailed Results: Detailed results report domain-wise scores, physical pass rates, interaction diagnostics, geometry diagnostics, and model-level submetrics.
B.1 Domain-wise Results
Table 6 reports domain-wise WorldOlympiad scores on the same-scene subset across gaming, robotics, and general domains, including physical, geometric, interaction, semantic, and overall evaluation.
- B.1 Domain-wise Results: Table 6 evaluates gaming, robotics, and general domains using physical faithfulness, 3D consistency, interaction fidelity, semantic alignment, and overall score.The table includes CLIP-augmented interaction fidelity, raw and calibrated CLIP semantic alignment, and the overall score.
- B.1 Domain-wise Results: The overall score is the equal-weight average of physical faithfulness, 3D consistency, and interaction fidelity.This aggregation defines the All score reported in Table 6.
B.2 Fine-grained Physical Results
This section reports fine-grained physical evaluation on the same-scene subset. Results are presented both as aggregate pass rates by physical dimension and as pass rates for individual physical questions.
- Table 7 reports physical pass rates aggregated by physical dimension on the same-scene subset.
- Table 8 further breaks down same-scene physical results into individual physical questions.
B.3 Fine-grained Interaction Results
Fine-grained interaction diagnostics separate local action following, boundary smoothness, and long-range consistency in stitched videos. The interaction score aggregates these diagnostics, while raw CLIP scores are calibrated into a bounded auxiliary score.
- B.3 Fine-grained Interaction Results: The chunk score measures local caption and action following.
- B.3 Fine-grained Interaction Results: The transition score measures boundary smoothness between adjacent chunks, while the global score measures long-range consistency over the stitched video.
- B.3 Fine-grained Interaction Results: Raw CLIP scores are calibrated into a bounded auxiliary score with fixed thresholds, and the interaction score is the aggregate metric reported in Table 6.
B.4 Fine-grained Geometry Results
This section reports fine-grained geometry diagnostics on the same-scene subset, separating reconstruction-video quality, rendered meta-view quality, and camera-trajectory consistency. It also relates the aggregate 3D consistency score to the geometry metric reported in Table 6.
- Srecon measures Gaussian-splat reconstruction-video quality, while Smeta measures rendered meta-view image quality.
- Straj measures camera-trajectory consistency, and the 3D consistency score is the aggregate geometry metric reported in Table 6.
- Table 10 evaluates fine-grained geometry diagnostics on the same-scene subset.
B.5 Model-level Fine-grained Results
This section aggregates fine-grained diagnostics at the model-category level, covering geometry and interaction. It reports model-level 3D consistency and interaction submetrics in Tables 11 and 12.
- Geometry: Table 11 reports model-level 3D consistency submetrics.
- Interaction: Table 12 reports model-level interaction submetrics.
C Case Study
WorldOlympiad uses representative qualitative case studies to diagnose failure modes beyond generic video quality, comparing models under identical source prompts or reference contexts. The cases span gaming, robotics, and real-world videos, highlighting geometry consistency, interaction fidelity, physical plausibility, and temporal coherence.
- The case studies use the same source prompt or reference context across models, isolating model behavior from prompt variation.
- Gaming case study: Gaming cases diagnose geometry consistency and interaction fidelity, including scene stability under camera movement and failures in action following or cross-chunk transitions.
- Robotics case study: Robotics cases jointly evaluate physical plausibility, scene-level geometric consistency, and instruction following, including unsupported floating objects that violate gravity and support constraints.
- General case study: Real-world cases apply all three evaluation dimensions to plausible object motion, spatial and semantic consistency over time, camera motion, and long-range visual coherence.
D Human Preference Study Details
The human preference study compares anonymized video pairs across sampled prompts using five annotators and evaluates alignment between human rankings and WorldOlympiad’s automatic ranking. Human and automatic rankings show strong agreement, with only two adjacent model-pair swaps.
- Annotation protocol: Five annotators compare anonymized generated videos for 20 sampled prompts, judging visual quality, physical plausibility, temporal coherence, and interaction fidelity.Model names are hidden, and ties are allowed when videos are indistinguishable or have balanced strengths and weaknesses.
- Annotation protocol: The study produces 560 prompt-level pairwise comparisons and 2,800 individual preference labels across 28 unordered model pairs per prompt.Each comparison receives independent labels from all five annotators.
- Score aggregation: Each comparison is labeled A, B, or Tie, with wins scored 1, ties 0.5, and losses 0 before averaging annotator labels.Model-level preference rates are computed from the aggregated pairwise outcomes, using Nm = 140 valid comparisons per model.
- Rank correlation: 0.95 Spearman rank correlation is obtained across eight annotated models, indicating strong agreement between human preference and WorldOlympiad automatic rankings.Human ranks sort Shuman descending, while WorldOlympiad ranks sort the automatic overall evaluation score Sauto descending.
- Rank correlation: Rank disagreements occur only for two adjacent pairs: LongLive versus Yume-1.5, and Matrix-Game 2.0 versus WoW.These swaps have limited effect on overall correlation while identifying borderline cases where human and rubric-based automatic scores differ.