PaperlayerView exact revision on arXiv

Source-linked AI summary

WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation

Kaining Ying, Hengrui Hu, Siyu Ren, Jiamu Li, Fengjiao Chen, Ziwen Wang, Xuezhi Cao, Xunliang Cai, Henghui Ding

arXiv:2605.25874v1cs.CV

TL;DR

Existing interactive world-model benchmarks provide fragmented coverage, limiting systematic comparison and failure diagnosis. WBench introduces a comprehensive multi-turn benchmark across five dimensions and finds that no model performs strongly across all dimensions.

  • Problem

    Existing benchmarks cover interactive world-model competencies only partially, making fair comparison and failure diagnosis difficult across visual quality, controllability, memory, and physics.

  • Method

    WBench evaluates multi-turn interactions across five dimensions using diverse settings, four interaction types, unified navigation controls, and 22 automatic metrics.

  • Results

    No model dominates all five dimensions, while navigation is largely independent of other dimensions and interaction types degrade unevenly across turns.

  • Takeaways & Limitations

    The benchmark provides diagnostic insights into model strengths, weaknesses, and open challenges for interactive world-model development.

  • Takeaways & Limitations

    The test set focuses on discrete action sequences rather than continuous control, and physical evaluation partly relies on LMMs whose reliability may degrade for subtle effects.

Abstract

from arXiv · show

Interactive world models are advancing rapidly, yet existing benchmarks cover only part of the required competencies, leaving no unified standard for systematic evaluation. To fill this gap, we introduce WBench, a comprehensive multi-turn benchmark for interactive world model evaluation along five dimensions, namely video quality, setting adherence, interaction adherence, consistency, and physics compliance. WBench contains 289 test cases and 1,058 interaction turns, where each case specifies a world setting and a multi-turn interaction sequence, covering diverse scenes, styles, subjects, and both first- and third-person perspectives, together with four interaction types, including navigation, subject action, event editing, and perspective switching. For navigation, WBench unifies text, 6-DoF pose, and discrete-action control, enabling evaluation of models with different native input interfaces. Evaluation uses 22 automatic sub-metrics that combine specialist vision models with large multimodal models, and all metrics are validated against human judgments. Across 20 state-of-the-art models, we find that no single model performs strongly across all dimensions. We provide detailed diagnostic insights into the characteristic strengths, weaknesses, and open challenges of each model. Code and data are available at https://github.com/meituan-longcat/WBench.

1 Introduction

WBench addresses fragmented evaluation of interactive world models with a unified, comprehensive multi-turn benchmark spanning five complementary dimensions. It standardizes diverse settings, interactions, and control paradigms, then evaluates 20 models to reveal differentiated strengths and failure patterns.

  • Motivation: Existing evaluation is fragmented across selected demonstrations and task-specific protocols, hindering fair comparison and failure diagnosis across visual quality, controllability, memory, and physics.The paper frames interactive world models as requiring complementary Renderer, Director, Controller, Memory, and Engine roles, which existing benchmarks cover only partially.
  • Benchmark design: WBench defines each test case with a world setting and a multi-turn interaction sequence spanning diverse scenes, subjects, styles, and perspectives.The benchmark includes navigation, subject action, event editing, and perspective switching interactions.
  • Benchmark design: 289 cases and 1,058 turns support full-benchmark evaluation, while a shared navigation subset of 158 cases enables comparison across native control interfaces.Navigation is represented as text, camera pose, and discrete action; all 20 models use the shared subset, while text-prompted I2V models use the full benchmark.
  • Findings: No model dominates all five dimensions, while navigation is largely independent, physical correctness correlates with rendering quality, and navigation is the most fragile interaction type over turns.The experiments also distinguish camera control from perspective consistency and identify perspective, scene type, and subject category as structured sources of benchmark difficulty.
  • Evaluation: The benchmark unifies five evaluation dimensions with 22 fine-grained sub-metrics and applies a fully automatic evaluation pipeline to 20 models.The pipeline establishes diagnostic baselines and supports fair cross-paradigm comparison.

2 Related Work

Prior work spans increasingly scalable video generation models and interactive video world models driven by observations and actions. Existing evaluation benchmarks largely assess non-interactive quality, alignment, or physical commonsense, while world-model suites target specialized capabilities such as camera-conditioned generation or decision-oriented quality.

  • Video Generation Models: Video generation progressed from U-Net diffusion models to scalable Diffusion Transformers trained with flow matching on large-scale data.This progression produced longer, higher-resolution, and more temporally coherent outputs.
  • Video Generation Models: Recent frontier systems, including Sora 2, Kling 3.0, Veo 3, and Wan 2.7, advance cinematic quality and prompt adherence.
  • Interactive Video World Models: Interactive video world models synthesize next frames from the current observation and an action signal, enabling closed-loop environment simulation.Earlier systems also appeared in robotic manipulation and autonomous driving, including UniSim, IRASim, and GAIA-1.
  • Interactive Video World Models: Recent systems explore natural-language actions, camera-controlled navigation, and real-time keyboard-and-mouse control, with Matrix-Game 3.0 using explicit memory for long-horizon consistency.HY-World 1.5 and LingBot-World emphasize navigation and geometric consistency, while Hunyuan-GameCraft and Matrix-Game systems support action-conditioned control.
  • World Model Evaluation: Existing benchmarks divide between non-interactive suites evaluating quality, text alignment, or physical commonsense and world-model benchmarks targeting camera trajectories or decision-oriented quality.Non-interactive examples include VBench, EvalCrafter, and VideoPhy; WorldScore and WorldModelBench represent specialized world-model evaluations.

3 WBENCH Dataset

WBENCH structures each test case around a defined world setting and a multi-turn interaction sequence, covering diverse environments, subjects, styles, perspectives, and interaction types. Its 289 cases and 1,058 turns emphasize navigation and multi-turn temporal evaluation, with sequences spanning 2–9 turns.

  • Dataset Structure: Each WBENCH case decomposes evaluation inputs into a World Setting W defining the initial state and an interaction sequence I spanning T consecutive turns.The interaction sequence specifies user control signals across consecutive turns.
  • World Settings: World settings specify scene, style, perspective, and subject, while initial frames are generated, web-collected, or manually captured and manually verified.Supported styles include realistic, cartoon, anime, cinematic, CG, and oil painting; perspectives are first- or third-person.
  • Interactions: Interactions comprise navigation, subject action, event editing, and perspective switching, with navigation supporting translational and rotational controls that can be composed into compound actions.The same key drives the camera in first-person mode and the subject in third-person mode.
  • Dataset Composition: 289 cases span 1,058 interaction turns, with first-person cases at 62%, third-person cases at 38%, and navigation comprising 57% of interactions.Subject action, event editing, and perspective switching account for 20%, 17%, and 6%, respectively.
  • Dataset Composition: Cases span six scene categories and diverse subjects and styles, led by nature scenes at 31%, urban environments at 21%, humans among explicit subjects at 64%, and photorealistic rendering at 52%.The remaining scene categories are indoor, works, fantasy, and sports; other subject categories include animals, robots, vehicles, and miscellaneous objects.
  • Multi-turn Interaction Depth: Each case contains 2–9 interaction turns averaging 3.7, with four-turn cases comprising 51% and longer 5–9-turn cases comprising 12%.The multi-turn structure probes temporal consistency and long-horizon coherence beyond single-turn benchmarks.

4 WBENCH Evaluation Suite · C.8 Background Consistency. · P.2 Visual Plausibility.

WBench evaluates interactive world models across five dimensions using 22 normalized sub-metrics, combining geometric estimation, specialist vision models, and multimodal scoring. Its consistency and physical assessments cover temporal, geometric, appearance, causal, and low-level visual plausibility.

  • 4 WBENCH Evaluation Suite: WBench decomposes evaluation into five complementary dimensions with 22 fine-grained sub-metrics, each linearly rescaled to [0, 100].Higher scores indicate better performance and enable direct comparison across dimensions.
  • 4 WBENCH Evaluation Suite: Video quality uses six sub-metrics covering aesthetics, imaging, flickering, dynamics, smoothness, and normalized human preference.Five metrics come from VBench, while HPSv3-Norm is percentile-normalized.
  • 4 WBENCH Evaluation Suite: Setting adherence separates scene fidelity from subject fidelity, scoring visible and offscreen elements alongside appearance and motion attributes.VLMs assess whether described scene components appear consistently and whether subjects match specified visual and movement characteristics.
  • 4 WBENCH Evaluation Suite: Interaction adherence evaluates navigation with geometric pose estimation and the other interaction types with structured per-turn VLM criteria.Event editing and subject action use five binary checks, averaged across turns and scaled to a 100-point score.
  • 4 WBENCH Evaluation Suite: Consistency measures stability of scene geometry, object appearance, and perspective anchoring as cameras move and interactions accumulate.Its sub-metrics include roundtrip spatial similarity, cut-free segment continuity, subject centroid stability, reprojection-based geometric and photometric consistency, and subject feature similarity.
  • C.8 Background Consistency.: Background consistency measures mean pairwise CLIP cosine similarity between consecutive frames to capture temporal stability of background appearance.The metric follows VBench.
  • C.8 Background Consistency.: The Physical dimension assesses both high-level causal fidelity and low-level visual plausibility against declared physical rules.Causal fidelity uses uniformly sampled frames across turns in a two-stage VLM protocol with three-point grading.
  • P.2 Visual Plausibility.: Visual plausibility uses a fine-tuned Qwen3-VL-30B-A3B to score geometric distortion, object penetration, and unnatural deformation on a normalized [0, 100] scale.The model produces a continuous [1] score and complements causal fidelity by detecting pervasive artifacts without case-specific questions.

5 Experiments · 6 Conclusion · Appendix Contents

Experiments evaluate 20 models across five dimensions and show that no model or paradigm dominates uniformly. Cross-dimension analyses expose decoupled capabilities, structured difficulty, multi-turn degradation, and strong alignment between automated and human evaluations, while the conclusion notes current limitations.

  • 5.1 Evaluated Models and Protocol: 20 models are evaluated across three paradigms, with text-driven systems covering all 289 cases and camera-controlled and action-conditioned systems restricted to 158 navigation cases.Text-driven models accept four interaction types; the other paradigms focus on navigation.
  • 5.2 Per-Dimension Results: Text-driven models lead setting adherence and physical compliance, while camera-controlled and action-conditioned models lead navigation and text-driven models remain competitive in video quality.Wan 2.7 leads setting adherence at 91.4 and physical compliance at 71.8; camera-controlled and action-conditioned paradigms score 76.0 and 77.7 on interaction versus 67.6 for text-driven models.
  • 5.2 Per-Dimension Results: Perspective switching is the hardest semantic interaction, averaging 30.7, whereas LingBot-World achieves the highest overall consistency at 89.9.Camera-controlled models lead geometric consistency, but text-driven models lead perspective consistency.
  • 5.3 Cross-Dimension Analysis: Navigation is largely independent of video quality, consistency, and physical compliance, with correlations of r=−0.12, r=−0.05, and r=−0.15, respectively.Physical scores instead correlate strongly with video quality at r=0.84 and consistency at r=0.72.
  • 5.3 Cross-Dimension Analysis: Open-source models lead specific capabilities, including HY-World 1.5 at 87.5 for navigation, LingBot-World at 89.9 for consistency, and Matrix-Game 3.0 at 83.5 for action-conditioned navigation.These results demonstrate capability-specific competitiveness from architectural and training choices.
  • 5.3 Cross-Dimension Analysis: Navigation degrades by 33 points from turn 1 to turn 4+, while event editing and subject action decline by 13 and 9 points, respectively.Perspective switching remains nearly flat at +2, and dedicated world models degrade less than text-based prompting.
  • 5.4 Human Preference Alignment: Automated WBENCH scores align strongly with human preferences across ten aspects, with every aspect reaching Spearman ρ ≥0.94 and four reaching ρ = 1.00.The validation uses blind pairwise comparisons from 400 crowdsourced annotators.
  • 6 Conclusion · Appendix Contents: WBENCH shows that current world models have not unified high-quality rendering, controllability, consistency, and physical correctness, while its limitations include discrete actions, partly LMM-based physical evaluation, and limited domains.The authors identify continuous control, additional domains, and real-time evaluation as promising extensions; full results appear in Appendix D.1.

A Additional Dataset Statistics and Analysis · B Evaluated Models

The appendix expands WBench’s dataset analysis through benchmark comparisons, galleries, and navigation statistics, then documents evaluated-model configurations, inference speed, and web-based evaluation protocols.

  • A Additional Dataset Statistics and Analysis: A.1 presents an extended benchmark comparison for WBench.
  • A.2 Dataset Gallery: The dataset gallery covers scene and style, perspective and subject, subject action and event editing, and perspective switching examples.
  • A Additional Dataset Statistics and Analysis: A.3 describes the design and distribution of navigation tasks.
  • B Evaluated Models: B.1 documents per-model configuration for text-driven, camera-controlled, and action-conditioned models.
  • B.1.4 Inference Speed Analysis: The appendix includes an inference speed analysis for the evaluated models.
  • B.2 Web-Based Evaluation Protocol (Genie 3 & Happy Oyster): It also specifies a web-based evaluation protocol for Genie 3 and Happy Oyster.

C Per-Metric Evaluation Details … A Additional Dataset Statistics and Analysis

The supplied material outlines per-metric evaluation details spanning video quality, setting and interaction adherence, consistency, and physical evaluation, followed by additional experimental-results subsections. It provides section headings but no substantive findings, numerical results, or content for the other named merged sections.

  • C Per-Metric Evaluation Details: The per-metric appendix covers video quality through aesthetic quality, imaging quality, temporal flickering, dynamic degree, motion smoothness, and HPSv3-Norm.The supplied passages list these as subsections C.2–C.2.6.
  • C Per-Metric Evaluation Details: Setting adherence is divided into scene adherence and subject adherence.These are listed as subsections C.3.1 and C.3.2.
  • C Per-Metric Evaluation Details: Interaction adherence includes NavScore, event editing adherence, subject action adherence, and perspective switching adherence.These are listed as subsections C.4.1–C.4.4.
  • C Per-Metric Evaluation Details: Consistency evaluation covers subject, background, spatial, segment, perspective, and reconstruction consistency.These are listed as subsections C.5.1–C.5.6.
  • C Per-Metric Evaluation Details: Physical evaluation consists of causal fidelity and visual plausibility.These are listed under section C.6.
  • D Additional Experimental Results: Additional experimental results include full split results on text-driven models and a human-preference annotation platform and protocol.The supplied passages identify these as sections D.1 and D.2, without reporting their findings.

A.1 Extended Benchmark Comparison

Table 3 compares WBench with surveyed benchmarks across perspectives, interaction types, and five evaluation dimensions. Existing benchmarks include quality-oriented suites with multi-dimensional metrics and physics-focused datasets spanning physical laws and domains.

  • Benchmark scope: WBench’s comparison framework distinguishes first- and third-person perspectives, four interaction types, and five evaluation dimensions.The interaction types are navigation, subject action, event editing, and perspective switching; the dimensions are video quality, setting adherence, interaction adherence, consistency, and physics compliance.
  • Quality-oriented benchmarks: VBench dissects text-to-video quality into 16 hierarchical dimensions with tailored prompts.It is presented as a representative quality-oriented benchmark.
  • Quality-oriented benchmarks: EvalCrafter scores around 700 prompts with 17 objective metrics calibrated against user preferences.It is another representative quality-oriented suite surveyed in Table 3.
  • Physics-focused benchmarks: PhyGenBench contains 160 prompts covering 27 physical laws across four domains.It is identified as physics-focused work.
  • Physics-focused benchmarks: VideoPhy-2 targets action-centric physical reasoning as a physics-focused benchmark.The supplied passage identifies its action-centric physics focus but does not provide additional quantitative details.

A.2 Dataset Gallery … B.1.4 Inference Speed Analysis

The appendix showcases WBENCH’s coverage across scenes, styles, perspectives, subjects, interactions, and navigation, then details evaluated model configurations and inference-speed measurement. Navigation uses perspective-dependent controls and diverse trajectories, while models are grouped by text, camera, or action conditioning.

  • A.2 Dataset Gallery; A.2.1 Scene and Style Gallery: WBENCH’s gallery presents all initial-frame cases and covers six scene categories, each paired across photorealistic and stylized renderings.The categories are nature, urban, indoor, workspace, fantasy, and sports/game.
  • A.2.1 Scene and Style Gallery: The style gallery spans realistic, anime, cartoon, oil painting, ink wash, flat, and pencil-sketch renderings.
  • A.2.2 Perspective and Subject Gallery; A.2.3 Subject Action and Event Editing Examples; A.2.4 Perspective Switching Prompts Gallery: Perspective cases distinguish disembodied first-person, embodied first-person, and third-person views, while subject cases cover humans, animals, vehicles, robots, and other objects.Temporal subject actions and event edits are represented through condensed example phrases, and perspective switching covers same-subject, multi-subject, and scope-mode transitions.
  • A.2.4 Perspective Switching Prompts Gallery: Perspective-switching prompts test coherent world maintenance during runtime reconfiguration, including transitions between narrow first-person and broader third-person views.
  • A.3 Navigation Design and Distribution: Navigation uses WASD and arrow keys with perspective-dependent semantics: first-person controls translate or rotate the camera, whereas third-person controls move the subject or orbit the camera.
  • A.3 Navigation Design and Distribution: 62.8% of navigation actions are translational and 37.0% rotational, with sequences spanning eight cardinal and diagonal directions, indoor, outdoor, and fantasy scenes, and text, 6DoF-pose, or discrete-action interfaces.Forward motion is the most frequent translational action at 29%, while lateral A/D motions occur at 12% each and backward S at 11%.
  • B Evaluated Models; B.1 Per-Model Configuration; B.1.1 Text-driven Models; B.1.2 Camera-controlled Models; B.1.3 Action-conditioned Models: The evaluated models are organized by native interaction interface into text-driven, camera-controlled, and action-conditioned groups, with iterative I2V used when models lack native action inputs.Camera-controlled models consume 6DoF pose sequences, while action-conditioned models use discrete or continuous signals and full action history.
  • B.1.4 Inference Speed Analysis: Inference speed is measured as wall-clock generation time divided by produced-video duration; values below 1 indicate faster-than-real-time generation.Local models are measured on a single NVIDIA A100 80GB GPU, with procedures differing for text-driven versus camera- and action-controlled models.

B.2 Web-Based Evaluation Protocol (Genie 3 & Happy Oyster) · C Per-Metric Evaluation Details

WBench evaluates Genie 3 and Happy Oyster through an automated browser-based interaction pipeline because neither system exposes a public API or downloadable weights. The pipeline fills inputs, waits for loading, executes scripted interactions, and downloads the resulting video.

  • B.2 Web-Based Evaluation Protocol (Genie 3 & Happy Oyster): The evaluation pipeline has four steps: filling the prompt, initial image, and perspective; waiting for world loading; executing interactions; and downloading the recorded video.The browser-use agent follows this sequence from an initial image and text prompt.
  • B.2 Web-Based Evaluation Protocol (Genie 3 & Happy Oyster): Each interaction turn lasts 5 seconds during scripted execution.After world loading completes, the agent runs the interaction script and then downloads the recorded video.
  • B.2 Web-Based Evaluation Protocol (Genie 3 & Happy Oyster): Genie 3 and Happy Oyster require web-based automation because they provide neither public APIs nor downloadable weights.Claude Code’s browser-use capability controls Chrome through structured commands for programmatic evaluation at scale.

C.1 Metric Quick Reference … C.4 Interaction Adherence

The appendix defines a unified 22-sub-metric evaluation framework spanning video quality, setting adherence, and interaction adherence. It specifies expert-model and VLM-based procedures for assessing visual quality, world-setting realization, and interactive behavior.

  • C.1 Metric Quick Reference: 22 evaluation sub-metrics are unified in a quick-reference table, with each metric linked to its detailed methodology and evaluation examples.The appendix organizes per-sub-metric definitions and examples by evaluation dimension.
  • C.2 Video Quality: Six Video Quality sub-metrics use expert models and produce scores in [0, 100], with higher values indicating better quality.The metrics cover aesthetic quality, imaging quality, temporal flickering, dynamic degree, motion smoothness, and normalized HPSv3 reward.
  • C.2.1 Aesthetic Quality: Aesthetic Quality samples frames at 2 FPS, applies CLIP ViT-L/14 and a LAION Aesthetic linear head, then averages and rescales scores from [0, 10] to [0, 100].The final score is the mean over sampled frames.
  • C.2.2 Imaging Quality: Imaging Quality uses MUSIQ on frames sampled at 2 FPS and resized so the longer edge is at most 512, producing per-frame scores in [0, 100].The metric evaluates resized sampled frames with MUSIQ.
  • C.2.3 Temporal Flickering: Temporal Flickering compares consecutive frame pairs using pixel-level MAE and inverts the result so lower flickering receives a higher score.The metric is based on frame-to-frame pixel differences.
  • C.2.4 Dynamic Degree: Dynamic Degree samples frames at 8 FPS and computes RAFT optical flow for consecutive pairs, averaging each pair’s top 5% magnitude values.The formulation additionally uses a dynamic threshold and a minimum count of dynamic pairs.
  • C.2.5 Motion Smoothness: Motion Smoothness follows VBench by interpolating even-indexed frames from odd-indexed neighbors with AMT-S and comparing predicted versus actual frames using pixel MAE.This evaluates interpolation error between neighboring frames.
  • C.2.6 HPSv3-Norm: HPSv3-Norm applies linear percentile normalization to per-frame HPSv3 rewards using p1 = 5.21 and p99 = 8.66.The raw reward is computed per frame and normalized from per-video mean scores across evaluated models.

C.4.1 NavScore … C.6 Physical

The evaluation framework combines adaptive trajectory-based navigation scoring, question-based interaction-adherence measures, seven temporal-consistency sub-metrics, and physical-dimension metrics. These components assess whether generated videos follow navigation and interaction instructions while maintaining coherent appearance, structure, continuity, perspective, reconstruction, and physical plausibility.

  • C.4.1 NavScore: NavScore compares predicted camera trajectories with action-conditioned synthetic ground truth, resampling both to K = 20 arc-length points and using normalized ATE as the primary metric.Ground truth adapts to each turn’s action type, while RPE is retained only as an auxiliary diagnostic because non-uniform motion makes velocity-matched trajectories unreliable.
  • C.4.1 NavScore: Qualitatively, Happy Oyster and HY-World 1.5 follow instructed directions and receive high NavScores despite different motion amplitudes, whereas HY-Video 1.5 reverses rotations and is penalized for directional error.Adaptive ground truth reflects predicted motion magnitude, distinguishing wrong direction from scale mismatch.
  • C.4.2 Event Editing Adherence: Event-editing adherence asks five progressive binary questions per turn, awards one point per matching response, and rescales the case mean from [0, 5] to [0, 100].High-scoring outputs show the instructed state transition, while failures leave the scene unchanged or introduce unrelated entities.
  • C.4.3 Subject Action Adherence: Subject-action adherence uses the same five-question scoring rule with templates assessing whether the subject initiates and completes the instructed action.Successful generations complete the prescribed action; failures leave the subject idle or stop at a partial gesture.
  • C.4.4 Perspective Switching Adherence: Perspective-switching adherence is evaluated as a distinct interaction-adherence dimension, alongside navigation, event editing, and subject action.The supplied qualitative comparison is presented in Figure 19.
  • C.5 Consistency: Consistency comprises seven sub-metrics measuring frame-level temporal coherence, including subject, background, spatial, segment, perspective, and reconstruction consistency.Subject and background consistency use consecutive-frame feature similarity; spatial consistency gates return-frame similarity against motion; segment continuity detects cuts; perspective consistency penalizes disappearance and drift.
  • C.5.4 Segment Continuity / C.5.5 Perspective Consistency / C.5.6 Reconstruction Consistency: Segment, perspective, and reconstruction consistency detect cuts, target instability, and 3D reprojection errors using TransNetV2, SAM2 tracking, and depth-based camera-pose estimation.Reconstruction combines geometric consistency, which captures structural distortion, with photometric consistency, which captures texture flicker and color shifts after alignment.
  • C.6 Physical: The physical dimension uses two sub-metrics: VLM-based causal fidelity and fine-tuned Qwen3VL visual plausibility.These metrics assess physical compliance through complementary causal and visual judgments.

C.6.1 Causal Fidelity · C.6.2 Visual Plausibility · D Additional Experimental Results

The paper evaluates causal fidelity through global and scene-aware physics tracks, while visual plausibility is learned from human ratings using a fine-tuned Qwen3-VL-30B-A3B scorer. Causal-fidelity results show heterogeneous physics failure modes, whereas the visual-plausibility scorer achieves strong alignment with human judgments.

  • C.6.1 Causal Fidelity: Causal fidelity uses Doubao-Seed-2.0-lite at 3 fps with global rendering-physics and causal-consistency scoring plus case-specific physics-dimension scoring, each on a 0–3 scale.Track 2 selects a subset of seven dimensions per case, and case scores average evaluated turns before normalization to [0, 1].
  • C.6.1 Causal Fidelity: Physics dimensions cover fluid and smoke, collisions and clipping, surface tracks, deformation and destruction, wind, reflections and lighting, and human motion and expression.The dimensions are selected according to the scene environment and interaction, with separate 0–3 rubrics for each phenomenon.
  • C.6.1 Causal Fidelity: Track 2 averages scores over manually verified, case-level dimension selections shared across models; per-turn fidelity averages Track 1 and Track 2 when applicable.If no dimension is selected, the score falls back to Track 1; case scores are then normalized to [0, 1].
  • C.6.1 Causal Fidelity: Reflection & Lighting is nearly saturated for most models, while Deformation & Destruction varies substantially despite activation in only 4 of 50 Track 2 cases.The results indicate heterogeneous failure modes and axis-dependent tier gaps, with Wan 2.7 leading on most dimensions.
  • C.6.2 Visual Plausibility: Visual plausibility measures appearance quality, temporal consistency, object structure, motion coherence, and common-sense physical reasonableness using fine-tuned Qwen3-VL-30B-A3B.Videos are rated across five ordered categories from Perfect to Bad, corresponding to scores from 5 to 1.
  • C.6.2 Visual Plausibility: Approximately 6K internally annotated videos train the visual-plausibility scorer, with three independent expert ratings averaged into each ground-truth score.The annotation set spans diverse subjects, scenes, visual styles, camera motions, and interaction patterns.
  • C.6.2 Visual Plausibility: The scorer converts probabilities over Perfect, Good, Fair, Poor, and Bad into a continuous expected score and is optimized against averaged human ratings with mean squared error.This preserves the ordinal structure of the categories while producing a continuous prediction.
  • C.6.2 Visual Plausibility: 0.92 PLCC is achieved against ground-truth labels, demonstrating strong alignment between the trained visual-plausibility scorer and human preferences.The reported metric is the Pearson Linear Correlation Coefficient.

D.1 Full Split Results on Text-Driven Models · D.2 Human-Preference Annotation Platform and Protocol · E Broader Impact

The appendix reports full-split text-driven results, details a human-preference annotation protocol and platform, and describes WBench’s broader impact and data practices. Together, these sections document benchmark coverage, evaluation procedures, model trade-offs, and responsible use.

  • D.1 Full Split Results on Text-Driven Models: Table 10 evaluates nine text-driven models across all 289 test cases, covering navigation, event editing, subject action, perspective switching, and full-split consistency metrics.The reported scores use gated spatial, geometric, and photometric consistency metrics, with all values scaled from 0 to 100.
  • D.1 Full Split Results on Text-Driven Models: Kling 3.0 and Wan 2.7 achieve the strongest interaction adherence, while perspective switching remains difficult with an average score of 30.7.The interaction-adherence strength is especially evident for event editing and subject action.
  • D.1 Full Split Results on Text-Driven Models: LongCat-Video and HunyuanVideo lead consistency metrics but show limited dynamic degree, indicating a trade-off between motion magnitude and temporal stability.Wan 2.7 leads causal fidelity at 83.3 versus the 74.7 average, while visual plausibility scores cluster.
  • D.2 Human-Preference Annotation Platform and Protocol: The annotation platform presents blind side-by-side video comparisons with synchronized playback controls and four responses: A better, B better, Tie, or Discard.Each task compares one model pair for a single evaluation dimension, with randomized left-right assignment to reduce position bias.
  • D.2 Human-Preference Annotation Platform and Protocol: Nine dimension-level task sheets cover interaction adherence, setting, aesthetic quality, spatial consistency, and physics compliance across fixed model and case pools.Most dimensions enumerate 45 pairs among ten sampled models, whereas interaction adherence enumerates 6 pairs among four eligible text-driven models.
  • D.2 Human-Preference Annotation Platform and Protocol: The study contains 13,515 pairwise comparison tasks completed by a 400-person crowdsourced pool across 8 evaluated models and 10 evaluation dimensions.Quality procedures include triple redundancy, gold-standard checks, completion-time filtering, and expert sampled review.
  • E Broader Impact: WBench diagnoses interactive world-model capabilities without generating or distributing synthetic media, using synthetic or openly licensed imagery without identifiable real individuals.Its fine-grained failure modes in physics, controllability, and consistency are intended to support more reliable world models and downstream applications.
Loading 2605.25874v1…