PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors

Keming Wu, Yijing Cui, Wenhan Xue, Qijie Wang, Xuan Luo, Zhiyuan Feng, Zuhao Yang, Sudong Wang, Sicong Jiang, Haowei Zhu, Zihan Wang, Ping Nie, Wenhu Chen, Bin Wang

arXiv:2605.10434v1cs.CV

TL;DR

Video benchmarks have largely emphasized perceptual quality rather than whether generated videos correctly predict future world states. WorldReasonBench reframes evaluation as world-state prediction and introduces human-aligned reasoning and preference benchmarks. Across generators, it reveals a persistent gap between visual plausibility and world reasoning, with dynamic-phase failures driving open-source deficits.

  • Problem

    Existing benchmarks lack open-domain, end-to-end tests of whether video generators can infer and simulate future world-state evolution beyond visual quality.

  • Method

    WorldReasonBench evaluates action-conditioned future videos with process-aware QA, multidimensional quality assessment, and expert preference data for reward-model evaluation.

  • Results

    ScorePR reaches Spearman ρ=0.955 against human Elo, while open-source generators trail closed-source systems by roughly twofold in reasoning and quality because of dynamic-phase failures.

  • Takeaways & Limitations

    Progress in world-aware video generation requires mechanism-level reasoning and information preservation, not merely more visually polished outputs.

  • Takeaways & Limitations

    The taxonomy excludes counterfactual queries, multi-agent dynamics beyond two actors, numerical physics grounding, long-horizon chains, and non-English prompts.

Abstract

from arXiv · show

Commercial video generation systems such as Seedance2.0 and Veo3.1 have rapidly improved, strengthening the view that video generators may be evolving into "world simulators." Yet the community still lacks a benchmark that directly tests whether a model can reason about how an observed world should evolve over time. We introduce WorldReasonBench, which reframes video generation evaluation as world-state prediction: given an initial state and an action, can a model generate a future video whose state evolution remains physically, socially, logically, and informationally consistent? WorldReasonBench contains 436 curated test cases with structured ground-truth QA annotations spanning four reasoning dimensions and 22 subcategories. We evaluate generated videos with a human-aligned two-part methodology: Process-aware Reasoning Verification uses structured QA and reasoning-phase diagnostics to detect temporal and causal failures, while Multi-dimensional Quality Assessment scores reasoning quality, temporal consistency, and visual aesthetics for ranking and reward modeling. We further introduce WorldRewardBench, a preference benchmark with approximately 6K expert-annotated pairs over 1.4K videos, supporting pair-wise and point-wise reward-model evaluation. Across modern video generators, our results expose a persistent gap between visual plausibility and world reasoning: videos can look convincing while failing dynamics, causality, or information preservation. We will release our benchmarks and evaluation toolkit to support community research on genuinely world-aware video generation at https://github.com/UniX-AI-Lab/WorldReasonBench/.

1 Introduction

WorldReasonBench reframes video-generation evaluation as open-domain world-state prediction, testing whether generated futures remain physically, socially, logically, and informationally consistent. It addresses gaps in existing benchmarks with structured reasoning evaluation and an expert-calibrated preference benchmark.

  • Motivation: Existing benchmarks emphasize perceptual quality, motion smoothness, prompt alignment, or narrower reasoning settings rather than end-to-end open-domain world-state prediction.Prior efforts target answer-verifiable cognition, procedural diagnostics, embodied control, or single-event causality, but do not evaluate future world evolution on open-domain content.
  • Benchmark framing: WorldReasonBench asks models to generate future videos whose state evolution remains physically, socially, logically, and informationally consistent after an observed initial state and action.The benchmark spans four reasoning dimensions organized into 22 dimension-specific subcategories.
  • Motivation: Visually impressive videos can still violate basic dynamics, such as making a dropped apple accelerate upward or follow a linear rather than parabolic trajectory.Standard quality metrics may reward realism while missing these world-model failures.
  • Evaluation resources: Approximately 6K expert-annotated pairs over 1.4K videos support pair-wise and point-wise reward-model evaluation in the preference-based calibration benchmark.This preference benchmark is designed to provide expert calibration for ranking and reward modeling.

2 Related Work

Prior work increasingly frames video generators as world simulators, but capability demonstrations do not establish robust world understanding. Existing benchmarks mainly assess perceptual quality or prompt alignment, while reasoning-oriented evaluations cover limited task slices and lack structured reasoning verification.

  • Video generation models as world simulators: Video generators are increasingly viewed as world simulators because commercial systems have improved long-horizon coherence, controllability, and realism.Recent studies also suggest zero-shot learning and reasoning-like behavior in selected settings.
  • Video generation models as world simulators: Capability demonstrations alone do not establish robust world understanding, as physical-law analyses identify failures even in strong models.
  • Benchmarks and automatic evaluation for video generation: Existing video benchmarks primarily evaluate perceptual quality, prompt alignment, aesthetics, or compositionality using reference metrics such as FID, FVD, and LPIPS.
  • Benchmarks and automatic evaluation for video generation: Reasoning-oriented benchmarks address isolated slices including embodied task success, answer-verifiable puzzles, procedural processes, and single-event causality rather than structured reasoning verification.

3 WorldReasonBench

WorldReasonBench frames video generation as predicting how an observed world should evolve after an action, using structured reasoning cases and complementary evaluation protocols. Its human-aligned extensions verify process faithfulness, score multiple video qualities, and support reward-model calibration through expert preferences.

  • Problem formulation: WorldReasonBench evaluates whether a generator produces a future video that faithfully realizes the state evolution implied by an initial state and intended action.The formulation contrasts world-state prediction with merely producing realistic-looking videos.
  • Data curation pipeline: A three-stage VLM-assisted pipeline generates structured captions, reasoning-aware prompts, and ground-truth QA pairs for each curated benchmark case.Qwen3.5 produces captions, Qwen3.5-27B generates prompts, and Gemini3.1-Pro generates QA pairs with expected answers and labels.
  • Benchmark construction: The benchmark organizes world reasoning into four dimensions and 22 interpretable subcategories, with structured QA covering factual, reasoning, detail, and temporal verification.QA proportions are factual 28.4%, reasoning 27.1%, detail 24.7%, and temporal 19.7%, with easy, medium, and hard difficulty levels.
  • WorldRewardBench: Approximately 6K balanced preference pairs over 1.4K unique videos form WorldRewardBench, enabling pair-wise and point-wise reward-model evaluation against human preferences.Annotations rate reasoning quality, temporal consistency, and visual aesthetics on a 1–5 scale, then construct filtered pairwise preferences.
  • Evaluation: Process-aware Reasoning Verification combines video-grounded structured QA with binary judging to assess both final-state correctness and plausible transition processes.Its reasoning-gap analysis contrasts static outcome performance with dynamic performance, while ScorePR discounts models that mainly succeed on static questions.

4 Experiments

Experiments evaluate eleven video generators with process-aware and multi-dimensional metrics, revealing a robust closed-source advantage and persistent weaknesses in world-state reasoning. Process-aware QA aligns more closely with human preferences than pairwise VLM judging, while reward-model protocols show complementary strengths in agreement and calibration.

  • Evaluation settings: Eleven generators are evaluated using Qwen3.5-27B at 4 FPS, with ScorePR, AccQA, phase diagnostics, process completeness, ∆RG, and weighted quality score S(v).The systems comprise five closed-source and six open-source models; reward-model alignment uses pairwise agreement and Spearman ρ.
  • Cross-model results: 32.4–39.8 versus 14.4–17.9 overall ScorePR and 50.3–59.4 versus 21.3–30.5 S(v) show a roughly two-fold closed-source advantage over open-source models.Every open-source overall-ScorePR confidence interval lies below every closed-source interval, with upper bound ≤23.1 versus lower bound ≥26.4.
  • Reasoning difficulty: Logic Reasoning is hardest, with the best closed-source ScorePR at 31.7, while Information-Based errors concentrate in World Mechanics, Material Change, and Data Reading.Open-source models gain 9.9–14.8 absolute QA points (+56–85% relative) from explicit hints, compared with +10.3 points (+29%) for Sora2-8s.
  • Human alignment: ScorePR and AccQA correlate with human Elo at Spearman ρ=0.955 and 0.927, exceeding pairwise VLM-judge Elo at ρ=0.804.ScorePR avoids the judge’s fixed-eight-frame duration mismatch and matches human ordering up to one rank swap.
  • Reward-model evaluation: Pairwise judging reaches 74.35% agreement without ties, whereas point-wise scoring reaches the best ρ=0.655, making the protocols complementary.Vanilla point-wise scoring achieves ρ=0.626 and 67.63% accuracy without ties with one call, while 4 FPS provides the best cost–accuracy trade-off.
  • Judge robustness: Across Qwen3.5 variants, pairwise agreement varies by at most 2.3 percentage points and point-wise ρ by at most 0.064, indicating cross-setting robustness.Across families, Gemini-3.1-Flash trails Qwen by approximately 10 percentage points pairwise but tracks it point-wise at ρ=0.568.

5 Conclusion … D Full Two-Component WorldReasonBench Results

WorldReasonBench evaluates future video world-state prediction across four reasoning dimensions using structured QA, quality assessment, and preference comparisons. Its results identify persistent reasoning and information-preservation challenges beyond visual plausibility, while the released benchmarks and toolkit support further evaluation.

  • 5 Conclusion: WorldReasonBench contains 436 cases with structured QA annotations across four reasoning dimensions and 22 subcategories, alongside WorldRewardBench’s approximately 6K expert-annotated pairs over 1.4K videos.The benchmarks are released with an evaluation toolkit for auditing reward models, calibrating judges, and extending reasoning evaluation.
  • 5 Conclusion: Closed- and open-source generators differ by roughly a factor of two on reasoning and quality, with dynamic-phase failures explaining the open-source deficit rather than static appearance.Logic Reasoning and Information-Based content remain the most challenging dimensions for both generators and judges.
  • A Representative Examples of WorldReason-Bench: Representative examples span World Knowledge, HumanCentric, Logic Reasoning, and Information-Based Reasoning, pairing an input image with a generation prompt.The examples are organized around physical, social, cultural, natural-world, human-behavior, logical, and information-preservation requirements.
  • B Evaluation Prompts: Evaluation prompts cover Process-aware Reasoning Verification, Multi-dimensional Quality Assessment, and pair-wise comparison as distinct evaluation components.The components correspond to QA-based reasoning verification, point-wise scoring, and pair-wise preference comparison.
  • B.1 Process-aware Reasoning Verification Prompts: Process-aware Reasoning Verification uses a two-stage QA pipeline: a VLM answers from visible video evidence, then a judge assigns binary correctness against ground truth.Multi-dimensional Quality Assessment produces point-wise scores for reasoning quality, temporal consistency, and visual aesthetics, while pair-wise comparison estimates preference between candidate videos.
  • C Reasoning Taxonomy Details: The taxonomy defines World Knowledge with 127 full-set cases, Human-Centric Reasoning with 78, Logic Reasoning with 131, and Information-Based Reasoning with 100.These dimensions cover physical, social, and cultural evolution; human behavior and interaction; structured logical relations; and faithful preservation or transformation of explicit information.
  • C Reasoning Taxonomy Details: The 22 subcategories include material and public-world changes, object handling and social scenes, quantitative and spatial reasoning, and data reading and process timelines.Additional categories cover world mechanics, earth cycles, living-world changes, skilled actions, experimental science, visual editing, knowledge media, and creative expression.
  • D Full Two-Component WorldReasonBench Results: Full WorldReasonBench results are separated by evaluation component and top-level reasoning dimension, reporting outcome QA accuracy (%) and three raw quality axes on a 1–5 scale.The quality axes are reasoning quality, temporal consistency, and visual aesthetics.

D.1 Auxiliary Process-Aware Metric Definitions … D.4 Multi-dimensional Quality Assessment

The appendix defines auxiliary process-aware diagnostics and expands evaluation across all 22 subcategories and multiple quality axes. These analyses show that visual plausibility and aggregate quality can exceed genuine reasoning, especially for Logic Reasoning and Information-Based tasks.

  • D.1 Auxiliary Process-Aware Metric Definitions: ScorePR is the headline process-aware metric because it preserves ranking discriminability while emphasizing temporal and mechanistic correctness.Two auxiliary diagnostics are additionally defined for ablations and detailed error analysis.
  • D.1 Auxiliary Process-Aware Metric Definitions: The bottleneck composite score penalizes failures in any reasoning phase, but its geometric-mean construction can be overly conservative for headline ranking.It is useful for identifying severe reasoning failures because it collapses when any phase approaches zero.
  • D.2 Full Subcategory-Level Results: The full 22-subcategory results report both ScorePR and S(v), exposing category-specific strengths and failure modes.ScorePR = Acc0.8_dyn, while S(v) is linearly mapped from [5] to [0,100].
  • D.2 Full Subcategory-Level Results: Closed-source systems lead on both ScorePR and S(v), with Veo3.1-Fast strongest on World Knowledge, Seedance2.0 leading overall and Information-Based reasoning, and Sora2 competitive on Human-Centric process scores.The advantage is not uniform across reasoning dimensions, and the gap between S(v) and ScorePR is substantial in several subcategories.
  • D.3 Process-aware Reasoning Verification: World Knowledge and Human-Centric categories show the clearest closed-source advantage, while Logic Reasoning remains difficult for all generators.Information-Based performance is polarized: structured timeline and creative-expression cases can be strong, whereas exact data reading and visual editing remain brittle.
  • D.3 Process-aware Reasoning Verification: Open-source models occasionally approach closed-source performance on narrow subcategories such as Data Reading, but their broader averages remain much lower.The shortfall is attributed to failures in temporal and mechanistic consistency across the broader set.
  • D.4 Multi-dimensional Quality Assessment: The quality-score breakdown separates reasoning quality, temporal consistency, and visual aesthetics, showing that temporal and visual scores are generally higher than reasoning quality.This explains how a model can obtain a strong S(v) while underperforming on process-aware reasoning, especially in Logic Reasoning and Information-Based categories.
  • D.4 Multi-dimensional Quality Assessment: Because visually coherent videos can still show sharply lower reasoning-quality scores, the three quality axes should be reported separately rather than reduced to an aggregate score.The gap is largest in Logic Reasoning and Information-Based categories.

E Full Frame-Rate Ablation Results

The frame-rate ablation evaluates six closed-source models across 2, 4, and 8 FPS using overall QA accuracy and reasoning gap, with full results adding category and reasoning-phase analyses. Results support 4 FPS as the default because it improves average accuracy substantially over 2 FPS while avoiding 8 FPS’s small gain and higher token cost.

  • Ablation setup: Tables 23 and 24 compare six closed-source models across 2, 4, and 8 FPS using overall QA accuracy, reasoning gap, per-category QA accuracy, and reasoning-phase scores.The full results also include selected 4 FPS open-source results and average rows used to select the default setting.
  • Result analysis: 34.9% to 37.2%: moving from 2 FPS to 4 FPS improves the six-model average accuracy and yields a lower reasoning gap than 2 FPS.The gains occur on most closed-source models.
  • Result analysis: 37.6%: 8 FPS slightly raises average accuracy, but the gain is small relative to its additional visual-token cost and does not consistently improve every model.Approximate visual-token costs per 5-second video are 4.5k, 9.0k, and 12.2k for 2, 4, and 8 FPS, respectively.

F Point-wise Scoring Ablation Details … J WorldRewardBench Post-processing Details

The appendix details WorldRewardBench’s scoring, human annotation, ranking, and post-processing procedures. Together, these analyses show that point-wise scoring, expert supervision, and confidence-aware filtering are designed to improve reliable world-state reasoning evaluation.

  • F Point-wise Scoring Ablation Details: On 5,969 WorldRewardBench pairs, Vanilla Thinking achieves the highest Spearman ρ of 0.626 and best w/o-ties accuracy of 67.63%.SDE lowers reasoning↔temporal-consistency correlation from 0.770 to 0.384, but does not improve rank correlation or induced accuracy.
  • H Expert Human Annotation Protocol: WorldRewardBench was annotated by fifteen trained annotators who judged whether videos realized prompt-implied transitions rather than merely appearing visually appealing.Annotators had diverse video-generation and multimodal-evaluation backgrounds and received one-hour training before formal annotation.
  • H.1 Annotators and Privacy: Annotators provided only scoring decisions voluntarily, and released files contain anonymized identifiers without personally identifying information.The annotations were used to construct aggregate video-level scores and pairwise preference labels.
  • H.2 Annotation Interface and Scoring Rubric: Each assignment showed an input image, prompt, and eight anonymized full videos, with hidden model identities and randomized ordering.Videos could be replayed, and each was scored from 1–5 on Reasoning Quality, Temporal Consistency, and Visual Aesthetics, with Reasoning Quality primary.
  • H.3 Disagreement Detection: Disagreement triggers additional annotation rather than discarding ratings, and final aggregated scores achieve Krippendorff’s α = 0.744, ICC(2,k)= 0.936, and mean pairwise Spearman ρ = 0.784.High-disagreement videos receive at least four valid ratings before aggregation; the aggregate score weights Reasoning Quality, Temporal Consistency, and Visual Aesthetics as 0.4, 0.3, and 0.3.
  • H.4 QA Quality Audit: An independent audit examines approximately 300 stratified QA pairs for answerability and ground-truth correctness using two trained auditors.The sample is balanced across four reasoning dimensions and four question types.
  • I WorldRewardBench Human Scoring Breakdown: Seedance2.0 leads the human weighted ranking, followed by Kling, Veo3.1-Fast, and Wan2.6, while category-specific strengths support category-weighted supervision.Systems can be stronger on Human-Centric or World Knowledge scenes yet lose ground on Logic Reasoning and Information-Based cases.

K Full-Set WorldRewardBench Results

On the full 5,969-pair WorldRewardBench, pair-wise comparison aligns better with human judgments than point-wise scoring, especially for Information-Based reasoning. The benchmark covers 1,432 unique videos, 130 tasks, and 11 generators under the natural distribution of score gaps.

  • Benchmark setting: 5,969 pairs span 1,432 unique videos, 130 tasks, and 11 generators in the full-set benchmark.The setting preserves the natural score-gap distribution, including many easy high-margin pairs.
  • Implementation details: 99.6% and 99.9% parse rates are reported for Qwen3.5-27B-Thinking and Qwen3.5-27B-Instruct, respectively, in pair-wise evaluation.The Instruct setting disables the extended thinking chain.
  • Agreement results: 67.74% overall agreement is achieved by Qwen3.5-27B pair-wise comparison, versus 57.85% for point-wise scoring.These are full-set human-alignment agreement results on the original benchmark.
  • Agreement results: The largest pair-wise versus point-wise gap occurs in Information-Based reasoning, where exact text and data preservation make single-video scoring less reliable.The result supports pair-wise comparison for fine-grained preference recovery and point-wise scoring for calibrated per-video feedback.

L Subcategory-Level WorldRewardBench Results

Table 33 provides a subcategory-level WorldRewardBench breakdown using the same reward-model ordering and Pair/Point protocol split as the main-text table. Direct pair-wise judging generally outperforms point-wise induced comparison, while Qwen3.5-27B and Gemini-3.1 remain competitive on many point-wise subcategories.

  • Breakdown design: Table 33 expands each top-level reasoning dimension into constituent subcategories for finer-grained diagnosis.The breakdown preserves the main-text table’s reward-model ordering and Pair/Point protocol split.
  • Protocol comparison: Direct pair-wise judging consistently outperforms point-wise induced comparison across nearly all categories, especially Logic Reasoning and Information-Based cases.This pattern is reported in the subcategory-level reward results.
  • Point-wise methods: Qwen3.5-27B and Gemini-3.1 remain competitive across many point-wise subcategories.Their competitiveness is reported relative to other point-wise methods.
  • Point-wise methods: The 4 FPS point-wise setting improves some Information-Based rows but does not close the gap to direct comparison.The improvement is limited to some rows and remains insufficient to match direct pair-wise judging.

M Weight Design and Sensitivity

The reported ScorePR and S(v) weight settings produce stable model rankings under reasonable alternatives while preserving interpretability and alignment with the human annotation protocol. ScorePR’s α=0.8 best recovers human Elo, and S(v)’s (0.4, 0.3, 0.3) weights lie on a broad high-performing plateau.

  • Take-aways: Both metrics yield stable model rankings under weight perturbations, supporting the reported settings’ interpretability and consistency with the human annotation rubric.The ScorePR setting preserves AccQA as the dominant signal, while the S(v) setting remains aligned with the human protocol.
  • ScorePR Weight Sensitivity: ScorePR uses α=0.8 to keep QA accuracy dominant while applying a second-order penalty to outcome-hacking through dynamic-phase performance.The multiplicative dynamic term gently down-weights models with strong QA accuracy but weak dynamic phases; α→1 recovers AccQA, while α→0 yields the dynamic phase score.
  • ScorePR Weight Sensitivity: ρ=0.955 for α=0.8, the highest value among probed ScorePR aggregators; pure AccQA reaches ρ=0.927, while pure dynamic scoring and the min-bottleneck reach ρ=0.827.Across the grid, ρ ranges from 0.83 to 0.96, with arithmetic and geometric means between these alternatives.
  • S(v) Weight Sensitivity: 67.5% of the 231 S(v) simplex points achieve ρ≥0.95, with the full grid spanning ρ∈[0.81, 1.00].The grid varies (wr, wc, wa) in increments of 0.05 and ranks eleven models against human Elo.
  • S(v) Weight Sensitivity: The paper S(v) weights (0.4, 0.3, 0.3) achieve ρ=0.973, matching equal weighting and remaining within 0.027 of the best simplex points.Pure consistency (0, 1, 0) is the sole clear underperformer, with ρ=0.809, indicating that temporal consistency alone is not a reliable model-level proxy for human preferences.

N Statistical Significance and Rank Stability

Bootstrap analyses support a statistically robust closed-source advantage and Seedance2.0’s dominance, while limiting finer-grained ordering claims because uncertainty is wider for smaller dimensions and subcategories. Joint resampling shows the two source tiers never exchange ranks.

  • Bootstrap protocol: B=2000 case-level bootstrap resamples produce 95% confidence intervals for AccQA, ScorePR, and S(v), while jointly resampling all 12 models estimates rank stability.ScorePR combines AccQA and temporal-reasoning phase scores; S(v) aggregates per-video ratings using weights 0.4, 0.3, and 0.3.
  • Limitations: The 22 subcategories average approximately 20 cases each, so their estimates have higher sampling variance and are used only for qualitative comparisons, not rank claims.Rank claims must also be supported by dimension-level confidence intervals.
  • Overall confidence intervals: 23.1, Wan2.2-14B’s largest open-source overall-ScorePR upper bound, remains below 26.4, Wan2.6’s smallest closed-source lower bound.The same closed-over-open ordering holds for AccQA and S(v).
  • Per-dimension confidence intervals: Logic Reasoning is the least settled closed/open dimension: Wan2.2-14B’s CI overlaps Kling’s, unlike the stronger separation reported for World Knowledge and Information-Based.Human-Centric has the widest per-dimension ScorePR intervals because it has the smallest case pool.
  • Rank stability: 89.3%, Seedance2.0’s probability of rank 1, accompanies a 95% rank interval of [1, 2], while closed-source and open-source intervals remain separated into [6] and [7, 12].The rank distribution is computed from joint bootstrap resampling of overall ScorePR.
  • Interpretation: Strict ordering among the remaining five closed-source models and among open-source generators above UniVideo is not statistically supported; the paper therefore reports tied clusters.Missing-coverage symbols indicate unevaluated dimensions or protocols rather than zero performance.

O Extended Evaluation of Open-Source Generators on the Full WorldReasonBench Benchmark … R Licenses for Existing Assets

The appendices extend evaluation across the full WorldReasonBench, document reproducibility resources, discuss broader impacts, and specify licenses for models, tools, data, and released assets.

  • O Extended Evaluation of Open-Source Generators on the Full WorldReasonBench Benchmark: On the full 436-case benchmark, Wan2.2-14B leads AccQA and S(v), while HunyuanVideo-1.5 is strongest on ScorePR, statistically tied with Wan2.2-14B.UniVideo and LTX2.3 form the bottom of the open-source tier; results match the qualitative ordering in Table 37.
  • O Extended Evaluation of Open-Source Generators on the Full WorldReasonBench Benchmark: Logic Reasoning is the worst dimension for every open-source model, whereas World Knowledge is the best on the full benchmark.Logic Reasoning has ScorePR values of 10.8–13.7 and S(v) values of 13.4–27.1; World Knowledge reaches ScorePR 23.5 and S(v) 50.2.
  • P Compute Resources: Open-source generation and on-premise VLM judging used NVIDIA H100 80GB GPUs, while closed-source generators and judges were accessed through commercial APIs.The local environment used Linux, NVLink-equipped 8-GPU nodes, CUDA 12.x, and PyTorch 2.x.
  • P Compute Resources: 436 cases across two instruction regimes yielded approximately 872 generations per open-source generator, with each model producing a 5-second 480p–720p clip.The six generators followed their official inference recipes and used image-plus-text inputs.
  • P Compute Resources: The reported experiments consumed about 400 H100·hours for on-premise VLM judging and about 1,500 H100·hours for open-source video generation.Closed-source generation and judge calls were paid through APIs and excluded from the H100-hour budget; inference scripts, serving configurations, and seeds will be released.
  • Q Broader Impacts: The benchmarks aim to expose visually polished but semantically wrong generations, especially failures in Logic Reasoning and Information-Based content.WorldRewardBench’s paired human-preference data is presented as reusable evaluation support.
  • Q Broader Impacts: The evaluation suite releases no generative model, but could indirectly strengthen future video generators and thereby carry deepfake, fabricated-evidence, and impersonation risks.The passage identifies concrete mitigations, but the supplied text truncates their details.
  • R Licenses for Existing Assets: The paper uses existing models, APIs, datasets, libraries, and imagery under their respective licenses, and releases WorldReasonBench and WorldRewardBench under CC-BY 4.0.Released materials include prompts, taxonomy, QA pairs, preference pairs, and evaluation scripts; image references use URLs or hashes rather than re-hosted pixel data.

S Limitations

The benchmarks rely on VLM-based QA construction and judging, while their evaluation claims are qualified by ranking uncertainty, bounded taxonomy coverage, and limited reward-model validation. The authors mitigate some risks through human auditing and calibrated process/outcome measures, and identify extensions for future work.

  • VLM dependency in QA construction and judging: VLM-based QA construction and judging are audited on a stratified sample of ∼300 QA pairs, achieving Cohen κ=0.78 and rejecting or revising 7.8% of items.The headline metrics are produced by Qwen3.5-27B; the passage also notes additional independent checks for systematic VLM bias.
  • Cross-model evaluation set and ranking uncertainty: The shared evaluation set makes closed- and open-source comparisons identical at the case level, but only Seedance2.0 has strict within-closed-tier ranking support.Bootstrap analysis supports the closed-versus-open separation, while the other five closed-source models are reported as a tied cluster.
  • Scope of the reasoning taxonomy and coverage: WorldReasonBench covers four dimensions and 22 subcategories for initial-state-conditioned future prediction, excluding counterfactuals, broader multi-agent dynamics, numerical physics ground truth, and longer event chains.The authors do not claim the taxonomy is exhaustive.
  • Scope of the reasoning taxonomy and coverage: Extending the taxonomy along these excluded axes is designated as community-driven future work, supported by open-sourcing the construction pipeline.The stated future directions include broader reasoning coverage beyond the current scope.
  • Reward-model evaluation scope: Reward-model evaluation covers five judges under pair-wise and point-wise protocols, but does not validate reward-model training, generator finetuning, or comprehensive GPT-5.4 evaluation.GPT-5.4 evaluation currently covers a 595-video subset.
  • Hint-gain interpretation: The Easy/Difficult hint-gain split is descriptive rather than evidence of latent world reasoning, whereas ScorePR and sdyn/AccQA provide process-versus-outcome attribution calibrated against human Elo.Ceiling effects, prompt length, and instruction-following capacity may each enlarge the observed asymmetry.
Loading 2605.10434v1…