PaperlayerView exact revision on arXiv

Source-linked AI summary

YoCausal: How Far is Video Generation from World Model? A Causality Perspective

You-Zhe Xie, Yu-Hsuan Li, Jie-Ying Lee, Kaipeng Zhang, Yu-Lun Liu, Zhixiang Wang

arXiv:2605.30346v1cs.CV

TL;DR

Whether video diffusion models truly understand causality remains unclear, especially because existing benchmarks rely largely on synthetic or controlled data. YoCausal evaluates causal cognition using temporally reversed real-world videos and finds that arrow-of-time perception does not equal causal understanding, with models still substantially below human performance.

  • Problem

    Whether video generation models understand causality remains unresolved, while synthetic and controlled benchmarks limit assessment of real-world generalization.

  • Method

    YoCausal uses temporally reversed real-world videos as counterfactuals and evaluates models with RSI for time-direction perception and CCI for causal cognition.

  • Results

    Across 13 state-of-the-art video diffusion models, perceiving the arrow of time is not equivalent to understanding causality, and a substantial human-model gap remains.

  • Takeaways & Limitations

    YoCausal provides a scalable real-world benchmark showing that current video diffusion models remain short of causal cognition required for world modeling.

  • Takeaways & Limitations

    RSI is ineffective for temporally symmetric events, and denoising-loss computation requires model weights, limiting evaluation of closed-source models.

Abstract

from arXiv · show

As video diffusion models (VDMs) advance toward world models, a key question arises: do they truly understand causality, or merely overfit to statistical temporal patterns? Existing benchmarks mostly rely on synthetic data, limiting real-world generalization due to the sim-to-real gap. We present YoCausal, a two-level benchmark inspired by the Violation of Expectation (VoE) paradigm from cognitive science. By temporally reversing real-world videos at zero cost as natural counterfactual samples, YoCausal establishes an arbitrarily extensible evaluation protocol. Level 1 introduces the Reverse Surprise Index (RSI), quantifying arrow-of-time perception via denoising loss. Level 2 introduces the Causality Cognition Index (CCI), which leverages a VLM to stratify datasets into causal and non-causal subsets, disentangling genuine causal reasoning from temporal bias. Evaluation of 13 state-of-the-art VDMs reveals that perceiving the arrow of time does not imply understanding causality, and a significant gap persists relative to human-level causal cognition.

1 Introduction

YoCausal asks whether realistic video generation models understand intuitive cause-and-effect rather than merely temporal patterns. It introduces a scalable, real-world two-level benchmark that separates arrow-of-time perception from causal cognition and reveals a persistent human-model gap.

  • Motivation: Video generation models are promising world-model candidates, but whether they understand causality remains an open question.The paper distinguishes realistic spatio-temporal modeling from genuine understanding of cause-and-effect mechanisms.
  • Motivation: Existing physics-focused benchmarks rely on synthetic data or limited controlled recordings, creating a sim-to-real gap in real-world assessment.The authors argue that world models must comprehend broader causality beyond physical-law adherence.
  • Two-Level Framework: Level 1 introduces the Reverse Surprise Index (RSI), measuring how often denoising loss assigns lower likelihood to reversed than forward videos.RSI operationalizes surprise under a generative model through denoising loss.
  • Two-Level Framework: Level 2 defines the Causality Cognition Index (CCI) as RSI(Dc) − RSI(Dnc) to distinguish causal cognition from arrow-of-time perception.The dataset is partitioned into causal and non-causal subsets, Dc and Dnc.
  • Benchmark Design: YoCausal uses temporally reversed real-world videos as zero-cost counterfactuals in an arbitrarily extensible benchmark, with annotators judging 1,200 videos for a human reference.The design avoids dependence on fixed synthetic scenes or controlled recordings.
  • Findings: Evaluation across 13 state-of-the-art video diffusion models finds that arrow-of-time perception does not equal causal understanding and that a significant human-model gap remains.The study also reports partial correlation with intuitive physics but no correlation with aesthetic quality.

2 Related Work

Related work advances video diffusion toward world-model applications while evaluating generation through increasingly diverse temporal, physical, counterfactual, and causal benchmarks. YoCausal differs by using appearance-agnostic denoising likelihoods, real-world videos, and causal cognition evaluation rather than discriminative judgments alone.

  • Video Diffusion Models: Video diffusion has progressed from UNet architectures to Diffusion Transformers and commercial systems, increasing interest in world-model and interactive-simulation applications.The supplied passage frames whether current VDMs acquire these capabilities as an open question.
  • Video Generation Evaluation: Video-generation evaluation spans distribution metrics, multidimensional suites, temporal benchmarks, physical commonsense, and counterfactual or compositional reasoning.Existing approaches use VLM-judge templates or pixel-level comparisons, whereas YoCausal relies on denoising likelihoods.
  • Intuitive Physics and Violation-of-Expectation Paradigm: The violation-of-expectation paradigm measures cognition through surprise to counterfactuals and motivates diagnostics for causal world models grounded in intuitive theories.This tradition originates in perceptual causality and developmental studies of infant core knowledge.
  • Intuitive Physics and Violation-of-Expectation Paradigm: Prior physics-law benchmarks rely on synthetic data or controlled recordings, whereas YoCausal uses any real-world video at zero cost, is extensible, and achieves the highest listed scene diversity.The table notes that scene coverage and video count grow continuously as new subsets are added.
  • Arrow of Time and Causality in Video Understanding: Earlier arrow-of-time and causal-reasoning efforts evaluate discriminative tasks, while YoCausal additionally explores causal cognition in video-generation models.Prior work includes self-supervised temporal directionality, video question answering, and language counterfactual benchmarks.

3 Method

YoCausal builds an extensible real-world benchmark from temporally reversed videos and evaluates models at two levels: arrow-of-time perception and causal cognition. Its CCI separates reversed-causality sensitivity from statistical temporal-pattern sensitivity by comparing causal and non-causal video subsets.

  • Dataset Construction: YoCausal uses zero-cost temporal reversal of real-world videos, enabling arbitrarily scalable evaluation across diverse scenes without synthetic rendering or controlled setups.New thematic subsets can be added seamlessly to the benchmark.
  • Dataset Construction: The benchmark organizes videos into extensible thematic subsets covering general daily life, physics, human actions, and animal actions.These subsets are denoted DGeneral, DPhysics, DHuman, and DAnimal.
  • Level 1: RSI: RSI compares denoising losses for forward and reversed versions of each video under identical noise and sampled timesteps, measuring correct arrow-of-time perception.The method uniformly samples K=10 diffusion timesteps and applies identical Gaussian noise to both sequences.
  • Level 1: RSI: RSI alone cannot distinguish surprise caused by reversed causality from surprise caused by reversed temporal direction, motivating a second evaluation level.This limitation makes RSI insufficient as a standalone measure of causal cognition.
  • Level 2: CCI: CCI compares RSI on causal and non-causal subsets, isolating sensitivity to reversed-causality cues beyond statistical temporal patterns.A causally aware model should show higher RSI on Dc than on Dnc; the causal/non-causal split is automated with a VLM for scalability.
  • Level 2: CCI: CCI must be interpreted jointly with RSI: high RSI and low CCI indicate statistical arrow-of-time perception, while high CCI and low RSI make CCI unreliable.The method introduces an aggregate ranking combining both metrics.

4 Experiment

YoCausal evaluates 13 open-source video diffusion models with RSI and CCI, finding that arrow-of-time perception remains substantially below humans and does not guarantee causal cognition. Aggregate and entropy-controlled analyses further assess overall capability and support the benchmark’s validity.

  • Level 1 RSI: Humans achieve the highest RSI across all subsets except Human Action, while several models exceed the 50% random-guess baseline with 90% confidence but remain significantly below human performance.Higher-fidelity models such as LTX-Video-13B and Wan2.1/2.2-14B tend to score higher.
  • Level 1 RSI: RSI varies across domains because subsets differ in reversal-cue strength and training-data biases, while some models fall below 50%, indicating local smoothness without an internalized arrow of time.DPhysics contains unambiguous reversed anomalies, whereas models often perform well on DHuman because human-activity videos are abundant online.
  • Level 2 CCI: Humans achieve the highest CCI, several models attain positive CCI with 90% confidence, and high-RSI LTX-Video-2B/13B and Video models can score poorly on CCI.Top CCI performers concentrate in the Wan and CogVideo families, while negative CCI reflects treating reversed causal and non-causal sequences as equally anomalous.
  • Aggregate ranking: The aggregate causality score sums RSI and CCI ranks, with lower totals better and RSI rank breaking ties, providing a holistic basis for cross-metric analysis.The aggregate ranking is intended as a single measure of overall causal cognition because robust understanding requires strong performance on both indices.
  • Cross-metric analysis: The aggregate ranking correlates with human preference (τ = 0.3333), LikePhys (τ = 0.5111), release date (τ=0.596), and parameters (τ=0.688), but not aesthetic quality (τ = 0.0000).These results indicate alignment with human causal judgments, a relationship to physical understanding without reducibility, no visual-appeal confounding, and associations with scaling and architectural evolution.
  • Entropy-controlled analysis: After entropy matching, RSI scores closely track the full dataset and most models exceeding 50% retain that performance, indicating RSI is not confounded by low-level entropy cues.Because CCI is defined via RSI, this entropy invariance also supports the validity of YoCausal’s causal-cognition measurement.

5 Conclusion and Limitation

YoCausal introduces a benchmark for evaluating causal cognition in video diffusion models using temporally reversed real-world videos as natural counterfactuals. Its experiments show that perceiving the arrow of time is not equivalent to understanding causality, while RSI is ineffective for temporally symmetric events and requires model-weight access.

  • Conclusion: YoCausal is presented as the first benchmark for evaluating causal cognition in VDMs, using temporally reversed real-world videos as natural counterfactual samples.The protocol is described as arbitrarily extensible and free from synthetic data or controlled settings.
  • Conclusion: Experiments across 13 open-source VDMs reveal that perceiving the arrow of time is not equivalent to understanding causality.
  • Limitation: RSI is ineffective for temporally symmetric events, such as Newton’s cradle, because forward and reversed sequences are visually near-identical.
  • Limitation: Computing denoising losses requires model-weight access, limiting external evaluation of closed-source models.Developers can still apply YoCausal internally to diagnose and improve causal cognition.

Appendix · Appendix Overview

The appendix complements the main manuscript with additional details and complete numerical results. It covers dataset construction, model configurations, preprocessing, and formalizations of the RSI and CCI algorithms.

  • Appendix Overview: The appendix provides additional details and complete numerical results that complement the main manuscript.
  • Appendix Overview: Appendix A.1 describes the dataset construction protocol.
  • Appendix Overview: Appendix A.2 details inference configurations and specifications for all 13 evaluated models.
  • Appendix Overview: Appendix A.3 presents the unified video preprocessing pipeline.
  • Appendix Overview: Appendix A.4 formalizes the Reverse Surprise Index (RSI) algorithm.
  • Appendix Overview: Appendix A.5 formalizes the Causality Cognition Index (CCI) algorithm.

A.1 Details on Dataset Construction

YoCausal comprises four causally diverse subsets built from established real-world video collections. The benchmark contains 1,232 videos and can be extended at zero additional cost through temporal reversal.

  • Dataset composition: The benchmark contains four thematic subsets sourced from established real-world video collections, with clips uniformly trimmed to specified durations.The subsets cover general everyday events, physical phenomena, human actions, and animal behaviors.
  • General Subset: DGeneral samples 500 three-second videos from Moments in Time, spanning 339 categories across human activities, animal behaviors, natural phenomena, and object state changes.Its semantic diversity supports evaluation of general everyday causality.
  • Physics Subset: DPhysics includes 132 videos from Physics IQ, using one viewpoint per scenario and the first five seconds of each clip.The scenarios cover gravity, fluid motion, and collision dynamics, with temporally asymmetric causal cues.
  • Human Action Subset: DHuman samples 400 three-second clips from Kinetics-400, one per action class, covering diverse goal-directed human interactions.Examples include swimming, playing guitar, and handshaking.
  • Animal Action Subset: DAnimal samples 200 three-second videos from Animal Kingdom to test causal cognition across non-human agents and natural-world dynamics.Its behavior taxonomy includes foraging, aggression, courtship, and locomotion across diverse species.
  • Scale and Extensibility: 1,232 videos comprise the current benchmark, which remains extensible because any real-world video can be added at zero additional cost via temporal reversal.The scale reflects computational constraints from evaluating 13 video diffusion models, several with billions of parameters.

A.2 Models Setting

The evaluation covers 13 state-of-the-art open-source text-to-video diffusion models spanning architectures and parameter scales, using each model’s official recommended inference defaults. Classifier-free guidance is omitted because evaluation directly compares predicted and sampled noise rather than performing full denoising generation.

  • Model Coverage: 13 state-of-the-art open-source text-to-video diffusion models span diverse architectures and parameter scales.The evaluated models include AnimateDiff, CogVideoX, Mochi-1-preview, HunyuanVideo, Wan2.1, Wan2.2, and LTX-Video variants.
  • Evaluation Settings: Inference configurations follow each model’s official recommended defaults for output resolution, frame count, and FPS.These settings are summarized in Table A.2 alongside model-specific parameters and video compression features.
  • Evaluation Settings: Classifier-free guidance is not applied because evaluation directly computes MSE between predicted noise ˆϵ and sampled Gaussian noise ϵ at each timestep.The procedure evaluates denoising predictions without performing full denoising generation.

A.3 Video Preprocessing

Because models require different spatial and temporal inputs, the evaluation uses a unified preprocessing pipeline with resolution adaptation, FPS resampling, and long-video handling. Videos are transformed to each model’s prescribed input specifications to support fair and consistent evaluation.

  • A.3 Video Preprocessing: The unified pipeline comprises resolution adaptation, FPS resampling, and long-video handling to normalize model-specific input requirements.Models differ in spatial resolution, frame rate, and temporal length under their officially recommended settings.
  • Resolution Adaptation: For fixed-resolution models, videos are rescaled so the shorter side matches the target dimension, then center-cropped to the specified resolution and aspect ratio.Models supporting multiple aspect-ratio buckets use a different adaptation strategy, as described for the Wan and HunyuanVideo series.
  • FPS Resampling: FFmpeg resamples every source video to each model’s prescribed frame rate, matching its temporal sampling configuration.This standardizes the frame rate across inputs while following each model’s training setup.
  • Long Video Handling: When a video exceeds a model’s maximum frame window, it is partitioned along the temporal axis into consecutive frame-window-sized segments instead of being truncated.Partitioning prevents evaluation from relying only on a partial video segment.

A.4 Details on RSI Algorithm

The RSI algorithm measures arrow-of-time perception by comparing denoising losses for forward and reversed videos under matched evaluation conditions. It averages losses across sampled timesteps and declares correct temporal-direction recognition when the forward loss is lower, with 50% as chance.

  • RSI definition: RSI quantifies arrow-of-time perception as the Level-1 metric through the procedure formalized in Algorithm 1.The procedure comprises Stage 1 and Stage 2.
  • Timestep and noise sampling: K=10 timesteps are uniformly sampled from [1, T], excluding fully clean and fully noised boundary states, with Nϵ=1 noise sample per timestep shared across forward and reversed sequences.Identical noise makes the two sequences face the same denoising difficulty.
  • Prompt conditioning: Both sequences use the original forward-video caption, avoiding confounding causal understanding with following an unrealistic reversed-video instruction.Meaningful caption conditioning provides a sharper low-SNR denoising signal than a null prompt.
  • Computational setting: K=10 and Nϵ=1 are computational choices for evaluating 13 models on 1,232 videos, while larger values would better approximate expected denoising loss.The authors encourage increasing both sampling counts when computational resources permit.
  • Decision criterion: RSI averages denoising losses across sampled timesteps and noise samples, marking correct temporal-direction recognition when L(θ; x_f) < L(θ; x_r); 50% is chance level.RSI is the proportion of videos satisfying this inequality.

A.5 Details on CCI Algorithm … A.13 Scaling Law and Generational Evolution in Causal Cognition

YoCausal defines CCI by contrasting RSI on VLM-identified causal and non-causal video subsets, then validates the benchmark’s robustness, human alignment, limitations, and scaling trends. Across these analyses, prompt choice and VLM judge have limited influence, while implicit causality remains outside coverage and larger, newer models correlate with stronger aggregate rankings.

  • A.5 Details on CCI Algorithm: CCI(D) is defined as RSI(Dc) − RSI(Dnc), isolating causal cognition by contrasting VLM-identified causal and non-causal video subsets.Dc contains salient cause-and-effect interactions, while Dnc excludes them; the algorithm computes RSI for each subset before taking their difference.
  • A.5 Details on CCI Algorithm: A VLM automatically partitions videos by whether they contain observable cause-and-effect interactions, enabling scalable CCI computation without manual annotation.The classifier prompt defines causality as Event A visibly causing Event B, including physical and logical event sequences.
  • A.6 Prompt Bias: Null-prompt ablation produces only negligible RSI and CCI changes while preserving discriminative signs, indicating the benchmark signal is not primarily caused by text-video misalignment.The study therefore retains forward prompts because they reduce low-SNR randomness while introducing little text misalignment.
  • A.7 VLM Reliability: Kendall’s τ=0.7613 and F1-score=82.76% show close VLM-human annotation agreement, while Cohen’s d=0.057 < 0.2 indicates negligible motion-magnitude differences between subsets.These results support semantic rather than low-level-motion-based VLM partitioning.
  • A.9 Limitation: Implicit Causality: Implicit causality, including visually subtle or non-perceptual relationships, is largely excluded from Dc and therefore not captured by the current benchmark.The authors argue that robotic manipulation, interactive games, and autonomous-driving simulations predominantly involve explicit, perceptually salient causal events covered reliably by the partitioning.
  • A.11 Details on Human Preference: The human preference study ranks six representative model families across 60 prompts, using three independent participant rankings per prompt and allowing ties.The study evaluates plausibility of causal interactions in 360 generated videos.
Loading 2605.30346v1…