PaperlayerView exact revision on arXiv

Source-linked AI summary

RISE-Video: Can Video Generators Decode Implicit World Rules?

Mingxin Liu, Shuran Ma, Shibei Meng, Xiangyu Zhao, Zicheng Zhang, Shaofeng Zhang, Zhihang Zhong, Peixian Chen, Haoyu Cao, Xing Sun, Haodong Duan, Xue Yang

arXiv:2602.05986v1cs.CVcs.AI

TL;DR

Whether TI2V models can reason over implicit world rules remains under-evaluated beyond metrics focused on perceptual quality and temporal coherence. RISE-Video introduces a reasoning-oriented benchmark and LMM judging pipeline, and evaluation finds persistent reasoning limitations across current systems.

  • Problem

    Whether TI2V models can reason over implicit world rules remains under-evaluated, as existing metrics largely emphasize perceptual quality and temporal coherence.

    Source: §1 · pp. 1–2

  • Method

    RISE-Video evaluates eight reasoning domains using 467 human-annotated samples, four complementary metrics, and an automated LMM judging pipeline.

    Source: §1 · p. 2

  • Results

    Across 11 TI2V models, higher-level reasoning remains difficult; even the best performer, Hailuo 2.3, reaches only 22.5% accuracy.

    Source: §4.1 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The LMM judging pipeline may provide a scalable, reliable, and cost-effective alternative to large-scale human assessment.

    Source: §1 · p. 2

Abstract

from arXiv · show

While generative video models have achieved remarkable visual fidelity, their capacity to internalize and reason over implicit world rules remains a critical yet under-explored frontier. To bridge this gap, we present RISE-Video, a pioneering reasoning-oriented benchmark for Text-Image-to-Video (TI2V) synthesis that shifts the evaluative focus from surface-level aesthetics to deep cognitive reasoning. RISE-Video comprises 467 meticulously human-annotated samples spanning eight rigorous categories, providing a structured testbed for probing model intelligence across diverse dimensions, ranging from commonsense and spatial dynamics to specialized subject domains. Our framework introduces a multi-dimensional evaluation protocol consisting of four metrics: \textit{Reasoning Alignment}, \textit{Temporal Consistency}, \textit{Physical Rationality}, and \textit{Visual Quality}. To further support scalable evaluation, we propose an automated pipeline leveraging Large Multimodal Models (LMMs) to emulate human-centric assessment. Extensive experiments on 11 state-of-the-art TI2V models reveal pervasive deficiencies in simulating complex scenarios under implicit constraints, offering critical insights for the advancement of future world-simulating generative models.

1 Introduction

Existing TI2V evaluation emphasizes perceptual quality and temporal coherence, leaving implicit reasoning underassessed. RISE-Video introduces a human-annotated benchmark and evaluation pipeline to probe whether models follow implicit world rules.

1 Introduction

  • Existing evaluations largely emphasize perceptual quality and temporal coherence, leaving a scarcity of protocols focused on implicit reasoning in TI2V.The paper motivates a dedicated reasoning-oriented diagnostic framework.Source: §1 · pp. 1–2
  • RISE-Video provides 467 human-annotated samples across eight reasoning domains to evaluate whether TI2V models internalize and execute implicit world rules.The domains span experiential, commonsense, temporal, societal, perceptual, spatial, subject-specific, and logical reasoning.Source: §1 · p. 2
  • RISE-Video defines four metrics—Reasoning Alignment, Temporal Consistency, Physical Rationality, and Visual Quality—for assessing reasoning and video quality.Its evaluation addresses cognitive and physical constraints as well as visual plausibility.Source: §1 · p. 2
  • An automated LMM judging pipeline uses reasoning-aware questions and prompts, with high alignment to human judgments for scalable assessment.The authors present LMM-based evaluation as a cost-effective alternative to large-scale human assessment.Source: §1 · p. 2
  • Evaluation of 11 representative TI2V models reveals systematic reasoning limitations across current systems.The benchmark targets implicit reasoning beyond explicit textual instructions.Source: §1 · pp. 2–3

2 Related Work

Video-generation research has diversified in its architectures and capabilities, while evaluation has advanced beyond perceptual metrics but still leaves higher-level reasoning insufficiently assessed.

2.1 Video Generation Models

  • Video-generation research centers on diffusion [5] [6] [7] [11] [12] [30], progressing from temporal modules [3] [9] to space–time and 3D-VAE designs [2], multimodal generation and editing [16] [24], and large-scale systems [32] [37].Lumiere generates whole clips in one pass; CogVideoX supports longer, higher-resolution, better text-aligned videos, while production systems push duration, realism, and controllability.Source: §2.1 · p. 3

2.2 Evaluation of Video Generation Models

  • Benchmarks progressed from coarse perceptual measures [10] [26] to VBench’s 8 categories and 16 dimensions [14], physics-commonsense [1] [18] [20] [27] and video-dynamics tests [8] [17] [41], and human-aligned LMM judging [31] [38] [42] [45].Despite this broader evaluation, existing benchmarks mainly assess perceptual quality and temporal coherence, falling short on higher-level reasoning.Source: §2.2 · p. 4

3 Method

RISE-Video organizes 467 human-annotated samples into eight reasoning-knowledge categories and evaluates generated videos through four complementary metrics. Its methods combine LMM-based judgments with task-specific assessment procedures and weighted and strict overall scores.

3.1 Data Construction

  • The benchmark defines eight reasoning-knowledge categories and comprises 467 samples curated and annotated by human experts.The categories target distinct aspects of reasoning needed to understand or generate videos under structured constraints.Source: §3.1 · pp. 4–5
  • Commonsense, subject, perceptual, and societal knowledge cover everyday cause and effect, discipline-specific concepts, visual attributes and occlusion, and social and cultural contexts.Examples include physical principles, size and position, emotion recognition, behavioral norms, and cultural customs.Source: §3.1 · p. 4
  • Logical, experiential, spatial, and temporal knowledge test rule-based tasks, intentions and procedures, 3D spatial relations, and event ordering across multiple timescales.Spatial knowledge is inspired by RISEBench [44]; temporal tasks also include events presented in reverse order.Source: §3.1 · pp. 4–5

3.2 Evaluation Metrics

  • Reasoning Alignment uses knowledge-aware yes/no questions scored from 0–1, with frame sampling adapted to whether a task requires event progression or a final state.Progression tasks use uniform sampling at 2 fps, while final-state tasks use a lower sampling rate.Source: §3.2 · p. 5
  • For schematic puzzles, specialized checks assess maze trajectories, symmetry-grid alignment, or board-game states instead of relying solely on standard LMM judging.Maze scores reflect wall-crossing and target-reaching constraints; symmetry scores use discretized grid accuracy, and board-game judgments receive the final frame with a ground-truth reference.Source: §3.2 · pp. 5–6, Figure 4 · p. 6
  • Temporal Consistency scores stability of non-instructed elements from 1–5, while Physical Rationality scores physical realism from 1–5 and applies only to physically grounded environments.Both use uniformly sampled frames; Physical Rationality considers gravity, object permanence, collisions, fluid motion, and coherent environmental reactions, excluding abstract puzzles and symbolic reasoning.Source: §3.2 · pp. 6–7
  • Visual Quality rates clarity and technical integrity on a 1–3 scale using six uniformly sampled frames, excluding the first and last frames.Low-clarity images are manually super-resolved before evaluation to distinguish native resolution from technical blur.Source: §3.2 · p. 7
  • Overall evaluation reports a Weighted Score using metric weights of 0.4, 0.25, 0.25, and 0.1, and Accuracy requiring full scores on all four dimensions.Accuracy is normalized to a 100-point scale.Source: §3.2 · p. 7

4 Experiments

Across 11 closed- and open-source TI2V models, reasoning remains challenging: Hailuo 2.3 leads accuracy and Reasoning Alignment, while category-specific and temporal failures persist. Human-rating comparisons find GPT-5 generally aligns best across metrics, with exceptions for Temporal Consistency and Visual Quality.

4 Experiments

  • The evaluation covers 11 representative closed- and open-source TI2V models, using GPT-5 to judge Reasoning Alignment, Temporal Consistency, and Physical Rationality, and GPT-5-mini for Visual Quality.The evaluated systems include six closed-source models and five open-source models or variants.Source: §4 · p. 7

4.1 Main Results

  • Hailuo 2.3 leads accuracy at 22.5%, followed by Veo 3.1 at 22.3% and Sora 2 at 21.3%; it also exceeds Wan 2.6 in Reasoning Alignment by 6.6%.Sora 2 shows a strength in Temporal Consistency, while open-source models generally underperform closed-source systems in reasoning and visual quality.Source: §4.1 · p. 8
  • Models perform better on Perceptual Knowledge than Logical Capability; in the Gold Miner task, none correctly captures the stone along the hook trajectory, while only Veo 3.1 and Hailuo 2.3 infer bottle-cap removal.Veo 3.1 changes the hook shape and Kling 2.6 moves the gold without contact, illustrating difficulty with rule-based decisions.Source: §4.1 · p. 8, Table 2 · p. 8
  • Kling 2.6 often produces near-static videos and misses required transformations, whereas Wan 2.6 and Hailuo 2.3 show stronger instruction following and dynamic behavior.Veo 3.1 and Sora 2 are less responsive to dynamic instructions; Veo 3.1 also fails to adapt a chameleon’s color to a branch, and Sora 2 and Veo 3.1 show temporal discontinuities.Source: §4.1 · pp. 8–9

4.2 Ablation Study

  • Five expert annotators provide averaged human ratings for MAE and STD comparisons; ratings retain native scales of 0–1 for RA, 1–5 for Consistency and PR, and 1–3 for VQ.MAE measures the mean absolute difference from the aggregated human consensus, while STD is the standard deviation of absolute errors.Source: §4.2 · p. 9
  • GPT-5 aligns most robustly with human preferences across most metrics, while Qwen3-VL-235B has lower Temporal Consistency error but tends to accept inconsistent outputs; gpt-5-mini aligns more tightly on Visual Quality than Gemini-3-Flash and Qwen3-VL-235B.The Qwen3-VL-235B high-score bias can undermine differentiation between high-quality outputs and those with severe defects.Source: §4.2 · pp. 9–10, Table 3 · p. 9

5 Conclusion

RISE-Video evaluates TI2V models on diverse reasoning requirements beyond perceptual fidelity, using an LMM judging pipeline aligned with human judgments. Evaluation of 11 representative models finds persistent difficulty with higher-level and implicit reasoning despite strong perceptual quality.

5 Conclusion

  • Despite strong perceptual quality, 11 representative TI2V models struggle with higher-level and implicit reasoning, exposing a gap between visual realism and rule-consistent generation.The authors hope RISE-Video will support more rigorous evaluation and inspire reasoning-aware model design and training.Source: §5 · p. 10
  • RISE-Video organizes data into eight complementary reasoning categories and assesses TI2V models across four dimensions beyond perceptual fidelity.The benchmark is designed to evaluate whether generated videos satisfy diverse reasoning requirements.Source: §5 · p. 10
  • An automated LMM-based judging pipeline enables scalable, fine-grained evaluation while maintaining high alignment with human judgments.Source: §5 · p. 10

Appendix · A.1 Data Source

The benchmark’s input images are primarily drawn from generated images, permissively licensed websites, and manually curated RISEBench samples adapted for TI2V reasoning tasks.

A.1 Data Source

  • High-quality image-generation models provide images selected for visual fidelity and diversity in downstream TI2V tasks.Source: §A.1 · p. 14
  • Suitable RISEBench samples are manually curated and adapted for TI2V reasoning tasks.Source: §A.1 · p. 14

A.1.1 Privacy-Preserving Image Stylization

The dataset uses image stylization on tasks where real-person appearance is unnecessary for reasoning evaluation, reducing privacy risks while preserving task-relevant information.

A.1.1 Privacy-Preserving Image Stylization

  • Stylization removes identifiable details from images of real individuals while retaining structural and semantic information needed for valid reasoning evaluation.It is applied only when preserving the individuals’ original appearance is not essential to the task.Source: §A.1.1 · p. 14

A.2 Prompt for Judgement

The judgement prompts operationalize four evaluation dimensions: reasoning alignment, temporal consistency, physical rationality, and visual quality. They constrain what judges assess and specify dimension-specific scoring rubrics and output formats.

Reasoning Alignment

  • Reasoning Alignment without reference asks the judge to answer questions from the provided video frames alone, with a reason for each Yes-or-No answer.The prompt requires answers in a JSON list and prohibits guessing or hallucination.Source: §A.2 · p. 14
  • Reasoning Alignment with reference compares a video’s final frame with a ground-truth image for the region or attribute specified by a focus question, using a 1–5 similarity scale.The judge is instructed to assess only visible evidence and not guess or hallucinate.Source: §A.2 · p. 15

Temporal Consistency

  • Temporal Consistency uses a 1–5 scale to rate unintended visual and semantic changes across frames, while excluding changes required or implied by the instruction.The rubric distinguishes minor discrepancies, noticeable inconsistencies, multiple significant issues, and severe continuity breaks.Source: §A.2 · pp. 15–16

Physical Rationality

  • Physical Rationality applies an amplitude-aware 1–5 rubric, from broken scenes and large physical failures to minor artifacts and physically seamless motion.It considers errors in both the main subject and background, including clipping, distortion, object popping, and unnatural motion.Source: §A.2 · p. 17

Visual Quality

  • Visual Quality judges sampled frames on sharpness, artifacts, distortion, lighting, and color using a strict 1–3 scale, while ignoring artistic blur and partial occlusion.The endpoints range from unusable severe technical failure to excellent, sharp, artifact-free imagery.Source: §A.2 · p. 18

A.3 Analysis on the Judge Models

The section presents qualitative comparisons of different judge models on temporal consistency, physical rationality, and visual quality.

A.3 Analysis on the Judge Models

  • Figure 6 qualitatively compares different judge models on temporal consistency using a prompt to generate the natural next moment after a scene.The instruction asks for a reasonable continuation showing what naturally happens next.Source: Figure 6 · p. 19, §A.3 · p. 20
  • Figure 7 qualitatively compares different judge models on physical rationality, reported on a 1–5 scale.The illustrated experimental process concerns generating a knee-jerk reflex.Source: Figure 7 · p. 20, §A.3 · pp. 20–21
  • Figure 8 qualitatively compares different judge models on visual quality.Source: Figure 8 · p. 21

A.4 More Vsualizations

The appendix presents qualitative TI2V examples across knowledge tasks, illustrating prompts that test implicit world rules and one described perceptual outcome.

Perceptual Knowledge

  • On a Perceptual Knowledge task, the vehicle becomes fully visible as it continues moving forward.Figure 9 presents qualitative generation examples from different TI2V models on this task.Source: §A.4 · p. 22, Figure 9 · p. 23

Commonsense and Experiential Knowledge

  • Commonsense and Experiential Knowledge examples prompt videos of toe-joint changes under long-term high uric acid and hands taking a letter out of an object.Figures 10 and 12 show qualitative generation examples for these task categories.Source: §A.4 · pp. 24, 29, Figure 10 · p. 25, Figure 12 · p. 30

Temporal Knowledge

  • The Temporal Knowledge prompt asks models to show time reversing over five seconds, with examples labeled for HunyuanVideo-1.5-720P-I2V and its cfg-distill variant.Figure 11 presents qualitative examples from different TI2V models on Temporal Knowledge tasks.Source: §A.4 · pp. 26–28, Figure 11 · p. 28

Logical, Societal, and Spatial Knowledge

  • Other examples test logical water flow into correct containers, placing Thanksgiving’s iconic main food on a plate, and preserving a left-to-right cookie shape order.These are the Logical Capability, Societal Knowledge, and Spatial Knowledge prompts, respectively.Source: §A.4 · pp. 31, 33, 35

Subject Knowledge

  • The Subject Knowledge prompt asks for the dynamic process after phenolphthalein is added to sodium hydroxide solution.Figure 16 presents qualitative generation examples from different TI2V models on Subject Knowledge tasks.Source: §A.4 · p. 37, Figure 16 · p. 38

Related papers

Loading 2602.05986v1…