PaperlayerView exact revision on arXiv

Source-linked AI summary

RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligence

Xuming He, Zehao Fan, Hengjia Li, Fan Zhuo, Hankun Xu, Senlin Cheng, Di Weng, Haifeng Liu, Can Ye, Boxi Wu

arXiv:2512.02622v1cs.CV

TL;DR

Video-generation benchmarks largely emphasize perception and understanding, leaving rule-based reasoning insufficiently decomposed and evaluated. RULER-Bench addresses this gap with a six-category benchmark spanning T2V and I2V, plus a four-metric checklist protocol. It covers 40 tasks and 622 instances, and its MLLM evaluation aligns with human judgments on 85% of checklist questions while exposing persistent rule-coherence limitations.

  • Problem

    Existing video-generation benchmarks primarily assess perception and understanding, while rule-based reasoning lacks fine-grained decomposition and systematic evaluation.

    Source: §1 · p. 2

  • Method

    RULER-Bench conceptualizes reasoning as cognitive rule-based prediction and evaluates T2V and I2V videos across six rule categories using checklist-based assessment.

    Source: §1 · p. 2

  • Results

    RULER-Bench covers 40 tasks and 622 instances, while its checklist evaluation aligns with human judgments on 85% of questions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments reveal substantial limitations in rule coherence across state-of-the-art video-generation models, indicating room for improved reasoning capability.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recent advances in video generation have enabled the synthesis of videos with strong temporal consistency and impressive visual quality, marking a crucial step toward vision foundation models. To evaluate these video generation models, existing benchmarks primarily focus on factors related to visual perception and understanding, like visual aesthetics, instruction adherence, and temporal coherence. However, the rule-based reasoning capabilities of video generation models remain largely unexplored. Although recent studies have carried out preliminary explorations into whether video models can serve as zero-shot learners, they still lack a fine-grained decomposition of reasoning capabilities and a comprehensive evaluation protocol. To address this gap, we introduce RULER-Bench, a benchmark designed to evaluate the reasoning ability of video generation models from the perspective of cognitive rules. Built upon two fundamental paradigms: text-to-video and image-to-video, RULER-Bench covers 40 representative tasks spanning six rule categories with 622 high-quality annotated instances. For the evaluation of each generated video, we construct a checklist covering four metrics and leverage GPT-o3 to assign scores to each question, achieving 85% alignment with human judgements. Extensive experiments show that the state-of-the-art model achieves only 48.87% on the rule coherence metric, highlighting significant room for improvement in the reasoning capability of next-level video models. We expect that the insight obtained from RULER-Bench will facilitate further development of reasoning-aware video generation, advancing video generation models toward vision foundation intelligence.

1. Introduction

Video generation models have largely advanced in visual quality and instruction adherence, but their rule-based reasoning remains insufficiently evaluated. RULER-Bench addresses this gap with a six-category benchmark and checklist protocol, revealing persistent rule-coherence limitations.

Motivation

  • RULER-Bench targets the unresolved question of whether next-level video generation models possess reasoning abilities needed for vision foundation intelligence.Existing models can generate visually coherent clips while differing substantially in their ability to infer and depict underlying phenomena.Source: §1 · p. 2
  • Existing studies and benchmarks lack fine-grained reasoning decomposition and systematic coverage beyond physics and commonsense reasoning.Prior work explores perception, modeling, manipulation, and reasoning, while other benchmarks remain limited to physical laws or commonsense dimensions.Source: §1 · p. 2

Contribution

  • The benchmark conceptualizes reasoning as cognitive rule-based prediction across Nature, Society, and Virtuality, organized into six rule categories.The categories are Vision, Science, Semantics, Hypothesis, Game, and Humanity.Source: §1 · p. 2
  • RULER-Bench covers 40 tasks and 622 high-quality instances across text-to-video and image-to-video generation.The benchmark uses a hierarchical organization within the six rule categories and distributes instances across tasks.Source: §1 · p. 2

Evaluation

  • Its checklist protocol evaluates Instruction Following, Visual Consistency, Visual Fidelity, and Rule Coherence, with MLLM judgments aligning with human annotations on 85% of questions.The protocol uses discrete good, medium, and bad ratings for checklist questions.Source: §1 · p. 2

Findings

  • Experiments on 10 state-of-the-art models reveal substantial limitations across rule types, with all models consistently limited in rule coherence.These results identify room for improving the reasoning capability of next-level video models.Source: §1 · p. 2

2. Related Work

Video-generation research has progressed from perceptual quality toward reasoning, but existing benchmarks do not provide a systematic framework for evaluating rule-based reasoning across generation settings.

Video Generation Models

  • Diffusion and autoregressive video-generation systems have improved visual consistency, aesthetic quality, and instruction adherence through larger models and high-quality data.Recent systems such as Sora2, Veo3, and Wan have driven this progress.Source: §2 · p. 2
  • Recent reasoning-oriented methods address physical dynamics or visual reasoning, yet benchmarks still lack systematic evaluation of video-model reasoning.NewtonGen models Newtonian motion, while V-Chain injects visual reasoning signals into generation.Source: §2 · p. 3

Benchmarks for Video Generation Methods

  • Existing benchmarks typically specialize in one task type or a narrow capability, leaving systematic rule-based reasoning underexplored across T2V and I2V.RULER-Bench unifies these two paradigms to evaluate diverse reasoning dimensions.Source: §2 · p. 3

3. RULER-Bench

RULER-Bench formulates video reasoning as prediction from cognitive rules, constructs diverse T2V and I2V data, and applies layered quality control to preserve rule correctness and visual coverage.

3.1. Rule-Based Task Formulation

  • RULER-Bench defines video reasoning as inferring implicit rules from inputs, predicting possible outcomes, and presenting them through generated videos.The formulation is grounded in Nature, Society, and Virtuality and supports six rule categories.Source: §3.1 · p. 3
  • The six categories cover real-world visual and scientific rules, socially consistent behavior and semantics, and game or hypothetical virtual-environment reasoning.They are organized into Natural, Social, and Virtual domains.Source: §3.1 · p. 3
  • The benchmark decomposes these categories into 40 distinct tasks for finer-grained evaluation.Detailed task descriptions are provided in the appendix.Source: §3.1 · p. 3

3.2. Data Construction

  • Data construction combines human annotation with GPT-5 generation across T2V and I2V paradigms, using manually curated seed samples to guide reliable sample creation.T2V instances pair prompts with implicit explanations, while I2V data integrates web-sourced and generated images.Source: §3.2 · pp. 3–4
  • For Game samples, ground-truth final-frame images and manually written implicit explanations address MLLM difficulty in representing underlying game strategies.This procedure is used to capture the relevant rules accurately.Source: §3.2 · p. 4

3.3. Quality Control and Data Refinement

  • Quality control deduplicates similar prompts, verifies prompt-explanation-image consistency, applies human refinement, filters privacy or ethical risks, and pads inputs to avoid aspect-ratio information loss.The process validates rule correctness, task diversity, scenario typicality, and visual-content preservation.Source: §3.3 · pp. 4–5

4. Evaluation Protocol

RULER-Bench evaluates videos with four complementary metrics and an interpretable checklist protocol, whose questions are generated and manually verified for broad coverage.

4.1. Evaluation Metrics

  • The evaluation protocol combines Instruction Following, Visual Consistency, Visual Fidelity, and the newly introduced Rule Coherence metric.Rule Coherence measures adherence to implicit scene-specific rules rather than only perceptual quality.Source: §4.1 · p. 5
  • Instruction Following measures semantic alignment with the prompt, independently of whether the generated video follows implicit rules.Visual Consistency concerns unchanged element identities, while Visual Fidelity concerns clarity, stability, and freedom from artifacts.Source: §4.1 · p. 5
  • Rule Coherence assesses whether videos use implicit rules to predict or infer the expected visual phenomena.The communicating-vessels example requires depicting equal final water levels in both arms of a U-shaped tube.Source: §4.1 · p. 5
  • Each sample receives a checklist of metric-specific questions rated as good, medium, or bad, enabling more interpretable and human-aligned judgments with o3 as evaluator.The checklist pipeline is designed to assess rule-based reasoning through multiple objective questions.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2. Checklist Construction and Quality Control

  • GPT-5 generates checklist questions from each sample’s image, prompt, and implicit explanation, covering distinct perspectives across all four metrics.The questions are designed to assess diverse aspects of generated-video content systematically.Source: §4.2 · p. 5
  • Manual verification finalized 6,500 checklist questions for 622 samples, checking accuracy, clarity, consistency, and completeness.This verification provides comprehensive coverage across the four evaluation metrics.Source: §4.2 · p. 5

5. Experiments and Evaluations

RULER-Bench evaluates 10 video models across six rule categories and four metrics, revealing substantial weaknesses in rule coherence and image-based reasoning. Prompt enhancement, case studies, and human-alignment tests further analyze these limitations and the evaluation protocol.

5.1. Main Results

  • 48.87 on Rule Coherence for Veo3.1, versus approximately 80 on Visual Consistency and Visual Fidelity, demonstrates a sharp decline as evaluated cognitive demands increase.Rule Coherence is the lowest-performing metric for all video models; open-source models generally perform below closed-source models.Source: Table 2 · p. 6, §5.1 · p. 7
  • 80 is the highest reported category average for Veo3.1 on Humanity and Semantics, while every model scores below 15 on Game Rule Coherence.Within Humanity, the dress task reaches an average of 84.9, indicating substantial variation between tasks in the same category.Source: §5.1 · p. 7, Figure 4 · p. 7
  • 65.55 on T2V tasks versus 48.56 on I2V tasks shows a substantial reasoning-performance gap when models must reason over input images.Game and Vision are the weakest categories and consist entirely of I2V instances.Source: §5.1 · p. 7

5.2. Analysis

  • Prompt enhancement substantially improves Veo3.1 and Sora2 on Rule Coherence, yet explicit expected outcomes do not eliminate their difficulty generating rule-consistent clips.The Game category is excluded from the prompt-enhancement comparison because many samples rely heavily on ground-truth images.Source: §5.2 · p. 7, Table 4 · p. 7
  • Case studies show model-specific strengths: Sora2 succeeds on game strategy, while Veo3.1 and PixelVerse-v5 correct visual anomalies.The examples also expose failures in metaphor interpretation, AED placement, clothing reasoning, and visual-context reasoning.Source: §5.2 · p. 8, Figure 5 · p. 8
  • GPT-o3 achieves 0.8512 accuracy and leads the other MLLMs across all three reported correlation metrics in human-alignment validation.The validation compares MLLM outputs with human annotations on 813 checklist questions from 80 generated videos.Source: Table 3 · p. 6, §5.2 · p. 8

6. Conclusion

The paper introduces RULER-Bench as a benchmark for rule-based reasoning in video generation and applies it to 10 state-of-the-art models. Its benchmark and analyses aim to support reasoning-aware video generation toward vision foundation intelligence.

6. Conclusion

  • RULER-Bench contains 622 high-quality instances spanning 40 tasks across six rule categories and uses a checklist-based protocol with four evaluation metrics.The benchmark is designed for fine-grained evaluation of reasoning capabilities in video models.Source: §6 · p. 8
  • Experiments on 10 state-of-the-art video generation models provide detailed analyses of their rule-based reasoning performance.The paper frames these analyses as insights for advancing reasoning-aware video generation toward vision foundation intelligence.Source: §6 · pp. 8–9

Supplementary Material

The supplementary material provides task explanations, implementation details, additional task settings, evaluation materials, visualizations, limitations, future work, and social-impact discussion.

Supplementary Material

  • The appendix covers detailed task explanations, model configurations, video-to-video explorations, annotation and user-study results, prompts, cross-task results, visualizations, limitations, future work, and social impact.These materials are organized across Sections A through I.Source: Appendix Contents · p. 13

A. Detailed Tasks Explanations

RULER-Bench defines 40 tasks across six rule categories spanning nature, society, and virtuality. The task suite tests visual, scientific, hypothetical, strategic, semantic, and human-convention reasoning.

A.1. Vision Rule

  • The Vision category contains 10 tasks testing physical, spatial, temporal, appearance, numerical, geometric, and anomaly-related visual reasoning.Tasks require models to transform or predict image content while preserving relevant spatial-temporal or physical properties.Source: §A.1 · p. 13

A.2. Science Rule

  • The Science category spans 10 tasks that require models to simulate natural laws, scientific processes, biological behavior, environmental change, mathematics, and medical procedures.Its tasks include physics, chemistry, biology, Earth, math, medicine, and life scenarios.Source: §A.2 · pp. 13–14

A.3. Hypothesis Rule

  • The Hypothesis category tests zero-shot reasoning under two kinds of supplied rules: subjective hypothetical scenarios and deliberately modified objective laws.Models must apply the prompt-defined rule instead of relying solely on real-world logic or physics.Source: §A.3 · p. 14

A.4. Game Rule

  • The Game category contains 10 constrained reasoning tasks involving planning, legal moves, tactical patterns, deduction, pathfinding, and spatial assembly.Examples include number sliding, matchsticks, Chinese Chess, Gomoku, Go, Sudoku, chess, Minesweeper, mazes, and puzzles.Source: §A.4 · pp. 14–15

A.5. Semantics Rule

  • The Semantics category evaluates whether models infer contextual and figurative meaning through definition, metaphor, and idiom tasks.The tasks specifically distinguish intended conceptual meaning from literal or surface-level interpretations.Source: §A.5 · p. 15

A.6. Humanity Rule

  • The Humanity category contains eight tasks covering social norms, cultural practices, emotions, safety, sports, and transportation conventions.The suite includes culturally appropriate dress, food, festivals, social behavior, safety, sport, and traffic scenarios.Source: §A.6 · p. 15

B. Implementation Detail

Videos follow official model configurations, with durations of 5–10 seconds and resolutions of either 720P or 480P.

B. Implementation Detail

  • 5–10-second videos at 720P or 480P follow each model’s official configuration.Open-source models are implemented using the Diffusers framework.Source: B. Implementation Detail · p. 15

C. Explorations on Video-to-Video tasks

The paper extends evaluation to V2V tasks, validates checklist construction, and compares automated evaluation with human preferences. These analyses show sporadic V2V reasoning success, stronger visual metrics with input-video references, and over 85% human alignment for GPT-o3.

C. Explorations on Video-to-Video tasks

  • Input videos yield substantially higher Visual Consistency and Visual Fidelity than corresponding I2V instances, likely by providing strong visual references.The reported improvement concerns visual alignment and temporal consistency rather than rule-based reasoning quality.Source: C. Explorations on Video-to-Video tasks · p. 15
  • Wan2.1 14B usually struggles with plausible V2V reasoning, producing expected results only sporadically.The V2V set covers eight Vision tasks and 81 samples; quantitative results are reported in Table A3.Source: C. Explorations on Video-to-Video tasks · pp. 15–16, Table A3 · p. 16

D.1. Checklist Validation

  • Checklist validation revises questions for accuracy, clarity, relevance, consistency, and completeness to support reliable judgments.Annotators receive metadata such as the prompt and implicit explanation when assessing each checklist question.Source: §D.1 · p. 16

D.2. User Study

  • GPT-o3 achieves over 85% alignment with human preferences and outperforms the compared MLLMs across KRCC, SRCC, PLCC, and ACC.The human-alignment study samples 80 generated videos with their checklist questions.Source: §D.2 · pp. 16–17, Table A2 · p. 16

E. Prompt for Generation and Evaluation

  • GPT-5 generates prompts, implicit explanations, initial scenarios, and checklists from task metadata, images, and instructions for the generation and evaluation pipelines.The pipeline uses separate prompts for image-to-scenario, scenario-to-image, checklist generation, and four-metric evaluation.Source: Figure A2 · p. 16, E. Prompt for Generation and Evaluation · p. 17

F. Experimental Results per Task

Across tasks, Veo3.1 performs best and closed-source models consistently outperform open-source models, while prompt enhancement helps but does not yield reliable rule-based reasoning. Additional visualizations cover eight representative tasks.

F. Experimental Results per Task

  • Veo3.1 performs strongly across multiple tasks, and closed-source models consistently outperform open-source models.Table A4 summarizes evaluation results across tasks in the six rule categories.Source: F. Experimental Results per Task · p. 17
  • Prompt enhancement produces observable gains, but current video models still fall short of reliable rule-based reasoning.The conclusion applies across the evaluated tasks rather than to a single category.Source: F. Experimental Results per Task · p. 17

G. More Visualization Results

  • Qualitative results visualize model outputs for Anomaly, Safety, Earth, Life, Physics, Chemistry, Emotion, and Color tasks.These figures provide task-level visualizations rather than additional quantitative comparisons.Source: G. More Visualization Results · p. 17

H. Limitations and Future Works

The benchmark’s main scope limitation is the lack of one unified evaluation across T2V, I2V, and V2V, while future expansion could add broader reasoning scenarios. The authors also note potential misuse risks requiring ethical oversight.

H. Limitations and Future Works

  • RULER-Bench does not jointly evaluate T2V, I2V, and V2V because existing models support different task combinations.The current evaluation therefore focuses on the settings supported by the benchmark’s model coverage.Source: H. Limitations and Future Works · p. 17
  • Future work could broaden task coverage to GUI interaction, classical algorithms, and table reasoning.The authors identify this as a potential expansion in both breadth and depth.Source: H. Limitations and Future Works · p. 17

I. Social Impacts

  • Stronger video-generation abilities may increase misuse risks, motivating ethical deployment practices and appropriate oversight.The paper frames this as a social-impact concern accompanying progress in generation quality and rule-based reasoning.Source: §I · p. 17

Related papers

Loading 2512.02622v1…