Source-linked AI summary
V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models
Yang Luo, Xuanlei Zhao, Baijiong Lin, Lingting Zhu, Liyao Tang, Yuqi Liu, Ying-Cong Chen, Shengju Qian, Xin Wang, Yang You
TL;DR
Generative video models increasingly exhibit reasoning-like behavior, but reliable quantitative evaluation remains underdeveloped. V-ReasonBench introduces a unified benchmark using last-frame scoring across four reasoning dimensions and finds distinct strengths and systematic failure modes across six state-of-the-art models.
Problem
Quantitative evaluation of generative video reasoning remains underexplored and poorly standardized, motivating more reliable assessment.
Method
V-ReasonBench uses scalable last-frame scoring across four reasoning dimensions to evaluate video reasoning with reproducible, answer-verifiable tasks.
Results
Six evaluated video models show distinct dimension-wise strengths and systematic failure modes rather than uniformly high reasoning performance.
Takeaways & Limitations
V-ReasonBench provides a reproducible foundation for benchmarking video reasoning beyond visual fidelity and for advancing more human-aligned models.
Takeaways & Limitations
VLM-based automatic evaluation remains unreliable for grid-structured and spatially dense scenes requiring fine-grained visual understanding.
Abstract
from arXiv · showhide
Recent progress in generative video models, such as Veo-3, has shown surprising zero-shot reasoning abilities, creating a growing need for systematic and reliable evaluation. We introduce V-ReasonBench, a benchmark designed to assess video reasoning across four key dimensions: structured problem-solving, spatial cognition, pattern-based inference, and physical dynamics. The benchmark is built from both synthetic and real-world image sequences and provides a diverse set of answer-verifiable tasks that are reproducible, scalable, and unambiguous. Evaluations of six state-of-the-art video models reveal clear dimension-wise differences, with strong variation in structured, spatial, pattern-based, and physical reasoning. We further compare video models with strong image models, analyze common hallucination behaviors, and study how video duration affects Chain-of-Frames reasoning. Overall, V-ReasonBench offers a unified and reproducible framework for measuring video reasoning and aims to support the development of models with more reliable, human-aligned reasoning skills.
1. Introduction
Generative video models show emerging zero-shot reasoning abilities, but these capabilities lack standardized quantitative evaluation. V-ReasonBench addresses this gap with last-frame assessment across four reasoning dimensions and reveals distinct strengths and limitations among six models.
- V-ReasonBench addresses the underexplored and poorly standardized problem of quantitatively evaluating reasoning in generative video models.
- The Chain-of-Frame paradigm treats generated videos as sequences of reasoning steps and evaluates the final frame as the model’s answer.This avoids annotating every intermediate step, reducing annotation effort, computational cost, and noisy supervision.
- The benchmark combines mask-based, grid-based, and lightweight VLM-based evaluation to match different task structures and improve scoring reliability.
- V-ReasonBench provides deterministic last-frame scoring across four complementary reasoning classes using synthetic and curated scenarios with controlled difficulty and perceptual variation.
- Across six video models, evaluations reveal clear dimension-wise strengths and limitations, while also examining duration effects, model contrasts, and recurring video hallucinations.
2. Related Works
Recent video-generation systems achieve high visual quality and text alignment, but their training data, model designs, and evaluation procedures are largely undisclosed.
- Commercial video-generation systems produce impressive visual quality and text alignment, while their training data, model designs, and evaluation procedures remain undisclosed.
3. V-ReasonBench Pipeline
V-ReasonBench is a unified, scalable pipeline for evaluating video reasoning through answer-verifiable image-pair tasks and deterministic last-frame scoring. It spans four reasoning dimensions and uses task-specific automated evaluators to produce consistent pass@k decisions.
- 3.1. Design Principles: All tasks determine the answer from the generated video’s last frame, enabling unambiguous and scalable evaluation with pass@k as a unified metric.
- 3.2. Reasoning Tasks and Formulations: The benchmark covers four reasoning dimensions through tasks targeting structured problem-solving, spatial cognition, pattern-based inference, and physical dynamics.Examples include arithmetic, code execution, Sudoku, Tic-Tac-Toe, shape fitting, symmetry, block sliding, communicating vessels, and temperature-induced deformation.
- 3.3. Dataset Construction: The dataset uses initial–final image pairs, with approximately 90% of instances programmatically synthesized to control state transitions, reasoning coverage, and perceptual variation.
- 3.3. Dataset Construction: V-ReasonBench contains 326 reasoning instances, 652 paired images, and 9,780 generated videos, with five videos produced for each instance-model pair.
- 3.4. Limitations of VLM-Based Evaluation: VLM-only evaluation can fail on grid-structured scenes because small cells, thin boundaries, and subtle spatial relationships are difficult to recognize reliably.
- 3.4. Limitations of VLM-Based Evaluation: Mask-based, grid-based, and lightweight VLM-based evaluators produce numerical final-frame scores that become binary pass decisions through task-specific thresholds.
4. Experiments
Experiments evaluate six video models across V-ReasonBench’s reasoning dimensions and validate the benchmark’s automated scoring against human judgments. Results show dimension-specific strengths, with strong agreement between automatic and human pass/fail decisions.
- Experimental Setup: The evaluation covers six commercial video models using five videos per prompt, default parameters, and approximately five-second outputs.The tested models are Sora-2, Veo-3.1, Hailuo-02, Vidu-Q2, KlingAI-2.5-Turbo-Pro, and Seedance-1.0-Lite.
- Model Performance: Sora-2 leads Structured Problem-Solving (72.00), Spatial Cognition (36.76), and Pattern-based Inference (40.00), while Hailuo-02 and Vidu-Q2 lead Physical Dynamics (36.67 each).The results indicate that models exhibit distinct strengths rather than uniformly high performance across reasoning dimensions.
- Human Preference Alignment: Automatic pass/fail decisions agree with human judgments at 97.09% average accuracy across models and reasoning categories.Human alignment was measured on 120 sampled videos per reasoning task, assessed independently by six graduate-level examiners.
- Human Preference Alignment: Disagreements mainly occur in visually ambiguous cases, including nearly symmetric configurations and partially occluded physics scenarios.Human evaluators show slightly higher tolerance for minor perceptual deviations in these cases.
- Human Preference Alignment: The benchmark’s scoring protocol is presented as a scalable, reproducible assessment framework aligned with human reasoning preferences.This conclusion follows the reported human–automatic agreement study.
5. Discussions
The discussion identifies systematic failures in preserving structure and causal consistency, while showing that temporal modeling can benefit process-heavy reasoning without guaranteeing reliable intermediate steps.
- 5.1. Reasoning Patterns in Video Generation: Some models prioritize visual enhancement over structural accuracy, adding textures or objects in sparse scenes and thereby reducing reasoning performance.The discussion attributes this tendency to creative biases and training or decoding choices that reward visual richness and motion.
- 5.2. Influence of Duration on Video Reasoning: Longer Chain-of-Frame durations do not consistently improve reasoning and can introduce irrelevant content or hallucinated objects in final frames.The authors relate this to attention drift and temporal mis-binding as sequence length expands.
- 5.3. Video Models vs. Image Models: Video models outperform image-only reasoning on dynamics and multi-step causal tasks by modeling transitions, while image models remain reliable on static symbolic and text-based tasks.CoF intermediate frames act as visual checkpoints for procedural reasoning, although temporal transitions may remain physically inconsistent.
- 5.4. Hallucination in Video Reasoning: Correct final answers can conceal physically impossible intermediate trajectories, such as wall-crossing maze paths or Newton’s-cradle sequences violating momentum conservation.These temporal hallucinations break causal consistency even when endpoint evaluation is correct.
- 5.4. Hallucination in Video Reasoning: Because intermediate-step judging is difficult and unreliable, the benchmark favors tasks where process errors necessarily produce incorrect terminal states.This design makes endpoint scoring more dependable for scalable evaluation.
6. Conclusion
V-ReasonBench provides a unified, reproducible framework for evaluating video reasoning beyond visual fidelity, exposing distinct model strengths and systematic failure modes.
- 6. Conclusion: V-ReasonBench evaluates Chain-of-Frame video reasoning with scalable last-frame scoring across four reasoning dimensions.Experiments on six state-of-the-art models reveal a gap between generation quality and true reasoning ability.
- 6. Conclusion: The benchmark offers a reproducible foundation for advancing more human-aligned video reasoning models.
7. Additional Details for Each Task
The benchmark’s task suite spans structured problem-solving, spatial cognition, and pattern-based inference through procedurally generated or curated visual reasoning problems with defined solution procedures.
- Structured Problem-Solving: Structured problem-solving tests step-by-step execution, numerical manipulation, and strategic thinking through arithmetic, code execution, Sudoku, and Tic-Tac-Toe.Tasks use visual inputs and require either computation, constraint satisfaction, program tracing, or optimal move selection.
- Task Illustrations: The visual task illustrations show initial states, intermediate reasoning or construction stages, and completed outputs for representative benchmark problems.The figures cover arithmetic, code execution, Sudoku, Tic-Tac-Toe, shape fitting, symmetry, color connection, and sequence completion.
- Spatial Cognition: Spatial cognition evaluates geometric arrangement, symmetry completion, and non-intersecting path construction through Shape Fitting, Visual Symmetry, and Color Connection.The tasks vary shape complexity, symmetry type, grid size, and number of color pairs.
- Pattern-based Inference: Pattern-based inference measures inductive and abstract reasoning through Sequence Completion, Analogy Solving, and Rule Following.These tasks require models to infer deterministic transformations or relationships from examples and apply them to new inputs.
8. Complete Task-Level Results
Complete task-level results are reported as pass@5 scores for all 13 reasoning tasks, complementing the benchmark’s dimension-level comparisons with granular model-specific strengths and weaknesses.
- 8. Complete Task-Level Results: Tab. 3 and Fig. 22 provide pass@5 performance for every evaluated model across all 13 VReasonBench tasks.The task-level breakdown complements dimension-level aggregation in the main paper.
9. Limitation of VLM-Based Evaluation
VLMs frequently misread fine-grained details in grid-structured and spatially dense scenes, limiting their reliability as evaluators for these tasks.
- VLMs frequently misread cell boundaries, object adjacency, and subtle geometric relations in grid-structured and spatially dense scenes.
10. More Reasoning Patterns Demonstrations
Additional demonstrations show that video models often preserve a rough board concept while sacrificing exact cell-level structure, especially when simple inputs are transformed into visually rich scenes.
- Seedance-1.0-Lite replaces tic-tac-toe cells with new symbols or removes board parts while retaining a rough 3×3 board-game structure.
- These visually rich transformations violate strict X/O matching requirements despite being harmless from a storytelling perspective.
- Seedance-1.0-Lite and Vidu-Q2 similarly lose exact cell-level structure when converting plain boards into cartoon, chessboard, or textured scenes.
- The pattern is linked to training data emphasizing rich open-domain visuals and containing few clean, diagram-like examples.
- Figure 22 summarizes six video models across 13 reasoning tasks using scores rescaled within each reasoning dimension.
11. More Hallucination in Video Reasoning
Failure cases span visual symmetry, rule following, block sliding, and tic-tac-toe, showing that video models can introduce distracting content or alter required structures.
- Visual symmetry, rule following, and block sliding examples expose failures on transformations, localized grid updates, and physical positioning tasks.
- Seedance-1.0-Lite adds characters, logos, chess pieces, and tokens that alter tic-tac-toe board configurations under strict cell-wise evaluation.
- Other generations preserve only rough grids while visual effects overwrite cells or obscure and replace the board with abstract textures and digits.