Source-linked AI summary
Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks
Cheng Yang, Haiyuan Wan, Yiran Peng, Xin Cheng, Zhaoyang Yu, Jiayi Zhang, Junchi Yu, Xinlei Yu, Xiawu Zheng, Dongzhan Zhou, Chenglin Wu
TL;DR
Video reasoning lacks a comprehensive benchmark despite video models’ potential for spatial-temporal reasoning. This paper introduces VR-Bench for maze-solving evaluation and finds that fine-tuning, video-based reasoning, and diverse test-time sampling improve performance and reliability.
Problem
Existing video-generation benchmarks largely assess visual quality and temporal coherence while neglecting models’ reasoning capabilities.
Method
The paper introduces VR-Bench, a visual trace reasoning benchmark with 7,920 procedurally generated maze videos spanning varied difficulties, textures, and maze scenarios.
Results
Fine-tuned video models consistently outperform leading vision-language models on spatial reasoning, generalize across diverse scenarios, and improve reliability by roughly 10–20% through diverse test-time sampling.
Takeaways & Limitations
Reasoning via video shows scalable potential for spatial reasoning by combining temporal continuity, spatial perception, and test-time sampling.
Abstract
from arXiv · showhide
Video Models have achieved remarkable success in high-fidelity video generation with coherent motion dynamics. Analogous to the development from text generation to text-based reasoning in language modeling, the development of video models motivates us to ask: Can video models reason via video generation? Compared with the discrete text corpus, video grounds reasoning in explicit spatial layouts and temporal continuity, which serves as an ideal substrate for spatial reasoning. In this work, we explore the reasoning via video paradigm and introduce VR-Bench -- a comprehensive benchmark designed to systematically evaluate video models' reasoning capabilities. Grounded in maze-solving tasks that inherently require spatial planning and multi-step reasoning, VR-Bench contains 7,920 procedurally generated videos across five maze types and diverse visual styles. Our empirical analysis demonstrates that SFT can efficiently elicit the reasoning ability of video model. Video models exhibit stronger spatial perception during reasoning, outperforming leading VLMs and generalizing well across diverse scenarios, tasks, and levels of complexity. We further discover a test-time scaling effect, where diverse sampling during inference improves reasoning reliability by 10--20%. These findings highlight the unique potential and scalability of reasoning via video for spatial reasoning tasks.
1. Introduction
The paper introduces reasoning via video, in which next-frame generation represents spatial and temporal reasoning, and VR-Bench evaluates this ability through maze-solving videos. Experiments report gains from fine-tuning, stronger performance than text-reasoning VLMs on complex tasks, and improved reliability through diverse test-time sampling.
- Reasoning via video: Reasoning via video represents intermediate reasoning through sequential frame generation, preserving spatial continuity and temporal dynamics rather than textual continuation.The paradigm is motivated by video’s ability to encode motion, spatial consistency, and temporal causality across frames.
- Benchmark and dataset: VR-Bench contains 7,920 procedurally generated maze videos spanning five maze types, varied difficulty levels, and diverse visual textures.The benchmark targets spatial planning, dynamic tracking, and multi-step reasoning through trajectory-based tasks.
- Empirical findings: Fine-tuned video models improve across VR-Bench tasks, generalize across difficulty, visual style, and task shifts, and outperform VLMs on high-complexity reasoning tasks.The reported comparison includes stronger stability as task difficulty increases and superior performance across diverse scenarios and tasks.
- Empirical findings: Diverse test-time sampling improves video-model reasoning reliability by an average of 10–20% as models explore multiple reasoning trajectories.The paper describes this as a test-time scaling effect analogous to scaling behavior observed in language models.
- Motivation: VR-Bench addresses limitations of prior video-generation evaluations, which mainly measure visual quality, temporal coherence, or human preference rather than reasoning trajectories.The benchmark is intended to provide a systematic testbed for video reasoning through maze-solving tasks.
1. Exact Match (EM)
Exact Match measures whether a generated trajectory exactly follows the shortest valid path.
- Exact Match (EM): Exact Match counts a trajectory as correct only when every generated step matches the shortest optimal valid path.A single deviation from the optimal solution is treated as incorrect.
2. Success Rate (SR)
Success Rate (SR) measures whether a generated trajectory reaches the designated goal region, while related metrics assess path progress, efficiency, and rule adherence.
- SR is binary: 1 indicates successful goal attainment and 0 indicates failure to reach the target position.
- Precision Rate measures the proportion of consecutively correct steps along the optimal path, providing a softer measure of progress than exact matching.
- Step Deviation measures path-length redundancy relative to the optimal trajectory, with smaller values indicating greater efficiency.
- Rule Compliance: Rule compliance evaluates motion continuity, temporal consistency, trajectory rationality, maze consistency, and subject–maze interaction through five binary VLM judgments.
- Rule Compliance: Maze Fidelity quantifies background stability across frames, with higher values indicating better preservation of the static maze layout.
4. Experiment
The experiment evaluates video and vision–language models on VR-Bench, then fine-tunes an open-source video model to test whether reasoning abilities emerge and generalize across settings.
- The evaluation compares proprietary and open-source video models with representative VLMs on the proposed VR-Bench.
- Wan2.2-TI2V-5B is fine-tuned on VR-Bench to assess the emergence and cross-setting generalization of reasoning capabilities.
- Wan-R1 training uses the first scene from each of five game types, with separate easy-only and mixed-difficulty settings and an 80%/20% train–validation split.
- The implementation uses LoRA fine-tuning on Wan2.2-TI2V-5B for five epochs, with 512×512 resolution and 193-frame videos.
- Difficulty-generalization results compare the baseline and easy-trained model across Easy, Medium, and Hard levels using EM, SR, PR, and SD.
5. Insights and Discussions
The evaluation finds that fine-tuned video models achieve strong maze-solving performance, generalize across difficulty and visual conditions, and benefit from diverse test-time sampling.
- Wan-R1 Outperforms Prior Models on VR-Bench: Wan-R1 achieves top performance across nearly all VR-Bench tasks and metrics, including perfect SR of 100.0 on Trap and 3D mazes.It also improves EM by +65.3 on 3D and reduces SD by 100.1 on Soko relative to Wan2.2-TI2V-5B.
- Reasoning via Video Scales Better: As maze difficulty increases, VLM performance declines more sharply than video-model performance on Irregular Maze and Trapfield.Figure 4 compares PR and SR curves across difficulty levels, with results for other maze types reported in the Appendix.
- Test-Time Scaling for Video Models: Diverse test-time sampling improves Wan-R1 performance by roughly 10–20% as K increases from 1 to 16 across maze difficulties.The gains arise from exploring multiple rollouts generated under perturbed sampling noise, including on Hard mazes.
- Difficulty Generalization: Fine-tuning only on Easy mazes transfers to Medium and Hard levels across all five maze tasks.The results indicate that the model learns a transferable maze-solving procedure rather than merely memorizing small layouts.
- Texture Generalization: Fine-tuning on Raw textures improves performance on unseen styles, including +50.0 EM and +78.5 PR on Skin3 for 3D Maze.The pattern holds across all five task domains.
6. Conclusion
The paper introduces VR-Bench to evaluate whether video models can reason through video generation using maze-solving tasks. Fine-tuned video models show strong spatial reasoning and outperform leading VLMs, while test-time scaling reveals further potential.
- VR-Bench is a comprehensive maze-solving benchmark for assessing the spatial reasoning ability of video models.
- Experiments show that fine-tuned video models exhibit strong spatial reasoning and consistently outperform leading vision-language models.
- The benchmark currently focuses on maze-centric spatial reasoning, with future work planned for broader problem-solving and embodied reasoning scenarios.
7. Experiment Details
The experiments standardize video inputs and outputs across models, define task-specific action spaces and training procedures, and visualize tracked trajectories across maze types.
- 7.1. Implementation Details of Baselines: Video-model inputs and outputs are standardized through model-specific padding, cropping, and temporal alignment procedures.Fixed or adjustable video durations are aligned for fair comparison across models.
- 7.1. Implementation Details of Baselines: VLM baselines predict action sequences that are executed in a simulator to verify trajectory validity against optimal references.
- 7.1. Implementation Details of Baselines: The five maze types use task-specific action spaces, including four-directional moves, irregular graph-node transitions, and six-directional 3D movements.
- 7.2. Training Details: Each maze category has a separately fine-tuned model trained on 80% of first-skin data across Easy, Medium, and Hard levels.
- 7.3. Visualization: Trajectory tracking visualizations overlay ground-truth and generated paths across temporally ordered frames for all five maze types.The green polyline denotes the ground-truth trajectory and the blue polyline denotes the tracked generated trajectory.
8. Evaluation Details
The evaluation aligns generated and ground-truth videos, extracts and normalizes target trajectories, selects matching references, and checks whether VLM-based judgments correspond to human preferences.
- 8.1. Path Matching: Generated and ground-truth videos are resized and temporally resampled to a unified specification before comparison.
- 8.1. Path Matching: An object tracker extracts frame-by-frame target centers, reusing the last valid center when tracking fails at a sampled frame.
- 8.1. Path Matching: Pixel trajectories are normalized and resampled by physical distance to produce speed-invariant, sparsity-controlled trajectory pairs.
- 8.1. Path Matching: Each generated clip is matched to the candidate ground-truth trajectory with the highest trajectory-length-consistency score.The evaluation records the matched identity, summary statistics, and visual overlays.
- 8.2. Human Alignment: A large-scale human preference study evaluates five diagnostic dimensions to verify the VLM-as-Judge assessment.The dimensions include Motion Continuity, Temporal Consistency, Trajectory Rationality, Structural Consistency, and Interactional Rationality.
9. Additional Analysis
Additional analyses show that performance generally declines with maze difficulty, while test-time scaling improves trajectory quality across maze structures and metrics. Video models remain more robust than VLMs on complex, long-horizon spatial reasoning tasks.
- Larger test-time sample counts generally improve Exact Match, Success Rate, Precision Rate, and Step Deviation across maze types and difficulty levels.
- In Maze3D, increasing K improves EM and PR while SR remains near 1.0 and SD decreases, separating goal reaching from fine-grained trajectory precision.
- Sokoban remains challenging despite steady gains with K, while Trapfield shows moderate EM and PR improvements with saturated SR and stable SD.
- Test-time scaling is most useful in structurally complex or interaction-heavy environments, with diminishing returns on easy levels.
- Performance generally declines from Easy to Hard across all maze types and metrics, most sharply for Irregular and Trapfield mazes.
- Video models remain more robust as complexity increases, whereas VLMs degrade sharply on Irregular, Trapfield, and 3D mazes.
10. Failure cases
Failure cases are visualized through initial, intermediate, and trajectory-comparison frames. Consistent color conventions distinguish predicted paths, ground-truth paths, starts, and destinations.
- Each failure example uses an initial frame, an intermediate frame, and a trajectory comparison to expose the model’s reasoning failure.
- Green denotes the ground-truth trajectory, blue the predicted path, yellow the start, and red the final destination.
- These visualization conventions are applied consistently across Figures 10–20.
11. Prompt Template
The prompt templates standardize maze-specific movement rules, trajectory behavior, and visual consistency requirements for video models and VLMs. They cover diverse maze structures while enforcing task-specific spatial and interaction constraints.
- Unified templates cover five maze types and provide separate variants for video-generation constraints and VLM trajectory interpretation.
- Video prompts specify valid movement space, expected trajectories, and frame-level temporal and spatial consistency, while VLM prompts define evaluation context and reasoning instructions.
- Failure cases: Failure examples illustrate unreachable, inefficient, obstacle-crossing, visually inconsistent, and interactionally invalid trajectories under these constraints.
- Regular Maze: Regular Maze requires a red circle to follow the white path to the green square without crossing blue walls under a static top-down camera.
- Irregular Maze: Irregular Maze requires a green circle to follow the white path to the red circle without crossing black regions under a static top-down camera.
- 3D Maze: 3D Maze requires a yellow ball to climb the gray cube pathway and stop on the red goal without touching blue or non-gray regions.
- Trapfield: Trapfield requires a blue circle to reach the green goal along gray paths while avoiding red crossed trap areas.