Source-linked AI summary
A Very Big Video Reasoning Suite
Maijunxian Wang, Ruisi Wang, Juyi Lin, Ran Ji, Thaddäus Wiedemer, Qingying Gao, Dezhi Luo, Yaoyao Qian, Lianyu Huang, Zelong Hong, Jiahui Ge, Qianli Ma, Hang He, Yifan Zhou, Lingzi Guo, Lantao Mei, Jiachen Li, Hanwen Xing, Tianqi Zhao, Fengyuan Yu, Weihang Xiao, Yizheng Jiao, Jianheng Hou, Danyang Zhang, Pengcheng Xu, Boyang Zhong, Zehong Zhao, Gaoyun Fang, John Kitaoka, Yile Xu, Hua Xu, Kenton Blacutt, Tin Nguyen, Siyuan Song, Haoran Sun, Shaoyue Wen, Linyang He, Runming Wang, Yanzhi Wang, Mengyue Yang, Ziqiao Ma, Raphaël Millière, Freda Shi, Nuno Vasconcelos, Daniel Khashabi, Alan Yuille, Yilun Du, Ziming Liu, Bo Li, Dahua Lin, Ziwei Liu, Vikash Kumar, Yijiang Li, Lei Yang, Zhongang Cai, Hokin Deng
TL;DR
Video reasoning lacks large-scale training resources despite its potential for spatiotemporal reasoning beyond text. VBVR introduces a 200-task dataset and verifiable benchmark, then reports early generalization to unseen tasks in scaling studies. The suite also exposes persistent gaps in process faithfulness and long-horizon control.
Problem
Video reasoning remains underexplored because large-scale training data, standardized splits, and controlled scaling protocols are limited.
Method
VBVR combines a cognitive-architecture-based dataset with 200 tasks, rule-based benchmark scoring, and systematic scaling evaluations.
Results
The study reports early signs of emergent generalization to unseen reasoning tasks as model scale increases.
Takeaways & Limitations
VBVR provides a large-scale foundation for studying generalizable video reasoning with reproducible and verifiable evaluation.
Takeaways & Limitations
Video reasoning models still exhibit process unfaithfulness and breakdowns in long-horizon interactive control.
Abstract
from arXiv · showhide
Rapid progress in video models has largely focused on visual quality, leaving their reasoning capabilities underexplored. Video reasoning grounds intelligence in spatiotemporally consistent visual environments that go beyond what text can naturally capture, enabling intuitive reasoning over spatiotemporal structure such as continuity, interaction, and causality. However, systematically studying video reasoning and its scaling behavior is hindered by the lack of large-scale training data. To address this gap, we introduce the Very Big Video Reasoning (VBVR) Dataset, an unprecedentedly large-scale resource spanning 200 curated reasoning tasks following a principled taxonomy and over one million video clips, approximately three orders of magnitude larger than existing datasets. We further present VBVR-Bench, a verifiable evaluation framework that moves beyond model-based judging by incorporating rule-based, human-aligned scorers, enabling reproducible and interpretable diagnosis of video reasoning capabilities. Leveraging the VBVR suite, we conduct one of the first large-scale scaling studies of video reasoning and observe early signs of emergent generalization to unseen reasoning tasks. Together, VBVR lays a foundation for the next stage of research in generalizable video reasoning. The data, benchmark toolkit, and models are publicly available at https://video-reason.com/ .
1. Introduction
VBVR addresses the limited study of reasoning in video models by introducing large-scale training data and a verifiable evaluation framework grounded in structured visual tasks.
- Video models have emphasized visual realism, while reasoning over spatial structure, physical dynamics, and long-range causality remains comparatively underexplored.
- VBVR-Dataset organizes video reasoning around five cognitive faculties and covers 200 tasks designed through a standardized task template.
- VBVR-Bench uses verifiable, rule-based scorers to provide systematic, reproducible, and explainable evaluation of video reasoning models.
- VBVR-Dataset is reported to exceed prior video reasoning benchmarks by multiple orders of magnitude across key dimensions and to provide large-scale training data.
2. Related Works
Prior video models largely target creative production, while emerging reasoning studies remain constrained by limited standardized training data and controlled scaling protocols.
- Many video generation models are optimized for creative production rather than explicit relational, causal, or counterfactual reasoning.
- Recent studies examine video as a reasoning substrate through zero-shot behaviors, generation-as-reasoning paradigms, and multi-step diagnostic suites.
- The field lacks standardized large-scale training splits and controlled ablation protocols for reproducible scaling studies focused on reasoning correctness.
3. Dataset
VBVR builds a cognitive-architecture-based dataset from executable task generators, scaling validated video reasoning instances through distributed generation and quality control.
- Cognitive Architecture: VBVR organizes task design around five foundational cognitive faculties, including perception, transformation, and spatiality.
- Cognitive Architecture: Sample tasks instantiate abstract faculties as deterministic, executable, and verifiable video-based reasoning problems with scalable variation.
- Data Statistics: VBVR compares 200 tasks with prior benchmarks, reserving 50 tasks as a hidden set while planning 150 for public release.
- Generation Pipeline: Task development follows design approval, task-specific generator implementation, and large-scale distributed generation with quality control.
- Task Design: Proposals must satisfy information sufficiency, deterministic solvability, video dependency, visual clarity, parametric diversity, and technical feasibility.
- Generator Implementation: Approved designs become parameterized generators that compute solutions and ground-truth outputs through standardized interfaces and validation hooks.
- Generator Outputs: Each generated sample includes initial and final frames, a prompt, and a ground-truth solution trajectory supporting supervision of reasoning paths.
- Large-Scale Generation: The system generates one million training samples across 100 tasks and 7,500 test samples across 150 tasks using disjoint random seed ranges and automated validation.
4. Benchmark
VBVR-Bench evaluates video reasoning across in-domain and novel tasks with deterministic, rule-based scoring designed for reproducible diagnosis. Its evaluations align with human preferences, while model performance and capability relationships reveal substantial differences and remaining challenges.
- Evaluation Design: VBVR-Bench evaluates 100 tasks across 50 in-domain and 50 entirely novel tasks to test both in-domain and out-of-domain generalization.The novel split measures whether models solve reasoning challenges without prior exposure to similar structures.
- Evaluation Design: Each test task uses a dedicated rule-based scorer that measures interpretable aspects such as spatial position, color, object identity, paths, or logical outcomes.Scoring can incorporate geometric, physical, and deductive constraints and aggregate multiple aspects into a weighted score.
- Human Alignment: VBVR-Bench evaluations closely match human perceptions across splits by comparing automatic and human preference win ratios.Human win ratios come from pairwise preferences, whereas benchmark win ratios count per-sample automatic-score wins among models.
- Model Performance: 0.685 is the overall score of VBVR-Wan2.2, an 84.6% relative improvement over its base model after fine-tuning on VBVR-Dataset.VBVR-Wan2.2 attains the best performance across all evaluated categories, especially Spatiality and Perception.
- Limitations and Diagnosis: A considerable gap to human performance remains, with long-horizon temporal reasoning and robust symbolic manipulation identified as persistent challenges.The benchmark also supports analysis of performance variability and rating noise across domains.
5. VBVR-Wan2.2 Analysis
VBVR-Wan2.2 shows data-scaling gains, including transfer to held-out task families, while qualitative results emphasize controllable execution and emerging multi-step behavior. Persistent saturation, process unfaithfulness, and long-horizon identity failures define important limits.
- Scaling behavior: VBVR-Wan2.2 improves in-domain performance from 0.412 to about 0.771 at 400K samples, after which gains plateau and slightly fluctuate.The authors associate this saturation with logical-constraint, temporal-consistency, rendering-noise, and temporal-drift challenges.
- Scaling behavior: ID performance rises from 0.412 to 0.760 and OOD performance from 0.329 to 0.610 as training data increases, leaving a persistent 15% generalization gap.The results indicate transfer beyond memorized patterns, but scaling within fixed task distributions is insufficient for robust systematic generalization.
- Qualitative analysis: On held-out OOD tasks, VBVR-Wan2.2 can match or surpass Sora 2 when success requires verifiable manipulation under stable scenes.Same-task, same-sample comparisons are used so the reported improvements reflect transfer to novel task structures rather than memorization.
- Qualitative analysis: VBVR-Wan2.2 performs minimal-edit deletion, pivot-based rotation, and constrained relocation, whereas Sora 2 can add edits, rotate the wrong elements, or mark without manipulating.These examples support a controllability-first pattern for precise, constraint-following execution.
- Qualitative analysis: On O-49 and O-11, VBVR-Wan2.2 exhibits consistent completion policies and occasional understand→act→adjust behavior on unseen task structures.The authors interpret these behaviors as qualitative evidence of transferred execution primitives, scene-level coherence, and multi-step planning, while noting imperfect final answers.
- Limitations and failure modes: VBVR-Wan2.2 can mimic plausible procedures without following the ground-truth method and can duplicate or flicker over long horizons, leaving process faithfulness and stable identity unresolved.The paper identifies stronger process-level supervision and evaluation as well as long-horizon control as open challenges.
6. Conclusion
VBVR presents a large-scale video-reasoning dataset, a verifiable benchmark, and scaling studies reporting early signs of emergent generalization. The suite supports reproducible assessment of generalizable video reasoning.
- Conclusion: VBVR-Dataset and VBVR-Bench provide large-scale training data and comprehensive, verifiable, reproducible assessment for video reasoning.The paper reports that systematic scaling studies show early signs of emergent generalization.
A. Details of Cognitive Architecture
VBVR organizes video reasoning around five cognitive faculties—perception, transformation, spatiality, abstraction, and knowledge—drawing on philosophical, cognitive-science, and neuroscience perspectives.
- VBVR consolidates its cognitive architecture into five faculties: perception, transformation, spatiality, abstraction, and knowledge.
- Perception extracts form from sensory input, while transformation actively manipulates, recombines, and synthesizes mental representations.
- Spatiality is treated as a foundational capacity for representing places and geometric relationships.
- Abstraction distills embodied experiences into generalizable knowledge by selecting essential features and forming prototypes.
- The architecture frames knowledge as emerging through perception, memory, experience, and abstraction of universal principles.
B. Data Curation
The appendix documents the data-curation pipeline underlying VBVR-Bench, including five construction stages, quality assurance, and infrastructure for large-scale generation.
- The appendix details VBVR-Bench’s five-stage curation process, quality-assurance mechanisms, and large-scale data-generation infrastructure.
B.1. Overview of Data Curation Pipeline
VBVR-Bench converts cognitive task concepts into a standardized evaluation framework through a five-stage pipeline emphasizing generalization, reasoning, quality control, extensibility, and cognitive inspiration.
- Overview: The VBVR-Bench pipeline systematically transforms cognitive task concepts into a standardized evaluation framework.
- Pipeline Flow: Stage 1 selects training and testing tasks from more than 300 candidates using dual review and six quality criteria.
- Design Principles: The benchmark supports both in-domain generalization to unseen samples and cross-task generalization to unseen task types.
- Design Principles: VBVR emphasizes reasoning-first evaluation rather than primarily assessing generation quality.
- Quality and Extensibility: Peer review and specialized implementation review provide industrial-scale quality assurance, while modular infrastructure supports extension with new tasks and models.
B.1.3. Task Selection and Filtering
Task selection filters proposals for sufficient information, deterministic solutions, video-based reasoning, visual clarity, scalability, and technical feasibility, excluding tasks that are ambiguous, excessively difficult, or poorly suited to video.
- Candidate Generation and Review: More than 300 task candidates were generated through iterative brainstorming before structured peer review and refinement.
- Selection Criteria: Reviewers applied six standards covering information sufficiency, deterministic solvability, video dependence, visual clarity, parametric diversity, and technical feasibility.
- Information and Solvability: Tasks must provide all reasoning cues in the first frame and prompt, with a unique or explicitly defined success criterion.
- Video-Based Reasoning: The benchmark includes both dynamic and static reasoning tasks, provided they are suitable for video-generation models.
- Clarity, Scale, and Feasibility: Tasks require distinguishable visual elements, sufficiently large parameter spaces, and implementability in the rendering pipeline.
- Rejection Cases: Excluded examples include logic chains too long to visualize, precise physics problems lacking deterministic solutions, and excessively difficult Sudoku tasks.
B.2. Details of Task Implementation
VBVR implements its 200 reasoning tasks as standardized, parameterized generators with shared interfaces, configurable sampling, automated validation, and fixed visual specifications. Independent repositories and systematic diversity mechanisms support extensibility, reproducibility, and large-scale task construction.
- Standardized Generator Architecture: All task generators inherit a standardized BaseGenerator interface for initialization, sample generation, validation, and saving.The shared template provides consistent interfaces and output formats across tasks.
- Standardized Generator Architecture: Each sample uses a standard directory containing an initial frame, prompt, target state or goal, and ground-truth solution video.
- Parameterization and Output: Generators expose parameter ranges, difficulty levels, and visual settings through configuration files.Examples include grid sizes, object counts, 512×512 resolution, and 24 fps.
- Parameterization and Output: The template supports consistency, quality assurance, and extensibility through shared interfaces, validation checks, and reusable utilities.Validation includes checks such as non-occluded objects and existence of valid solutions.
- Code Management Infrastructure: Independent task repositories provide separate versioning and modular updates while shared core dependencies preserve common utilities.Each task can maintain its own commits and release cycles without affecting other generators.
B.3. Large-Scale Distributed Generation of Training Data
VBVR-DataFactory uses a distributed serverless pipeline to generate and validate million-scale training data from hundreds of task generators. The system combines parallel execution, fault tolerance, monitoring, secure storage, and standardized model-evaluation infrastructure.
- Distributed Generation: VBVR-DataFactory orchestrates parallel generation of one million training samples using more than 300 task generators.It provides cloud workflow, organization, and quality-assurance components for large-scale production.
- Distributed Generation: AWS Lambda distributes generation across up to 990 concurrent workers, while SQS retries, dead-letter queues, and CloudWatch monitoring provide fault tolerance.
- Generation Workflow: A one-million-sample run typically completes in approximately 2–4 hours at a total cost of roughly $800–1200.
- Generation Workflow: The pipeline uses validated queue messages to configure generators, sample counts, seeds, numbering, and output formats before execution and validation.
- Quality Assurance: Quality assurance validates solvability, visual clarity, file integrity, and sample diversity throughout generation, with monitoring for failures and anomalies.
- Storage and Security: Training data are stored in encrypted private S3 buckets with access controls and audit logging, and the samples are fully synthetic with no personally identifiable information.
- Benchmark Infrastructure: VBVR-Bench evaluates video-generation systems through standardized inference and rule-based scoring across a 100-task benchmark.The framework provides a unified abstraction spanning 29 video-generation systems and evaluates eight state-of-the-art image-to-video models.
- Benchmark Infrastructure: Generated videos are matched to ground-truth resolutions, with padding and cropping procedures accommodating fixed-resolution closed-source models.
C.2. Details of Human Preference Analysis
The human preference analysis combines relative pairwise judgments with absolute per-video ratings, then compares human-derived model rankings with automatic benchmark scores. Additional residualized correlations and domain-wise distributions characterize capability dependencies and task variability.
- Human Scoring Protocol: Each video is independently annotated five times by trained annotators, with samples randomly shuffled before assignment.
- Human Scoring Protocol: Relative evaluation compares two videos from the same prompt, while absolute evaluation rates each video for task completion, reasoning logic, and visual quality.Absolute ratings use a 5-point Likert scale, with higher scores indicating better performance.
- Score Aggregation: Relative scores produce model win ratios, while absolute dimension scores are averaged and used to cross-verify or revise contradictory pairwise judgments.
- Score Aggregation: The analysis compares human-derived win ratios with automatic VBVR-Bench win ratios and measures their correlation across models.
- Residualized Capability Correlation: Category scores average per-sample Overall ratings within each model and cognitive category, while the general factor averages scores across all evaluated samples.
- Residualized Capability Correlation: Residualization regresses each category score on the model-level general factor and retains residuals representing strength beyond overall model quality.
- Residualized Capability Correlation: Pearson correlations between category residuals form a 5 × 5 capability-dependency matrix visualized as a heatmap.Spearman ρ is also reported as an auxiliary robustness analysis.
- Domain-wise Analysis: Domain-wise boxplots show task-level score distributions, with each point representing a task mean within a domain.