PaperlayerView exact revision on arXiv

Source-linked AI summary

VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis

Jinho Park, Youbin Kim, Hogun Park, Eunbyung Park

arXiv:2605.22570v1cs.CVcs.AI

TL;DR

Existing benchmarks provide limited fine-grained evidence about MLLMs’ spatio-temporal reasoning because they rely mainly on static images or passively curated videos. VGenST-Bench actively synthesizes controlled photorealistic videos and QA data through a multi-agent pipeline, taxonomy, and hierarchy. Results show sharply weaker performance at higher reasoning levels and a substantial gap between models and humans, while transfer from synthetic to natural video remains unguaranteed.

  • Problem

    Static and passively curated benchmarks inadequately isolate fine-grained spatio-temporal reasoning and remain vulnerable to contamination, shortcuts, and narrow coverage.

  • Method

    VGenST-Bench uses actively synthesized videos, a multi-agent generation pipeline with human quality control, a 3×2×2 taxonomy, and a hierarchical task suite.

  • Results

    The strongest model achieves 85.9% versus 99.0% for humans, while accuracy collapses sharply from L1 to L3 and reformulated questions expose hidden shortcuts.

  • Takeaways & Limitations

    Generation-driven benchmark construction provides a viable foundation for diagnosing spatio-temporal reasoning limitations in MLLMs.

  • Takeaways & Limitations

    Performance on this synthetic distribution is not claimed to predict performance on naturally captured video, and generator priors may introduce bias.

Abstract

from arXiv · show

Spatio-temporal reasoning is a core capability for Multimodal Large Language Models (MLLMs) operating in the real world. As such, evaluating it precisely has become an essential challenge. However, existing spatio-temporal reasoning benchmark datasets primarily rely on static image sets or passively curated video data, which limits the evaluation of fine-grained reasoning capabilities. In this paper, we introduce VGenST-Bench, a video benchmark that employs generative models to actively synthesize highly controlled and diverse evaluation scenarios. To construct VGenST-Bench, we propose a multi-agent pipeline incorporating a human quality control stage, ensuring the quality of all generated videos and QA pairs. We establish a comprehensive 3x2x2 video taxonomy, encompassing Spatial Scale, Perspective, and Scene Dynamics to span diverse scenarios. Furthermore, we design a hierarchical task suite that decouples low-level visual perception from high-level spatio-temporal reasoning. By shifting the paradigm from passive curation to active synthesis, VGenST-Bench enables fine-grained diagnosis of spatio-temporal understanding in MLLMs.

1 Introduction

VGenST-Bench addresses limitations of static and passively curated benchmarks by actively synthesizing controlled videos for fine-grained spatio-temporal evaluation. It combines a multi-agent construction pipeline, a 3×2×2 taxonomy, and hierarchical questions, with experiments showing sharp reasoning deficits in current MLLMs.

  • Current MLLMs still struggle to infer object positions, orientations, and attributes across time and changing viewpoints.
  • Passively curated video benchmarks remain vulnerable to data contamination, shortcut exploitation, limited scalability, and narrow scenario coverage.
  • VGenST-Bench actively synthesizes photorealistic evaluation videos and questions through a multi-agent pipeline followed by human quality control.
  • The benchmark spans 12 categories from a 3×2×2 taxonomy of spatial scale, perspective, and scene dynamics, with one dedicated reasoning task per category.
  • Its three-level hierarchy separates visual perception, scene understanding, and spatio-temporal reasoning for fine-grained diagnosis.
  • Experiments show performance degrades sharply from L1 to L3, and even the strongest model remains substantially below human performance.

2 Related Work

Prior benchmarks progress from static images to video and 3D datasets, but passive sourcing limits controllability and diversity while exposing evaluations to contamination. Synthetic data offers ground-truth control but may sacrifice visual realism.

  • Static image benchmarks cannot capture state transitions across the temporal dimension.
  • Video and repurposed 3D benchmarks provide visual richness but are passively curated, limiting evaluation diversity and controllability while increasing contamination risk.
  • Synthetic evaluation data affords precise ground-truth control but suffers from a visual realism gap for models trained on photorealistic content.

3 VGenST-Bench

VGenST-Bench organizes controlled spatio-temporal scenarios across a 3×2×2 taxonomy and a three-level question hierarchy. Its construction pipeline converts themes into validated scene graphs, temporal scenarios, generated videos, and quality-checked QA variants.

  • Video Taxonomy and Task Design: The 3×2×2 taxonomy crosses figural, vista, and environmental scales with egocentric or exocentric perspectives and static or dynamic scenes.
  • Video Taxonomy and Task Design: The taxonomy defines distinct task cells, while Table 2 organizes the benchmark’s twelve tasks by spatial scale, perspective, and scene dynamics.
  • Level Design: The question hierarchy progresses from L1 visual perception to L2 scene understanding and L3 spatio-temporal reasoning.
  • Task–QA Applicability: A task–QA applicability matrix excludes undefined question types and preserves balanced coverage across the twelve tasks.
  • Question Reformulation for Robust Evaluation: Each base MCQ is reformulated into three variants to reduce option-level shortcuts in evaluation.
  • Dataset Construction: The pipeline sequentially produces scene graphs, scenarios, videos, and QA pairs from themes, with structured representations fixing visual context and spatio-temporal ground truth.
  • Human Quality Control: Two validators retain generated videos and base QA pairs only when they pass visual-fidelity, scenario-adherence, and answer-validity checks.

4 Experiments

Experiments show that VGenST-Bench separates strong low-level perception from weaker higher-order spatio-temporal reasoning, while reformulated evaluations expose shortcuts hidden by standard multiple-choice accuracy.

  • Main Results: 99.0% human accuracy exceeds Gemini 3 Flash’s 85.9% average by more than 13pp across the benchmark.Most open-source models score below 70%, although Kimi-K2.6 and Gemma-4-31B-it reach approximately 71.0% and 72.5%.
  • Hierarchical Analysis: 90.2% on L1 falls to 65.3% on L2 and 36.4% on L3 for GPT-5.4 mini, while humans decline only from 99.4% to 97.9%.The consistent L1-to-L3 degradation indicates bottlenecks in integrating spatial and temporal cues beyond single-frame perception.
  • Robustness Analysis: Vanilla accuracy systematically exceeds circular accuracy, indicating that models exploit position bias and answer-choice priors.Circular evaluation tests correctness across all cyclic answer-choice permutations.
  • Robustness Analysis: V2 None-of-these reformulation causes dramatic accuracy drops across all models, whereas V1 maintains base accuracy.The result indicates that models often rank answer options instead of verifying answers against the video.
  • Robustness Analysis: Open-ended evaluation exposes the L1-to-L3 performance hierarchy more sharply than closed-form multiple-choice evaluation.This suggests that standard multiple-choice accuracy can overestimate spatio-temporal reasoning capability.
  • Failure Analysis: A representative Direction Estimation failure correctly tracks visual events but reverses the final egocentric transformation.The model identifies the initial orientation, camera turn, and final view before selecting the wrong spatial direction.

5 Conclusion

The conclusion presents VGenST-Bench as a generation-driven benchmark for diagnosing MLLM spatio-temporal reasoning. It argues that controllable video synthesis can support benchmark construction beyond a fixed dataset.

  • Conclusion: VGenST-Bench uses video generative models to evaluate spatio-temporal reasoning in MLLMs.The benchmark shifts evaluation toward generated videos rather than relying only on naturally collected footage.
  • Conclusion: The strongest model trails the 99.0% human ceiling by over 13pp, while L1-to-L3 collapse and reformulations expose hidden reasoning shortcuts.None-of-these and open-ended variants reveal weaknesses that closed-form MCQ accuracy can conceal.
  • Conclusion: The released dataset, generation pipeline, and evaluation suite are intended to support future research on spatio-temporal reasoning.The authors frame the work as more than a single benchmark.

A.1 Benchmark Design Rationale

The benchmark design rationale favors controllable synthetic video because natural footage limits coverage and precise scenario specification. The authors also define scope boundaries and future expansion paths.

  • Why generated videos: Naturally collected videos constrain coverage because rare combinations of spatial scale, viewpoint, and scene dynamics are scarce in public corpora.Balancing such combinations through curation is described as difficult and labor-intensive.
  • Why generated videos: Improved generator realism, controllability, and temporal horizons could make VGenST-Bench a scalable framework rather than a fixed video collection.The authors expect the benchmark’s usable task space to expand as generation improves.
  • Task scope: The benchmark restricts base MCQs to qualitative tasks and excludes noisy quantitative estimates such as absolute distance and metric size.The rationale is that monocular-video estimates can admit a band of acceptable human answers rather than one exact target.
  • Limitations: Performance on VGenST-Bench measures reasoning under a synthetic distribution, so transfer to naturally captured video remains an empirical question.The authors also note that generator-specific visual, cultural, and physical priors may propagate into generated videos.
  • Broader impact: Synthetic-benchmark adoption could tune models to synthetic distributions while causing drift from real-world reasoning.The authors present this as a motivation for future work rather than a reason to reject synthetic evaluation.
  • Future directions: The pipeline is intended to scale through longer, higher-resolution, and more controllable generation, while expanding taxonomy cells and task counts.Future directions include longer temporal reasoning, finer physical interactions, and more interacting agents.
  • Domain transfer: The pipeline could transfer to autonomous driving, robotics, and surgical video where structured specifications exist but real-world capture is constrained.Scene-graph-driven generation is proposed for rare or safety-critical scenarios that are difficult to collect at scale.

B.1 Image Benchmark Datasets

Image benchmarks established visual question answering and perception evaluation, but their single-image format excludes temporal change, motion-conditioned spatial reasoning, and viewpoint dynamics.

  • Image benchmark foundations: VQA unified free-form visual question answering over natural images, while GQA extended the paradigm using scene-graph-derived questions.These datasets formed early foundations for evaluating visual reasoning in vision-language models.
  • Image benchmark landscape: MME, SEED-Bench, MMStar, What’s Up, and BLINK broaden static-image evaluation across perception, cognition, grounding, spatial relations, and multi-view tasks.Their coverage includes depth, correspondence, and relational understanding but remains image-based.
  • Scope boundary: Single-image benchmarks cannot evaluate temporal change, motion-conditioned spatial reasoning, or viewpoint dynamics.These omissions motivate the transition to video benchmarks for spatio-temporal reasoning.

B.2 Video Benchmark Datasets

Video benchmarks extend evaluation beyond static images by testing temporal, causal, perceptual, and spatio-temporal reasoning across increasingly comprehensive task suites.

  • Early video benchmarks: Video question-answering benchmarks pair temporal reasoning with activity videos, television clips, or short causal sequences.ActivityNet-QA and TVQA+ use activity and television videos, while NExT-QA emphasizes causal and temporal reasoning.
  • Comprehensive video evaluation: Comprehensive suites broaden coverage through varied video durations, categories, and unified multiple-choice tasks.Video-MME spans short, medium, and long videos across six categories; MVBench covers twenty temporal-understanding tasks; TempCompass targets fine-grained temporal sensitivity.
  • Spatio-temporal benchmarks: Recent benchmarks explicitly target spatio-temporal reasoning through egocentric spatial intelligence, incremental scene exploration, and quantitative motion or pose measurement.VSI-Bench, OST-Bench, and STI-Bench emphasize complementary settings and measurements.

B.3 Synthetic Benchmark Datasets

Synthetic benchmarks provide controlled visual and temporal environments, while VGenST-Bench uses video generation specifically to synthesize spatial-reasoning scenarios with scene-graph-derived ground truth.

  • Synthetic visual reasoning: Procedurally generated datasets such as CLEVR, CLEVRER, and CATER extend controlled compositional reasoning from static scenes into temporal object motion and event structure.These datasets use synthetic or rendered environments to control visual configurations and temporal events.
  • Synthetic video benchmarks: VideoCogQA, Video-MSR, and VideoNIAH use synthetic generation for abstract cognitive tasks, multi-step spatial reasoning, or long-context retrieval.Their construction media include programmatic game engines and synthetic insertion into videos.
  • VGenST-Bench: VGenST-Bench uses video generation as a synthesis medium for spatial reasoning, deriving ground truth directly from the scene graph that drives generation.This design supports controlled coverage of spatial configurations that are difficult to sample from naturally collected videos.

C.1 Dataset Statistics

VGenST-Bench contains 1,200 videos and 33K QA pairs organized across a 3 × 2 × 2 taxonomy, a three-level reasoning hierarchy, and controlled task-specific scenario designs.

  • Dataset scale: 1,200 videos and 33K QA pairs span the 3 × 2 × 2 taxonomy and the L1/L2/L3 QA hierarchy.Each of the 12 tasks contributes 100 generated videos retained after two-stage human quality control.
  • Generative sources: Videos use 10 distinct video-generation models across image-to-video, text-to-video, and start-end-to-video paradigms, with 1,100 images drawn from four text-to-image models.Authors initially selected outputs for prompt fidelity and visual quality before human quality control.
  • Video taxonomy: The taxonomy varies spatial scale, perspective, and scene dynamics, yielding 12 task cells covering figural, vista, and environmental reasoning.The three scales range from local object configurations to extended spaces requiring integration across viewpoints or trajectories.
  • Task design: Task-specific descriptions define local configurations, room-scale layouts, and extended environments requiring viewpoint, depth, landmark, or trajectory integration.Examples include figural attribute mapping, vista direction estimation, and environmental directional-signage grounding.
  • QA hierarchy: The QA hierarchy progresses from L1 visual perception to L2 scene understanding and L3 spatio-temporal reasoning.L1 isolates recognition and attributes from individual frames, while higher levels require integration across frames and temporal structure.

C.5 Task–QA Applicability Matrix

The benchmark restricts QA generation to task–QA combinations that are well-defined, then reformulates each base MCQ to reduce option-level shortcuts within a multi-agent construction pipeline.

  • Base MCQ: Each base MCQ uses one correct answer and semantically plausible distractors before reformulation variants are applied.The base format establishes the comparison set for the subsequent robustness tests.
  • Reformulation variants: The None-of-these distractor variant tests commitment when the correct answer remains available alongside an explicit reject option.The added option is incorrect, so the original correct answer remains among the choices.
  • Reformulation variants: The None-of-these answer variant removes the correct option and tests whether models reject all remaining distractors.Here, “None of these” becomes the correct answer.
  • Reformulation variants: The open-ended variant removes all options and scores free-form responses against ground truth using an LLM-as-judge protocol.This variant is intended to eliminate option-level priors and assess answer production rather than selection.
  • Construction pipeline: Four agent modules transform themes into scene graphs, temporal scenarios, videos, and QA pairs, with generator–validator loops and human verification.Scenario validation checks fidelity, first-frame compliance, temporal flow, ground-truth determinacy, and task-rule compliance.

D.3 Human Quality Control

VGenST-Bench uses sequential human review to verify generated videos and QA pairs, retaining only items independently judged valid by both annotators. The process combines calibrated reviewers, intersection-based decisions, regeneration of rejected videos, and strong agreement rates.

  • Two-stage pipeline: Two-stage quality control reviews generated videos first and their associated base MCQs second.Video QC checks physical plausibility, generation artifacts, and prompt–video drift; QA QC evaluates question quality after video approval.
  • Validator pool: Twelve calibrated graduate-student validators review tasks in overlapping pairs to distribute workload while preserving cross-task consistency.Each task is assigned to adjacent annotators under a circular pairing scheme, so neighboring task pairs share one reviewer.
  • Decision rule: An item is retained only when both annotators independently mark it valid; authors jointly resolve disagreements.This intersection rule applies to both video and QA review, while split cases receive author re-inspection.
  • Video QC results: 90.4% of 1,200 generated videos pass both reviews, with task-level retention ranging from 78% to 97%.The lowest retention is LS_E_EXO_STA at 78/100; rejected videos are regenerated against the same scene graph to preserve a balanced final benchmark.
  • QA QC results: 93 of 9,800 reviewed MCQs are rejected, leaving 9,707 base MCQs in the benchmark.The high pass rate is attributed to QA generation conditioned on scene graphs and scenarios, with few-shot guidance for templates, distractors, and answer derivation.
  • Inter-annotator agreement: Video QC reaches 92.9% raw agreement with κ = 0.58, while QA QC reaches 99.6% raw agreement with κ = 0.79.The stronger QA agreement is described as consistent with constrained few-shot generation producing well-formed MCQs.

E Video Quality Human Study

The blind human study evaluates VGenST-Bench against real-source video benchmarks for perceptual quality and authenticity. Results show a scene-comprehensible but visibly synthetic benchmark, with realism traded for systematic control.

  • Study design: The study compares VGenST-Bench with VSI-Bench, EgoExoBench, and Perception Test across photorealism, temporal coherence, and scene comprehensibility.Evaluators rank four clips within each of 50 comparison sets, using lower mean rank as better.
  • Evaluation protocol: Three non-expert evaluators independently rank clips and judge each as real or AI-generated.The real-vs-fake task also measures false-positive judgments on real benchmark clips and Fleiss’ κ for consistency.
  • Ordinal ranking results: VGenST-Bench ranks lowest on photorealism at 3.64, but its gap to the worst real-source baseline narrows to 0.57 ranks on scene comprehensibility.The corresponding gaps are 1.25 ranks for photorealism and 0.80 ranks for temporal coherence.
  • Authenticity judgments: VGenST-Bench clips are judged fake in 63.3% of cases, versus 12.7–26.0% false-positive rates for the real-source baselines.Evaluators therefore reliably identify the generated clips, although real benchmark videos are also mistaken for AI output.
  • Interpretation: The benchmark is intended to support spatio-temporal reasoning through scene comprehensibility rather than to deceive viewers with photorealism.The authors describe visual realism as the cost of systematic control over taxonomy, scene graphs, and scenarios.
  • Participants: The annotator pool includes 25 volunteers partitioned into disjoint QC, video-study, and human-baseline groups.The groups contain 12 QC annotators, 3 video-quality evaluators, and 10 human-baseline participants.
Loading 2605.22570v1…