Source-linked AI summary
V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
Zixu Cheng, Jian Hu, Ziquan Liu, Chenyang Si, Wei Li, Shaogang Gong
TL;DR
Existing Video-LLM benchmarks emphasize object presence without adequately testing relational spatio-temporal reasoning. V-STaR introduces RSTR, a coarse-to-fine CoT dataset, and the LGM metric; experiments on 14 Video-LLMs reveal uneven grounding and reasoning capabilities.
Problem
Existing benchmarks mainly assess object presence, providing limited evidence about whether Video-LLMs understand spatial-temporal relations or rely on co-occurrence biases.
Method
V-STaR decomposes video reasoning into RSTR questions that evaluate what, when, where, and CoT logic using coarse-to-fine reasoning data and LGM scoring.
Results
Across 14 Video-LLMs, many models perform well on what but struggle to ground answers in time and location, with GPT-4o, Gemini-2-Flash, and Qwen2.5-VL ranking highest on the what-where-when chain.
Takeaways & Limitations
Consistent performance across what-when-where reasoning remains important because weaknesses in earlier steps can propagate to later tasks.
Abstract
from arXiv · showhide
Human processes video reasoning in a sequential spatio-temporal reasoning logic, we first identify the relevant frames ("when") and then analyse the spatial relationships ("where") between key objects, and finally leverage these relationships to draw inferences ("what"). However, can Video Large Language Models (Video-LLMs) also "reason through a sequential spatio-temporal logic" in videos? Existing Video-LLM benchmarks primarily focus on assessing object presence, neglecting relational reasoning. Consequently, it is difficult to measure whether a model truly comprehends object interactions (actions/events) in videos or merely relies on pre-trained "memory" of co-occurrences as biases in generating answers. In this work, we introduce a Video Spatio-Temporal Reasoning (V-STaR) benchmark to address these shortcomings. The key idea is to decompose video understanding into a Reverse Spatio-Temporal Reasoning (RSTR) task that simultaneously evaluates what objects are present, when events occur, and where they are located while capturing the underlying Chain-of-thought (CoT) logic. To support this evaluation, we construct a dataset to elicit the spatial-temporal reasoning process of Video-LLMs. It contains coarse-to-fine CoT questions generated by a semi-automated GPT-4-powered pipeline, embedding explicit reasoning chains to mimic human cognition. Experiments from 14 Video-LLMs on our V-STaR reveal significant gaps between current Video-LLMs and the needs for robust and consistent spatio-temporal reasoning.
1. Introduction
Video reasoning requires sequentially integrating what objects are present, when events occur, and where objects are located, but existing benchmarks inadequately assess this relational process. V-STaR addresses the gap with RSTR, coarse-to-fine CoT evaluation, and a GPT-4-powered dataset construction pipeline.
- Humans first identify relevant moments, establish spatial-temporal relationships, and use them to infer answers.
- Video reasoning is harder than text reasoning because it must connect object recognition with spatial and temporal relationships.
- Existing benchmarks often measure object names without testing relational reasoning, allowing answers based on pre-trained co-occurrence biases.
- Many datasets cover what, when, or where separately, but fail to measure their integration through CoT-style spatio-temporal reasoning.
- V-STaR introduces RSTR to jointly evaluate object presence, event timing, object location, and the CoT logic connecting these outputs.
- The benchmark uses a fine-grained dataset with explicit reasoning chains and coarse-to-fine CoT tasks for systematic assessment.
2. Related Works
Related work has advanced Video-LLMs for video question answering and explored temporal or spatial understanding, while the V-STaR pipeline decomposes generated reasoning into grounding challenges.
- Video-LLMs answer diverse video questions, with open-source systems showing competitive results against proprietary models across benchmarks.
- Recent studies have investigated temporal and spatial understanding, including approaches that incorporate temporal or spatial grounding.
- V-STaR’s pipeline has GPT-4 generate a spatio-temporal CoT chain and independent temporal or spatial RSTR grounding questions.
3. V-STAR Benchmark
V-STaR evaluates Video-LLMs through Reverse Spatio-Temporal Reasoning, decomposing video understanding into what, when, where, and their reasoning relationships. Its dataset combines grounded videos, coarse-to-fine CoT question chains, and metrics that assess both individual components and cross-chain consistency.
- Benchmark design: RSTR evaluates what objects are present, when events occur, where objects are located, and how models construct CoT logic.The task is designed to expose reasoning processes that direct-answer benchmarks may conceal behind pre-trained co-occurrence biases.
- Dataset construction: The dataset uses videos with spatial and temporal grounding from existing datasets, supplemented by diverse YouTube videos ranging from 3 minutes to 1 hour.The collection process targets diversity in video duration and content before constructing coarse-to-fine reasoning questions.
- Question design: Each sample contains one spatio-temporal CoT chain and two RSTR chains: “what-when-where” and “what-where-when”.Later questions incorporate earlier ground-truth answers, reducing error propagation and enabling more independent evaluation of temporal and spatial reasoning.
- Metric design: Individual “what”, “when”, and “where” questions use Acc, mean temporal IoU, and mean visual IoU, respectively.Arithmetic Mean and modified logarithmic Geometric Mean aggregate the three dimensions while addressing limitations of isolated metrics and zero-valued geometric means.
- Dataset statistics: The dataset comprises 2094 videos totaling 64.12 hours across 9 domains, with temporal moments and spatial annotations covering grounded events and objects.Temporal moments average 9.06 seconds and 19.3% of video duration; spatial annotations include 342 objects and 16,793 bounding boxes.
- Benchmark comparison: Unlike most prior benchmarks, V-STaR evaluates complete spatio-temporal reasoning chains rather than focusing primarily on “what” questions or covering dimensions independently.The comparison states that TVQA+ covers all three dimensions but does not model their inner spatio-temporal reasoning relationship.
4. Experiments
Experiments evaluate 14 Video-LLMs across V-STaR’s spatio-temporal reasoning tasks and find strong object-answering performance alongside inconsistent grounding and weak joint reasoning. Performance also varies across reasoning chains, video lengths, and domains.
- Setting and Metrics: 14 Video-LLMs were evaluated, including commercial, generic open-source, time-aware, and segmentation models.The evaluation included GPT-4o, Gemini-2-Flash, 12 open-source models, and their official configurations.
- Quantitative Results: GPT-4o, Gemini-2-Flash, and Qwen2.5-VL rank among the strongest models, while Trace, TimeChat, and Oryx-1.5 occupy the bottom ranks on the “what-when-where” chain.Their LGM scores are 39.15/36.14/35.20 for the top three and 13.78/14.47/16.05 for the bottom three.
- Quantitative Results: Open-source models show substantial cross-task imbalance: Video-CCAM-v1.2 leads VQA accuracy but struggles with temporal and spatial grounding, whereas Qwen2.5-VL is more balanced.Video-CCAM-v1.2 reaches 59.35% VQA accuracy but has 1.50% m tIoU and fails spatial understanding; Qwen2.5-VL leads open-source overall performance.
- Quantitative Results: Removing temporal grounding as a prerequisite generally reduces spatial grounding, with Qwen2.5-VL’s m vIoU falling to 2.00%.The same chain also reports a substantial VTimeLLM temporal-grounding decline to 5.96% when prompts contain excessive spatial information.
- Quantitative Results: Longer videos remain difficult: current models show declines across durations, and several models obtain mLGM scores below 15.The results attribute the challenge to maintaining long-range dependencies and reasoning continuity across extended sequences.
- Quantitative Results: Joint spatio-temporal reasoning remains limited, with the best combined accuracies reaching only 4.68% in Chain 1 and 2.24% in Chain 2.Qwen2.5-VL leads Chain 1 and Gemini-2 leads Chain 2 for combined reasoning under the reported IoU thresholds.
- Qualitative Analysis: Qualitative results show that models often analyze frames independently, overlooking dynamic relationships and treating objects as static.The five-model comparison also reveals differing spatial-temporal strengths and degradation patterns across reasoning chains.
5. Conclusion
V-STaR introduces a benchmark for comprehensively assessing Video-LLM spatio-temporal reasoning through coarse-to-fine CoT questions and a new LGM metric. Experiments on 14 Video-LLMs provide insights into current reasoning capabilities and future improvements.
- Conclusion: V-STaR combines coarse-to-fine CoT questions with a new Logarithmic Geometric Mean metric to evaluate Video-LLM spatio-temporal reasoning.The benchmark is designed for comprehensive assessment of spatio-temporal reasoning ability.
- Conclusion: Experiments on 14 Video-LLMs provide insights into their reasoning capabilities and future improvements.