Source-linked AI summary
VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification
Jiahao Meng, Tan Yue, Qi Xu, Haochen Wang, Zhongwei Ren, Weisong Liu, Yuhao Wang, Renrui Zhang, Yunhai Tong, Haodong Duan
TL;DR
VideoZeroBench targets the gap between high video-QA scores and verified fine-grained understanding by evaluating answers together with their spatio-temporal evidence. It builds a manually annotated long-video benchmark and applies a five-level protocol that progressively tightens grounding requirements. Models reach at most 17.0% in standard QA and 1.0% when correct answers must also have accurate spatio-temporal grounding.
Problem
Existing video benchmarks can report high answer accuracy without resolving fine-grained capabilities or verifying the spatio-temporal evidence supporting predictions.
Method
VideoZeroBench provides 500 challenging questions over long videos, with annotated temporal intervals and spatial boxes evaluated through five progressively stricter levels.
Results
17.0% is Gemini-3-Pro’s highest Level-3 accuracy, while the best Level-5 accuracy is only 1.0%, with many models scoring zero.
Takeaways & Limitations
Answer correctness does not reliably imply genuine understanding because models frequently fail to ground predictions in authentic spatio-temporal evidence.
Abstract
from arXiv · showhide
Recent video multimodal large language models achieve impressive results across various benchmarks. However, current evaluations suffer from two critical limitations: (1) inflated scores can mask deficiencies in fine-grained visual understanding and reasoning, and (2) answer correctness is often measured without verifying whether models identify the precise spatio-temporal evidence supporting their predictions. To address this, we present VideoZeroBench, a hierarchical benchmark designed for challenging long-video question answering that rigorously verifies spatio-temporal evidence. It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence. To disentangle answering generation, temporal grounding, and spatial grounding, we introduce a five-level evaluation protocol that progressively tightens evidence requirements. Experiments show that even Gemini-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting (Level-3). When grounding constraints are imposed, performance drops sharply: No model exceeds 1% accuracy when both correct answering and accurate spatio-temporal localization are required (Level-5), with most failing to achieve any correct grounded predictions. These results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning, revealing that grounded video understanding remains a bottleneck for long-video QA. We further analyze performance across minimal evidence spans, atomic abilities, and inference paradigms, providing insights for future research in grounded video reasoning. The benchmark and code will be made publicly available.
1 Introduction
VideoZeroBench addresses inflated video-QA scores by testing fine-grained capabilities and whether answers are supported by precise spatio-temporal evidence. Its hierarchical evaluation reveals that current models rarely combine correct answers with reliable grounding.
- Motivation: Current video benchmarks can exceed 80% accuracy while leaving unresolved fine-grained abilities and failing to verify supporting spatio-temporal evidence.These protocols can treat answer correctness as genuine understanding without checking whether predictions are grounded in the correct evidence.
- Benchmark: VideoZeroBench contains 500 open-ended questions across 13 domains, with long videos averaging 667.1 seconds and tasks spanning 11 atomic abilities.Questions require precise fixed-format answers and often combine multiple abilities under complex spatio-temporal conditions.
- Evaluation: The benchmark progressively evaluates answering, temporal grounding, and spatial grounding, culminating in predictions tied to verifiable spatio-temporal evidence.Level-5 is described as the objective of trustworthy video understanding.
- Results: 17% is the maximum Level-3 accuracy reported for Gemini-3-Pro, while all models score below 1% under Level-5 grounding requirements.Many models achieve zero at Level-5, showing that correct answers rarely coincide with authentic supporting evidence.
- Findings: Analyses identify failed evidence grounding, fine-grained spatial intelligence, and needle-in-a-haystack temporal search as central weaknesses.Thinking-with-video paradigms provide some improvements but remain limited by spatio-temporal grounding precision.
2 VideoZeroBench
VideoZeroBench is constructed from manually curated long videos and verified questions requiring subtle evidence discovery. Its five-level protocol progressively removes evidence hints and adds temporal and spatial grounding requirements.
- Dataset Construction: 138 manually curated long videos are retained after screening for meaningful event progression, non-trivial scene transitions, and visual complexity.Most videos range from 5 to 20 minutes, dispersing supporting evidence across longer temporal contexts.
- Dataset Statistics: The benchmark spans 13 video domains and 11 atomic capabilities, with distributions reported for video length and minimal evidence span.Capabilities are grouped into Detailed Perception, Spatial&Temporal Reasoning, and Semantic&Cross-Modal Reasoning.
- Question Construction: Annotators use dense captions only as auxiliary references, then watch videos fully to construct precise fixed-format questions requiring subtle or non-salient evidence integration.The auxiliary captions do not determine ground-truth labels.
- Evidence Annotation: Temporal intervals and timestamp-linked normalized bounding boxes are manually recorded when questions depend on distinct spans or specific spatial regions.The annotation process retains 500 high-difficulty, logically consistent questions after verification.
- Dataset Statistics: 500 verified questions include 442 with temporal evidence and 372 with spatial grounding evidence, while average spatial boxes cover 6.8% of the frame.The dataset totals 25.57 hours of video and emphasizes long-horizon reasoning and fine-grained evidence grounding.
- Five-Level Evaluation: The five-level hierarchy removes evidence hints progressively, then requires explicit temporal and spatial evidence prediction to isolate reasoning and grounding failures.Levels 1–3 evaluate answering under different evidence conditions, while later levels add temporal and spatial precision.
- Five-Level Evaluation: Level-4 verifies correct answers with temporal overlap above 0.3, and Level-5 additionally requires visual overlap above 0.3 for spatial grounding.Level-5 combines answer correctness, temporal IoU, and visual IoU, using fixed key-frame timestamps for box prediction.
3 Evaluation on VideoZeroBench
VideoZeroBench evaluates video MLLMs across progressively stricter answer and spatio-temporal grounding requirements. Performance falls sharply under grounding constraints, exposing weaknesses in fine-grained spatial reasoning, temporal evidence search, and multimodal integration.
- Overall benchmark results: 17.0% is Gemini-3-Pro’s highest standard QA accuracy at Level-3, while the best Level-5 accuracy is only 1.0%.Level-5 requires both correct answers and accurate spatio-temporal localization.
- Overall benchmark results: Performance decreases consistently as evaluation constraints tighten from Level-1 through Level-5.Providing evidence produces higher scores than standard QA, while removing spatial hints causes further declines.
- Model categories: Most proprietary systems outperform open-source models, whereas no video-specific reasoning model achieves non-zero Level-5 accuracy.Open-o3-Video is the only video-specific reasoning model with non-zero Level-4 accuracy.
- Atomic abilities and failure cases: Small-object perception and spatial orientation are especially difficult, with Gemini-3-Pro scoring 11.7% and 11.8%, respectively.Driving videos also show notably low accuracy, consistent with cluttered scenes and complex relative positions.
- Input modality: Full-video input improves audio perception by 18.5% over frame-only input but decreases small-object perception by 3.9 percentage points.The result indicates a modality-dependent trade-off in Gemini-3-Pro’s Level-3 performance.
4 Related Works
Existing video MLLM benchmarks evaluate dynamic-scene understanding, multimodal integration, and long-video reasoning across diverse domains and task categories.
- Most video MLLM benchmarks assess understanding of dynamic scenes, multimodal signal integration, and reasoning over long videos.
- Video-MME and MVBench cover diverse domains and task categories, including action understanding and event reasoning.
- Video-MMMU, LongVideoBench, and CG-Bench extend evaluation toward instructional content, long-form reasoning, and temporal clue-grounding.
5 Conclusion
VideoZeroBench evaluates whether video MLLMs can answer questions and identify the correct temporal and spatial evidence. Experiments show sharp performance declines under grounding constraints and expose weaknesses in detailed visual reasoning.
- VideoZeroBench is a hierarchical benchmark testing both question answering and identification of correct temporal and spatial evidence.
- Performance drops sharply once grounding constraints are imposed, with weaknesses in small-object perception, fine-grained temporal search, spatial reasoning, and multi-evidence integration.
- The findings motivate greater emphasis on detailed visual grounding and more effective ways of thinking with videos.
A Appendix
The appendix supplements the main paper with implementation details, annotation descriptions, extended analyses, human evaluation, and qualitative failure cases.
- Section A.1 provides additional experimental details, while Section A.2 describes annotation details.
- Section A.3 analyzes frame sampling strategies, test-time scaling, and the effect of video length.
- Sections A.4 and A.5 report a human performance study and additional qualitative visualizations and error cases.
A.1 Additional Experimental Details
Additional experimental details describe video input and frame sampling, score computation, and the presentation of frame-sampling ablations.
- Gemini models receive compressed base64-encoded videos, whereas other models use uniformly sampled frames at 1 FPS with model-specific limits.
- Table 6 reports an ablation of frame-sampling strategies using Level-3 accuracy.
- Level-4 and Level-5 grounding scores are set to 0 for invalid outputs or missing annotations, with accuracy computed over all 500 questions.
A.2 Additional Annotation Details
Before manual annotation, Gemini generates dense captions and question insights to help annotators understand videos and identify potential question directions.
- Gemini generates dense captions and question insights as auxiliary reference information before manual annotation.Human annotators design and verify all final questions and answers.
A.3 Additional Analysis
Additional analyses examine frame sampling, test-time scaling, video duration, and evaluation materials to identify factors affecting long-video understanding.
- Frame Sampling Strategy: Increasing the frame budget does not consistently improve Level-3 accuracy across sampling settings.The analysis varies frame caps at 1 FPS and also evaluates 2 FPS with up to 768 frames.
- Video Length: VideoZeroBench includes analyses grouped by video duration, including short videos under 5 minutes, medium videos from 5–15 minutes, and long videos over 15 minutes.The duration grouping is reported for Level-3 accuracy.
- Frame Sampling Strategy: Temporal grounding is proposed as a better direction than simply increasing frame budgets because it concentrates sampling on informative segments and reduces redundant frames.The method dynamically resamples predicted evidence windows at higher density.
- Test-Time Scaling: 15.0% Pass@5 (Any) versus 7.8% greedy decoding shows that parallel scaling raises the chance of obtaining one correct answer, while 1.8% Pass@5 (All) indicates uncertain reasoning paths.Sequential localize-then-answer scaling instead declines to 7.0%.
- Test-Time Scaling: Sequential scaling declines to 7.0% because incorrect initial evidence windows can redirect later sampling toward irrelevant frames.This result is linked to weak intrinsic temporal localization.
A.4 Human Performance Study
A human study on a 50-question subset finds that participants substantially outperform frontier proprietary models, especially on fine-grained perception capabilities.
- Overall Performance: 67.6% human accuracy exceeds Gemini-3-Pro at 22.0% and Gemini-2.5-Pro at 20.0% on the 50-question subset.The subset spans all 13 categories and includes Chinese and English questions.
- Atomic Capabilities: Humans achieve much higher accuracy than models on Counting, Small-Object, Spatial Orientation, and Object Tracking.These tasks require subtle visual-cue identification and object tracking across frames.
- Study Conditions: Participants typically require around one hour to complete the fine-grained questions carefully.
- Atomic Capabilities: Gemini models perform relatively better on World Knowledge, while human Audio Perception performance is relatively low.The passage suggests participants may overlook audio cues or struggle to convert them into precise textual responses.
A.5 More Visualization and Error Cases
Representative VideoZeroBench examples illustrate recurring reasoning failures involving temporal localization, spatial grounding, hallucinated support, and fine-grained perception.
- The examples highlight incorrect temporal localization, inaccurate spatial grounding, and answers that lack the supporting evidence despite being correct.They also cover weaknesses in counting, small-object perception, and spatial orientation.
B Ethics Statement
VideoZeroBench uses publicly accessible videos solely for academic research, with privacy protections in displayed visualizations and an explicit boundary against harmful surveillance applications.
- Videos are collected from publicly accessible sources and used solely for academic research.
- Identifiable information, including human faces, is blurred or masked in the paper’s visualizations.
- The benchmark should not support systems that violate privacy or enable harmful surveillance applications.