PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

Jiahao Meng, Tan Yue, Qi Xu, Haochen Wang, Zhongwei Ren, Weisong Liu, Yuhao Wang, Renrui Zhang, Yunhai Tong, Haodong Duan

arXiv:2604.01569v1cs.CVcs.MM

TL;DR

Existing video benchmarks can report high answer accuracy without verifying the spatio-temporal evidence behind answers, leaving fine-grained abilities unclear. VideoZeroBench evaluates long-video QA with a five-level evidence-grounding protocol, and performance drops sharply under grounding constraints, with a best Level-5 accuracy of 1.0%.

  • Problem

    Existing video benchmarks can report high answer accuracy without resolving which fine-grained abilities remain limited or verifying that answers are supported by correct spatio-temporal evidence.

    Source: §1 · p. 3

  • Method

    VideoZeroBench pairs 500 open-ended questions across 13 domains with temporal and spatial evidence and evaluates answering and grounding through a five-level hierarchy.

    Source: §1 · p. 3, §2.3 · p. 7

  • Results

    Performance drops sharply once grounding constraints are imposed; at Level-5, the best accuracy is 1.0% (Gemini-3-Pro), and only about one-third of models score above zero.

    Source: §3.2 · p. 8, §5 · p. 15

  • Takeaways & Limitations

    The findings support prioritizing detailed visual grounding and understanding in future research on video reasoning.

    Source: §5 · p. 15

Abstract

from arXiv · show

Recent video multimodal large language models achieve impressive results across various benchmarks. However, current evaluations suffer from two critical limitations: (1) inflated scores can mask deficiencies in fine-grained visual understanding and reasoning, and (2) answer correctness is often measured without verifying whether models identify the precise spatio-temporal evidence supporting their predictions. To address this, we present VideoZeroBench, a hierarchical benchmark designed for challenging long-video question answering that rigorously verifies spatio-temporal evidence. It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence. To disentangle answering generation, temporal grounding, and spatial grounding, we introduce a five-level evaluation protocol that progressively tightens evidence requirements. Experiments show that even Gemini-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting (Level-3). When grounding constraints are imposed, performance drops sharply: No model exceeds 1% accuracy when both correct answering and accurate spatio-temporal localization are required (Level-5), with most failing to achieve any correct grounded predictions. These results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning, revealing that grounded video understanding remains a bottleneck for long-video QA. We further analyze performance across minimal evidence spans, atomic abilities, and inference paradigms, providing insights for future research in grounded video reasoning. The benchmark and code will be made publicly available.

1 Introduction

VideoZeroBench targets the gap between video QA accuracy and evidence-grounded understanding with a challenging benchmark and a five-level evaluation framework. Experiments show that models rarely combine correct answers with precise spatio-temporal grounding.

1 Introduction

  • VideoZeroBench addresses a gap left by saturated QA benchmarks and existing grounding tests, which do not adequately probe fine-grained spatio-temporal reasoning in long videos.VideoMME [10], Video-MMMU [19], and LongVideoVench [45] exceed 80%, while V-STAR [4], Know-Show [34], and ToG-Bench [47] focus mainly on short videos or simple perception; the video domain also lacks a fair benchmark for thinking-with-videos approaches.Source: §1 · p. 3
  • VideoZeroBench comprises 500 open-ended questions across 13 domains, averaging 667.1 seconds per video and covering 11 atomic abilities.Questions are manually annotated, intentionally challenging, and require precise answers; many demand multiple abilities, including counting, small-object perception, spatial orientation discrimination, and multi-segment causal dependency.Source: §1 · p. 3
  • Its five-level framework separates answering from grounding, progressing from video QA with evidence hints to correct answers with temporal and spatial localization.Level-1 supplies temporal and spatial evidence, Level-2 temporal evidence, and Level-3 no hints; Level-4 requires correct answers and temporal segments, while Level-5 adds spatial localization.Source: §1 · pp. 3–4
  • Gemini-3-pro achieves no more than 17% accuracy at Level-3, while every model scores below 1% at Level-5 and many score zero.Level-3 is standard video QA without evidence hints; Level-5 requires correct answers and accurate temporal and spatial grounding.Source: §1 · p. 4
  • Across categories, abilities, evidence spans, inference paradigms, and input modalities, analyses find that answer correctness often fails to reflect understanding and grounding precision is the main bottleneck.The reported conclusions identify fine-grained spatial intelligence and needle-in-a-haystack temporal search as primary challenges; thinking-with-video offers some improvement but remains limited by grounding precision.Source: §1 · p. 4

2 VideoZeroBench

VideoZeroBench pairs manually curated long videos with verified questions and explicit temporal and spatial evidence. Its five-level protocol progressively tests answering, temporal grounding, and spatial precision.

2.1 Dataset Construction

  • 138 long, complex videos yield 500 manually constructed, cross-verified questions requiring subtle clue discovery or evidence integration, with temporal intervals and spatial boxes annotated where relevant.LLM-generated captions and question insights serve only as references; annotators watch each video in full, and reviewers re-watch videos to check answerability, determinism, and evidence quality.Source: §2.1 · pp. 4–6

2.2 Dataset Statistics

  • The 500 questions include 220 English and 280 Chinese items; 442 involve temporal evidence and 372 spatial grounding, with average bounding-box area of 6.8% of the frame.The 138 videos total 25.57 hours, average 667.14 seconds, and range from 30 to 3037 seconds; most last 5–20 minutes.Source: §2.2 · p. 6
  • The benchmark organizes questions into 11 atomic capabilities across detailed perception, spatial and temporal reasoning, and semantic and cross-modal reasoning, and labels single-frame, short-term, and long-range evidence spans.The capabilities include counting, small-object perception, OCR, action and event perception, spatial orientation, object tracking, scene transitions, multi-segment dependency, world knowledge, and audio perception.Source: Figure 2 · p. 5, §2.2 · p. 6

2.3 Five-Level View of VideoZeroBench

  • Levels 1–3 test QA with both evidence types, temporal hints only, or no hints; Levels 4–5 additionally require correct temporal grounding and then spatial localization.Level 4 combines Level-3 answer correctness with tIoU > 0.3; Level 5 also requires vIoU > 0.3, and fixed key-frame timestamps isolate spatial localization from temporal shifts.Source: §2.3 · pp. 7–8

3 Evaluation on VideoZeroBench

The five-level evaluation shows a wide gap between answering video questions and grounding answers in precise spatio-temporal evidence. Analyses identify persistent challenges in fine-grained perception, evidence retrieval, and grounding, while testing whether iterative reasoning and different inputs close those gaps.

3.1 Settings

  • Across 17 proprietary, open-source, and video-specific models, proprietary systems generally outperform others, Qwen3-VL-235B leads open-source models, and video-specific reasoning models show weak grounding.Gemini-3-Pro leads overall from Level-3 to Level-5; no video-specific reasoning model has non-zero Level-5 performance, and only Open-o3-Video has non-zero Level-4 accuracy.Source: §3.1 · p. 8, §3.2 · p. 10

3.2 Benchmark Results

  • 17.0% is Gemini-3-Pro’s highest Level-3 accuracy, while the best Level-5 accuracy is just 1.0%, achieved by Gemini-3-Pro.Only about one-third of models score above zero at Level-5, which requires both correct answers and spatio-temporal grounding.Source: §3.2 · p. 8, Table 2 · p. 9
  • Level-1 evidence hints raise Seed-2.0 and Qwen3-VL-235B to 25.4% and 28.4%, but removing spatial hints lowers accuracy and tighter grounding constraints further degrade performance.Even with evidence provided, the highest accuracy remains below 30%; at Level-5, Seed-2.0 has the highest vIoU at 21.8%, while most models have near-zero accuracy.Source: §3.2 · pp. 9–10

3.3 Discussion

  • 11.7% on small-object perception and 11.8% on spatial orientation are Gemini-3-Pro’s scores, while most models achieve below 8% on counting.Counting requires precise localization and aggregation across frames, including handling repeated appearances and occlusion; driving scenes also show notably low accuracy.Source: §3.3 · p. 12
  • Single-frame-answerable questions are not easier than short- or long-range questions because critical fleeting frames in long videos may be missed by uniform sampling.For these needle-in-a-haystack cases, retrieving the right frame can be more difficult than reasoning over it.Source: §3.3 · p. 12

Q3: Does the agentic thinking-with-videos paradigm exactly bring

  • 1.4, 1.0, and 1.8 percentage-point gains at Levels 1–3 from three-round zoom-in reasoning yield no improvement at Levels 4–5.The VideoChat-R1.5 ablation indicates that iterative temporal refinement alone cannot compensate for inadequate spatial grounding.Source: Table 3 · p. 12, Q3: Does the agentic… · p. 13
  • Above 40% accuracy for Gemini-3-Pro and Gemini-2.5-Pro with zoomed temporal segments and spatial crops contrasts with their 17.0% and 16.4% full-video Level-3 scores.The comparison controls total visual tokens and suggests evidence localization is a primary weakness.Source: Table 4 · p. 12, Q3: Does the agentic… · p. 13
  • 18.5% higher audio-perception performance but 3.9 percentage points lower small-object perception with full-video versus frame-only input show a modality tradeoff for Gemini-3-Pro.Without visual or audio input, the model still answers a small fraction correctly; audio-only improves audio perception over the no-input setting, but remains far below full-video performance.Source: Q3: Does the agentic… · pp. 13–14, Table 5 · p. 13

4 Related Works

The section positions VideoZeroBench against three lines of prior work: video MLLM benchmarks, detailed spatial benchmarks, and video reasoning with explicit grounding. It argues that none rigorously verifies spatio-temporal evidence for difficult questions in long videos.

Video MLLM Benchmarks

  • VideoZeroBench probes harder open-ended questions with a hierarchical protocol than existing video MLLM benchmarks, which cover dynamic-scene understanding, multimodal reasoning, and long-form tasks [10] [19] [22] [45] [58].Video-MME [10] and MVBench cover diverse domains and tasks; Video-MMMU [19], LongVideoBench [45], and CG-Bench [3] extend evaluation toward knowledge-intensive, long-form, and temporal-clue-grounded questions.Source: §4 · p. 14

Detailed Spatial Benchmarks

  • Existing video spatial benchmarks assess localization or spatial intelligence, but do not jointly test difficult question answering and explicit evidence localization in long, cluttered videos [4] [34] [47] [46] [21].V-STaR [4], Know-Show [34], and ToG-Bench [47] evaluate when and where objects or events occur, while VSI-Bench, SpatialTree [46], and MMSI-Video-Bench [21] examine spatial relations and motion direction.Source: §4 · p. 14

Thinking With Videos

  • Video reasoning approaches make evidence explicit through iterative temporal zooming or timestamp and bounding-box outputs, but no benchmark rigorously verifies key spatio-temporal evidence on genuinely difficult problems [48] [7] [40].VideoChat-R1.5 [48] and VideoZoomer [7] refine relevant intervals iteratively; Open-o3-Video [40] emits timestamps and bounding boxes in its reasoning trace without external tools.Source: §4 · p. 14

5 Conclusion

VideoZeroBench evaluates whether video MLLMs can answer questions and identify the correct temporal and spatial evidence. Experiments reveal sharp performance drops under grounding constraints and weaknesses across several visual reasoning abilities.

5 Conclusion

  • VideoZeroBench is a challenging, hierarchical benchmark that evaluates whether video MLLMs can answer questions and identify the correct temporal and spatial evidence.Source: §5 · pp. 14–15
  • Performance drops sharply under grounding constraints, with weaknesses in small-object perception, fine-grained temporal search, spatial reasoning, and multi-evidence integration.The authors argue that future research should emphasize detailed visual grounding and understanding, and explore more effective ways of thinking with videos.Source: §5 · p. 15

A Appendix

The appendix supplements the main paper with implementation and annotation details, additional analyses, a human performance study, and qualitative examples of model failures.

A.1–A.2

  • Sections A.1–A.2 provide additional experimental and annotation details.The overview does not specify the details themselves.Source: A Appendix · p. 19

A.3

  • Section A.3 analyzes frame-sampling strategies, test-time scaling, and the effect of video length.Source: A Appendix · p. 19

A.4–A.5

  • Section A.4 reports a human performance study on a VideoZeroBench subset, while A.5 presents qualitative visualizations and error cases illustrating typical video MLLM failure patterns.Source: A Appendix · p. 19

A.1 Additional Experimental Details

The appendix specifies video input and inference configurations, grounding prompt formats, and scoring rules to support reproducibility and comparison across evaluation levels.

Video Input and Frame Sampling

  • Most non-Gemini models uniformly sample video at 1 FPS, usually up to 384 frames, with lower limits for Seed-2.0-Pro, Qwen3.5-397B, GPT-5.2, and InternVL3.5 [44].Gemini videos are instead compressed below 100 MB and sent directly to the API; sampled frames preserve the original aspect ratio.Source: §A.1 · p. 19

Inference Setup

  • Proprietary models and Qwen3.5-397B use API inference, while other open-source models use vLLM; temperature is 0, and models retain native chain-of-thought or receive final-answer-only prompts.These settings are intended to ensure deterministic outputs and reproducibility.Source: §A.1 · p. 19

Thinking-with-videos Paradigm Configurations

  • VideoChat-R1.5 [48] uses key_ratio = 1.5 and three reasoning rounds, while Video-o3 [53] uses adaptive-pixel-budget dynamic sampling for at most eight rounds before forced final output.For VideoChat-R1.5, frames near predicted key timestamps are sampled more densely than other video segments.Source: §A.1 · p. 19

Prompt Templates Across Five Levels

  • Prompts are adapted to model-specific timestamp and coordinate formats; Qwen3-VL uses normalized 0–1000 box coordinates, whereas Qwen2.5-VL uses absolute pixel coordinates.An output parser extracts temporal intervals and spatial boxes in the corresponding formats.Source: §A.1 · p. 19

Score Computation

  • Invalid temporal intervals or spatial boxes, and questions lacking the relevant grounding annotation, receive IoU 0 at Level-4 or Level-5; both accuracies use all 500 questions.This denominator supports direct comparability with the other evaluation levels.Source: §A.1 · p. 20

A.2 Additional Annotation Details

Gemini-generated captions and question insights provide reference material, while human annotators design and verify all final questions and answers. The annotation interface combines video, guidance, reference information, and tools for querying Gemini.

A.2 Additional Annotation Details

  • Gemini generates dense captions and question insights as reference material, but human annotators design and verify all final questions and answers.The prompts used to generate the auxiliary descriptions are shown in Fig. 8.Source: §A.2 · p. 20
  • The annotation interface combines a video player, Gemini-generated reference information, annotation guidelines, and fields for questions, answers, and evidence.It also supports direct API calls to Gemini models during annotation.Source: §A.2 · p. 20
  • Annotators can query Gemini to understand current video MLLM capability boundaries and design challenging questions that expose potential weaknesses.Source: §A.2 · p. 20

A.3 Additional Analysis

Additional analyses show that more frames do not reliably improve answering, test-time scaling has mixed results, and accuracy generally falls as videos get longer.

Effect of Frame Sampling Strategy

  • Level-3 accuracy improves from 48 to 96 frames, then quickly saturates; larger budgets can add irrelevant-frame noise, favoring denser sampling of informative grounded segments.The analysis concludes that simply increasing the frame budget is not an effective strategy for long-video understanding.Source: §A.3 · pp. 20–21

Effect of Test Time Scaling

  • 15.0% Pass@5 (Any) versus 7.8% greedy decoding, but Pass@5 (All) was 1.8% and sequential scaling reached 7.0%.Sequential scaling first localizes evidence windows, then resamples them; weak localization can direct sampling toward irrelevant frames and filter out genuine cues.Source: §A.3 · pp. 21–22, Table 7 · p. 22

Effect of Video Length

  • For most models, Level-3 accuracy decreases with video duration, while Gemini-3-Pro, Video-R1 [9], and VideoChat-R1.5 [48] show greater long-video robustness.The passage attributes longer-video difficulty to a larger search space, more distractors, and more complex reasoning.Source: Figure 6 · p. 21, §A.3 · p. 22

A.4 Human Performance Study

On a 50-question subset, humans substantially outperformed Gemini models, particularly on fine-grained visual tasks. Models performed relatively better on World Knowledge, while human Audio Perception accuracy was low.

A.4 Human Performance Study

  • The study sampled 50 questions across all 13 video categories, with 27 Chinese and 23 English questions, and recruited 20 participants independent of the annotators.Each participant answered at most 10 questions; participants’ birth years ranged from the 1970s to the 2000s.Source: §A.4 · pp. 22–23
  • Human accuracy was 67.6%, compared with 22.0% for Gemini-3-Pro and 20.0% for Gemini-2.5-Pro on the subset.The study reports that humans outperformed frontier proprietary models.Source: §A.4 · p. 23
  • Humans substantially outperformed models on Counting, Small-Object, Spatial Orientation, and Object Tracking, but typically needed around one hour to complete ten questions.These tasks require identifying subtle visual cues and tracking objects across frames, which remain challenging for current video MLLMs.Source: §A.4 · p. 23
  • Gemini models performed relatively better on World Knowledge, while human Audio Perception performance was low; fine-grained visual perception remains the models’ primary bottleneck in long videos.The passage suggests participants may overlook audio cues or find it difficult to express auditory information precisely in text.Source: §A.4 · p. 23

A.5 More Visualization and Error Cases

VideoZeroBench examples illustrate that video MLLMs often fail to ground their answers in precise temporal and spatial evidence, and struggle with fine-grained visual reasoning.

A.5 More Visualization and Error Cases

  • Examples in Fig. 9, 10, and 11 show incorrect temporal localization, inaccurate spatial grounding, and correct answers without supporting evidence.The latter cases are hallucinations in which models answer correctly but do not identify the evidence.Source: §A.5 · p. 23
  • The examples also expose weaknesses in counting, small-object perception, and spatial orientation.Source: §A.5 · p. 23

B Ethics Statement

VideoZeroBench uses publicly accessible videos solely for academic research, masks identifiable information in paper visualizations, and is not intended to enable privacy violations or harmful surveillance.

Ethics Statement

  • The benchmark videos come from publicly accessible sources and are used solely for academic research.Source: B Ethics Statement · p. 23
  • Human faces and other identifiable information are blurred or masked in the visualizations shown in the paper to mitigate privacy risks.Source: B Ethics Statement · p. 23
  • The benchmark is intended for video-understanding research, not for developing systems that violate privacy or enable harmful surveillance.Source: B Ethics Statement · p. 23

Related papers

Loading 2604.01569v1…