Source-linked AI summary
VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
Fan Zhang, Guangming Yao, Jinyang Wu, Hao Wu, Zheng Lian, Xinyu Geng, Jingdong Chen, Yi Yuan, Pheng-Ann Heng
TL;DR
Conventional single-turn video benchmarks are nearing saturation, motivating evaluation of more capable multimodal systems. VideoGAIA introduces a tool-augmented, multi-turn benchmark, yet all evaluated models remain below 60% accuracy.
Problem
Conventional single-turn video benchmarks are approaching 90% accuracy, creating a need for evaluation that better reflects next-generation multimodal systems.
Method
VideoGAIA uses a multi-stage human-model co-design pipeline to create 271 human-verified tasks requiring iterative video grounding, tool use, and multimodal evidence integration.
Results
All 20 evaluated frontier MLLMs score below 60% accuracy, with Seed2.0-Pro achieving 58.30%.
Takeaways & Limitations
VideoGAIA reveals substantial limitations in video grounding, cross-modal reasoning, and reliable tool use among current frontier MLLMs.
Takeaways & Limitations
The benchmark does not comprehensively cover multilingual scenarios or depend heavily on audio information.
Abstract
from arXiv · showhide
Video understanding is a fundamental task for evaluating the capabilities of multimodal large language models (MLLMs). However, existing leading models have already achieved approximately 90% accuracy on the Video-MME leaderboard, suggesting that conventional single-turn video understanding tasks are becoming increasingly saturated and insufficient for assessing the intelligence of advanced MLLMs. Towards this end, we introduce VideoGAIA, an agentic video understanding benchmark for general artificial intelligence (AI) assistants. Moving beyond one-shot video question answering, VideoGAIA formulates video understanding as a multi-turn, tool-augmented interaction process, where models must iteratively perceive videos, invoke external tools, gather complementary information, and integrate multimodal evidence across turns. VideoGAIA contains 271 model-human co-designed tasks covering diverse and complex real-world scenarios. Each video-question-answer instance is independently verified by three human experts to ensure both correctness and appropriate difficulty. All evaluated MLLMs, including frontier models such as GPT-5.5 and Kimi-K3, achieve less than 60% accuracy on VideoGAIA, highlighting its value as a high-quality and timely benchmark for evaluating next-generation MLLMs. We hope that VideoGAIA will facilitate the transition from conventional video understanding toward agentic video understanding.
1 Introduction
VideoGAIA reframes video understanding as agentic, tool-augmented interaction because conventional single-turn benchmarks are nearing saturation. It introduces a human-verified benchmark and unified evaluation showing that current frontier MLLMs remain far from solving agentic video understanding.
- Existing benchmarks assess multimodal perception, temporal reasoning, and object recognition, while leading MLLMs approach 90% accuracy on widely adopted video understanding benchmarks.This saturation suggests conventional single-turn video question answering is becoming nearly solved.
- The benchmark evaluates 20 frontier MLLMs under a unified ReAct-style agent loop equipped with web search, page visit, and video-thinking tools.The shared setting is designed to support fair comparison across models.
- 30%–60% accuracy is achieved by all evaluated models, while the strongest model reaches only 58.30%, revealing substantial room for developing agentic video understanding.These results are substantially below conventional video understanding performance and indicate that agentic video understanding remains far from solved.
- VideoGAIA shifts from single-turn question answering toward agentic video understanding, using video content as evidence for accomplishing concrete tasks.The paradigm emphasizes moving beyond merely answering questions about videos.
- VideoGAIA contains 271 human-verified video-agent tasks across six real-world categories, constructed through multi-stage human-model co-design and reviewed independently by at least three experts.The pipeline includes model-assisted annotation and quality control, while expert review targets correctness, difficulty, multimodal reasoning, and tool-use requirements.
2 Related Work
Prior video understanding benchmarks assess temporal, long-context, multimodal, and domain capabilities, while newer work frames video understanding as interactive evidence seeking with retrieval, memory, inspection, and tool-use strategies.
- Conventional video understanding benchmarks: Existing benchmarks cover temporal perception, long-context comprehension, multimodal reasoning, and domain knowledge across diverse video understanding settings.MVBench emphasizes temporal understanding, while Video-MME spans diverse durations, domains, and modalities.
- Interactive video understanding: Recent approaches formulate video understanding as iterative evidence seeking through informative visual selection, structured memory, retrieval, and learned tool use.VideoAgent uses iterative visual evidence selection with structured memory or retrieval tools; related systems explore multi-granular retrieval and coarse-to-fine inspection.
3 VideoGAIA
VideoGAIA is constructed through evidence-grounded, tool-augmented task generation followed by multi-stage quality control and human verification. The final benchmark contains 271 open-ended tasks spanning six categories and emphasizing multi-step, cross-modal reasoning.
- Data collection and task construction: 100K public videos are processed at 1 FPS into segments of at most 24 frames, with up to four segments and approximately 96 frames per video.Gemini-3.1-Pro generates frame-level descriptions and OCR extracts visible text.
- Data collection and task construction: Tasks combine two to five reusable video anchors with target attributes, candidate answers, and externally retrieved webpage evidence.Claude-Opus-4.7 extracts anchors, GPT-5.5 proposes task attributes and answers, and Serper and Jina provide search and webpage content.
- Quality control: Quality control requires at least two reasoning hops, explicit evidence support, multiple video anchors, and at least three reasoning steps linking video evidence with web information.The rubric also requires at least two web-search or full-page extraction steps, visually described entities, a unique canonical answer, and a concise, grammatical question.
- Quality control: 847 candidate tasks remain after model-based filtering, and each is independently verified by three human annotators for correctness and answer uniqueness.Annotators solve tasks from scratch using the video and web searches before determining whether proposed answers are correct and unique.
- Dataset statistics: 271 tasks span six categories—Geography (63), Daily Life (61), Culture (58), History (44), Technology (39), and Economics (6)—with 209 L2 tasks and 77.12% of the benchmark at L2.The benchmark also contains 62 L1 tasks and preserves open-ended evaluation without multiple-choice cues.
4 Experiments
Experiments evaluate 20 frontier MLLMs with a unified ReAct-style agent loop combining video reinspection, web search, and page visits. Performance remains below 60% overall, declines sharply on harder tasks, and is limited primarily by visual perception errors.
- Experimental Setup: 20 frontier MLLMs are evaluated with a unified ReAct-based agent loop that interleaves reasoning with web search, page visits, and video reinspection.Each task begins with 20 uniformly sampled video frames, permits up to 40 agent steps, and allows up to 20 additional frames from a selected temporal segment.
- Overall Results: 58.30% is the highest overall accuracy, achieved by Seed2.0-Pro, followed by Qwen3.7-Plus at 56.09% and Kimi-K3 at 54.61%; no model reaches 60%.These results hold despite access to both video reinspection and open-web tools.
- Overall Results: Claude-Opus-4.7 and Claude-Opus-4.8 score 34.69% and 36.90%, below Claude-Opus-4.6 at 47.97%, indicating non-monotonic performance across model versions.Category leadership also varies: Seed2.0-Pro leads Geography and Technology, while Kimi-K3 leads Daily Life.
- Difficulty Analysis: 22.70%–29.75% point gaps separate L1 and L2 for the three strongest models, with Seed2.0-Pro falling from 75.81% to 53.11%, Qwen3.7-Plus from 79.03% to 49.28%, and Kimi-K3 from 77.42% to 47.85%.Every evaluated model declines from L1 to L2, showing that L2 contains substantially harder trajectories.
- Tool Analysis: 60.5%–90.6% of tool calls are web searches, page visits never exceed 14.0%, and Kimi-K3 allocates 25.5% to video thinking.Gemini-3.1-Pro makes 2,557 calls, 90.6% of them searches.
- Error Analysis: 36.8%–48.0% of incorrect predictions are dominated by video-perception errors, making visual grounding the largest error source for every audited model.The analysis covers all 615 incorrect predictions from five representative models and notes that wrong visual anchors can contaminate downstream searches.
5 Conclusion · A More Related Works · A.1 Multimodal AI Agents
VideoGAIA advances agentic video understanding through human-verified, tool-augmented tasks requiring iterative evidence grounding and cross-turn integration. Its evaluation shows that all 20 frontier MLLMs remain below 60% accuracy, motivating rigorous testing of practical video agents.
- 5 Conclusion: VideoGAIA is a benchmark advancing agentic video understanding.It is designed as a rigorous testbed for developing video agents.
- 5 Conclusion: 271 human-verified video tasks span six real-world categories.Each task is verified by humans, supporting evaluation across practical scenarios.
- 5 Conclusion: The benchmark requires models to iteratively ground visual evidence, use external tools, and integrate information across turns.These requirements move beyond passive or one-shot video understanding toward agentic interaction.
- 5 Conclusion: 20 frontier MLLMs were evaluated under a unified evaluation.The evaluation covers a broad set of frontier multimodal large language models.
- 5 Conclusion: Below 60% accuracy was achieved by all evaluated models.This result indicates that the benchmark remains challenging for current frontier MLLMs.
- A.1 Multimodal AI Agents: Multimodal agents extend language-model agents by actively manipulating visual evidence during reasoning.The related work frames multimodal agents as extending planning, tool use, memory, and feedback beyond passive generation.
B Implementation Details · B.1 System Prompt for Video Agents · B.2 System Prompt for Judge Models
The implementation specifies a tool-augmented video-agent protocol that interleaves visual reasoning, targeted external information gathering, and concise tagged responses. A separate judge prompt evaluates answers by semantic equivalence and emits a binary correctness label.
- B.1 System Prompt for Video Agents: Video agents interleave reasoning with exactly three tools: VideoAnalysisAction, WebSearchAction, and ExtractUrlContentAction.The prompt also forbids inventing tools and requires the action field to match an available tool name exactly.
- B.1 System Prompt for Video Agents: Agents first reason from uniformly sampled coarse frames and invoke VideoAnalysisAction only for insufficient evidence, closer segment inspection, or additional segment frames.VideoAnalysisAction is restricted to visual analysis and must not request audio, transcription, speech, music, or sound.
- B.1 System Prompt for Video Agents: WebSearchAction handles open-web searches, while ExtractUrlContentAction is used only after a promising URL has been found.The examples illustrate searching for external verification of a named entity and then extracting a specific fact from a relevant page.
- B.1 System Prompt for Video Agents: Each assistant turn uses exactly one action or a final answer, with observations summarized after tool calls and outputs constrained to required tags, valid JSON actions, and concise answers.The protocol separates <action> and <answer> turns and disallows markdown, code fences, multiple actions, and legacy tool-call fields.
- B.1 System Prompt for Video Agents: When evidence remains insufficient near the step limit, the agent should provide the best available answer, including “unknown” when the requested detail cannot be identified confidently.The prompt also demonstrates concise final answers after observations become sufficient.
- B.2 System Prompt for Judge Models: Judge models assess whether a predicted answer correctly answers the question using semantic equivalence rather than exact string matching.They accept aliases, abbreviations, translations, official or common names, formatting differences, and noncontradictory extra explanation.
- B.2 System Prompt for Judge Models: For numeric answers, judges require the same value while allowing normal rounding, unit conversion, or formatting differences.Empty, merely hedged, irrelevant, or contradictory predictions are marked incorrect.
- B.2 System Prompt for Judge Models: The judge returns only one character: 1 for a correct prediction or 0 for an incorrect prediction, without JSON, explanations, markdown, or other text.This output constraint makes evaluation a binary label rather than a textual assessment.
C More Experimental Results · C.1 Efficiency Analysis
The efficiency analysis examines whether longer or more expensive ReAct trajectories improve task performance, measuring response length and output-only computation cost across all 271 tasks.
- C.1 Efficiency Analysis: The analysis tests whether longer or more expensive ReAct trajectories yield better task performance.Response length is defined as the mean visible output tokens per task across all turns.
- C.1 Efficiency Analysis: 271 tasks are used to calculate total output-token computation cost.The cost sums output tokens across tasks and applies each model’s public standard output-token price.
C.2 Additional Visualization
Additional visualizations show that VideoGAIA models have distinct domain-specific strengths rather than uniform dominance, while expanded analyses reveal consistent reliance on web search and varied use of other tools.
- Category-level performance: Radar plots across six domains show that no evaluated model dominates uniformly, with markedly different performance profiles.Figure 10 compares all 20 evaluated frontier video agents.
- Tool usage and errors: 53.1%–88.1% of calls use web search across every additional model, while page visits and targeted video inspection vary considerably.Figures 11 and 12 extend the analyses to the remaining 15 models and complete coverage of all evaluated systems.
C.3 Additional Case Study
Figures 13–18 provide complete trajectories from each VideoGAIA domain, illustrating how successful agents ground visual clues and use targeted searches or video revisiting, whereas failed trajectories confuse related facts.
- Trajectory analysis: Figures 13–18 present one complete trajectory from each VideoGAIA domain.The case study compares successful and failed interaction trajectories across the benchmark’s domains.
- Successful trajectories: Seed2.0-Pro and Qwen3.7-Plus ground distinctive visual clues before issuing focused web searches in successful cases.These trajectories combine visual evidence with targeted external information gathering.
- Successful trajectories: Kimi-K3 revisits the relevant video segment to resolve the color of the racing buoys.The example shows iterative video inspection resolving a detail that required returning to the footage.
- Failed trajectories: Failed trajectories remain superficially plausible but confuse closely related facts.The passage contrasts plausible presentation with incorrect factual discrimination.
D Limitations and Social Impact
VideoGAIA’s limitations include incomplete coverage of multilingual and audio-dependent scenarios and evaluation with only a simple ReAct-style agent loop. Nevertheless, the benchmark aims to advance agentic video understanding by encouraging agents that perceive continuous visual information, seek complementary evidence, and act on it.
- Limitations: VideoGAIA does not comprehensively cover multilingual scenarios or depend heavily on audio information.
- Limitations: The benchmark uses a simple ReAct-style agent loop for consistent, controlled evaluation across models.
- Limitations: VideoGAIA does not evaluate more sophisticated agent systems such as Codex or OpenClaw, leaving broader modalities, languages, and advanced harnesses for future work.Such systems may provide stronger context management, tool orchestration, and long-horizon planning capabilities.
- Social Impact: VideoGAIA could benefit video understanding and multimodal-agent research by encouraging attention to video agents beyond systems operating primarily on static images and text.The paper frames real-world visual information as continuous, dynamic, and temporally evolving, requiring agents to perceive it, seek complementary evidence, and act upon it.