PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-Browser: Towards Agentic Open-web Video Browsing

Zhengyang Liang, Yan Shu, Xiangrui Liu, Minghao Qin, Kaixin Liang, Nicu Sebe, Zheng Liu, Lizi Liao

arXiv:2512.23044v2cs.CV

TL;DR

Agentic web research still has a modality gap for video: existing approaches either incur high context costs or lose fine-grained visual details. Video-Browser combines metadata filtering with targeted visual perception, and Video-BrowseComp results report improved accuracy with lower token consumption than direct visual inference.

  • Problem

    Video research agents must search across videos and verify visual evidence, but direct inference has prohibitive context costs while text-centric summaries lose fine-grained details.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    Video-Browser uses Pyramidal Perception to filter candidates with metadata, localize relevant moments, and apply high-fidelity perception to selected video windows.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    37.5% relative improvement in accuracy and 58.3% lower token consumption are reported compared with direct video inference on Video-BrowseComp.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors present Video-Browser as a foundation for verifiable and efficient video agents in open-web research.

    Source: §6 · p. 8

    Video-BrowseComp has a modest scale of 210 samples, reflecting the trade-off between expert-validation rigor and computational accessibility.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The evolution of autonomous agents is redefining information seeking, transitioning from passive retrieval to proactive, open-ended web research. However, a significant modality gap remains in processing the web's most dynamic and information-dense modality: video. In this paper, we first formalize the task of Agentic Video Browsing and introduce Video-BrowseComp, a benchmark evaluating open-ended agentic browsing tasks that enforce a mandatory dependency on videos. We observe that current paradigms struggle to reconcile the scale of open-ended video exploration with the need for fine-grained visual verification. Direct visual inference (e.g., RAG) maximizes perception but incurs prohibitive context costs, while text-centric summarization optimizes efficiency but often misses critical visual details required for accurate grounding. To address this, we propose Video-Browser, a novel agent leveraging Pyramidal Perception, filtering with cheap metadata and zooming in with expensive visual perception only when necessary. Experiments demonstrate that our approach achieves a 37.5% relative improvement while reducing token consumption by 58.3% compared to Direct visual inference, establishing a foundation for verifiable open-web video research. We open-source all codes, benchmark at {https://anonymous.4open.science/r/VideoBrowser} and {https://github.com/chrisx599/Video-Browser}.

1 Introduction

The paper identifies a gap in agentic web research: video is often treated as a clip to perceive rather than a source to search and explore. It introduces a video-dependent browsing benchmark and an agent that balances efficient search with fine-grained visual verification.

1 Introduction

  • Video-BrowseComp evaluates open-ended agentic browsing with mandatory video dependence and difficulty levels spanning explicit retrieval to cross-source reasoning.The task is framed as iterative decision-making rather than video question answering.Source: §1 · p. 1
  • Video-Browser uses Pyramidal Perception to filter irrelevant videos with lightweight metadata and apply high-fidelity visual perception only when needed.This targets the context costs of direct inference and the visual-detail loss of text-centric summarization.Source: §1 · p. 2
  • 37.5% relative improvement in accuracy and 58.3% lower token consumption versus direct video inference are reported for Video-Browser on Video-BrowseComp.The authors say this balances open-web scalability with fine-grained verification.Source: §1 · p. 2

2 Related Works

Prior video-understanding work largely evaluates passive perception of curated clips, while web-browsing agents have expanded from text to static multimodal information. The paper highlights the remaining challenge of actively searching video while balancing context cost and visual detail.

2.1 Video Understanding

  • Video-understanding models and benchmarks commonly query curated clips in closed-world settings, limiting evaluation of external verification and cross-source reasoning.Research-oriented questions can require combining video evidence with external knowledge or searching across multiple videos.Source: §2.1 · p. 2

2.2 Web Browsing Agents

  • Direct video inference incurs prohibitive context costs, while text-centric summaries reduce context but lose fine-grained visual details needed to verify complex queries.Text- and image-based browsing agents have advanced multimodal information seeking, but the cited work identifies a video-specific trade-off.Source: §2.2 · p. 2

3 Benchmark

Video-BrowseComp frames video browsing as open-ended information seeking with multimodal evidence verification. Its design requires video evidence, stages questions by search and reasoning complexity, and uses expert-built, validated questions.

3.1 Task Definition

  • Video browsing is defined as open-ended information seeking in which an agent plans web actions, retrieves video timestamps or reads subtitles, and produces an answer.The task is constrained by multimodal evidence verification.Source: §3.1 · pp. 2–3

3.2 Principles and Scope

  • The benchmark requires video evidence, pairs hard-to-find questions with short verifiable answers, and enforces a single objective answer per question.These principles aim to prevent text-only shortcuts and support unambiguous evaluation.Source: §3.2 · p. 3

3.3 Difficulty Stratification

  • The three difficulty levels progress from explicit retrieval, through indirect video identification, to questions requiring information from multiple sources.At Level 3, evidence from one source is a prerequisite for locating the next.Source: §3.3 · p. 3

3.4 Dataset Construction

  • Annotators explore videos, formulate indirect compositional questions, and construct Level 3 questions from complementary evidence across related videos.They document evidence chains or authoritative source URLs for traceability.Source: §3.4 · p. 3

3.5 Dataset Statistics

  • The final benchmark contains 210 validated questions across eight genres; Film accounts for 29% and Sports 21%, while Level 1 has 125 samples, Level 2 has 62, and Level 3 has 23.Figure 1 presents category distribution on the left and difficulty distribution on the right.Source: §3.5 · p. 4, Figure 1 · p. 4

4 Method

Video-Browser coordinates planning, video observation, and answer synthesis in an iterative framework. Its Pyramidal Perception progressively filters candidates, localizes relevant moments, and decodes selected windows at high fidelity.

4.1 Framework Formulation

  • The Planner decomposes a query into subtasks and searches iteratively, the Watcher returns focused video evidence and observations, and the Analyst synthesizes the final answer from accumulated evidence.When evidence is insufficient or ambiguous, the Planner can refine queries until the subtasks are resolved or the iteration limit is reached.Source: §4.1 · p. 4

4.2 Pyramidal Perception

  • Metadata-only relevance filtering prunes noisy video candidates before visual inference, reserving perceptual resources for promising results.The stage assesses titles and snippets at zero visual inference cost.Source: §4.2 · p. 5
  • Sparse localization combines full transcripts with sampled frames to propose time windows that may contain relevant visual evidence.These focused glimpses give the Planner feedback without fully decoding every video.Source: §4.2 · p. 5
  • Targeted visual decoding reprocesses prioritized windows at high frequency so the Analyst can resolve details missed during sparse scanning.The framework overview identifies the Planner, Watcher, and Analyst and depicts the Watcher’s Pyramidal Perception mechanism.Source: §4.2 · p. 5, Figure 2 · p. 5

5 Experiments

Experiments assess Video-Browser on Video-BrowseComp across performance, efficiency, test-time scaling, and architectural ablations.

5.1 Metrics

  • Overall Accuracy is judged by semantic equivalence to the ground truth, while Calibration Error measures alignment between confidence and empirical accuracy.Lower Calibration Error indicates less overconfidence and a more trustworthy agent.Source: §5.1 · pp. 5–6

5.2 Baselines

  • Baselines comprise tool-free MLLMs using internal knowledge and search-augmented services that search the internet during inference.The evaluation includes model families such as GPT, Gemini, and Qwen.Source: §5.2 · p. 6

5.3 Performance

  • 26.19% accuracy is state of the art; Video-Browser (GPT-5.1) improves 71.8% over Search-Augmented GPT-5.1 (15.24%) and outperforms the o4-mini Deep Research baseline.Its calibration profile is comparable to search-augmented models despite tackling more challenging visual queries.Source: §5.3 · pp. 6–7

5.5 Test-time Scaling

  • Accuracy rises from 10.0% to 14.8% with 1–5 candidate videos, peaks at 19.0% with 5 search loops, and increases from 10.5% to 15.7% with 4–64 Watcher frames.These sweeps vary search breadth, reasoning depth, and perceptual granularity, respectively.Source: Figure 3 · p. 7, §5.5 · p. 7

5.4 Efficiency

  • 58.3% fewer total tokens than Direct Visual Inference (GPT-5.1) accompanies 32,627 tokens and the highest accuracy, 26.19%.Summarization reduces context to 919 tokens, but its total processing cost remains 75,728 tokens and accuracy is 18.57%.Source: §5.4 · p. 7, Table 2 · p. 7

5.7 Qualitative comparison

  • Figure 4 presents a qualitative comparison of three paradigms; the paper also points to additional case studies in Appendix B.3.The implementation details, module prompts, and configurations are provided in Appendices B.1 and B.2.Source: Figure 4 · p. 8, §5.7 · p. 8, §5.8 · p. 8

5.6 Ablation Study

  • Removing Stages 1 or 3 lowers accuracy to 9.52% or 10.48%; removing Stage 2 raises tokens from 19,505 to 44,961 and context about 7.5×; transcript-only accuracy is 7.14% versus 12.86% for the full model.The stages respectively ensure relevance, efficiency, and precision.Source: Table 3 · p. 8, §5.6 · p. 8

5.8 Reproducibility

  • Implementation details, module prompts, and configurations are provided in Appendices B.1 and B.2.The paper directs readers to those appendices for reproducibility information.Source: §5.8 · p. 8

6 Conclusion

The conclusion presents Video-BrowseComp as a benchmark requiring video evidence and Video-Browser as a hierarchical perception agent, reporting improved accuracy with lower token use than direct visual inference.

6 Conclusion

  • Video-BrowseComp enforces mandatory video dependency to evaluate agentic browsing rigorously.Video-Browser uses Pyramidal Perception to hierarchically filter content and address the perception-context dilemma.Source: §6 · p. 8
  • Video-Browser improves accuracy by 37.5% while reducing token consumption by 58.3% versus direct visual inference.The paper presents this as a foundation for verifiable and efficient video agents.Source: §6 · p. 8

Limitations

Video-BrowseComp contains 210 samples, a deliberate scale choice balancing expert annotation rigor with computational accessibility.

Limitations

  • 210 samples make Video-BrowseComp modest in scale relative to traditional single-turn QA datasets.The paper describes this as a deliberate trade-off rather than an accidental shortfall.Source: Limitations · p. 9
  • Expert validation is costly because open-ended answers require checking answer uniqueness.The benchmark prioritizes sample difficulty and quality over magnitude.Source: Limitations · p. 9
  • Agentic video browsing requires multi-step reasoning, web retrieval, and frequent video decoding, making a massive benchmark costly for the research community.The authors position the result as a high-fidelity “Golden Test Set” for affordable comparisons.Source: Limitations · p. 9

Ethical Considerations

The paper describes ethical safeguards and appendix procedures for benchmark construction, evaluation, and category-level analysis.

Ethical Considerations

  • The benchmark uses only publicly available sources, avoids tasks identifying private individuals, and honors takedown requests.Released materials should provide question–answer pairs and pointers rather than host video content.Source: Ethical Considerations · p. 9

Appendix

  • The appendix covers benchmark details and agent details, including samples, annotators, evaluation, prompts, analysis, and case studies.The listed sections include A.1–A.6 and B.1–B.3.Source: Appendix · p. 12

A.1 Benchmark Samples

  • Figure 5 provides examples of benchmark samples.Source: §A.1 · p. 12

A.2 Annotators

  • Eight student volunteers with Master’s or PhD degrees and relevant experience helped curate the benchmark under a two-stage quality-control pipeline.The experience covered LLM usage, web search, and video understanding.Source: §A.2 · p. 12
  • The benchmark annotators gave informed consent for research use and public release of collected data.Source: §A.2 · p. 12

A.3.1 LLM-based Accuracy Judge

  • An automated GPT-5-mini judge checks whether predictions convey the same factual information as short reference answers.The paper motivates this choice because rigid string matching can fail on minor formatting differences.Source: §A.3.1 · p. 12

A.3.2 Calibration Error (CE)

  • Calibration Error averages absolute gaps between empirical accuracy and predicted confidence across five equally spaced bins, weighted by bin size.Each bin’s accuracy and average confidence are compared.Source: §A.3.2 · p. 12

A.4 Evaluation Prompts

  • The appendix includes the evaluation prompts and judge process in Figure 6.Source: §A.4 · p. 12

A.5.2 Cost-Efficiency Analysis (Token Usage)

The cost-efficiency analysis compares accuracy with average token use per query and highlights the trade-off between the two.

A.5.2 Cost-Efficiency Analysis (Token Usage)

  • Figure 9 relates average token usage per query to accuracy, framing their trade-off.The upper-right quadrant is identified as the ideal high-accuracy, low-token-use region.Source: §A.5.2 · p. 13, Figure 9 · p. 13
  • High-performing agents such as o4-mini-deep-research use significantly more tokens, while gpt-4o-mini is token-efficient but less capable on complex video tasks.The paper attributes the higher token use likely to extensive iterative browsing and multi-step reasoning.Source: §A.5.2 · p. 13
  • The stated development target is to move agents toward higher accuracy with lower token usage.Source: §A.5.2 · p. 13

A.6 Benchmark Case Studies

The case studies show that search-augmented models can miss visual evidence when text metadata is absent, conflict with visuals, or span multiple sources.

A.6 Benchmark Case Studies

  • In a double-overtime NBA example, GPT-4o and Gemini-2.5-Pro refuse to identify the Houston Rockets and Oklahoma City Thunder because the game narrative is absent from textual logs.The visual evidence shows both teams, but the specific sequence is not indexed in search results.Source: §A.6 · p. 13
  • For technical fouls and baseline altercations, GPT-5.1 with search defaults to Unknown when textual search lacks temporal grounding.Source: §A.6 · p. 13
  • For a space-module oven display, o4-mini-deep-search reports 190°C from a news report instead of the visually shown 85.The retrieved design specification conflicts with the display’s visual value.Source: §A.6 · p. 13
  • On a query linking Loki, Iron Man, and a food stain, GPT-5.1 identifies a cheeseburger while Gemini-2.5-Pro hallucinates an Avengers: Infinity War scene.The example illustrates difficulty maintaining context across disparate video sources.Source: §A.6 · p. 13

B.1 Experiment Settings

The appendix specifies experimental settings and prompt references, then illustrates Video-Browser’s workflow and compares it qualitatively with direct visual inference and text-centric summarization. The case studies show both successful video-grounded answering and contrasting baseline failures on fine-grained attributes and complex events.

B.1 Experiment Settings

  • The main-results setup uses five search loops, three retrieved videos per iteration, 16 sparse frames, and dense 1-FPS sampling capped at 32 frames per window.Efficiency, scaling, and ablation experiments use different fixed settings to isolate token use or vary video count, search loops, and Watcher frames.Source: §B.1 · pp. 13–14

B.2 Prompts

B.3 Case Studies

  • The case studies cover an end-to-end workflow, comparisons with two baselines, and typical failure cases.The workflow and comparative examples are described in the following subsections.Source: §B.3 · p. 14

B.3.1 End-to-End Workflow Analysis

  • After a broad query returns no results, the Planner grounds the entities as Charles Barkley and Yao Ming, refines its search, and the Watcher selects three relevant videos.The Planner then confirms the answer through sparse scanning and zooms in on relevant timestamps before returning 20 points.Source: §B.3.1 · p. 14

B.3.2 Paradigm Comparison

  • For the pen-cap query, summarization misses the small visual detail and direct inference fails to identify it, while Video-Browser’s zoom-in identifies a red cap.In the basketball case, the baselines fail to match the described sequence, while Video-Browser localizes the 2024 NBA Finals moment and identifies Jayson Tatum.Source: §B.3.2 · pp. 14–15

B.3.3 Failure Case Analysis

The failure analysis identifies errors in distinguishing semantically similar candidates, recognizing attributes of small objects, retrieving absent information, and matching observed actions to the correct entity.

B.3.3 Failure Case Analysis

  • Semantically similar candidates can mislead selection, and tiny-object attributes can be hallucinated; current MLLMs also struggle with zero-shot OCR or logo recognition on low-resolution objects.Examples include confusing fermented shark with the target food and identifying a Coca-Cola can as Dr Pepper despite locating the scene and object.Source: §B.3.3 · p. 15
  • When a statistic is neither spoken in the transcript nor shown in the visual summary, the agent reports it as unstated rather than retrieving external box scores.A separate case shows correct basketball plays attributed to the wrong player, producing a factually correct answer for the wrong entity.Source: §B.3.3 · p. 15
Loading 2512.23044v2…