Source-linked AI summary
BrowseComp-$V^3$: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents
Huanyao Zhang, Jiepeng Zhou, Bo Li, Bowen Zhou, Yanzhe Shan, Haishan Lu, Zhiyong Cao, Jiaoyang Chen, Yuqian Han, Zinan Sheng, Zhengwei Tao, Hao Liang, Jialong Wu, Yang Shi, Yuanpeng He, Jiaye Lin, Qintong Zhang, Guochen Yan, Runhao Zhao, Zhengpin Li, Xiaohan Yu, Lang Mei, Chong Chen, Wentao Zhang, Bin Cui
TL;DR
Existing multimodal browsing benchmarks provide limited task complexity, evidence accessibility, and evaluation granularity, motivating a more comprehensive assessment of deep search. The paper introduces BrowseComp-V3 and OmniSeeker, and reports that state-of-the-art MLLMs remain below 40% Success Rate while humans achieve 68.03%.
Problem
Existing multimodal browsing benchmarks inadequately cover task complexity, publicly accessible evidence, and systematic process evaluation, limiting fair and reproducible assessment.
Method
BrowseComp-V3 combines deep cross-modal multi-hop tasks, publicly searchable evidence, expert-validated sub-goals, and annotated trajectories, alongside OmniSeeker’s integrated search and visual tools.
Results
Under 40% Success Rate for state-of-the-art MLLMs, compared with humans’ 68.03% average Success Rate on BrowseComp-V3.
Takeaways & Limitations
The benchmark exposes substantial current-model gaps in multimodal deep search and identifies multimodal information integration as a critical deficiency.
Takeaways & Limitations
The benchmark prioritizes temporally invariant, objective knowledge to reduce fluctuations from dynamic web content.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs), equipped with increasingly advanced planning and tool-use capabilities, are evolving into autonomous agents capable of performing multimodal web browsing and deep search in open-world environments. However, existing benchmarks for multimodal browsing remain limited in task complexity, evidence accessibility, and evaluation granularity, hindering comprehensive and reproducible assessments of deep search capabilities. To address these limitations, we introduce BrowseComp-$V^3$, a novel benchmark consisting of 300 carefully curated and challenging questions spanning diverse domains. The benchmark emphasizes deep, multi-level, and cross-modal multi-hop reasoning, where critical evidence is interleaved across textual and visual modalities within and across web pages. All supporting evidence is strictly required to be publicly searchable, ensuring fairness and reproducibility. Beyond final-answer accuracy, we incorporate an expert-validated, subgoal-driven process evaluation mechanism that enables fine-grained analysis of intermediate reasoning behaviors and systematic characterization of capability boundaries. In addition, we propose OmniSeeker, a unified multimodal browsing agent framework integrating diverse web search and visual perception tools. Comprehensive experiments demonstrate that even state-of-the-art models achieve only 36% accuracy on our benchmark, revealing critical bottlenecks in multimodal information integration and fine-grained perception. Our results highlight a fundamental gap between current model capabilities and robust multimodal deep search in real-world settings.
1 Introduction
BrowseComp-V3 addresses limitations in multimodal browsing benchmarks by combining complex cross-modal deep search, publicly searchable evidence, and process-oriented evaluation. It also introduces OmniSeeker, a tool-integrated browsing framework for these tasks.
- Motivation: Existing benchmarks often use shallow two-hop retrieval, inaccessible evidence, and final-answer-only evaluation, limiting fair and reproducible capability assessment.These shortcomings hinder diagnosis of failure modes and delineation of model capability boundaries.
- Benchmark Design: BrowseComp-V3 contains 300 challenging questions spanning 24 sub-domains and requiring deep, multi-level, cross-modal reasoning across web pages.Critical evidence is interleaved across textual and visual modalities, limiting text-only shortcuts and reliance on parametric knowledge.
- Benchmark Design: All critical evidence is publicly searchable, with manually annotated gold-standard trajectories to support fairness and reproducibility.The benchmark also provides expert-validated intermediate sub-goals for fine-grained process evaluation.
- Evaluation: Process-oriented evaluation systematically characterizes multimodal browsing agents’ capabilities and limitations beyond final-answer accuracy.This mechanism supports more comprehensive analysis of intermediate reasoning behavior.
- Agent Framework: OmniSeeker integrates diverse web search and visual perception tools in a unified multimodal browsing agent framework.The framework is reported to rival state-of-the-art closed-source systems and improve open-source model performance.
2 Related Work
Related work shows that multimodal browsing agents build on advances in MLLMs, tool augmentation, and open-world search, while existing multimodal benchmarks remain comparatively simple. BrowseComp-V3 is positioned within this shift toward deeper, integrated browsing tasks.
- Multimodal Large Language Models: MLLMs support visual question answering, grounding, OCR, and multimodal reasoning, but struggle to acquire up-to-date information in knowledge-intensive queries.Tool-augmented frameworks address this by enabling dynamic retrieval and incorporation of external knowledge.
- Tool-Augmented Browsing Agents: Tool-enhanced browsing agents increasingly use supervised fine-tuning and reinforcement learning to improve retrieval, reasoning, and decision-making in dynamic web environments.
- Multimodal Browsing Benchmarks: Traditional multimodal browsing benchmarks decouple visual understanding from text retrieval and focus on simple two-hop tasks, where performance has largely saturated.BrowseComp provides a text-only open-world navigation paradigm that informs multimodal task construction.
3 The BrowseComp-V3 Dataset
BrowseComp-V3 is constructed through explicit design principles and a multi-stage quality-control pipeline to support complex, reproducible, process-level multimodal search evaluation. Its questions cover balanced domains and multiple cross-modal and task-complexity dimensions.
- Design Principles: The dataset follows three principles: increased task complexity, process-oriented granular evaluation, and high reliability and reproducibility.These principles target limitations in benchmark complexity, evaluation dimensions, and evidence accessibility.
- Design Principles: Search depth is extended through multi-hop variation, while cross-modal interaction is organized into intra-region alignment, inter-region integration, and inter-image reasoning.
- Design Principles: Expert-validated sub-goals track intermediate evidence acquisition and support diagnosis of failure modes and model capability boundaries.
- Data Construction Pipeline: The construction pipeline has five stages, beginning with expert guidelines and exemplars and proceeding through exploratory annotation, verification, formatting, and expert quality control.Initial exemplars include visual inputs, queries, sub-goals, answers, and metadata.
- Data Construction Pipeline: Dual verification combines replicated human trajectory checks with SOTA multimodal-model filtering to retain samples requiring long-tail knowledge or complex reasoning.
- Dataset Statistics: BrowseComp-V3 uses five balanced categories—Science, Technology, Society, Culture, and Life—and reports additional statistics on task levels and difficulty distributions.
4 Experiments
The experiments evaluate humans, tool-free and tool-augmented MLLMs, and OmniSeeker using result-level Success Rate and process-level Process Score. Results show a substantial human–model gap, benefits from tool augmentation and OmniSeeker, and the value of process-level evaluation.
- Experimental Setup: The benchmark evaluates four settings: human participants, tool-free MLLMs, tool-augmented MLLMs, and models using OmniSeeker.The unified protocol includes standardized browsing tools for OmniSeeker and separate tool-access conditions for MLLMs.
- Evaluation Metrics: Success Rate measures task completion, while Process Score measures the proportion of critical sub-goals completed during multi-step search and reasoning.Process Score uses ground-truth sub-goals required for each problem and the sub-goals achieved by the model or human.
- Main Results: 68.03% average Success Rate and 82.93% Process Score are achieved by humans, whereas no model exceeds 40% Success Rate.The gap is reported as evidence of benchmark difficulty and current model limitations in multimodal deep search.
- Main Results: Approximately 10% Success Rate is achieved by most models without tools, while tool augmentation substantially improves performance.The results indicate that parameterized knowledge alone does not adequately capture dynamic, cross-modal evidence chains on the open web.
- Main Results: OmniSeeker consistently improves all evaluated models, bringing performance to levels comparable with specialized proprietary systems.The framework provides a unified tool-calling setup integrating standardized search and visual perception tools.
- Main Results: Process Score typically exceeds Success Rate, indicating that models often complete individual sub-goals but fail to maintain logical consistency across long-sequence tasks.Fine-grained process evaluation is used to identify where and why models fail and to reveal capability boundaries.
5 Further Analysis
Further analysis shows that increasing task complexity exposes weaknesses in inter-region and inter-image reasoning, while multimodal integration, visual grounding, and long-horizon planning remain key bottlenecks. Additional interaction steps and consistent sampling improve performance, especially for larger models.
- Task Level: Performance declines substantially from Level 1 to Levels 2 and 3, revealing challenges in inter-region integration and inter-image relational reasoning.
- Search Depth: Human and model success rates decline with increasing search depth, but models degrade more gradually than humans.The authors attribute this discrepancy to models using internalized parametric knowledge to compensate for search complexity.
- Ability Boundaries: Multimodal integration is the primary bottleneck for models, whereas humans are mainly limited by TextSearch-related attention span and cognitive load.
- Test Time Scaling: Increasing the maximum number of interaction turns substantially improves performance, with Qwen3-VL-235B scaling more strongly than its 8B counterpart.The authors link this advantage to larger models’ stronger long-horizon reasoning and ability to use extra turns for iterative refinement.
- Test Time Scaling: Best-of-N scales most effectively among the three sampling strategies, continuously improving Qwen3-VL-235B performance as the number of independent samples increases.
- Failure Mode Analysis: Visual Grounding and Perception Failure dominate error distributions across four representative models, indicating difficulty extracting visual information from noisy web environments.
- Failure Mode Analysis: Closed-source models reduce perception and grounding errors, but long-horizon planning becomes the main bottleneck as multimodal capabilities improve.
6 Conclusion
BrowseComp-V3 is a benchmark for multimodal deep browsing and search that addresses limitations in task complexity, information searchability, and evaluation dimensions. Results show that leading MLLMs achieve under 40% success rate, while process evaluation and test-time scaling provide potential improvement pathways and OmniSeeker performs comparably to leading closed-source models.
- BrowseComp-V3 evaluates multimodal deep browsing with 300 curated and annotated questions targeting task complexity, information searchability, and evaluation dimensions.
- SOTA MLLMs achieve under 40% SR on BrowseComp-V3, revealing a substantial gap relative to human performance.
- Process-level evaluation and test-time scaling offer potential pathways for enhancing model capabilities, including through methodologies such as reinforcement learning.
- OmniSeeker achieves performance comparable to leading closed-source models, providing an open alternative for developing multimodal browsing agents.