Source-linked AI summary
MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
Shilong Li, Xingyuan Bu, Wenjie Wang, Jiaheng Liu, Jun Dong, Haoyang He, Hao Lu, Haozhe Zhang, Chenchen Jing, Zhen Li, Chuanhao Li, Jiayi Tian, Chenchen Zhang, Tianhao Peng, Yancheng He, Jihao Gu, Hui Huang, Donghao Zhou, Yuanxing Zhang, Jian Yang, Ge Zhang, Wenhao Huang, Zhaoxiang Zhang, Qiangpeng Yang, Shilei Wen
TL;DR
Text-focused browsing benchmarks leave visual queries and multimodal web evidence underrepresented. MM-BrowseComp introduces 400 challenging questions with verified reasoning checklists, and leading agents achieve low accuracy, including 24.25% for GPT-5-High with tools.
Problem
Text-focused benchmarks such as BrowseComp overlook visual queries and the need to extract evidence from multimodal web content.
Source: §1 · p. 1Method
MM-BrowseComp is a benchmark of 400 challenging hand-crafted questions with visual prompts, multimodal web evidence, and verified reasoning checklists.
Source: §1 · pp. 1–2Results
24.25% overall accuracy for GPT-5-High and 18.75% for Gemini-3-Pro with tools; all evaluated open-source agents score below 10%.
Source: §1 · p. 2Takeaways & Limitations
The conclusion identifies native multimodal reasoning as a critical driver for success on MM-BrowseComp.
Source: §5 · p. 9Reliance on human expert annotations for quality assurance limits benchmark scale.
Source: §6 · p. 9
Abstract
from arXiv · showhide
AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However, existing benchmarks such as BrowseComp primarily focus on textual content, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 400 challenging, hand-crafted questions designed to evaluate multimodal retrieval and reasoning capabilities. Unlike prior work, MM-BrowseComp incorporates visual prompts and necessitates the extraction of key evidence from web images and videos to complete questions, rendering text-only approaches insufficient. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of 27 state-of-the-art models reveals that even leading models like GPT-5-High with tools achieve only 24.25\% accuracy, highlighting the suboptimal multimodal browsing capabilities, establishing MM-BrowseComp as a rigorous new standard for the field.
1 Introduction
MM-BrowseComp addresses the gap left by text-focused browsing benchmarks with challenging multimodal questions and verified reasoning checklists. Evaluations find low accuracy overall, weaker performance on visual web evidence, and distinct capabilities among agent designs.
1 Introduction
- Existing BrowseComp (Wei et al., 2025) focuses on textual questions, overlooking visual queries and evidence embedded in multimodal web content.The paper motivates a framework for evaluating multimodal browsing capabilities.Source: §1 · p. 1
- MM-BrowseComp comprises 400 hand-crafted questions whose solutions require interleaved retrieval and reasoning over visual web evidence, with a verified checklist for each question.The checklist supports analysis beyond final-answer correctness.Source: §1 · pp. 1–2
- 24.25% overall accuracy for GPT-5-High and 18.75% for Gemini-3-Pro with tools; all evaluated open-source agents remain below 10%.The results characterize MM-BrowseComp as challenging for current agents.Source: §1 · p. 2
- Fine-grained checklist evaluation finds that models perform worse on image and video content than on text content from the web.Source: §1 · p. 2
- GPT-5-High directly processes images as a native multimodal agent, whereas current open-source agents primarily rely on external tools to understand them.Source: §1 · p. 2
- Agents using reflection and ReAct (Yao et al., 2022) outperform orchestrated agents by reducing reliance on sub-agent outputs and correcting errors.Source: §1 · p. 2
- Advanced closed-source agents show test-time scaling, while open-source models do not yet show this phenomenon and need stronger foundational capabilities.Source: §1 · p. 2
2 Related Works
The related-work section traces browsing agents’ development alongside benchmarks for browsing performance. It identifies a gap in prior benchmarks’ coverage of interleaved multimodal reasoning.
2 Related Works
- Browsing agents combine LLM or VLM capabilities with retrieval-augmented generation or web tools, while reinforcement learning is increasingly used to strengthen reasoning.Source: §2 · p. 2
- Many earlier browsing benchmarks feature easily retrievable information, while BrowseComp (Wei et al., 2025) tests retrieval across hundreds of web pages.Source: §2 · p. 3
- BrowseComp and derivatives focus on text or replace text-query entities with images, leaving interleaved multimodal reasoning—thinking with images—uncovered.The paper presents MM-BrowseComp as evaluating scenarios where inputs, reasoning, and answers require multimodal capabilities.Source: §2 · p. 3
3 MM-BrowseComp
MM-BrowseComp is constructed to test difficult multimodal browsing with verified evidence paths and answer criteria. Its three-stage validation filters candidate questions before finalizing a 400-question benchmark spanning five categories and 22 subtasks.
3.1 Data Construction Criteria
- The benchmark’s construction combines design principles for multimodal evaluation with standards intended to ensure robustness and integrity.Source: §3.1 · p. 3
3.1.1 Core Design Principles
- Essential evidence must primarily reside in images or videos and be absent from text sources; curation and validation check for at least one such evidence item and textual shortcuts.Source: §3.1.1 · p. 3
- Each question includes a fine-grained checklist of key evidence in its retrieval and reasoning path, helping distinguish methodical reasoning from lucky guessing.Source: §3.1.1 · p. 3
3.1.2 Foundational Quality Standards
- Questions must resist single-attempt solving by strong VLMs and five minutes of searching by an unfamiliar human annotator.The construction criteria identify Gemini-2.5-Pro and GPT-4o for the VLM check.Source: §3.1.2 · p. 3
- Answers are concise and objectively verifiable, should remain temporally stable, and are refined to isolate a unique valid answer.Annotators check for alternatives and tighten question constraints when multiple answers are possible.Source: §3.1.2 · p. 4
3.2 Validation
- Validation proceeds through pilot calibration, full-scale construction with secondary review, and tool-dependency screening followed by factual verification.Source: §3.2 · p. 4
- 400 questions remain after a three-stage validation process accepted, revised, or discarded the initial 550 candidates.263 candidates were accepted directly, 137 required revision, and 150 were discarded.Source: §3.2 · p. 5
3.3 Dataset Statistics
- The dataset contains 400 questions across five categories and 22 subtasks, with balanced task distribution and checklist items from multiple modalities in every question.Figure 2 presents task distribution and detailed benchmark statistics.Source: Figure 2 · p. 4, §3.3 · p. 5
4 Experiments
The experiments evaluate 27 models using answer correctness and checklist-based metrics, and find that performance varies with browsing access, modality, agent design, and test-time scaling.
4.1 Experimental Setup
- The evaluation covers 27 models across tool-free VLMs, tool-augmented VLMs, and open-source agents, using OA, SA, and AVG CS.SA requires both a correct final answer and completion of every associated checklist item; AVG CS measures average checklist completion.Source: §4.1 · p. 5
4.2 Main Results
- All tool-free VLMs achieve at most 10% OA, while GPT-5-High leads across evaluated models; its advantage is attributed largely to interleaved multimodal reasoning and function calls.Agent-R1 leads the open-source agents, particularly on AVG CS; the authors associate its relative advantage with its reflective, unified ReAct architecture.Source: §4.2 · pp. 5–6, Table 2 · p. 6
- Agent-R1 performs best among the evaluated open-source agents, particularly on AVG CS, and its unified thought-action-observation loop is described as more robust than orchestrated frameworks prone to subagent failure.The authors suggest its reflective architecture may account for this relative advantage.Source: §4.2 · pp. 6–7, Table 3 · p. 7
4.3 Modality-Specific Performance Analysis
- Most models perform better on textual than visual checklist items, with the visual gap attributed to acquisition, perception, tool, and reasoning-intent challenges.GPT-5-High directly loads images into its input after downloading them, unlike agents relying on external captioning tools.Source: §4.3 · p. 7
4.4 Test Time Scaling
- Gemini-3-Pro improves both OA and SA with multiple runs, whereas Agent-R1 gains only marginally in OA and shows no SA improvement.The authors interpret the latter pattern as evidence that Agent-R1's correct answers may be lucky guesses, and say its foundational capabilities need improvement.Source: Figure 3 · p. 7, §4.4 · p. 8
4.5 Failure Modes Analysis
- As closed-source models evolve, failure shifts from tool execution toward reasoning and visual hallucination; agent design also affects error patterns among open-source frameworks.Gemini-3-Pro has 0.6% tool-execution failures, while OWL has a 27.8% visual-hallucination rate despite sharing a Gemini-2.5-Flash reasoning backbone with the compared frameworks.Source: §4.5 · p. 8, Figure 4 · p. 8
5 Conclusion
The paper introduces MM-BrowseComp to evaluate deep reasoning in multimodal web browsing. Its experiments find that state-of-the-art models still struggle, while identifying native multimodal reasoning as a critical driver of success.
5 Conclusion
- MM-BrowseComp is a benchmark for evaluating the integration of deep reasoning with complex multimodal web browsing.It contains 400 human-annotated questions verified through a three-stage process for challenge and verifiability.Source: §5 · p. 9
- Experiments show that state-of-the-art models struggle with MM-BrowseComp tasks and identify native multimodal reasoning as a critical driver of success.The results point to opportunities to improve foundational and reasoning capabilities.Source: §5 · p. 9
- The authors expect MM-BrowseComp to catalyze research toward agents capable of navigating the complex multimodal web.Source: §5 · p. 9
6 Limitations
The authors identify limits in MM-BrowseComp's topic coverage, release security, and annotation-based scale.
6 Limitations
- Coverage of five major topics and 22 subtasks may need expansion as new multimodal browsing scenarios emerge.Source: §6 · p. 9
- Annotator subjectivity and possible data exposure after release may lead to leaks or shortcuts despite verification and encryption.Source: §6 · p. 9
- Reliance on human expert annotations for quality assurance limits the benchmark's scale.The authors plan periodic updates and automated synthesis to address these issues.Source: §6 · p. 9
A Distribution of Checklist Items by Modality
Checklist items span text, images, and video, with visual evidence making up more than half of the dataset and modality mix varying by category.
A Distribution of Checklist Items by Modality
- For 400 questions, checklist sources are Text (43.17%), Image (41.23%), and Video (15.60%); visual evidence accounts for 56.83% of items.The authors say this prevents high checklist scores through text-only reasoning alone.Source: A Distribution of Checklist Items by… · p. 12
- Technology and Geography have image-heavy checklists, Media has more video items, and Academics and Society have higher text proportions.The heterogeneous mix makes checklist metrics sensitive to capabilities across modalities and domains rather than specialization in one information type.Source: A Distribution of Checklist Items by… · p. 12
B.1 Implementation Details
The paper specifies model and agent configurations, a shared checklist-based evaluation protocol, and evaluator robustness checks. Human annotators also provide a reference point for benchmark difficulty.
B.1 Implementation Details
- Tool-free VLMs were evaluated without external tools, while tool-augmented VLMs used their respective official tools; GPT-5-High denotes GPT-5 evaluated with the reasoning_effort parameter set to high.Agent-R1 was equipped with a search engine, web browser, and analyzers for images, videos, and PDFs.Source: §B.1 · p. 12
- Table 4 organizes checklist items by required modality and sample counts, with percentages calculated row-wise.Source: Table 4 · p. 13
B.2 Evaluation
- A universal instruction asked agents to outline a roadmap, and an AI evaluator judged final-answer correctness and checklist fulfillment; GPT-4o-2024-11-20 was the main judge.The instruction template is shown in Table 5.Source: §B.2 · p. 13, Table 5 · p. 13
B.3 Evaluator Robustness
- Gemini-2.5-Flash judging produced slightly higher scores than GPT-4o judging, but rankings and primary conclusions remained unchanged.Table 6 reports Pass@1 evaluations judged by Gemini-2.5-Flash.Source: §B.3 · p. 13, Table 6 · p. 14
C Human Performance
- 59.50% Overall Accuracy, 49.75% Strict Accuracy, and 8.1 minutes average cost time were recorded for human annotators using standard browsing without AI assistants.The gap between Overall Accuracy and Strict Accuracy indicates that completing every checklist item remained demanding even when the final answer was correct.Source: C Human Performance · pp. 13–14
D Data Release
The paper releases the full MM-BrowseComp dataset to support future research and reproducibility, while describing measures intended to protect benchmark integrity and reduce data leakage.
D Data Release
- The full MM-BrowseComp dataset is released to facilitate future research and reproducibility.Source: D Data Release · p. 14
- Researchers and users are asked not to share dataset examples online in plain text or image format to maintain benchmark integrity and prevent data leakage.Source: D Data Release · p. 14
- The dataset is encrypted and embeds a unique canary string so developers can filter it from training corpora and prevent direct retrieval of ground-truth answers.Source: D Data Release · p. 14
E Failure Analysis
The error analysis finds distinct failure patterns across model-agent pairings and relates them to both backbone behavior and agent-system design.
E Failure Analysis
- GPT-4o-2024-11-20 showed a high proportion of Tool Execution Failure, attributed to frequent, proactive tool calls rather than necessarily weak tool-use capability.Source: E Failure Analysis · p. 14
- Gemini-2.5-Flash errors were often Incorrect Reasoning, reflecting logical errors during its reasoning process.Source: E Failure Analysis · p. 14
- Guessing or Fabrication was the most common failure mode for Qwen2.5-VL-72B-Instruct, suggesting a propensity to hallucinate.Source: E Failure Analysis · p. 14
F Reasoning Capability Analysis
The analysis characterizes the reasoning steps used by tool-augmented models and illustrates how an incorrect intermediate identification can coexist with a correct final answer.
F Reasoning Capability Analysis
- GPT-4o judged trajectories for tool-augmented GPT-5-High, Gemini-3-Pro, and Gemini-2.5-Pro, assigning each reasoning step to one of four primary categories.The categories were Causal Reasoning, Comparison, Abstraction, and Hypothesis Revision.Source: F Reasoning Capability Analysis · pp. 14–15
- 57.1–63.2% of reasoning steps were Causal Reasoning and 18.6–29.5% Comparison; Abstraction (7.9–13.1%) and Hypothesis Revision (4.9–11.3%) also appeared.The authors conclude that the benchmark calls for a combination of reasoning capabilities rather than one isolated skill.Source: Table 8 · p. 15, F Reasoning Capability Analysis · p. 15
- GPT-5-High used a higher proportion of Comparison than the Gemini series, while Gemini-2.5-Pro devoted 11.3% of its reasoning resources to Hypothesis Revision.The authors suggest comparison may contribute to GPT-5-High's leading Overall Accuracy, while frequent revision may indicate backtracking among weaker models.Source: F Reasoning Capability Analysis · p. 15
G Case Study
- The agent identified Pale-Axe Stalwart but mistook the Rusty-Gold Cart for a Brown Iron Cart; its final “two wheels” answer was correct despite unsound reasoning.The case is scored correct for Overall Accuracy but incorrect for Strict Accuracy.Source: Table 10 · p. 18, Table 11 · p. 19, Table 12 · p. 20