Source-linked AI summary
VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents
Zhengbo Zhang, Jinbo Su, Zhaowen Zhou, Changtao Miao, Yuhan Hong, Qimeng Wu, Yumeng Liu, Feier Wu, Yihe Tian, Yuhao Liang, Zitong Shan, Wanke Xia, Yi-Fan Zhang, Bo Zhang, Zhe Li, Shiming Xiang, Ying Yan
TL;DR
Existing multimodal browsing benchmarks often under-test visual reasoning or let visual evidence disappear from later search. VisBrowse-Bench combines expert-built, multi-hop image-text tasks with an iterative visual-search workflow; Claude-4.6-Opus achieves the highest reported accuracy, 47.6%.
Problem
Existing benchmarks insufficiently evaluate visual reasoning and often allow text-only browsing after an image query is resolved.
Source: §1 · pp. 1–2Method
VisBrowse-Bench comprises 169 expert-constructed VQA instances, and its agent workflow iteratively retrieves and reasons over visual and textual evidence during search.
Source: §1 · p. 3, §3 · p. 4, §5 · p. 11Results
47.6% accuracy is achieved by Claude-4.6-Opus, the best-performing model on VisBrowse-Bench; the authors also report significant improvement over direct answering with their workflow.
Source: §1 · p. 3Takeaways & Limitations
The benchmark requires visual evidence in the search process and reasoning across multiple images, with required evidence described as publicly accessible.
Source: §3.6 · p. 8In a reported failure case, the model finds key visual evidence but does not extract useful information from it and invokes the visit tool incorrectly.
Source: §5.3.2 · p. 20
Abstract
from arXiv · showhide
The rapid advancement of Multimodal Large Language Models (MLLMs) has enabled browsing agents to acquire and reason over multimodal information in the real world. But existing benchmarks suffer from two limitations: insufficient evaluation of visual reasoning ability and the neglect of native visual information of web pages in the reasoning chains. To address these challenges, we introduce a new benchmark for visual-native search, VisBrowse-Bench. It contains 169 VQA instances covering multiple domains and evaluates the models' visual reasoning capabilities during the search process through multimodal evidence cross-validation via text-image retrieval and joint reasoning. These data were constructed by human experts using a multi-stage pipeline and underwent rigorous manual verification. We additionally propose an agent workflow that can effectively drive the browsing agent to actively collect and reason over visual information during the search process. We comprehensively evaluated both open-source and closed-source models in this workflow. Experimental results show that even the best-performing model, Claude-4.6-Opus only achieves an accuracy of 47.6%, while the proprietary Deep Research model, o3-deep-research only achieves an accuracy of 41.1%. The code and data can be accessed at: https://github.com/ZhengboZhang/VisBrowse-Bench
1. Introduction
Existing multimodal browsing benchmarks often reduce visual queries to tool calls or let downstream search proceed through text alone. VisBrowse-Bench makes visual evidence integral to search and reasoning; the proposed workflow improves performance, though the best model reaches only 47.6% accuracy.
1. Introduction
- Existing benchmarks often test image-query tool invocation or allow text-only downstream browsing, leaving visual reasoning over evidence discovered during search unevaluated.The paper identifies active acquisition and integration of visual information across web pages as a capability needed in real-world retrieval.Source: §1 · pp. 1–2
- 47.6% accuracy is achieved by Claude-4.6-Opus, the best-performing model on VisBrowse-Bench, while most models achieve around 30%.The authors also report that their agentic workflow significantly improves performance over direct answering, while limitations remain.Source: §1 · p. 3, Figure 2 · p. 3
- VisBrowse-Bench targets this gap with 169 expert-constructed VQA instances across seven domains, requiring multimodal evidence integration and visual capabilities for task completion.Its design uses visually ambiguous seed entities and multi-hop chains requiring novel visual evidence alongside complementary but insufficient textual sources.Source: §1 · pp. 2–3
- The paper introduces a multi-turn visual information retrieval-reasoning workflow, and reports significantly better performance than direct answering while noting remaining performance limitations.The workflow encourages agents to actively collect and reason over visual information during search.Source: §1 · p. 3
2. Related Work
Prior multimodal browsing agents add visual capabilities to iterative web search, but benchmarks still lack rigorous evaluation of fine-grained image understanding. VisBrowse-Bench is proposed to address that gap using expert-constructed, publicly available data with visual evidence integral to search.
2.1. Multimodal Browsing Agents
- Earlier LLM browsing agents iteratively retrieved and integrated text but could not process visual content, reducing multimodal web environments to text-only representations.Later MLLM-based systems add multi-turn search, external tools, and image cropping, yet a benchmark dedicated to fine-grained image understanding remains missing.Source: §2.1 · p. 4
2.2. Multimodal Browsing Benchmarks
- Existing benchmarks variously permit reverse-image-search shortcuts, limit sustained multi-hop reasoning, or rely on visual evidence inaccessible through public tools.VisBrowse-Bench is proposed as expert-constructed from publicly available data, with visual evidence incorporated into search to activate visual reasoning.Source: §2.2 · p. 4
3. VisBrowse-Bench
VisBrowse-Bench combines image-text questions, multi-hop visual and textual evidence, expert validation, and an iterative multimodal search workflow. Its design targets visual reasoning throughout search, and its evidence is described as publicly accessible.
3. VisBrowse-Bench
- Questions span seven categories and pair a textual query with images containing initial entity information.The categories are Media, Life, Art, Geography, Technology, Sport, and Finance; two field experts collaboratively obtained each category’s question-answer pairs.Source: §3 · p. 4, Figure 3 · p. 5
3.1. Data Design Principles
- Questions require active processing of visual and textual modalities during query formulation and evidence acquisition, with visual cues structurally indispensable to answering.The design calls for spatial localization, pixel-level attribute analysis, and cross-image reasoning rather than substitutable captions or text matching.Source: §3.1 · p. 5
3.2. Data Collection
- Experts recursively connect entities through events and visual evidence to construct multi-hop chains containing at least two visual evidence blocks.Questions integrate the retrieved evidence and explicitly ground intermediate reasoning in visual and textual blocks.Source: §3.2 · pp. 5–6
3.3. Quality Validation
- Each instance is reviewed by two experts and must span at least three hops and two visual evidence blocks, while two experts independently verify solvability and answer uniqueness.Instances with inconsistent expert answers or critical evidence gaps are flagged for reconstruction or elimination.Source: §3.3 · p. 6
3.4. Dataset Statistics
- The dataset is balanced across seven main categories, covers 24 secondary categories, and has an average question length of 47.7 words.Each instance contains at least one image as a visual query.Source: Figure 4 · p. 6, §3.4 · p. 7
3.5. Agentic Framework
- The agent iteratively searches, reasons over returned evidence, and refines queries using text search, image search, reverse image search, image cropping, and webpage browsing.This closed loop interleaves visual and textual evidence acquisition until the agent has sufficient evidence to synthesize an answer.Source: §3.5 · p. 7
3.6. Discussion
- VisBrowse-Bench incorporates visual evidence into search and reasoning across multiple images; experts verified its data, and required search evidence is publicly accessible.The paper presents these as advancements for evaluating multimodal reasoning and visual-aware search.Source: §3.6 · p. 8
4. Experiments
The experiments compare models across progressively richer tool access, then analyze benchmark accuracy and tool-use patterns. Image search improves performance for models that use it correctly, while tool preferences vary across models.
4.1. Experimental Setups
- The evaluation includes closed-source, open-source, and Deep Research models, tested with direct answering, text search, or all framework tools.Accuracy is the metric; final answers are extracted by regular-expression matching and judged against ground truth using GPT-5.1.Source: §4.1 · p. 8
4.2. Results and Analysis
- 47.6% accuracy was achieved by the best-performing model, while the top open-source model reached 14.2%.Table 2 reports accuracy across models and the Direct answer, + TS, and + IS methods.Source: Table 2 · p. 9, §4.2 · p. 10
- Under direct answering, all models perform poorly, indicating that parametric knowledge is insufficient for complex queries requiring dynamically acquired evidence.Source: §4.2 · p. 9
- Text search improves all models to varying degrees, but gains are limited for most because visual evidence is not readily obtained through text retrieval alone.Source: §4.2 · p. 10
- Image search further improves model performance, but Claude-4.6-Sonnet’s incorrect image-search use causes framework failure and lower accuracy.The authors attribute the framework’s lower accuracy for Claude-4.6-Sonnet to its inability to use image-search tools correctly.Source: §4.2 · p. 10
4.3. Analysis of Tool Usage
- Claude-4.6-Opus uses the five tools relatively evenly and achieves the best performance; Kimi-K2.5 uses image search more often than Gemini-3.0-Pro and GPT-5.2.The authors report that Kimi-K2.5’s more frequent image_search use enables it to uncover more visual information through text queries and yields the greatest performance improvement under + IS.Source: §4.3 · p. 10
5. Conclusion
The conclusion identifies gaps in existing multimodal browsing benchmarks and presents VisBrowse-Bench and an agent workflow designed to evaluate search and visual reasoning together.
5. Conclusion
- Existing benchmarks inadequately test visual reasoning and overlook multimodal information integration during the reasoning process.Source: §5 · p. 10
- VisBrowse-Bench comprises 169 human-expert-constructed questions across seven categories, requiring multimodal fusion during search and visual understanding for task completion.Source: §5 · p. 11
- The agentic workflow uses text search, image search, reverse image search, image cropping, and webpage browsing to encourage active collection and reasoning over visual content.Source: §5 · p. 11
Appendix
The appendix documents the agent prompts and tool-use rules, analyzes tool-use patterns, and presents successful and failed visual-search trajectories, including a case where retrieved page content did not establish the visual detail in the answer.
5.1. Prompts of agentic framework
- The agent prompt directs the model to decompose visual questions, describe images in detail, explain tool calls, and return an answer within 15 turns.It also prescribes iterative tool-use cycles, image search for additional visual evidence, cropping to inspect regions, one tool call per round, and concise answers inside <answer> tags.Source: §5.1 · p. 14
5.2. Details of tool usage
- The benchmark requires iterative use of text, image, reverse-image, crop, and webpage tools because no single tool or modality suffices to resolve its multimodal, multi-hop questions.Table 3 reports tool-use frequencies and the accompanying analysis says strategic signatures correlate with performance outcomes; successful resolution requires refining visual hypotheses, verifying provenance, and corroborating with text.Source: §5.2 · p. 15, Table 3 · p. 15
5.3.1. Analysis of successful case
- Claude-4.6-Opus answered the Media-category case correctly as Tom Conti by cross-validating evidence gathered with five tools.The trajectory links the image’s upper-right person to Robert Downey Jr., then identifies the darkly dressed character in the Oppenheimer lake scene as Einstein, played by Tom Conti.Source: §5.3.1 · pp. 16, 20
5.3.2. Analysis of failed case
- In the failed case, Claude-4.6-Opus found key visual evidence but did not use visual reasoning to extract useful information, instead invoking webpage visit to analyze the image.The question asks for the tie color worn by a brand ambassador in a 2025 world-tour poster, but the trajectory shown does not reach an answer.Source: §5.3.2 · p. 20
- Although the page summary said it could not determine whether Jay Chou wore a tie or its color, the agent inferred blue from the poster’s blue theme and answered “blue.”The agent had searched for the 2025 Carnival World Tour poster, but the retrieved description mentioned only a blue brocade jacket and no tie.Source: §5.3.2 · p. 25