PaperlayerView exact revision on arXiv

Source-linked AI summary

MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents

Xijia Tao, Yihua Teng, Xinxing Su, Xinyu Fu, Jihao Wu, Chaofan Tao, Ziru Liu, Haoli Bai, Rui Liu, Lingpeng Kong

arXiv:2508.21475v3cs.AI

TL;DR

Existing multimodal browsing benchmarks often allow text-only shortcuts instead of sustained visual reasoning. MMSearch-Plus introduces a 311-task benchmark and agent framework built around spatial-temporal extrapolation, iterative retrieval, provenance verification, and SoM-based zooming; the strongest system reaches 36.0% accuracy, while SoM adds gains up to +3.9 points. The paper also reports dataset and domain limitations that bound how broadly the findings should be generalized.

  • Problem

    Existing multimodal browsing benchmarks rarely require sustained fine-grained visual reasoning, long-horizon tool use, or robust multimodal verification.

    Source: §1 · pp. 1–2

  • Method

    MMSearch-Plus constructs questions requiring spatial-temporal extrapolation to out-of-image facts and evaluates agents using iterative image-text search with Set-of-Mark zoom-in.

    Source: §1 · pp. 1, 3, §2.1 · p. 3, §2.5 · p. 5

  • Results

    The strongest agent achieves 36.0% accuracy, while adding SoM yields consistent gains up to +3.9 points.

    Source: §4 · p. 7, §7 · p. 10

  • Takeaways & Limitations

    MMSearch-Plus establishes a rigorous stress test showing that current multimodal agents still struggle with fine-grained visual retrieval and provenance-aware browsing.

    Source: §1 · p. 3, §7 · p. 10

    The data inherit public-web and domain-sampling biases, underrepresent non-English or low-visual-density pages, and focus mainly on image-text browsing.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Existing multimodal browsing benchmarks often fail to require genuine multimodal reasoning, as many tasks can be solved with text-only heuristics without vision-in-the-loop verification. We introduce MMSearch-Plus, a 311-task benchmark that enforces multimodal understanding by requiring extraction and propagation of fine-grained visual cues through iterative image-text retrieval and cross-validation under retrieval noise. Our curation procedure seeds questions whose answers require extrapolating from spatial cues and temporal traces to out-of-image facts such as events, dates, and venues. Beyond the dataset, we provide a model-agnostic agent framework with standard browsing tools and a set-of-mark (SoM) module, which lets the agent place marks, crop subregions, and launch targeted image/text searches. SoM enables provenance-aware zoom-and-retrieve and improves robustness in multi-step reasoning. We evaluated closed- and open-source MLLMs in this framework. The strongest system achieves an end-to-end accuracy of 36.0%, and integrating SoM produces consistent gains in multiple settings, with improvements up to +3.9 points. From failure analysis, we observe recurring errors in locating relevant webpages and distinguishing between visually similar events. These results underscore the challenges of real-world multimodal search and establish MMSearch-Plus as a rigorous benchmark for advancing agentic MLLMs.

1 INTRODUCTION

MMSearch-Plus addresses the limited visual and long-horizon reasoning demanded by existing multimodal browsing benchmarks. It introduces a benchmark and agent framework requiring localized cue extraction, noisy retrieval verification, and sustained multimodal tool use.

Benchmark design

  • MMSearch-Plus requires localized visual reasoning, robust verification under noisy retrieval, and multi-step interleaving of text and image search.Its tasks are designed so a single strong image search cannot bypass the visual reasoning requirements.Source: §1 · pp. 1–2
  • Spatial–Temporal Extrapolation asks models to transform localized spatial and temporal cues into out-of-image facts such as dates, events, or locations.The procedure targets subtle signals including micro-text, broadcast overlays, and layout signatures.Source: §1 · pp. 1–3

Agent framework

  • The model-agnostic framework combines visual tools and search engines with Set-of-Mark zoom-in reasoning for region-seeded retrieval.SoM supports cropping, OCR, and targeted searches while enabling analysis of localized visual extraction.Source: §1 · pp. 1, 3, Figure 1 · p. 2

Motivation

  • Existing multimodal browsing benchmarks often permit text-only heuristics because image search results can expose answer-bearing webpage text.This reduces multimodality to narrow image–source cross-validation rather than sustained visual reasoning.Source: §1 · p. 2

Contribution

  • MMSearch-Plus provides a high-difficulty benchmark and toolkit focused on sustained fine-grained visual reasoning rather than shortcut-based browsing.It differs from concurrent benchmarks in its data sources and agent framework emphasis.Source: §1 · p. 3

2 METHOD

The method constructs multimodal questions whose answers require spatial-temporal extrapolation beyond the visible frame, then evaluates agents using standardized browsing infrastructure and SoM-based visual grounding.

2.1 SPATIAL-TEMPORAL EXTRAPOLATION

  • Spatial–Temporal Extrapolation requires agents to infer unseen entities or adjacent events, localize the source event, and integrate evidence from diverse sources.Difficulty comes from expanding candidate search spaces and iteratively refining hypotheses.Source: §2.1 · p. 3

2.2 DATA COLLECTION

  • The dataset combines manually selected video keyframes and paper figures or tables, filtering for vague, unfamiliar, and non-directly-searchable visual content.Sensitive content is anonymized, and videos and papers are treated as events.Source: §2.2 · pp. 3–4
  • Adversarial filtering removes questions solvable without retrieval through cross-validation, image perturbation, and iterative refinement.The benchmark is regularly refreshed because model knowledge can make previously retrieval-dependent tasks solvable from memory.Source: §2.2 · p. 4

2.3 DATASET STATISTICS

  • MMSearch-Plus is approximately balanced across eight categories, uses short-answer questions, and labels tasks Easy or Difficult according to whether search is needed.Answers average 3.7 words.Source: §2.3 · pp. 4–5, Figure 3 · p. 4

2.4 BENCHMARK METRIC AND EVALUATION

  • Accuracy is the primary metric, with LLM-as-a-judge matching predictions against acceptable answers and manual validation agreeing completely with GPT-4o.The benchmark supplies acceptable answer variants for each question.Source: §2.4 · p. 5

2.5 SEARCH AGENT FRAMEWORK

  • The search agent interleaves text search, image search, and zooming while maintaining cached summaries and threaded context across long-horizon interactions.SoM presents human-verified bounding boxes with indices and supports precise region inspection.Source: §2.5 · p. 5

3 EXPERIMENT

The experiment compares closed- and open-source MLLMs under standardized no-search, image-search, text-search, full-rollout, and SoM-enhanced settings.

Models and protocol

  • The evaluation covers Gemini-2.5-Pro, o3, GPT-5, and Qwen-2.5-VL-72B-Instruct using a common search-and-reasoning pipeline.Human annotators were given at least 10 minutes per task, with a 20-minute cap.Source: §3 · p. 5

Search modes

  • The no-search condition answers from the question and image alone, while image search supplies cached summaries of the top-10 image results.Text search retrieves top websites over iterative rounds, and full rollout allows the agent to choose search tools.Source: §3 · pp. 5–6
  • Full Rollout + SoM adds indexed image overlays, subregion zooming, and image search over cropped regions, with a maximum of 20 tool calls.These capabilities extend the standard full-rollout setting with region-level visual retrieval.Source: §3 · p. 6

4 RESULTS

Full rollout substantially improves benchmark accuracy, while SoM adds consistent targeted gains; analyses show that tool-choice errors and limited image-search use remain important failure modes.

Main results

  • 36.0% accuracy is achieved by o3 with Full Rollout, compared with 23.8% for Gemini and 6.1% for Qwen.The o3 result is a +20.9-point gain over no search, while Gemini gains +13.2 points.Source: §4 · pp. 6–7

Failure analysis

  • On the easy split, full rollout loses 7.4 points for o3, and 9 of 10 inspected failures never invoked image search.The analysis attributes the gap mainly to miscalibrated tool-use decisions rather than over-retrieval.Source: §4 · p. 7
  • The benchmark’s easy label reflects memorization or direct image-hop solvability, not necessarily trivial multi-step retrieval.Rollout helps when progressive narrowing is needed but can fail when models skip necessary image search.Source: §4 · p. 7

Main results

  • SoM raises Full Rollout accuracy to 37.6% for o3, 27.7% for Gemini, and 7.1% for Qwen.The corresponding gains are +1.6, +3.9, and +1.0 points.Source: §4 · p. 7

Internal knowledge and tool use

  • On MMSEARCH+ lite, o3 reaches 31.4% with full rollout using only text search, while Sports gains from image search by 6.6 points with full rollout and 10.0 points with SoM.Most methods lie near the diagonal comparing full and lite benchmarks, indicating trends are not primarily driven by parametric memorization.Source: §4 · p. 7, Figure 4 · p. 7

5 ANALYSIS

The analysis examines how agents use search and visual tools, then characterizes their dominant failure modes and implications for reliable multimodal browsing.

5.1 OBSERVATIONS

  • Human and MLLM strengths differed by category: o3 reached 31.2% without search and exceeded the 20.3% human geography baseline, while humans usually outperformed MLLMs on Sports and Vlog.The authors describe this as different comparative advantages of humans and MLLMs.Source: §5.1 · p. 8
  • o3 sustained 10+ search rounds over ∼50+ retrieved items, whereas Gemini answered by Step 9 or earlier; more calls nevertheless did not imply correctness.Incorrect trajectories typically contained more search calls than correct ones.Source: §5.1 · p. 8
  • Closed-source models consistently produced valid tool calls, while Qwen made 421 invalid image-search calls across 45 of 311 tasks.The invalid calls inflated retries without improving quality.Source: §5.1 · p. 8
  • Gemini used image search after zooming in in 25.37% of cases, compared with 10.56% for Qwen and 2.87% for o3.These transition patterns align with Gemini using zoomed regions as retrieval queries, while o3 more often used zooming for visual inspection or continued text search.Source: §5.1 · p. 8
  • Text and image search benefited different categories: image search helped Film/TV, while additional text-search rounds helped Geography.For o3, text search through 10 rounds reached 37.0% accuracy versus 36.0% for full rollout overall.Source: §5.1 · pp. 8–9
  • Scene text was usually converted into textual queries through OCR rather than searched as image regions.Only 2 of 10 sampled o3 trajectories and 3 of 10 Gemini trajectories used only text search? Actually, o3 used only text search in 8 of 10 and Gemini in 7 of 10 sampled trajectories.Source: §5.1 · p. 9

5.2 ERROR ANALYSIS

  • The error taxonomy was manually curated for Gemini-2.5-Pro under full rollout, with one dominant label assigned to each erroneous prediction.Representative trajectories are provided in Figure 7.Source: §5.2 · p. 9, Figure 6 · p. 9
  • No relevant information found accounts for 51.1% of errors, while hallucination accounts for 11.5%, together representing 62.6% of failures.Retrieval failures often arise from noisy or overly specific queries, whereas hallucinations involve misattributing the image to an incorrect event and discounting contradictory evidence.Source: §5.2 · p. 9

6 RELATED WORK

Prior work spans visual reasoning, live visual knowledge, web agents, and image-grounded browsing, but MMSearch-Plus targets sparse visual cues that must drive multi-step retrieval and verification.

6 RELATED WORK

  • Classical vision–language benchmarks test compositional reasoning, while OK-VQA and InfoSeek add external knowledge or information-seeking components.InfoSeek relies on offline retrieval corpora that may overlap with modern MLLM pretraining data.Source: §6 · p. 9
  • LiveVQA derives questions from recent web content beyond model cutoff dates and reports gains from multimodal search augmentation.This distinguishes live visual knowledge evaluation from offline retrieval settings.Source: §6 · p. 10
  • Web-agent benchmarks study planning, browsing, and evidence gathering, while multimodal web-search benchmarks often emphasize entity matching or template-based grounding.MM-BrowseComp shows substantial headroom for frontier models using image or video evidence encountered during search.Source: §6 · p. 10
  • Related “thinking with images” work improves grounding or uses training methods to elicit image-first reasoning.These approaches complement benchmark efforts by targeting visual reasoning behavior directly.Source: §6 · p. 10
  • MMSearch-Plus targets sparse spatial–temporal cues that must become targeted queries and multi-step browsing plans.Tasks require selecting relevant regions, choosing image or text search, and gathering and verifying online evidence, making internal knowledge alone unlikely to suffice.Source: §6 · p. 10
  • Figure 7 provides representative case studies of high-frequency errors, including missing key information, unverified relevance, and misunderstood questions.Each row shows the prompt, ground-truth answer, and a shortened rollout.Source: Figure 7 · p. 10

7 CONCLUSION

MMSearch-Plus restores multimodal difficulty by requiring weak visual cues to propagate through noisy retrieval into out-of-image facts. Evaluation shows substantial remaining headroom and persistent challenges in cropping, evidence reuse, and verification.

7 CONCLUSION

  • The strongest tested agent, o3, achieved less than 40% accuracy, while Qwen-2.5-VL-72B-Instruct achieved 13.5%.These results indicate substantial headroom for multimodal browsing agents.Source: §7 · p. 10
  • MMSearch-Plus requires agents to extract weak localized visual cues and propagate them through noisy retrieval to out-of-image facts.Its Spatial–Temporal Extrapolation procedure preserves text-browsing difficulty while requiring fine-grained visual reasoning and provenance verification.Source: §7 · p. 10
  • Current systems struggle to decide when and how to crop and reuse visual evidence and to maintain verifiable evidence chains across long-horizon tool use.The benchmark is intended as a rigorous stress test and common yardstick for progress.Source: §7 · p. 10

ETHICS STATEMENT

The ethics statement addresses research integrity, public-data sourcing, privacy protections, and fairness practices, while acknowledging residual privacy risks and dataset limitations.

ETHICS STATEMENT

  • The project follows an ethics code emphasizing rigor, harm avoidance, transparency, fairness, intellectual labor, and privacy.These principles frame the benchmark’s data and evaluation practices.Source: ETHICS STATEMENT · p. 11
  • Data come from publicly accessible video platforms and open-access repositories, with licensing, attribution, and research-only use requirements.The pipeline preserves source URLs and treats collected items as events.Source: ETHICS STATEMENT · p. 11
  • Sensitive content such as faces and license plates is masked, while minors and private spaces are excluded when feasible.Residual risks remain, and the project provides an appeals and takedown channel.Source: ETHICS STATEMENT · p. 11
  • Manual sampling across predefined categories and regions, measurable distribution reporting, and use restrictions address platform skews and protected-class harms.The authors forbid uses that target protected classes or amplify inequities.Source: ETHICS STATEMENT · p. 11
  • The dataset inherits public-web and sampling-domain biases, under-represents some non-English or low-visual-density pages, and focuses mainly on image–text browsing.Specialized systems may behave differently, while videos and dynamic interfaces remain underexplored.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The paper supports independent verification through supplementary task examples and an anonymized dataset archive, while disclosing Grammarly-assisted polishing. An example illustrates the benchmark’s provenance-aware multimodal search process.

REPRODUCIBILITY STATEMENT

  • Thirty randomly selected task examples are included as supplementary material, alongside an anonymized dataset-use archive.The subset uses seed 42 and is provided because of a 100MB space constraint.Source: REPRODUCIBILITY STATEMENT · p. 11

A USE OF LLMS

  • The manuscript was polished with Grammarly after drafting, followed by manual review of all suggestions.Source: A USE OF LLMS · p. 15

B EXAMPLE FROM MMSEARCH-PLUS

  • A concert-photo task requires localized visual cues to identify the event and cross-validate the official schedule for the singer’s performance time.The answer is not explicitly present in the prompt or image, requiring targeted image and text searches under retrieval noise.Source: Figure 8 · p. 15

C COMPARISON WITH PREVIOUS BENCHMARKS

The paper positions MMSearch-Plus at the intersection of multimodal reasoning and long-horizon web search, targeting sparse visual evidence that must drive multi-step query formulation. Its framework adds human-curated region marks, cropping, and region-seeded image search to support this setting.

D RELATED WORK

  • Existing multimodal reasoning benchmarks are mainly self-contained and do not require dynamic open-web interaction or tool-mediated search.Source: D RELATED WORK · p. 16

C COMPARISON WITH PREVIOUS BENCHMARKS

  • The dataset complements prior multimodal browsing benchmarks by requiring visual evidence to support search rather than reducing multimodality to entity matching.Source: D RELATED WORK · pp. 16–17

D RELATED WORK

  • MM-BrowseComp reports approximately 29% accuracy for top models such as OpenAI o3 with tools, leaving substantial headroom for native multimodal browsing reasoning.MMSearch-Plus differs by focusing on sparse visual cues transformed into targeted queries rather than predominantly entity-level matching.Source: D RELATED WORK · pp. 16–17
  • MMSearch-Plus addresses the open problem of turning sparse visual evidence into targeted queries and multi-step plans for multimodal browsing.The paper formulates Spatial-Temporal Extrapolation and emphasizes grounding and cropping for fine-grained reasoning.Source: D RELATED WORK · p. 17
  • Text-browsing benchmarks test persistence and decomposition, while multimodal benchmarks emphasize perception-heavy reasoning.Recent datasets begin combining these capabilities but leave sparse-evidence query formulation unresolved.Source: D RELATED WORK · pp. 16–17

E DETAILED DATASET STATISTICS

IMPLEMENTATION DETAILS

  • Set-of-Mark represents key visual entities with indexed boxes and exposes zoom-in and image-search operations over cropped regions.The raw image and indexed overlay are presented together so regions can be referenced during browsing.Source: IMPLEMENTATION DETAILS · p. 17
  • Human-curated boxes reduce localization errors because baseline MLLMs were unreliable at proposing precise crops and long-tail entities challenge automatic mark generation.Source: IMPLEMENTATION DETAILS · p. 17

G EVALUATION DETAILS

The evaluation uses GPT-4o as the judge after consistency checks against Gemini-2.5-Pro, with optional lightweight rule-based verification for reproducibility.

G EVALUATION DETAILS

  • GPT-4o and Gemini-2.5-Pro produced consistent judge outputs, so subsequent experiments used GPT-4o.Source: G EVALUATION DETAILS · p. 18
  • The evaluation additionally provides lightweight rule-based verification methods as an optional reproducibility aid.Source: G EVALUATION DETAILS · p. 18
  • The judging procedure was selected after comparing two multimodal model judges rather than relying on a single initial choice.Source: G EVALUATION DETAILS · p. 18

H RESULTS ON MMSEARCH+

On MMSEARCH+ lite, o3 performs best with full rollout using text search only at 31.4% accuracy, while image search helps substantially for Sports. Agent traces further show that zoom is rarely followed immediately by image search, especially for o3.

H RESULTS ON MMSEARCH+

  • 31.4% accuracy is achieved by o3 with full rollout using text search only, the best-performing MMSEARCH+ lite setting reported.The lite benchmark removes tasks answerable without search by tested models.Source: H RESULTS ON MMSEARCH+ · p. 19
  • Sports accuracy for o3 increases by 6.6 points with full rollout and 10.0 points with full rollout plus SoM relative to text search only.The passage identifies Sports as a category where image search provides a substantial benefit.Source: H RESULTS ON MMSEARCH+ · p. 19
  • The reported MMSEARCH+ lite results are presented as end-to-end accuracy comparisons in Table 4 and Figure 10.The table marks best and second-best methods under each setting, while darker blue denotes higher accuracy.Source: Table 4 · p. 19, H RESULTS ON MMSEARCH+ · p. 19

I AGENT BEHAVIOR ANALYSIS

I.1 EPISODE RASTER PLOTS

  • Episode raster plots order the 40 longest episodes for each agent by length and encode sequential tool choices with colors.The tools include text search, image search, zoom-in, and answer; dark blue indicates early stopping before 20 rounds.Source: §I.1 · p. 20

I.2 TOOL TRANSITION MATRICES

  • The transition analysis tests whether cropped images are searched immediately after zoom by examining P(image|zoom).Source: §I.2 · p. 20
  • Immediate image search follows 25.37% of Gemini zooms, 10.56% of Qwen zooms, and 2.87% of o3 zooms.These correspond to 34/134, 15/142, and 18/627 zoom actions, respectively.Source: §I.2 · p. 20

I.3 ZOOM-IN BEHAVIOR ANALYSIS

The analysis compares agents’ zoom-in behavior and tool-use trajectories, showing that o3 zooms earlier and more often than Gemini and Qwen. It also characterizes search-call distributions and response length across correct and incorrect full-rollout trajectories.

I.3 ZOOM-IN BEHAVIOR ANALYSIS

  • o3 exhibits the most aggressive zooming behavior, while Qwen and Gemini are more conservative.The comparison is summarized across zoom-in patterns analyzed in Figures 13 and 14.Source: §I.3 · p. 20
  • o3 tends to zoom earlier in episodes than Gemini and Qwen.Source: Figure 13 · p. 21
  • o3 has the highest zoom-in frequency, with some episodes containing 10+ zoom operations.Source: Figure 14 · p. 22

J ROLLOUT TRAJECTORY STATISTICS

  • The trajectory analysis compares image and text search-call distributions with response-word counts across correct and incorrect full-rollout trajectories.Figure 15 organizes MLLMs by rows and separates search-call distributions from response-length relationships.Source: Figure 15 · p. 23, J ROLLOUT TRAJECTORY STATISTICS · p. 23

K GOOD CASES

This section presents successful cases in which agents combine visual cues with targeted retrieval to identify entities and locations, answer multimodal questions, and apply iterative planning. The cases also emphasize citation-backed responses that directly address the user’s question and exclude irrelevant content.

K GOOD CASES

  • Several retrieved pages matched the character visually but lacked publisher or 2024-release information, while one wrestling result contradicted the fantasy-game context.This illustrates the need to cross-validate visual matches against page semantics and factual content.Source: K GOOD CASES · pp. 25–27
  • Kepler Ghost is Kepler Interactive’s white-label publishing service, and its 2024 titles include Windblown, Unrailed 2: Back on Track, and Dark Hours.The service launched in 2023 and provides marketing and publishing support under the Kepler Ghost imprint.Source: K GOOD CASES · pp. 28–29
  • The pictured game is Clair Obscur: Expedition 33, published by Kepler Interactive [21, 22, 23, 24, 25].Multiple image-text matches connect the character to the game, while publisher sources identify Kepler Interactive.Source: K GOOD CASES · pp. 25, 29, 30
  • Excluding Kepler Ghost releases, Kepler Interactive published four games in 2024: Ultros, Pacific Drive, Flintlock: The Siege of Dawn, and Cat Quest III.The listed release dates are February 13, February 22, July 18, and August 8, 2024, respectively.Source: K GOOD CASES · pp. 28, 30
  • The reasoning strategy zooms into prominent flags and signs, uses Russian text searches after generic image results fail, and synthesizes visual and retrieval evidence.A billboard reading “Thai Spa Salon” confirmed the Russian-language context, while the subtropical scene supported the Sochi hypothesis.Source: K GOOD CASES · pp. 36–37
  • The agent concludes that the pictured city is Sochi, Russia, linking Russian signs, palm trees, and the flag-associated company to Sochi real-estate and construction records.Targeted searches returned Sochi-linked companies matching the flag text, although some records indicated liquidation [25, 30, 33].Source: K GOOD CASES · p. 37

L PROMPTS

  • The prompt requires multi-round decomposition and planning, with 3 to 5 text queries at a time and revised queries when initial retrieval is unhelpful.It instructs the agent to use tools when image content or concepts are uncertain rather than relying on unsupported confidence.Source: L PROMPTS · p. 37
  • When image targets are too small or unclear, the prompt directs the agent to request indexed crops from the annotation image and combine visual concepts with text for retrieval.This cropping procedure is presented as a fallback when image-search results are not ideal.Source: L PROMPTS · p. 38
  • After each tool-call round, agents must analyze results, reassess the original question and plan, then decide whether further calls are needed.When historical information already suffices, agents should summarize the reasoning and answer-acquisition process before returning the answer.Source: L PROMPTS · p. 38
  • Final answers must place the answer between <answer> and </answer> and include citations in the required [quote id] format.Text-search queries must be keywords, phrases, or sentences.Source: L PROMPTS · p. 38

Related papers

Loading 2508.21475v3…