PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning

Zhengbo Zhang, Changtao Miao, Jinbo Su, Zhaowen Zhou, Chunxia Zhang, Xukai Wang, Ruiqi Liu, Kaiyuan Zheng, Jiansheng Cai, Bo Zhang, Zhe Li, Shiming Xiang, Ying Yan

arXiv:2606.15231v1cs.AI

TL;DR

Existing multimodal search data lacks complex, realistic visual queries and required visual evidence in search paths. Visual-Seeker trains an agent to reason over fine-grained image entities and harvest visual evidence across multi-hop trajectories; it reports 39.6% average accuracy across five benchmarks, outperforming current multimodal deep-search agents.

  • Problem

    Existing multimodal search training data lacks complex, realistic visual queries and does not incorporate visual evidence into necessary search paths.

    Source: §2.2 · p. 3

  • Method

    Visual-Seeker uses an active visual reasoning data pipeline to generate multi-hop trajectories from complex entity-centric queries with strategically injected visual evidence.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    39.6% average accuracy across five benchmarks, outperforming current multimodal deep-search agents and competing with some proprietary models.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    The reported advantages are particularly evident on complex multi-entity queries and benchmarks requiring visual evidence during search.

    Source: §4.2 · pp. 8–9

    The supplied author-limitation excerpt is truncated after noting that knowledge-graph-based QA synthesis provides structured factual reasoning, so it does not specify the limitation.

    Source: §3.1.3 · p. 6

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have demonstrated impressive capabilities in many visual tasks, but they often struggle with factual grounding when confronted with complex, open-world scenarios. While recent multimodal deep search agents attempt to address this issue by utilizing external tools, the visual-native search paradigm remains underexplored. Existing methods primarily rely on simple images with explicit semantics and text-only evidence trajectories, limiting the agent's ability to perform multi-hop, cross-modal reasoning and search. To address these limitations, we propose Visual-Seeker, a visual-native multimodal deep search agent via active visual reasoning. Rather than treating vision as a static input, our agent actively attends to fine-grained visual details, dynamically harvests visual evidence throughout the search process. To unlock its visual-native potential, we design an active visual reasoning data pipeline and synthesize 5K high-quality multimodal trajectories for model training. Extensive experiments demonstrate the state-of-the-art performance across five challenging multimodal search benchmarks, even surpassing several proprietary models, validating robust visual-native reasoning and search in real-world web environments. The code and data can be accessed at: https://github.com/ZhengboZhang/Visual-Seeker.

1. Introduction

Visual-Seeker uses active visual reasoning to gather and integrate visual evidence during open-world cross-modal search, trained on 5K synthesized multimodal trajectories; the paper reports state-of-the-art performance across five benchmarks.

1. Introduction

  • Text-only search agents struggle with visual queries and web imagery, while recent multimodal extensions use MLLMs to handle image and text inputs for open-world visual question answering.The paper situates Visual-Seeker as a response to these limitations.Source: §1 · p. 1
  • Visual-Seeker achieves state-of-the-art performance on five multimodal search benchmarks, outperforming existing open-source and several proprietary models.The introduction reports this result without giving benchmark-specific scores.Source: §1 · pp. 2–3
  • The agent reasons over fine-grained properties and interconnected entities in complex real-world images, then proactively collects and integrates visual evidence across multi-hop search.It is designed to bridge passive visual perception and active cross-modal search in open-world environments.Source: §1 · pp. 2–3
  • Its three-stage data pipeline identifies seed entities and visual descriptions, expands search with dual-strategy random walks, then retrieves entity images and merges queries containing visual evidence.The pipeline produces 5K high-quality multimodal search trajectories for training Visual-Seeker.Source: §1 · pp. 2–3

2. Related Works

Prior work extends search agents from text-only retrieval toward multimodal tools, but existing multimodal methods remain limited in how they collect and use visual evidence.

2.1. Text-only Deep Search Agent

  • Text-only deep search agents iteratively reason and call tools to search real-world environments, but textual queries and documents cannot leverage multimodal sensory data.They aim to overcome pre-built knowledge-base limits of RAG [10], building on external-tool search [1] [2] and iterative reasoning-and-tool loops [3] [12] [13].Source: §2.1 · p. 3

2.2. Multimodal Deep Search Agent

  • Early multimodal agents use reverse image search and entity-visual queries, while recent methods add image cropping to locate target entities and reduce background interference.Early studies fuse textual QA with entity-visual queries to create fine-tuning data for image retrieval and multi-turn reasoning [4] [7]; recent work uses models’ visual grounding [14] [6] [5] [15].Source: §2.2 · p. 3
  • Existing multimodal search data lacks visual queries for complex, multi-entity real-world images and fails to make visual evidence necessary along the search path [16] [17].Visual reasoning and proactive visual-evidence gathering are also indispensable for solving complex problems through multi-turn search.Source: §2.2 · p. 3

3. Method

Visual-Seeker’s method synthesizes entity-centered, multi-hop multimodal search trajectories by injecting visual evidence, then trains an agent to collect and organize visual and textual evidence during search.

3.1. Active Visual Reasoning Data Pipeline

  • The pipeline builds multi-hop multimodal trajectories from complex entity-centric queries and strategically injects visual evidence to activate visual-native reasoning.Figure 2 depicts extracting and disambiguating entities from multi-entity images, extending search depth with dual-strategy random walks, and inserting visual evidence.Source: §3.1 · p. 3, Figure 2 · p. 4

3.1.1. Fine-grained Seed Entity Selection

  • From LiveVQA [18], the method jointly extracts visually grounded entities and captions, filters generic mentions and easy images, disambiguates polysemous entities, and selects 2K seeds.Filtering combines rules and MLLM verification; disambiguation uses the image, question, and surrounding entities to select among Wikipedia senses.Source: §3.1.1 · pp. 4–5

3.1.2. Multi-hop Trajectory Synthesis

  • Random walks from seed entities add backtracking branches and cycle-constrained, node-disjoint paths, then generate questions from connected subgraphs to create diverse, non-linear search trajectories.Backtracking models revisiting earlier assumptions; generated questions are constrained by the relationships in each sampled subgraph.Source: §3.1.2 · pp. 5–6

3.1.3. Visual Evidence Injection

  • To make agents actively acquire and interpret web-page visuals, the method retrieves candidate images for answer entities and adds two-hop QA pairs about image details to the original VQA.An MLLM extracts details not derivable from text alone; fuzzy search keywords become questions and visual details become answers.Source: §3.1.3 · p. 6

3.2. Visual-Seeker

  • Visual-Seeker uses supervised fine-tuning as a cold start, with teacher-generated multimodal search trajectories, rather than costly reinforcement learning.The trajectories are generated from the synthesized VQA data using the agentic workflow.Source: §3.2 · p. 6

3.2.1. Workflow and Tools

  • The agent uses five external tools in a ReAct-style loop and stops when it answers or reaches the 15-round interaction limit.Tools include text search, reverse-image search, image search, webpage visiting and summarization, and image cropping for fine-grained visual reasoning.Source: §3.2.1 · pp. 6–7

3.2.2. Supervised Fine-Tuning

  • Training mixes 5K trajectories: 3K VQA without visual injection, 800 with injection, 500 text-only QA, and 700 FVQA [4] trajectories, using Claude-4.6-Opus as teacher.The mixed data is trained with cross-entropy loss to teach multi-turn tool interaction and active collection and organization of visual and textual evidence.Source: §3.2.2 · p. 7
  • After visual evidence injection, search_image calls increase significantly, while pipeline-built queries show high text_search use indicative of multi-hop search.The authors interpret these tool-call patterns as evidence that the synthesized queries prompt deeper searches and multimodal evidence collection.Source: §3.2.2 · p. 7, Figure 3 · p. 7

4. Experiments

Experiments compare Visual-Seeker with other systems across five multimodal search benchmarks, then use data and tool ablations and tool-use analysis to examine its performance. Visual-Seeker averages 39.6% accuracy, and the reported ablations show benefits from multimodal training data and visual search tools.

4.1. Experimental Setup

  • The evaluation uses accuracy on five multimodal agentic-search benchmarks, an LLM judge, and direct-answer, agentic-workflow, and multimodal deep-search baselines.The benchmarks are MMSearch, MMSearch-Plus, BrowseComp-VL, MM-BrowseComp, and VisBrowse-Bench; MMSearch-Plus uses single-image samples.Source: §4.1 · p. 8

4.2. Main Results

  • Across models, agentic workflow improved results; Claude-4-Sonnet rose 145.8% from a direct-answer average of 13.1 across five benchmarks.The paper attributes the generally poor direct-answer performance to limited pretrained knowledge.Source: §4.2 · p. 8
  • Visual-Seeker shows improved fine-grained visual understanding on MMSearch-Plus and outperforms GPT-5 and Gemini-2.5-Pro on visually evidence-dependent MM-BrowseComp and VisBrowse-Bench.The paper describes MMSearch-Plus as involving complex multi-entity image queries, and the latter benchmarks as requiring visual evidence during search.Source: §4.2 · pp. 8–9
  • Visual-Seeker averages 39.6% accuracy across five benchmarks, outperforming all current multimodal deep-search agents, competing with some proprietary models, and nearly doubling Qwen3-VL-8B-Instruct (Agentic) on every benchmark.Table 1 presents comparisons across the five benchmarks.Source: §4.2 · p. 8, Table 1 · p. 9

4.3. Ablation Study and Analysis

  • Multimodal trajectories synthesized by the pipeline significantly benefit multi-hop search, yielding a 17.2% performance improvement on MMSearch-Plus; visual-evidence data also improves MM-BrowseComp and VisBrowse.Open-source multimodal query and text-query trajectories provide slight improvements by teaching tool calls and reasoning patterns.Source: §4.3 · p. 9
  • Removing either image_crop or search_image significantly reduced performance across benchmarks, with each tool’s largest decline on VisBrowse-Bench.Image cropping helps resolve complex entities that whole-image search may miss, while search_image supports collecting visual evidence; removing both weakens active visual grounding and evidence collection.Source: §4.3 · p. 10, Table 3 · p. 10
  • Tool interactions averaged 4.3 rounds on MMSearch versus 14.1 on MM-BrowseComp, while VisBrowse required more reverse_image_search and search_image calls.Text_search was the most-used tool across benchmarks because textual evidence dominates search trajectories; VisBrowse’s visual-tool use reflects its reliance on visual evidence from web pages.Source: §4.3 · p. 10, Figure 4 · p. 11

5. Conclusion

Visual-Seeker unifies fine-grained visual entity perception with active visual evidence harvesting across multi-hop search trajectories. Its active visual reasoning data pipeline produces 5K training trajectories, and the agent achieves state-of-the-art performance across five challenging benchmarks.

5. Conclusion

  • Visual-Seeker achieves state-of-the-art performance across five challenging benchmarks, especially where precise multi-entity grounding and cross-modal evidence integration are required.The conclusion attributes this capability to learning active visual reasoning from the synthesized training data.Source: §5 · p. 11
  • The agent unifies fine-grained visual entity perception with active visual evidence harvesting across multi-hop trajectories.This approach addresses text-only systems’ visual blindness and multimodal extensions’ treatment of vision as passive input.Source: §5 · p. 11
  • The active visual reasoning data pipeline extracts complex entities from multi-entity real-world images and strategically injects visual evidence, yielding 5K high-quality training trajectories.Source: §5 · p. 11

A. Appendix · A.1. Data Example

The appendix presents examples from a synthesized 5K multi-hop VQA dataset containing complex entity queries and visual evidence, contrasting examples without and with visual evidence injection.

A.1. Data Example

  • The pipeline synthesized a 5K multi-hop VQA dataset containing complex entity queries and visual evidence.The passage identifies Figures 5 and 6 as examples without and with visual evidence injection, respectively.Source: §A.1 · p. 14
  • Figure 5 presents a data example without visual evidence injection.The supplied passage does not describe the example's specific content.Source: Figure 5 · p. 14, §A.1 · p. 14
  • Figure 6 presents a data example with visual evidence injection.The supplied passage does not describe the example's specific content.Source: §A.1 · p. 14, Figure 6 · p. 15

A.2. Prompt

The prompt specifies a methodical visual question-answering agent that decomposes questions, gathers and cross-checks evidence with tools, and describes the image. It also includes a separate semantic-equivalence check for ground-truth and model answers.

  • The agent should break the visual question into sub-questions, solve them step by step, and summarize prior-round knowledge before deciding what to investigate next.It should describe the image in detail, treating each sub-image separately when there are multiple ones.Source: §A.2 · p. 15
  • The agent should iteratively use tools, explain each call’s purpose, cross-validate evidence, and make only one tool call per round.It may use search_image to find additional visual information and image_crop to inspect a specific region.Source: §A.2 · pp. 15–16
  • The agent must provide its final answer within 15 turns, whether or not it has collected all valid information.Source: §A.2 · p. 15
  • The final answer must be concise and enclosed in <answer></answer> tags.Source: §A.2 · p. 16
  • A separate evaluator compares the ground-truth answer with the model answer for semantic equivalence and outputs “yes” or “no.”Source: §A.2 · p. 16

A.3. Case Study

The VisBrowse-Bench case study illustrates visual evidence collection and fine-grained entity extraction through a multi-step search, identifying a banana as the fruit in the documentary poster.

A.3. Case Study

  • The case study uses VisBrowse-Bench because its tasks include fine-grained entity extraction and visual evidence collection.Source: §A.3 · p. 16
  • The model examined the necklace, identified Vivienne Westwood, and resolved ambiguity between two 2018 documentaries by selecting the film about the brand’s founder.It used a crop and image search to identify the necklace, then considered Westwood: Punk, Icon, Activist and McQueen before choosing the Westwood documentary.Source: §A.3 · pp. 17–18
  • The model identified a banana in the poster for the 2018 documentary Westwood: Punk, Icon, Activist.It reported that the poster shows Vivienne Westwood eating a banana, and noted that the Japanese poster text confirms it.Source: §A.3 · pp. 18–19

Related papers

Loading 2606.15231v1…