Source-linked AI summary
Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning
Zhengbo Zhang, Changtao Miao, Jinbo Su, Zhaowen Zhou, Chunxia Zhang, Xukai Wang, Ruiqi Liu, Kaiyuan Zheng, Jiansheng Cai, Bo Zhang, Zhe Li, Shiming Xiang, Ying Yan
TL;DR
Existing multimodal search agents underuse visual evidence in complex, multi-entity open-world searches. Visual-Seeker addresses this with active visual reasoning and achieves state-of-the-art performance across five multimodal search benchmarks, averaging 39.6% accuracy.
Problem
Existing multimodal search agents lack complex visual queries and visual evidence within the necessary search path for multi-hop reasoning.
Method
Visual-Seeker combines fine-grained visual entity perception with active visual evidence harvesting, trained on 5K trajectories synthesized by an active visual reasoning pipeline.
Results
39.6% average accuracy across five benchmarks makes Visual-Seeker state of the art among multimodal deep search agents and nearly twice as strong as Qwen3-VL-8B-Instruct (Agentic).
Takeaways & Limitations
Visual-Seeker is competitive with some proprietary models across the evaluated multimodal search benchmarks.
Takeaways & Limitations
The paper identifies a limitation in its knowledge-graph-based QA synthesis, but the supplied passage does not state it fully.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have demonstrated impressive capabilities in many visual tasks, but they often struggle with factual grounding when confronted with complex, open-world scenarios. While recent multimodal deep search agents attempt to address this issue by utilizing external tools, the visual-native search paradigm remains underexplored. Existing methods primarily rely on simple images with explicit semantics and text-only evidence trajectories, limiting the agent's ability to perform multi-hop, cross-modal reasoning and search. To address these limitations, we propose Visual-Seeker, a visual-native multimodal deep search agent via active visual reasoning. Rather than treating vision as a static input, our agent actively attends to fine-grained visual details, dynamically harvests visual evidence throughout the search process. To unlock its visual-native potential, we design an active visual reasoning data pipeline and synthesize 5K high-quality multimodal trajectories for model training. Extensive experiments demonstrate the state-of-the-art performance across five challenging multimodal search benchmarks, even surpassing several proprietary models, validating robust visual-native reasoning and search in real-world web environments. The code and data can be accessed at: https://github.com/ZhengboZhang/Visual-Seeker.
1. Introduction
Existing search agents are text-only and struggle with visual queries and web-based visual information, motivating Visual-Seeker, a visual-native agent that actively reasons over fine-grained image details and gathers multimodal evidence for cross-modal search.
- Motivation: Text-only autonomous search agents struggle with visual queries and visual information in web environments.These systems support multi-hop reasoning, tool use, and web navigation but operate only through text.
- Search setting: Multimodal search agents must aggregate multimodal web evidence and synthesize diverse cues through cross-modal reasoning to generate comprehensive answers.Real-world image queries often contain complex, entity-rich visual content requiring iterative search refinement.
- Approach: Visual-Seeker bridges passive visual perception and active cross-modal search in open-world environments.The agent enhances visual-native capabilities within the search trajectory through active visual reasoning.
- Capabilities: Visual-Seeker performs visual understanding in complex multi-entity scenes and proactively collects visual evidence for cross-modal search.Its design targets complex real-world images containing multiple entities and fine-grained properties.
- Training data: The Active Visual Reasoning data synthesis pipeline develops fine-grained visual perception and active visual-evidence collection during deep search.The work reports 5K high-quality trajectories generated from the pipeline.
2. Related Works
Related work progresses from text-only deep search agents to multimodal agents that use reverse image search and cropping tools, but existing methods still inadequately support proactive visual evidence gathering for complex real-world images.
- Text-only deep search: Text-only deep search agents use external tools in iterative reasoning-and-tool-call loops to search real-world environments beyond static or pre-built knowledge.These methods address time-truncated language-model knowledge and the limitations of pre-building a knowledge base with RAG.
- Multimodal deep search: Early multimodal agents use reverse image search to extract whole-image semantics, combine textual QA with entity-visual queries, and support image retrieval with multi-turn reasoning.Later multimodal methods introduced image-cropping tools to support visual search and reasoning.
- Limitations: Existing multimodal search agents lack visual queries for complex, multi-entity images resembling real-world scenes and insufficiently incorporate proactive visual evidence gathering.The related work emphasizes that visual reasoning and actively collecting visual evidence are indispensable for solving complex multi-turn search problems.
3. Method
Visual-Seeker’s method synthesizes visual-native multimodal search trajectories by extracting and disambiguating entities from complex images, expanding multi-hop reasoning over a knowledge graph, and injecting visual evidence that requires image search and pixel-level reasoning. The resulting data supports supervised fine-tuning and increases image-search tool use in agent trajectories.
- Active Visual Reasoning Data Pipeline: The active visual reasoning pipeline starts from complex entity-centric queries and strategically injects visual evidence to synthesize multi-hop multimodal search trajectories.It is designed to activate visual-native reasoning capabilities in multimodal large language models.
- Entity Recognition and Filtering: Joint visual-textual entity extraction identifies visually grounded entities, followed by generic-mention filtering, complex-image filtering, and context-aware disambiguation.The process uses LiveVQA images, questions, and reasoning processes, with MLLM verification and Wikipedia disambiguation pages.
- Entity Recognition and Filtering: 2K entities are retained as seed entities for synthesizing multi-hop VQA after entity recognition, filtering, and disambiguation.The seeds are extracted from query images containing complex semantic entities.
- Multi-hop QA Synthesis: Knowledge-graph random walks generate diverse, non-linear trajectories using backtracking and cycle constraints instead of naive linear chains.Backtracking creates tree-shaped reasoning branches, while cycle constraints enforce disjoint branches that later converge toward a common node.
- Visual Evidence Injection: Visual evidence injection retrieves candidate images and extracts fine-grained attributes to create two-hop QA pairs requiring text-to-image retrieval and visual perception.The extended two-hop questions are merged into the original VQA, addressing the limitation that knowledge-graph synthesis alone omits active visual evidence acquisition.
- Trajectory Characteristics: After visual evidence injection, search_image tool triggers increase significantly, while the pipeline’s high text_search proportion indicates multi-hop search requirements.The synthesized queries encourage agents to collect multimodal evidence from web environments.
4. Experiments
Experiments evaluate Visual-Seeker across five multimodal agentic search benchmarks using accuracy, comparing direct-answer, agentic-workflow, and multimodal deep-search methods. The model achieves 39.6% average accuracy, outperforming existing multimodal search agents and nearly doubling the Qwen3-VL-8B-Instruct Agentic baseline on every benchmark.
- Evaluation setup: Five benchmarks—MMSearch, MMSearch-Plus, BrowseComp-VL, MM-BrowseComp, and VisBrowse-Bench—are used for evaluation.MMSearch-Plus uses only single-image samples.
- Evaluation setup: Accuracy (%) is the evaluation metric, with Qwen3-235B-A22B-Instruct serving as the LLM-as-Judge model against ground truth.Prompt details are provided in Appendix A.2.
- Results: Direct-answer models perform poorly; Claude-4-Sonnet reaches only 13.1 average accuracy across five benchmarks.Direct answering relies on internal parametric knowledge without external tool access.
- Results: 145.8% increase is achieved by Claude-4-Sonnet after integration with the agent workflow, with all models showing significant improvement.The agentic workflow enables models to collect visual and textual evidence using the framework’s tools.
- Results: 39.6% average accuracy is achieved across five benchmarks, outperforming current multimodal deep search agents and nearly doubling Qwen3-VL-8B-Instruct (Agentic) on every benchmark.The comparison includes proprietary models, open-source models, and existing multimodal deep search agents.
- Results: Visual-Seeker is especially competitive on MMSearch-Plus, which contains complex multi-entity image queries, and shows improvements on benchmarks requiring visual evidence.The cited passage specifically identifies MM-BrowseComp and VisBrowse-Bench as requiring visual evidence.
Multimodal Deep Search Agent
Visual-Seeker’s multimodal deep search relies on active visual reasoning and visual-evidence tools, with interaction effort scaling by benchmark difficulty and performance exceeding GPT-5 and Gemini-2.5-Pro. Ablations further show that both core tools and multimodal training data contribute to effectiveness.
- The model outperforms the proprietary models GPT-5 and Gemini-2.5-Pro during the search process.
- Data Ablation: Adding open-source multimodal query and text query trajectories enables tool calls and reasoning patterns, producing a slight performance improvement.The data ablation incrementally adds four data types, including FVQA, pipeline-generated QA, and VQA variants with or without visual-evidence injection.
- Tool Ablation: Removing either image_crop or search_image significantly reduces performance across all benchmarks.The largest decline after removing image_crop occurs on VisBrowse.
- Analysis of Tool Usage: 4.3 average tool-interaction turns occur on MMSearch, compared with 14.1 on the more challenging MM-BrowseComp benchmark.The analysis also measures the distribution of different tool types across five benchmarks.
5. Conclusion
The paper identifies visual blindness in text-only search agents and passive vision in multimodal systems, then proposes Visual-Seeker as a visual-native agent with active visual reasoning and evidence harvesting.
- Conclusion: Existing search agents suffer from visual blindness or treat vision as a passive input.The paper formalizes these limitations across text-only and multimodal systems.
- Conclusion: Visual-Seeker unifies fine-grained visual entity perception with active visual evidence harvesting across multi-hop trajectories.It is presented as a visual-native multimodal deep search agent.
- Conclusion: The authors design an active visual reasoning data synthesis pipeline for extracting complex entities.The supplied passage introduces the pipeline but ends before specifying its full extraction procedure.
A. Appendix · A.1. Data Example
The appendix presents a synthesized 5K multi-hop VQA dataset with complex entity queries and visual evidence, illustrated through examples with and without visual-evidence injection. The examples require chaining visual identification with cross-domain factual associations.
- A. Appendix: 5K synthesized a multi-hop VQA dataset containing complex entity queries and visual evidence.The appendix describes the dataset as produced by the data synthesis pipeline.
- A.1. Data Example: Figures 5 and 6 contrast data examples without and with visual evidence injection.Figure 5 shows examples without injection, whereas Figure 6 shows examples with injection.
- A.1. Data Example: One question links a basketball player’s jersey number and college team to an NCAA champion, an NBA draft, and a later league season.The chain begins from a player in a black jersey numbered 45 and continues through college basketball, NCAA, draft, and league-season clues.
- A.1. Data Example: Another question identifies a man by the portfolio he holds and traces a defunct justice-related ministerial role through federal legal administration.It further references the Commonwealth’s chief law officer and the country’s second-highest-ranking law officer.
- A.1. Data Example: A music question identifies a woman by her stage name and connects her album single to dancehall, four-on-the-floor music, bowed-string ensembles, and Gustav Holst.The passage identifies the stage name as distinct from Katheryn Elizabeth Hudson and references her fifth studio album.
- A.1. Data Example: A television-and-politics question identifies an actress through an orphan role and Mallory O’Brien before asking about the founding era of a U.S. political party.The chain invokes Aaron Sorkin’s White House drama and its fictional Democratic presidency.
- A.1. Data Example: Additional examples connect a woman’s directing education to a theatre artist, a 2012 memorial, and a 1990s urban siege.The question starts from the woman in the blue dress on the right and follows several biographical and historical associations.
- A.1. Data Example: Another example connects a filmmaker in costume to a satirical biopic about a U.S. vice president, the 2003 conflict, and an international factfinding mission.The question also references a British government inquiry into pre-war intelligence and the mission’s conclusion about banned arms programs.
A.2. Prompt
The prompt defines a visual question-answering agent that methodically decomposes questions, describes images, gathers and cross-validates web evidence, and produces concise answers under strict tool-use constraints. It also specifies a separate evaluator that judges semantic equivalence between ground-truth and model answers.
- Reasoning process: The agent must decompose each visual question into sub-questions, summarize prior-round knowledge, and solve the sub-questions step by step.This establishes an iterative reasoning process centered on deciding what to investigate next.
- Visual analysis: The agent must describe the image in detail, separately describing each sub-image when multiple sub-images are present.Image description is required whether or not the final question can be answered.
- Interaction constraints: The agent must explain each tool call beforehand, make only one tool call per round, and provide the final answer within 15 turns.These rules constrain the search process and impose a fixed interaction budget.
- Evidence gathering: The agent begins with tool-selection cycles, may use search_image or image_crop for additional visual evidence, and must cross-validate information from multiple tools.The prescribed cycle is thinking, tool call, waiting for the response, then thinking about and answering the question.
- Output and evaluation: The final response must be concise and enclosed in <answer></answer> tags, while the quality assessor outputs yes or no for semantic equivalence.The evaluator compares the ground truth with the model answer for a given question.
A.3. Case Study
The case study analyzes Visual-Seeker’s visual-native ability on VisBrowse-Bench through a multi-step query linking necklace identification, documentary disambiguation, and poster evidence. The trajectory concludes that Vivienne Westwood is eating a banana.
- VisBrowse-Bench combines fine-grained entity extraction with visual evidence collection, motivating the case study of the model’s visual-native ability.
- The agent crops the image, identifies a pearl necklace with a possible Vivienne Westwood Orb logo, and uses reverse-image search for confirmation.
- After finding two 2018 documentaries, the agent resolves the ambiguity by selecting Westwood: Punk, Icon, Activist as the documentary about the brand’s founder.
- The documentary poster shows Vivienne Westwood eating a banana, with the Japanese poster text providing confirmation.