Towards Long-horizon Agentic Multimodal Search

Yifan Du; Zikang Liu; Jinbiao Peng; Jie Wu; Junyi Li; Jinyang Li; Wayne Xin Zhao; Ji-Rong Wen

Source-linked AI summary

Towards Long-horizon Agentic Multimodal Search

Yifan Du, Zikang Liu, Jinbiao Peng, Jie Wu, Junyi Li, Jinyang Li, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2604.12890v2Apr 14, 2026cs.CVcs.AI

TL;DR

Long-horizon multimodal search is constrained by heterogeneous inputs, context explosion, and possible loss of visual signals. LMM-Searcher stores visual assets externally under lightweight UIDs, loads them progressively, and trains a specialized agent with synthesized cross-modal trajectories; it scales to 100 turns and achieves strong open-source benchmark performance.

Problem
Heterogeneous multimodal inputs and their high token costs make context management difficult over long search horizons, while discarding intermediate images can lose important visual signals.
Method
LMM-Searcher stores visual assets in an external file system under textual UIDs, uses on-demand visual tools, and fine-tunes Qwen3-VL-Thinking-30A3B on synthesized cross-modal trajectories.
Results
LMM-Searcher scales to 100 turns and achieves state-of-the-art performance among open-source models across four multimodal search benchmarks.
Takeaways & Limitations
The framework provides an end-to-end approach for handling multimodal content and long-horizon interactions in open-source multimodal deep search agents.
Takeaways & Limitations
The model-merging setting fixes α = 0.8, and a rigorous study of model merging is left for future work.

Abstract

from arXiv · show

Multimodal deep search agents have shown great potential in solving complex tasks by iteratively collecting textual and visual evidence. However, managing the heterogeneous information and high token costs associated with multimodal inputs over long horizons remains a critical challenge, as existing methods often suffer from context explosion or the loss of crucial visual signals. To address this, we propose a novel Long-horizon MultiModal deep search framework, named LMM-Searcher, centered on a file-based visual representation mechanism. By offloading visual assets to an external file system and mapping them to lightweight textual identifiers (UIDs), our approach mitigates context overhead while preserving multimodal information for future access. We equip the agent with a tailored fetch-image tool, enabling a progressive, on-demand visual loading strategy for active perception. Furthermore, we introduce a data synthesis pipeline designed to generate queries requiring complex cross-modal multi-hop reasoning. Using this pipeline, we distill 12K high-quality trajectories to fine-tune Qwen3-VL-Thinking-30A3B into a specialized multimodal deep search agent. Extensive experiments across four benchmarks demonstrate that our method successfully scales to 100-turn search horizons, achieving state-of-the-art performance among open-source models on challenging long-horizon benchmarks like MM-BrowseComp and MMSearch-Plus, while also exhibiting strong generalizability across different base models. Our code will be released in https://github.com/RUCAIBox/LMM-Searcher.

1 Introduction

LMM-Searcher addresses the context explosion and visual-information loss that hinder long-horizon multimodal search by combining file-based context management, on-demand visual access, and specialized training. Across four benchmarks, it scales to 100-turn searches and achieves state-of-the-art performance among open-source models.

Motivation: Multimodal deep search must accumulate heterogeneous textual and visual evidence across long iterative reasoning horizons.High token costs make multimodal context explosion more severe as interactions increase.
Framework: LMM-Searcher stores visual assets externally, maps them to textual UIDs, and retrieves them progressively when fine-grained perception is needed.The framework uses a tailored fetch-image tool for active perception and on-demand loading.
Training: The data synthesis pipeline constructs complex cross-modal multi-hop queries and supports distillation of 12K high-quality trajectories for agentic training.These trajectories fine-tune Qwen3-VL-Thinking-30A3B into a specialized multimodal deep search agent.
Results: 22.3 and 32.9 are the success rates on MM-BrowseComp and MMSearch-Plus, respectively, before enabling 100-turn context management.With context management, performance reaches 30.1 and 34.8 on the same benchmarks.
Results: 30.1 and 34.8 are the reported 100-turn performances on MM-BrowseComp and MMSearch-Plus, establishing state-of-the-art results among open-source models.The framework also reaches 46.7 on MMSearch-Plus with Seed-1.8, indicating generalizability across base models.

2 Related Work

Related work progresses from language-based deep search with external retrieval to multimodal agents that use visual and linguistic tools or internalize interactive visual reasoning. The supplied passages position LMM-Searcher within this transition without detailing its comparison to those paradigms.

Language-based Deep Search Agent: Language-based deep search agents extend LLMs with external search and retrieval mechanisms to address knowledge-boundary limitations.Early systems used retrieval-augmented generation, while later systems equipped models with internet search tools.
Multimodal Deep Search Agent: Multimodal deep search agents equip multimodal models with external tools for complex real-world tasks.Earlier approaches included object detection, image segmentation, and OCR plugins.
Multimodal Deep Search Agent: Recent multimodal frameworks internalize interactive visual capabilities into model reasoning through the thinking-with-image paradigm.

3 Long-horizon Multimodal Context Management

LMM-Searcher decouples heavy visual perception from lightweight long-horizon reasoning by storing images externally, representing them with UIDs, and loading them through specialized tools only when needed.

File-based Multimodal Data Management: LMM-Searcher stores visual assets in an external file system and references them with lightweight textual UIDs instead of inserting raw multimodal content into context.This design preserves fine-grained perceptual capability while reducing context consumption.
File-based Multimodal Data Management: The file system defines a persistent one-to-one mapping f: I → U, assigning each visual asset i a unique identifier u = f(i).The mapping lets the agent locate and later load a target image precisely by UID.
File-based Multimodal Data Management: Webpage content is reorganized into textual summaries and image–caption pairs, with images represented by URLs.
Extended Agentic Tool Interface: The redesigned tools operate over UID-based visual references rather than eagerly loading images into the model context.Search, browsing, and visual-processing tools support this file-based representation.
Extended Agentic Tool Interface: fetch_image retrieves the image corresponding to a UID from external storage for detailed visual inspection.The tool serves as a bridge between the UID space and visual space.
Long-horizon Multimodal Search Workflow: The workflow intercepts search outputs, saves all visual items, replaces raw images with UIDs, and lets the agent fetch or process specific images on demand.This allows extensive search histories across dozens of turns without visual token bloat.

4 Agentic Training for Multimodal Search

The paper builds multimodal search training data by synthesizing visual questions, extending them into obfuscated multi-hop reasoning chains, and generating agent trajectories for supervised fine-tuning. The resulting training setup combines multimodal and textual search datasets while using model merging to address interaction-scale limitations.

Data synthesis: The pipeline synthesizes visual questions from multimodal webpages, extends their reasoning chains, and produces agent trajectories for model training.It targets queries requiring multimodal information to be read and comprehended throughout the search process.
Multimodal Webpage Query Synthesis: A multimodal webpage is parsed to identify a core entity and related image, then a visual question is constrained so webpage text alone cannot answer it.A clue linking the entity and image is combined with the visual question to form a single-hop question.
Multi-hop Reasoning Chain Extension: The multi-hop extension constructs a knowledge graph from the core entity, samples a subgraph, and inserts an explicit image before generating reasoning text leading back to the core entity.Graph construction iteratively expands entities through selected attributes and target entities.
Multi-hop Reasoning Chain Extension: Information irreversibility constraints prevent any single selected attribute from independently revealing the source entity, making each added edge necessary for the final reasoning chain.This design is intended to prevent shortcuts that directly retrieve the answer.
Model Training: The training data combines synthesized and open-source search datasets, filters queries answerable without search, and uses multi-turn supervised fine-tuning with tool responses masked from the loss.The trained checkpoint is merged with MiroThinker-1.7-mini using α = 0.8 to retain most multimodal capabilities while incorporating that model’s strengths.

5 Experiment

Experiments across four multimodal search benchmarks compare LMM-Searcher with baseline frameworks, analyze tool-use and interaction scaling, and ablate training data and the fetch-image tool. The results support competitive benchmark performance, long-horizon scaling, and benefits from synthesized queries and image fetching.

5.1 Experiment Setup: Evaluation covers MM-BrowseComp, VisBrowse-Bench, MMSearch-Plus, and MMSearch, with direct-answer, agent-workflow, and multimodal-search-agent baselines.The MMSearch-Plus comparison uses its single-image subset for fair comparison.
5.2 Overall Performance: LMM-Searcher-30B achieves 28.7 on MMSearch-Plus and improves consistently across all benchmarks when extended to 100 turns with context management.The 100-turn setting achieves state-of-the-art performance on MM-BrowseComp and MMSearch-Plus among open-source models.
5.2.2 Comparison with Other Frameworks: Identical models generally improve under the proposed framework, with Seed-1.8 gaining 13.7% on MMBC and 35.7% on MMSearch-Plus.More capable models benefit more substantially, while weaker visual agentic models show marginal gains.
5.2.2 Comparison with Other Frameworks: Synthesized queries trigger more visual search, image search, and fetch-image calls, indicating deeper multimodal webpage inspection than superficial retrieval.The training-data analysis compares tool-call distributions with those induced by existing open-source multimodal search queries.
5.3 Further Analysis: MMBC and VisBrowse continue improving at 100 turns, while the base model saturates around 20 turns and synthetic-data training plus model merging enhance scaling.The interaction analysis measures accuracy for samples solved correctly within each turn threshold.
5.3 Further Analysis: Adding synthesized queries produces further gains on MMBC and VisBrowse and the best average performance across all benchmarks, while removing fetch-image lowers performance across every benchmark.Without fetch-image, Seed-1.8 drops from 58.0 to 48.5 on VisBrowse and from 73.2 to 71.0 on MMSearch.

6 Conclusion

LMM-Searcher combines file-based visual representation, agentic visual tools, and synthesized training trajectories for multimodal deep search. Evaluations across four benchmarks report advanced open-source-agent performance and support the framework’s scalable design.

6 Conclusion: LMM-Searcher uses file-based visual representation and agentic visual tools to handle multimodal content and long-horizon interactions.The framework is presented as an open-source multimodal deep search agent for complex multimodal queries.
6 Conclusion: Evaluations on four multimodal deep search benchmarks show advanced performance among open-source search agents.The conclusion attributes the result to the combined end-to-end approach, scalable framework design, and data synthesis technique.

A Case Study

The case study traces LMM-Searcher’s multimodal, multi-hop investigation from identifying Want Want to counting trees in the China Times Group’s Wikipedia poster. The trajectory illustrates tool switching, visual inspection, and recovery from failed webpage scraping to obtain the answer 5.

Search and Recovery: When direct scraping of the Wikipedia page returns a 403 Forbidden error, the model searches for the relevant infobox image instead.This demonstrates recovery through an alternative image-search route after the initial webpage extraction attempt fails.
Visual Verification: The model uses a fetched image to inspect whether the retrieved logo is the requested poster, rejecting it before continuing the search.It recognizes that the first retrieved image is a logo and fetches it for visual analysis, then revises the hypothesis about the target image.
Final Answer: 5 trees are counted in the China Times Group newspaper poster, matching the trajectory’s final answer.The reasoning lists five tree locations and concludes that the headquarters image contains five distinct trees.
Entity Resolution: The model identifies Want Want’s mascot, links the company to China Times Group, and interprets the target as the newspaper’s Wikipedia poster.The trajectory resolves the entity and disambiguates which company and poster the question refers to.