PaperlayerView exact revision on arXiv

Source-linked AI summary

InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search

Bohan Hou, Jiuning Gu, Jiayan Guo, Ronghao Dang, Sicong Leng, Xin Li, Xuemeng Song, Jianfei Yang

arXiv:2605.07510v1cs.CVcs.CLcs.IR

TL;DR

Existing benchmarks do not evaluate visual evidence as an intermediate signal that guides later retrieval. InterLV-Search introduces a three-level benchmark for interleaved language-vision agentic search, and evaluations show current multimodal agents remain far from robust open-world interleaved search.

  • Problem

    Existing benchmarks largely confine visual evidence to inputs or treat retrieved visuals as answer endpoints rather than signals guiding subsequent retrieval decisions.

  • Method

    InterLV-Search provides 2,061 examples across three progressively challenging levels spanning visual evidence seeking, controlled interleaved search, and open-web search.

  • Results

    Current multimodal agents remain far from robust open-world interleaved search, with substantial gaps across levels and between single-chain and multi-branch tasks.

  • Takeaways & Limitations

    InterLV-Search shows that success requires locating visual evidence, using it as a search pivot, and maintaining coherent state across long or branching trajectories.

Abstract

from arXiv · show

Existing benchmarks for multimodal agentic search evaluate multimodal search and visual browsing, but visual evidence is either confined to the input or treated as an answer endpoint rather than part of an interleaved search trajectory. We introduce \textbf{InterLV-Search}, a benchmark for Interleaved Language-Vision Agentic Search, in which textual and visual evidence is repeatedly used to condition later search. It contains 2,061 examples across three levels: active visual evidence seeking, controlled offline interleaved multimodal search, and open-web interleaved multimodal search. Beyond existing benchmarks, it also includes multimodal multi-branch samples that involve comparison between multiple entities during the evidence search. We construct Level 1 and Level 2 with automated pipelines and Level 3 with a machine-led, human-supervised open-web pipeline. We further provide InterLV-Agent for standardized tool use, trajectory logging, and evaluation. Experiments on proprietary and open-source multimodal agents show that current systems remain far from solving interleaved multimodal search, with the best model below 50% overall accuracy, highlighting challenges in visual evidence seeking, search control, and multimodal evidence integration. We release the benchmark data and evaluation code at https://github.com/hbhalpha/InterLV-Search-Bench

1 Introduction

InterLV-Search targets interleaved multimodal search, where intermediate visual evidence guides subsequent retrieval decisions across textual and visual modalities. It provides 2,061 examples across three progressively challenging levels, including multi-branch comparisons, and evaluates current multimodal agents with InterLV-Agent, finding the best model below 50% overall accuracy.

  • Motivation: Unlike earlier benchmarks that largely restrict visual information to the initial input, InterLV-Search evaluates visual evidence acquisition during retrieval.Prior multimodal agentic search benchmarks focus largely on textual evidence acquisition, while visual browsing benchmarks treat visual evidence as part of browsing rather than an interleaved search trajectory.
  • Motivation: InterLV-Search requires intermediate visual evidence to guide later retrieval actions rather than serving only as an answer source.Agents dynamically switch between visual and textual evidence acquisition, with one modality determining subsequent retrieval actions in the other.
  • Benchmark: The benchmark contains 2,061 examples across three progressively challenging levels: active visual evidence seeking, offline interleaved multimodal search, and open-web interleaved multimodal search.These levels evaluate visual evidence seeking and offline and open-web interleaved multimodal evidence search.
  • Benchmark: Multi-branch examples require agents to compare multiple entities, explore alternative branches, gather multimodal evidence, and select a branch.This evaluates non-linear search control beyond prior single-chain multimodal search benchmarks.
  • Evaluation: Best-model accuracy remains below 50% overall, showing that current multimodal agents still struggle with interleaved search and evidence integration.The evaluation covers both proprietary and open-source multimodal agents using tool use.
  • Method: InterLV-Agent standardizes unified tool use, trajectory logging, and model comparison, while construction uses automated pipelines for Levels 1–2 and a machine-led, human-supervised process for Level 3.Levels 1–2 leverage multimodal entity data and knowledge-graph chains, whereas Level 3 uses web-capable agents to generate open-world question-answering pairs requiring interleaved multimodal evidence search.

2 InterLV-Search Benchmark

InterLV-Search organizes interleaved multimodal search into three progressively challenging levels, from visual evidence seeking to controlled multi-hop search and open-web execution. The benchmark contains 2,061 examples spanning diverse domains, with multi-hop and multi-branch cases in its later levels.

  • Benchmark Design: The three levels progress from acquiring missing visual evidence, through using visual evidence as multi-hop pivots, to executing interleaved search on the open web.This progression requires visual evidence seeking, evidence-to-query transitions, and ultimately the same search paradigm in an open-web setting.
  • Data Construction: Level 3 uses a semi-automated open-web pipeline, while Levels 1 and 2 are built with fully automated generation, composition, verification, and quality-filtering pipelines.Level 3 combines an automated web-search-capable generation agent with PhD-level human supervision because real webpages provide noisier evidence.
  • Level 1: Visual Evidence Seeking: Level 1 is a Search-to-VQA task requiring retrieval of an implicit target entity followed by image inspection to answer a fine-grained visual attribute question.The benchmark filters out instances solvable by standalone search, standalone VQA, commonsense guessing, metadata lookup, leakage, or ambiguous targets.
  • Level 2: Controlled Interleaved Search: Level 2 requires at least two rounds of visual evidence retrieval, converting an initial visual clue into a subsequent retrieval target before grounding the terminal image.It reuses MMKG-W entity-relation annotations to mine verifiable multi-hop paths and supports initial-visual-probed and intermediate-visual-probed constructions.
  • Benchmark Overview: 2,061 examples comprise 975 Level 1, 225 Level 2, and 861 Level 3 instances across diverse visual and open-web domains.Level 2 and Level 3 have average estimated chain lengths of 6.0 and 6.9 hops, respectively.
  • Benchmark Overview: 340 Level 3 examples, representing 39.5% of that level, are multi-branch cases designed to evaluate comparison between multiple entities during evidence search.These examples extend the benchmark beyond single-chain interleaved search.

3 InterLV-Agent

InterLV-Agent standardizes InterLV-Search evaluation through a reason–act–observe framework with unified multimodal tools, trajectory logging, and model comparison. It also supports controlled offline retrieval and two-level memory for later search levels.

  • 3 InterLV-Agent: InterLV-Agent standardizes evaluation with a reason–act–observe loop, unified tool use, trajectory logging, and model comparison.Its tools include image search, reverse image search, web search, webpage browsing, image cropping, and code execution.
  • 3 InterLV-Agent: For Levels 1 and 2, an offline multimodal retriever enables controlled retrieval over the benchmark corpus.
  • 3 InterLV-Agent: For Levels 2 and 3, two-level memory stores recent interaction rounds and summarizes past observations into compact history notes.

4 Experiment

Experiments show that InterLV-Search remains difficult without search, while tool use helps but does not eliminate weaknesses in complex multimodal search. Analyses identify visual-evidence retrieval, search topology, memory, and interleaved tool use as important determinants of performance.

  • Main results: Without tools, all models achieve limited accuracy, with the best Level 3 model reaching only 20.00%; tool use produces consistent gains for proprietary models.These results indicate that InterLV-Search cannot be reliably solved from parametric knowledge alone.
  • Main results: In Level 3, all models perform substantially worse on multi-branch examples than on single-chain examples, exposing limited robustness to complex search topologies.This supports evaluating non-linear multimodal search control rather than only simpler search chains.
  • Retrieval–answer decomposition: Acc.|Ret. is consistently higher than Acc.|UnRet., especially on Level 2, while limited Ret. R@5—particularly on Level 2—makes target-evidence localization a major bottleneck.Successful retrieval contributes substantially to correct answers, but retrieval recall remains limited.
  • Component ablations: Removing image search causes a significant Level 2 performance drop and can underperform the Direct baseline, whereas its Level 3 impact is smaller but still consistently negative.Richer textual evidence on the open web can partially substitute for visual signals, while visual retrieval remains beneficial; memory has a clearer effect on Level 3 because its trajectories are typically longer.
  • Tool use and trajectories: Level 2 tool use is dominated by image-related retrieval, whereas Level 3 relies more on web/text retrieval while still using image-related tools substantially.A large fraction of executed trajectories contain visual pivots, where visual evidence guides subsequent search rather than only final answering.

5 Conclusion … B Effect of Interaction Budget

The conclusion identifies interleaved multimodal search as exposing challenges beyond existing agentic-search benchmarks, including visual-evidence seeking, search pivots, and coherent state across long or branching trajectories. The supplied excerpt also lists related-work subsections on multimodal search agents and multimodal agentic-search benchmarks, but provides no substantive passages for the Appendix or interaction-budget section.

  • 5 Conclusion: InterLV-Search exposes challenges not captured by existing agentic-search benchmarks, beyond merely accessing external tools.The conclusion frames these challenges across benchmark levels and validates the benchmark design through retrieval–answer decomposition, tool-use analysis, and trajectory inspection.
  • 5 Conclusion: Success depends on locating the intended visual evidence, using it as a search pivot, and maintaining coherent search state across long or branching trajectories.
  • A Related Work: The related-work outline includes a subsection on multimodal search agents.
  • A Related Work: The related-work outline includes a subsection on multimodal agentic-search benchmarks.

C Details of Agentic Framework … A.1 Multimodal Search Agent

The paper presents an agentic framework with dedicated memory, tool, and prompt implementations, alongside case studies spanning all three benchmark levels and Level 3 success and failure analyses. Its related-work discussion motivates multimodal search agents while identifying the continuing limitation of text-centric evidence acquisition.

  • C Details of Agentic Framework: The agentic framework details its memory implementation in subsection C.1.
  • C Details of Agentic Framework: Subsection C.2 describes the framework’s tool implementation.
  • C Details of Agentic Framework: Subsection C.3 specifies the prompts used by the agentic framework.
  • D Case Study: The case study covers examples from Level 1, Level 2, and Level 3.
  • D Case Study: The case study also analyzes success and failure cases on Level 3.
  • A.1 Multimodal Search Agent: Multimodal search agents extend static multimodal reasoning with tool use, web browsing, image search, and iterative evidence acquisition for open-domain tasks.
  • A.1 Multimodal Search Agent: Existing agents remain largely text-centric: visual information is often only an initial input, while later evidence acquisition relies mainly on textual queries and webpage reading.

A.2 Multimodal Agentic Search Benchmark

Prior multimodal search benchmarks evaluate reasoning over user-provided or retrieved visual evidence, while InterLV-Search evaluates agents that actively acquire visual evidence and interleave it with later textual search. This setting targets open-web information seeking in which visual discoveries guide subsequent queries, pages, tools, or branch decisions.

  • Prior browsing benchmarks: Earlier browsing benchmarks such as BrowseComp emphasize difficult multi-hop web search, textual evidence synthesis, browsing depth, and final-answer correctness (Wei et al., 2025; Zhang et al., 2026a).
  • Prior multimodal search benchmarks: MMSearch and FVQA-Test require reasoning over user-provided images alongside external evidence, but visual information is typically pre-specified rather than actively sought during search (Wu et al., 2025; Jiang et al., 2024; Li et al., 2025a).
  • Prior multimodal browsing benchmarks: BrowseComp-VL and VDR-Bench add richer visual inputs, region-level inspection, cropping, and noisy web environments, yet mainly test interpretation of given or retrieved visual evidence (Zeng et al., 2026).
  • InterLV-Search: InterLV-Search requires agents to actively acquire visual evidence, use it to guide later search, and repeatedly transition between textual and visual evidence.Visual cues discovered during browsing can determine the next query, entity, page, tool call, or branch decision.

B Effect of Interaction Budget · C Details of Agentic Framework

Interaction budgets matter substantially more for open-web Level 3 than controlled offline Level 2, where gains saturate after a few interactions. InterLV-Agent operationalizes this search through a reason-act-observe loop, multimodal tools, bounded budgets, and two-level memory.

  • B Effect of Interaction Budget: Level 2 uses smaller interaction budgets, whereas Level 3 requires larger budgets for open-web search, branch exploration, and error recovery.This distinction reflects the shorter controlled offline chains in Level 2 and the broader search demands of Level 3.
  • B Effect of Interaction Budget: Level 2 accuracy improves from direct answering to a few tool interactions but saturates around 5–7 interactions because its evidence paths are fixed.The main challenge is following the intended evidence-to-query chain rather than repeatedly exploring alternative sources.
  • B Effect of Interaction Budget: Level 3 shows large gains when increasing the budget to 5 interactions, while larger budgets support webpage exploration, noisy-evidence recovery, and branch comparisons.The gains are not strictly monotonic for every model because extra tool calls can introduce distractors.
  • C Details of Agentic Framework: InterLV-Agent executes a reason-act-observe loop that selects tools under a limited interaction budget, observes results, and updates memory before the next step.The workflow begins with a user query and reflection on the current search state.
  • C Details of Agentic Framework: A lightweight two-level memory stores recent interactions and summarizes accumulated evidence and unresolved subgoals for subsequent search steps.Short-term memory handles recent context, while long-term memory preserves summarized evidence and open subgoals.
  • C Details of Agentic Framework: The framework integrates text-to-image, image-to-image, web-search, webpage-browsing, screenshot-browsing, image-cropping, and code-execution tools.These tools support multimodal evidence acquisition during agentic search.

C.1 Memory Implementation

InterLV-Agent uses a lightweight running memory updated from prior memory, the current tool query, and the returned observation. Its two-level design preserves recent interaction detail while compactly summarizing long-horizon search state.

  • C.1 Memory Implementation: The running memory is updated at each step from the previous memory, current tool query, and tool-returned result.This memory-update process supplies the next step with the latest interaction state.
  • C.1 Memory Implementation: Short-term memory stores recent interaction rounds, while long-term memory summarizes entities, evidence, visual clues, and unresolved subgoals.Recent rounds retain tool names, queries, and observations directly, whereas long-term summaries keep the context compact for extended interleaved trajectories.

C.2 Tool Implementation · C.3 Prompt of the Agentic Framework

InterLV-Agent unifies online open-web and offline controlled retrieval, while its agentic framework orchestrates iterative multimodal tool use, evidence-based state updates, and answer-equivalence evaluation. The framework uses specialized prompts to guide search decisions, compress running memory, and judge final answers against gold references.

  • C.2 Tool Implementation: InterLV-Agent provides a unified tool interface for online open-web search and offline controlled retrieval across benchmark levels.Offline tools support fixed-corpus evaluation, whereas online tools reproduce the same agentic search loop under realistic open-web conditions.
  • C.2 Tool Implementation: Online Level 3 tools combine image search, web search, reverse image search, webpage browsing, screenshot browsing, image cropping, and code execution.SerpAPI implements web and image search, Playwright enables screenshot browsing, and observations use the top five search results.
  • C.3 Prompt of the Agentic Framework: The agent prompt frames search as iterative decisions among web or image search, page inspection, cropping, code execution, summarization, and final answering.It requires concise tool-oriented actions, brief pre-search reflection, and a single done block when sufficient evidence is collected.
  • C.3 Prompt of the Agentic Framework: The prompt directs agents to inspect webpages when visual evidence or text retrieval is insufficient and to critically assess search results for relevance and reliability.Agents may answer directly from existing knowledge when appropriate rather than searching for every question.
  • C.3 Prompt of the Agentic Framework: The running-memory prompt replaces prior memory with a short, evidence-backed state snapshot that removes superseded, speculative, or irrelevant information.Its structured fields include the goal, status, blocking gap, confirmed facts, best candidates, dead ends, and useful key images.
  • C.3 Prompt of the Agentic Framework: The answer-equivalence judge evaluates only the predicted final answer against the gold answer, accepting equivalent wording, aliases, numerals, specificity, and harmless descriptive additions.It returns exactly YES for clearly matching answers and NO for clearly different answers, without using outside knowledge.

D Case Study … D.2 Level 2 Case

The case studies demonstrate interleaved multimodal search through single-chain and multi-branch trajectories, where visual evidence repeatedly grounds later textual, relational, and fine-grained visual reasoning. Level 1 emphasizes active image retrieval, while Level 2 uses visual evidence as intermediate pivots before terminal VQA.

  • D Case Study: One Level 2 chain identifies Hangzhou from a lake-and-pavilion skyline, Wuhan from a landmark panorama, and Galați through a sister-city relation before querying a final building detail.The search alternates visual descriptions and textual geographic or relational constraints.
  • D Case Study: A Level 3 single-chain case routes from Ariana Grande’s Eternal Sunshine through Wicked, Harriet, Glass Onion, and Almost Famous to identify the vehicle in Nomadland as a van.The trajectory’s intermediate outputs include Harriet → Leslie Odom Jr. → Glass Onion and Almost Famous → Frances McDormand → Nomadland.
  • D Case Study: The cases include both single-chain and multi-branch Level 3 trajectories, with the latter requiring comparison across multiple entities before selecting the branch for final visual interpretation.The multi-branch example starts from separate Amazon and World Wetlands Day pages and converges through official flags.
  • D.1 Level 1 Case: Level 1 begins with a textual information need, requires semantic reasoning to retrieve a matching image from a local collection, and ends with a fine-grained visual question.The retrieved image must be inspected for consistency with the question context before answering.
  • D.2 Level 2 Case: The Level 2 case interleaves multiple visual-search blocks with textual relation constraints, using landmark descriptions to identify cities before returning to a terminal image for fine-grained VQA.Its trajectory uses visual evidence as intermediate search pivots rather than only as the final VQA source.

D.3 Level 3 Case

Level 3 cases require agents to alternate visual localization and textual verification throughout open-web search. They cover multi-branch route comparison and long single-chain trajectories where visual evidence repeatedly redirects later searches.

  • Multi-branch case: Multi-branch cases require agents to localize candidate images or webpages, verify titles, years, and runtimes, compare textual evidence across routes, and retain the surviving branch.The routes begin from multiple visual clues tied to different film pages and continue from the comparison result.
  • Single-chain case: Single-chain cases begin from visually described music clues and repeatedly pivot through film posters, cast members, and film pages during a long interleaved trajectory.One example identifies an artist from a pastel-toned breakup album, uses textual evidence to route through Wicked, and then pivots on the poster and green-skinned co-lead.
  • Level 3 Case: Level 3 combines multi-branch cases that compare parallel routes before continuation with single-chain cases requiring long open-web trajectories driven by recurring visual pivots.Both patterns alternate visual localization with textual verification instead of treating images as fixed inputs or final VQA endpoints.

D.4 Success and Failure Cases on Level 3

Level 3 success requires visual evidence to steer multi-branch open-web search, preserve search state, compare routes, and return to the selected branch for terminal inspection. Failures arise when agents rely on broad textual search without grounding visual pivots, preventing correct branch recovery and final visual answers.

  • Successful case: The successful trajectory alternates image search, official-page text lookup, branch comparison, and terminal visual inspection rather than treating vision as an endpoint.This workflow grounds visual clues, recovers textual runtimes, preserves the selected branch, and performs a final visual question answering step.
  • Failed case: The Tate/Picasso and Berlinale/Niigata failure never visually grounds either branch into its local-symbol system and therefore answers blue instead of the correct purple border color.The supplied failure trace identifies the model answer as Blue and the correct answer as Purple.
  • Failed case: Another failure uses broad textual searches without grounding bus graphics, so it misses the COP30 guardian identity and Pará branch, confusing the flag’s red field with the correct white diagonal band.The model answer is Red, while the correct answer is White.
Loading 2605.07510v1…