InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search

1 Introduction

• Prior benchmarks focus on textual evidence or use retrieved visuals to answer local questions, overlooking how visual cues can determine what to search for next.Fine-grained cues such as logos, inscriptions, people, or spatial relations can reveal the next query, entity, webpage, tool call, or branch.Source: §1 · pp. 1–2
• Interleaved multimodal search requires recurrent visual and textual evidence segments, with later retrieval conditioned on earlier evidence.Evidence in one modality guides subsequent retrieval actions in the other, rather than serving only as input or final-answer support.Source: §1 · p. 2
• InterLV-Search spans active visual evidence seeking, controlled offline search, and open-web search, including multi-branch comparisons among entities.Its levels progress from seeking visual evidence from textual needs to multi-hop search in controlled and noisy, dynamic web settings; Level 3 includes branching beyond single-chain search.Source: §1 · pp. 2–3
• Automated Level 1 and 2 pipelines and a machine-led, human-supervised Level 3 process produce 2,061 examples; InterLV-Agent standardizes tool use and trajectory logging.The benchmark jointly covers text-to-visual search, visual multi-hop retrieval, recurrent vision–text interleaving, and multimodal multi-branch search.Source: §1 · p. 3
• The best evaluated model remains below 50% overall accuracy, even with tool use.Experiments with proprietary and open-source multimodal agents reveal ongoing difficulty with interleaved search and evidence integration.Source: §1 · p. 3

2 InterLV-Search Benchmark

• The benchmark progresses from visual evidence seeking to controlled interleaved search and then open-web search, using automated construction for Levels 1–2 and human-supervised curation for Level 3.Level 3 uses a semi-automated pipeline because real webpages and noisier evidence sources require human verification and refinement.Source: §2 · p. 3

2.1 Level 1: Active Visual Evidence Seeking

• Level 1 frames active visual evidence seeking as Search-to-VQA: agents retrieve an implicitly specified entity, inspect its image, and answer with a grounded attribute.Instances are composed from separate target-search and image-dependent VQA queries, then filtered for visual dependence, search specificity, answerability, leakage, and search-to-VQA coupling.Source: §2.1 · pp. 3–4

2.2 Level 2: Controlled Offline Interleaved Search

• Level 2 tests whether agents can use visual evidence as intermediate pivots across at least two retrieval rounds in controlled multi-hop search.It reuses MMKG-W entity-relation paths and creates initial-visual-probed and intermediate-visual-probed samples; intermediate probes use a visually similar bridge entity before KG traversal resumes.Source: §2.2 · p. 5

2.3 Level 3: Open-Web Interleaved Multimodal Search

• Level 3 extends interleaved search to noisy open-web sources and adds multi-branch tasks that compare evidence across parallel entity-specific routes.GPT-5.4-Thinking generates candidate questions and evidence chains; AI checks and PhD-level human review refine them, and filtering screens for factuality, leakage, source quality, and no-search solvability.Source: §2.3 · p. 6

2.4 Benchmark Statistics

• 2,061 examples span three levels; Level 3 includes 340 multi-branch examples (39.5%) for parallel-route search and evidence-based branch selection.The benchmark contains 975 Level 1, 225 Level 2, and 861 Level 3 examples; average estimated chain lengths are 6.0 hops for Level 2 and 6.9 for Level 3.Source: §2.4 · p. 6

• InterLV-Agent provides a reference framework for evaluation with a reason–act–observe loop, unified tool use, trajectory logging, and model comparison.Source: §3 · pp. 6–7
• Its tools include image and reverse-image search, web search, webpage browsing, image cropping, and code execution; Levels 1 and 2 also use an offline multimodal retriever for controlled corpus retrieval.Source: §3 · p. 7
• For Levels 2 and 3, InterLV-Agent uses two-level memory, with short-term memory for recent interaction rounds and long-term memory summarizing past observations into compact history notes.Source: §3 · p. 7

4.1 Experimental Settings

• Evaluation uses final-answer accuracy judged for semantic equivalence under shared InterLV-Agent protocols and level-specific budgets of 3, 7, and 10 interactions.For Level 3, at most the first three tool calls in an interaction are executed when multiple calls are issued.Source: §4.1 · p. 7

4.2 Main Results

• Tool-free models reach at most 20.00% on Level 3, while tool-use gains vary substantially and open-source search agents often fail to improve.Performance generally falls from Level 1 to Level 3, although some Level 3 +Tool scores exceed Level 2 with larger interaction budgets; all models fare worse on multi-branch than single-chain Level 3 examples.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 Target Retrieval–Answer Decomposition for Levels 1 and 2

• On Levels 1 and 2, answer accuracy is higher when the target entity is retrieved, but limited retrieval recall—especially on Level 2—remains a major bottleneck.The decomposition compares retrieval recall and answer accuracy for Gemini-3.1-Pro, GPT-5.4, and Claude-Sonnet-4.6.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 Further Analysis

• Removing image search substantially hurts Level 2 and also degrades Level 3, while memory matters more for Level 3’s longer, noisier trajectories.Sparse offline evidence makes image search important for finding visual pivots; richer web text can partly substitute for visual signals at Level 3, where memory helps track entities, cues, and subgoals.Source: §4.4 · pp. 8–9, Table 4 · p. 9
• Top models rely mainly on image-related retrieval at Level 2 and more on web/text retrieval at Level 3, where image tools still account for substantial usage.This pattern reflects Level 2’s visual–entity transitions and Level 3’s diverse but noisy open-web evidence.Source: §4.4 · p. 9, Figure 4 · p. 9
• A large fraction of executed trajectories contain visual pivots, with especially high rates at Level 2 and substantial rates at Level 3 despite open-web noise.The trajectory judge counts visual evidence as a pivot when it guides subsequent search rather than only the final answer.Source: §4.4 · p. 9, Figure 5 · p. 9

4.5 Case Study

• The case studies illustrate Level 1 image retrieval and Level 2 and Level 3 chains in which visual evidence guides later entity transitions.Level 2 alternates visual and textual evidence, while Level 3 extends interleaved chains across noisier open-web sources.Source: §4.5 · pp. 9–10, Figure 6 · p. 10

5 Conclusion

• Performance gaps across benchmark levels, single-chain and multi-branch tasks, and retrieved versus unretrieved cases show that current multimodal agents remain far from robust open-world interleaved search.The paper presents retrieval–answer decomposition, tool-use analysis, and trajectory inspection as validating InterLV-Search’s design.Source: §5 · p. 10
• Interleaved multimodal search requires agents to locate intended visual evidence, use it to steer search, and maintain coherent state across long or branching trajectories.The conclusion emphasizes that success depends on more than access to external tools.Source: §5 · p. 10
• InterLV-Search is intended to support future agents that more reliably acquire, connect, and act on multimodal evidence during search.Source: §5 · p. 10

A.1 Multimodal Search Agent

• Multimodal search agents extend static multimodal reasoning with tools, browsing, image search, and iterative evidence acquisition for open-domain evidence beyond the input or model parameters.They decompose queries into subgoals, inspect textual and visual results, then update subsequent actions. The passage cites prior work (Jiang et al., 2024; Wu et al., 2025; Li et al., 2025b).Source: §A.1 · p. 14
• Existing multimodal search agents often use visuals only as initial input for identifying, describing, or disambiguating entities, leaving later evidence acquisition largely text-driven.This limits visual evidence’s role as an active part of the search trajectory.Source: §A.1 · p. 14
• In open-web search, visual observations such as logos, inscriptions, posters, emblems, screenshots, and spatial layouts can reveal entities or constraints that determine what to search next.InterLV-Search evaluates whether agents retrieve and understand visual evidence and use it as a pivot in interleaved language–vision search.Source: §A.1 · p. 14

A.2 Multimodal Agentic Search Benchmark

• Earlier benchmarks emphasized multi-hop textual search, while MMSearch and FVQA-Test typically supplied visual information in the query image or auxiliary context (Wei et al., 2025; Zhang et al., 2026a; Wu et al., 2025; Jiang et al., 2024; Li et al., 2025a).They evaluated synthesis of textual evidence or reasoning over user-provided images with external evidence, rather than active visual evidence seeking.Source: §A.2 · pp. 14–15
• BrowseComp-VL and VDR-Bench add richer visual inspection, while BrowseComp-V 3 and VisBrowse introduce active image search; retrieved images often remain final VQA-style evidence (Geng et al., 2026; Zeng et al., 2026; Zhang et al., 2026a,b).These benchmarks underexplore visual evidence as a pivot that determines subsequent retrieval targets.Source: §A.2 · p. 15
• InterLV-Search requires agents to acquire visual evidence, use it to guide later search, and repeatedly transition between textual and visual evidence across potentially branching trajectories.Visual cues can determine the next query, entity, page, tool call, or branch decision.Source: §A.2 · p. 15

B Effect of Interaction Budget

• On Level 3, increasing the budget from direct answering to 5 interactions greatly improves accuracy for all models, and larger budgets help further despite non-monotonic gains.More interactions enable alternative webpage exploration, recovery from noisy evidence, and branch comparisons, but can add distractors when evidence is poorly filtered or integrated.Source: Figure 7 · p. 15, B Effect of Interaction Budget · p. 15
• On Level 2, accuracy gains from increasing the budget saturate around 5–7 interactions because fixed evidence paths mainly test following the intended evidence-to-query chain.Level 2 uses smaller budgets because its controlled offline chains are shorter.Source: B Effect of Interaction Budget · p. 15, Figure 7 · p. 15
• The budget ablation supports that InterLV-Search tests long-horizon interleaved search rather than shallow single-step retrieval.Level 3's stronger budget effect reflects active evidence acquisition in open-web search.Source: B Effect of Interaction Budget · p. 15

C Details of Agentic Framework

• The agent repeatedly reflects on its search state, selects a tool within a limited interaction budget, observes the result, and updates memory.The loop begins with a user query and uses updated memory to guide the next step.Source: C Details of Agentic Framework · p. 16
• Supported tools include text-to-image and image-to-image search, web search, webpage and screenshot browsing, image cropping, and code execution.Recent interactions and compact long-term summaries enable standardized tool use, trajectory logging, and evaluation across models.Source: C Details of Agentic Framework · p. 16

C.1 Memory Implementation

• At each step, a memory-update prompt uses the previous memory, current tool query, and returned observation to produce the next running memory.The passages denote these inputs as M_t−1, q_t, and o_t, respectively.Source: §C.1 · p. 16
• Short-term memory formats recent rounds with tool names, queries, and observations, while long-term memory summarizes entities, evidence, visual clues, and unresolved subgoals.The two-level design preserves recent local context without summarization loss while keeping longer search trajectories compact.Source: §C.1 · p. 16

C.2 Tool Implementation

• Online tools support image and web search, reverse image search, webpage text retrieval, and screenshot browsing for visual evidence.Image and web search use SerpAPI; screenshot browsing uses Playwright, and the model observes the top five search results.Source: §C.2 · p. 17
• Offline tools use Qwen3-VL-Embedding-2B to retrieve text, text-linked images, images from text queries, and visually similar images.These modes support visual evidence seeking and controlled interleaved search without live-web stochasticity.Source: §C.2 · p. 17
• Level 1 and 2 use fixed-corpus local retrieval, while Level 3 uses online tools to evaluate the same search loop under open-web conditions.InterLV-Agent provides a unified interface for both modes.Source: §C.2 · pp. 16–17

System Prompt

• The system prompt directs the agent to take concise, one-at-a-time search actions, reflect briefly before searches, assess relevance, and finish when evidence suffices.It specifies web and image search, page inspection, cropping, code execution, summarization, and final-answer actions.Source: System Prompt · pp. 17–18

Input Prompt

• The input prompt supplies available skills, recent observations, and—when present—running memory, then asks the agent to choose the next action or answer.Source: Input Prompt · pp. 18–19

Memory Prompt

• The memory prompt requires each update to replace prior memory with a short, evidence-backed state snapshot, dropping superseded or speculative content.Its format tracks goal, status, blocking gap, confirmed facts, candidates, dead ends, and useful images, omitting empty fields.Source: Memory Prompt · pp. 19–20

GPT Judge Prompt

• The judge compares only the predicted final answer with the gold answer, ignores reasoning and outside knowledge, and permissively accepts equivalent wording or forms.It returns YES for clearly equivalent answers and NO for clearly different ones.Source: GPT Judge Prompt · pp. 20–21

D Case Study

• A Level 2 case links city clues to a twin-city relation and a final image question about the color of a building’s central dome.The question moves from a lake-and-pavilion city to another provincial capital, then through a southeastern European port to a South Asian coastal metropolis.Source: D Case Study · p. 22
• A Level 3 single-chain case follows clues from a 2024 breakup album through a musical, biopic, mystery, and road movie to identify a vehicle.The album clue refers to Ariana Grande’s Eternal Sunshine.Source: D Case Study · p. 23
• A Level 3 multi-branch case begins with an Amazon investment page and a World Wetlands Day messages page, then compares flags before reading a motto.The question directs the search from the smaller visual-unit count to the phrase beneath the central figure.Source: D Case Study · p. 24

<answer> ORO EN PAZ – FIERRO EN GUERRA </answer>

• The multi-branch case returns “ORO EN PAZ – FIERRO EN GUERRA” as the phrase beneath the central figure on the selected flag.The trajectory compares visual-unit counts in two symbols and continues from the smaller one.Source: D Case Study · p. 24

D.1 Level 1 Case

• Level 1 asks agents to infer what image to seek from a textual need, retrieve it from a local collection, verify its fit, and answer from visual details.This case evaluates active visual evidence seeking, from text-based search reasoning through image-grounded answering.Source: §D.1 · p. 25

D.2 Level 2 Case

• Level 2 controlled search uses visual evidence as intermediate pivots, interleaved with textual relations, before returning to a terminal image for fine-grained VQA.The examples use skyline or landmark descriptions to ground intermediate city entities and sister-city and country constraints to guide textual search.Source: §D.2 · p. 25

D.3 Level 3 Case

• In the multi-branch case, agents compare textual metadata across visually grounded film routes, select a surviving branch, then inspect its target image.The case tests open-web search, branch-level control, and multimodal evidence integration.Source: §D.3 · p. 25
• In the single-chain example, visual clues repeatedly redirect a long film-and-cast search, and the answer depends on inspecting the terminal image.The trajectory moves from an album clue through Wicked, Cynthia Erivo, Harriet, and later film relations to a final road-movie composition.Source: §D.3 · p. 25
• Level 3 tests parallel-route comparison and long single-chain search, with visual evidence repeatedly pivoting subsequent open-web queries.Both patterns require alternating visual localization and textual verification, rather than treating images only as inputs or final VQA endpoints.Source: §D.3 · p. 25

D.4 Success and Failure Cases on Level 3

• These cases show Level 3 tests open-web search, long search-state maintenance, and branch comparison, requiring visual evidence as intermediate pivots rather than endpoint browsing or text-centric search.Agents must compare multimodal evidence across routes and continue searching from the selected branch.Source: §D.4 · pp. 25–26

Case 1: Successed Example

• The successful search visually grounded three film branches, compared runtimes 64/79/93 minutes to select Krakatoa, then inspected its target image and identified a volcanic eruption.The visual inspection focused on the natural event behind the title, not the lettering.Source: Case 1: Successed Example · pp. 27–28

Case 2: Failed Multi-Branch Example

• The failed agent used broad textual search without grounding either branch, missed counts 2 versus 7 and terminal inspection, and answered blue instead of purple.The correct left-half border color on the smaller-count Málaga branch was purple.Source: Case 2: Failed Multi-Branch Example · p. 31

Case 3: Failed Single-Chain Example

• The chain uses COP30 itinerary buses as a visual pivot through their guardian identity, recovers host city Belém and state branch Pará, then asks about the diagonal on the final flag.This illustrates a single-chain Level 3 bottleneck: the agent must use an intermediate visual clue to redirect its search path, not merely search for related webpages.Source: Case 3: Failed Single-Chain Example · pp. 31, 33
• Broad, unrelated web results never completed the route from the bus visual identity to COP30’s host city and the state flag.The agent relied on broad textual queries and did not use image search or visual inspection to identify the guardian identity on the buses; it consequently failed to recover the Pará branch.Source: Case 3: Failed Single-Chain Example · p. 33
• The model answered “Red” when the correct color of the flag’s diagonal band was “White.”It confused the flag’s red field with the diagonal band after failing to ground the intended visual route.Source: Case 3: Failed Single-Chain Example · p. 33