Source-linked AI summary
DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories
Chenlong Deng, Mengjie Deng, Junjie Wu, Dun Zeng, Teng Wang, Qingsong Xie, Jiadeng Huang, Shengjie Ma, Changwang Zhang, Zhaoxiang Wang, Jun Wang, Yutao Zhu, Zhicheng Dou
TL;DR
Existing multimodal retrieval systems evaluate query-image relevance in isolation, despite realistic visual histories containing temporal and cross-image dependencies. DeepImageSearch reframes retrieval as autonomous, multi-step exploration, constructs DISBench through model-assisted association mining and human verification, and finds substantial difficulty for current models, with the best model achieving 28.7 EM. The paper concludes that corpus-level contextual reasoning over visual histories remains an open problem.
Problem
Existing retrieval systems rely on independent semantic matching, overlooking structured dependencies across images in realistic visual histories.
Method
DeepImageSearch reformulates retrieval as agentic exploration, while DISBench uses model-driven context discovery, human verification, and a specialized baseline framework.
Results
28.7 EM is achieved by the best-performing model on DISBench, and experiments show current models struggle with long-horizon exploration and cross-event associations.
Takeaways & Limitations
Corpus-level contextual reasoning over visual histories remains an open problem and DISBench provides a testbed for advancing agentic retrieval.
Takeaways & Limitations
DISBench contains 122 queries across 57 users, smaller than conventional retrieval benchmarks, because strict human verification prioritizes reliability over scale.
Abstract
from arXiv · showhide
Existing multimodal retrieval systems excel at semantic matching but implicitly assume that query-image relevance can be measured in isolation. This paradigm overlooks the rich dependencies inherent in realistic visual streams, where information is distributed across temporal sequences rather than confined to single snapshots. To bridge this gap, we introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task. Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues. We construct DISBench, a challenging benchmark built on interconnected visual data. To address the scalability challenge of creating context-dependent queries, we propose a human-model collaborative pipeline that employs vision-language models to mine latent spatiotemporal associations, effectively offloading intensive context discovery before human verification. Furthermore, we build a robust baseline using a modular agent framework equipped with fine-grained tools and a dual-memory system for long-horizon navigation. Extensive experiments demonstrate that DISBench poses significant challenges to state-of-the-art models, highlighting the necessity of incorporating agentic reasoning into next-generation retrieval systems.
1. Introduction
Existing retrieval systems usually match queries to images independently, but complex intents in personal visual histories require contextual reasoning. DeepImageSearch reframes retrieval as active, multi-step exploration and introduces DISBench to evaluate it.
- Existing Paradigm: Independent instance matching scores each candidate image separately, forming the backbone of modern visual search.This paradigm measures query-image semantic relevance without considering relationships among images.
- Motivation: Complex personal-album queries rely on events, time, and companions that a single visual embedding may not capture.Reasoning-intensive methods improve semantic understanding but retain the independent-matching assumption.
- DeepImageSearch: DeepImageSearch reformulates retrieval as agentic exploration requiring autonomous planning, fine-grained perception, and evidence chains across scattered clues.The paradigm changes retrieval from one-shot matching into multi-step reasoning over the corpus.
- DISBench: DISBench uses a human-model collaborative pipeline in which vision-language models mine spatiotemporal associations before human verification.The pipeline addresses the cognitive load of manually discovering subtle cross-event connections.
- Findings: 28.7 EM is achieved by the best-performing model on DISBench, despite near-ceiling performance on conventional retrieval benchmarks.Error analysis identifies long-horizon exploration and cross-event association discovery as persistent difficulties.
- Contributions: The paper contributes a context-dependent retrieval paradigm, DISBench, and a specialized agent framework exposing capability gaps in long-horizon exploration.These contributions establish a benchmark and baseline for future research on visual-history retrieval.
2. Related Work
Existing multimodal benchmarks have expanded beyond semantic matching, but most still evaluate query-target relevance independently. DeepImageSearch instead evaluates corpus-level contextual reasoning over raw visual histories.
- Multimodal Retrieval: Multimodal representation learning has progressed from vision-text alignment to advanced multimodal foundation architectures.Benchmarks now cover broader capabilities, including compositional understanding and temporal video retrieval.
- Benchmark Gap: Most existing retrieval benchmarks overlook structured associations within data because they evaluate query-target relevance independently.DeepImageSearch addresses this limitation through corpus-level contextual reasoning over visual histories.
- Multimodal Agents: Multimodal-agent benchmarks evaluate planning and reasoning across web search, GUI manipulation, gaming, and embodied intelligence.DeepImageSearch extends this agentic evaluation perspective to context-aware image retrieval.
3. DISBench: The Proposed Dataset
DISBench formulates retrieval as context-aware set prediction over chronological visual histories, with queries requiring event localization or cross-event association. Its semi-automated construction combines model-based discovery with human verification.
- Task Definition: The task predicts a target subset R ⊆ C from a query Q and chronological visual history C, modeling relevance as P(R|Q,C).Each image includes visual content and metadata such as timestamps and GPS coordinates.
- Task Definition: Text-only queries may require locating visual anchors before performing spatiotemporal reasoning.A referenced logo, for example, can identify the relevant event that anchors subsequent retrieval.
- Query Types: Intra-Event queries locate an event before filtering targets, whereas Inter-Event queries compare recurring elements across the timeline.The two query types require event-level localization and cross-event association discovery, respectively.
- Dataset Criteria and Source: DISBench preserves temporal continuity and user-centric coherence by retaining photoset structure while hiding event boundaries during evaluation.The source data preserves users, photosets, and photos, with complete histories accumulated up to 2,000 photos per user.
- Data Construction: The construction pipeline parses visual clues and person attributes, mines associations through retrieval and verification, builds a heterogeneous memory graph, and samples subgraphs for query synthesis.Human annotators then verify candidate clues, reasoning chains, and contextual difficulty.
- Human Verification: 122 queries remain from 2,000 candidates after strict human filtering, yielding a 6.1% retention rate.The filtering requires distractors that cannot be resolved through direct semantic matching alone.
- Dataset Statistics: DISBench contains 122 queries across 57 users and 109,467 photos, with histories spanning 3.4 years on average.Queries target 3.84 images on average; 46.7% are Intra-Event and 53.3% are Inter-Event.
4. ImageSeeker: An Agentic Framework
ImageSeeker is a baseline agent for exploring large visual histories through coordinated retrieval, metadata reasoning, and visual verification. It uses targeted tools and memory mechanisms to sustain multi-step reasoning under context limits.
- Framework Overview: ImageSeeker provides a baseline agent framework tailored to visual-history exploration.Its design addresses the need to navigate large photo collections rather than rank images independently.
- Planning: Structured prompting guides query decomposition and constraint identification during planning.The design combines planning with tools that construct multi-step reasoning paths.
- Tools: The framework coordinates semantic retrieval, metadata reasoning, and visual verification through a specialized tool set.These capabilities support finding relevant photos, applying temporal or spatial constraints, and inspecting candidates directly.
- Tools: ImageSearch, GetMetadata, FilterMetadata, ViewPhotos, and WebSearch support retrieval, metadata filtering, visual inspection, and external-entity resolution.Saved photo subsets let agents perform subsequent retrieval or filtering within previously identified collections.
- Memory: Explicit state memory, global memory, local memory, and tool-use memory preserve subsets, goals, subgoals, findings, and failure experiences.This separation maintains direction and operational efficiency under strict context constraints.
5. Experiments
Experiments evaluate agentic and embedding-based retrieval on DISBench, alongside ablations, test-time scaling, and manual failure analysis. Results show that contextual retrieval remains difficult, with reasoning, cross-event association, and state management as central bottlenecks.
- Experimental Setup: The evaluation compares multimodal agents using ImageSeeker with conventional embedding retrievers on DISBench.Agentic models and embedding models are assessed under separate retrieval protocols, while Gemini-3-Flash-Preview with Qwen3-VL-Embedding-8B supports detailed analyses.
- Main Results: 55.0 F1 and 28.7 EM are achieved by the best-performing model, while complete target-set identification remains difficult.The low Exact Match reflects missed relevant images and false positives.
- Main Results: Inter-Event queries are substantially harder for stronger models, indicating that long-range cross-event association is a primary bottleneck after basic agentic capabilities.Switching embedding size from 2B to 8B has inconsistent effects, suggesting retrieval quality is not the systematic bottleneck.
- Direct Retrieval: Recall@3 around 10-14% and NDCG@5 only 13-17% show poor performance for embedding-based retrieval on DISBench.These limited scores are largely attributable to chance because independent matching cannot enforce context-dependent constraints across images.
- Ablation Study: Removing GetMetadata lowers F1 by 5.7 points, while removing explicit state memory harms Inter-Event queries more than Intra-Event queries.Metadata tools provide temporal and spatial constraints, and state memory supports accumulating evidence across exploration steps.
- Test-time Scaling: Best@k rises from 35.4 to 60.8 as parallel test-time instances increase, whereas Majority Voting substantially trails the Best@k ceiling.The results indicate latent task-solving potential but difficulty selecting the correct reasoning path.
- Error Analysis: Reasoning Breakdown accounts for 36–50% of sampled failures, with visual discrimination, episode misgrounding, and clue mislocalization also prominent.The analysis points to planning, constraint tracking, and state management as needed improvements beyond visual understanding alone.
6. Conclusion
DeepImageSearch reframes image retrieval as corpus-level contextual reasoning over visual histories and introduces DISBench with a semi-automated construction pipeline. Experiments show that state-of-the-art models still find this task challenging, leaving contextual reasoning over visual histories open for research.
- 6. Conclusion: DeepImageSearch advances image retrieval from independent semantic matching to corpus-level contextual reasoning over visual histories.The paper constructs DISBench through model-driven context discovery with human verification and provides a baseline agent framework for exploration.
Impact Statement
The paper presents contextual image retrieval as a foundation for assistants that help users navigate visual memories, while recognizing privacy considerations in personal photo analysis.
- The proposed retrieval paradigm supports intelligent assistants that help users navigate visual memories through natural language.Potential applications include memory assistance, family archive organization, and digital legacy preservation.
- Personal photo collections introduce privacy considerations that require safeguards when such technologies are deployed.The benchmark uses publicly licensed data, and the intended application is searching users’ own collections.
- The benchmark is constructed from publicly accessible YFCC100M images whose licensing permits research use.The released benchmark is intended to respect those licensing requirements.
B. Limitations
The paper’s limitations concern benchmark scale, single-source data, metadata assumptions, baseline scope, and implementation choices in data construction and verification.
- Benchmark Scale: DISBench contains 122 queries across 57 users, making it smaller than conventional retrieval benchmarks with thousands of queries.The authors attribute the 6.1% retention rate to strict quality standards and prioritize reliability over size.
- Data Source: Using only YFCC100M may introduce demographic biases toward Flickr users, although it improves consistency and reproducibility.The authors state that the pipeline is dataset-agnostic and can be applied to other photo collections.
- Metadata Availability: The task assumes timestamps and geographic coordinates are available, which may not hold in all real-world photo collections.Metadata imputation is identified as a possible way to handle missing values in deployment.
- Agent Framework: ImageSeeker is a modular baseline rather than a comprehensive solution, leaving reflection, backtracking, and learned planning for future work.Its modularity allows replacement of the retrieval backbone, memory mechanism, or planning strategy.
- Quality Control Criteria: Query synthesis requires visually ambiguous targets, contextual identifiability, and strong-to-weak reasoning flows verified through context-removal tests.These constraints make target uniqueness depend on events, persons, or temporal context rather than appearance alone.
D.1. Tool Specifications
The tool specification supports stateful, multi-step exploration by combining visual search, metadata filtering, web search, explicit subsets, compressed context, and structured query decomposition.
- State Management: ImageSearch and FilterMetadata let agents save named photo subsets and reuse them through search within or filter within parameters.This supports multi-step filtering and reasoning over previously identified photos.
- Tool Behavior: FilterMetadata normalizes location aliases and can fall back to geocoding when direct address matching fails.WebSearch returns ranked results with titles and snippets through the Serper API.
- Memory: Compressed Context Memory summarizes high-level goals, accumulated findings, current subgoals, and immediate action plans when context approaches its limit.The system uses session memory and working memory as the two summary components.
- Query Understanding: The query-understanding framework separates an episode, its stepwise breakdown and relations, and the target photos with visual and metadata constraints.This separation prevents anchor constraints from being conflated with target requirements.
- Agent Behavior: Agents are instructed not to transfer anchor-photo appearance to targets and to make best-effort inferences without requesting clarification.Temporal phrases primarily constrain time or location unless the query explicitly requires the referenced event in the results.
D.4. Experimental Configurations
The experiments standardize model access, retrieval backbones, interaction limits, memory compression, and evaluation across agentic and direct-retrieval settings.
- Backbone Models: Open-source models run locally through vLLM, while proprietary models are accessed through official APIs with identical tool interfaces.Models use default temperature settings.
- Embedding Models: Qwen3-VL-Embedding is evaluated at 2B and 8B parameters, with per-user indexes containing all photos in each visual history.Seed-1.6-Embedding is additionally evaluated for direct retrieval baselines.
- Hyperparameters: The agent permits 30 interaction turns and uses a 128K-token context limit that triggers compressed context memory.ImageSearch returns 20 results by default, and ViewPhotos accepts at most 20 photo IDs per call.
- Evaluation Protocol: Evaluation computes set-level Exact Match and F1 for agents, while retrieval baselines report MAP@k, Recall@k, and NDCG@k for k ∈{1, 3, 5, 10}.Each query runs independently with a fresh agent state until completion or the 30-turn limit.
E. Qualitative Examples
The qualitative traces show agents resolving temporal ambiguity successfully in one multi-event search, but failing when visual discrimination and retrieval coverage are insufficient.
- Case 1: Successful Multi-Event Disambiguation: The successful case identifies multiple fireworks events, checks their dates, prunes an empty branch, and narrows the target to photos from July 31.Three candidate events were identified; filtering produced 26 photos for August 5, none for June 6, and 8 for July 31.
- Case 1: Successful Multi-Event Disambiguation: Scoped searches found three high-confidence sea matches for July 31 but low similarity for August 5, whose inspected photos were parade or carnival scenes.Visual verification confirmed 3 of 8 July 31 candidates and 0 of 20 August 5 results.
- Case 1: Successful Multi-Event Disambiguation: The agent correctly concluded that August 5 was spent at town events rather than at the beach and returned three correct photo identifiers.The final output is marked correct.
- Case 2: Visual Discrimination Failure: In the failure case, the agent identified the earliest April 18 photo, recognized a distant church spire, and retrieved 30 close-up candidates.The anchor photo was timestamped 10:09, and two image-search variants each returned 30 candidates.
- Case 2: Visual Discrimination Failure: The agent incorrectly rejected a rank-12 image depicting the same church, stopped after confirming one candidate, and returned an incomplete result.The ground truth contained two identifiers, while the output contained only one; another target never appeared in search results.
- Case 2: Visual Discrimination Failure: The failure was attributed to both visual discrimination error and insufficient retrieval coverage, compounded by the absence of a location-based pre-filter.Because the building was very small in the anchor, pure embedding retrieval produced a large, noisy candidate set.