Source-linked AI summary
Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models
Yu Zeng, Wenxuan Huang, Zhen Fang, Shuang Chen, Yufan Shen, Yishuo Cai, Xiaoman Wang, Zhenfei Yin, Lin Chen, Zehui Chen, Shiting Huang, Yiming Zhao, Xu Tang, Yao Hu, Philip Torr, Wanli Ouyang, Shaosheng Cao
TL;DR
Existing benchmarks inadequately evaluate Vision-DeepResearch because textual shortcuts, model priors, near-duplicate whole-image retrieval, and shallow text search can replace realistic multimodal search. The paper introduces VDR-Bench, a 2,000-instance human-verified benchmark and a multi-round cropped-search workflow; experiments show the workflow substantially improves realistic visual retrieval performance.
Problem
Existing benchmarks do not reliably require genuine visual verification and instead use shortcut-prone or overly idealized visual and textual search settings.
Method
The paper constructs VDR-Bench with 2,000 human-verified instances and proposes multi-round cropped search for iterative visual query refinement.
Results
A simple multi-round cropped-search strategy substantially improves performance on realistic visual retrieval tasks.
Takeaways & Limitations
Realistic multimodal deep-research evaluation should require iterative visual search, entity-level verification, and cross-modal evidence aggregation.
Takeaways & Limitations
Experiments use a fixed search budget and identical interaction constraints for fair comparison.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have advanced VQA and now support Vision-DeepResearch systems that use search engines for complex visual-textual fact-finding. However, evaluating these visual and textual search abilities is still difficult, and existing benchmarks have two major limitations. First, existing benchmarks are not visual search-centric: answers that should require visual search are often leaked through cross-textual cues in the text questions or can be inferred from the prior world knowledge in current MLLMs. Second, overly idealized evaluation scenario: On the image-search side, the required information can often be obtained via near-exact matching against the full image, while the text-search side is overly direct and insufficiently challenging. To address these issues, we construct the Vision-DeepResearch benchmark (VDR-Bench) comprising 2,000 VQA instances. All questions are created via a careful, multi-stage curation pipeline and rigorous expert review, designed to assess the behavior of Vision-DeepResearch systems under realistic real-world conditions. Moreover, to address the insufficient visual retrieval capabilities of current MLLMs, we propose a simple multi-round cropped-search workflow. This strategy is shown to effectively improve model performance in realistic visual retrieval scenarios. Overall, our results provide practical guidance for the design of future multimodal deep-research systems. The code will be released in https://github.com/Osilly/Vision-DeepResearch.
1. Introduction
Existing Vision-DeepResearch benchmarks often allow text-only cues, model priors, or whole-image near-duplicate retrieval to replace genuine visual search. VDR-Bench addresses these gaps with a curated benchmark and multi-round cropped-search workflow for realistic multimodal reasoning.
- Motivation: Existing benchmarks are not visual-search–centric because textual cross-referencing and model world knowledge can answer questions without genuine visual verification.These shortcuts can make benchmark performance reflect memorization or textual reasoning rather than visual search.
- Motivation: Whole-image retrieval creates an idealized setting in which near-duplicate images and identifying metadata can make one-shot lookup sufficient.Real-world visual search is noisy, ambiguous, and iterative, often requiring entity-level localization rather than querying the full image.
- Benchmark and workflow: VDR-Bench contains 2,000 VQA instances designed to evaluate realistic multimodal search behavior.The benchmark targets conditions requiring visual search, textual reasoning, and multimodal evidence use.
- Benchmark and workflow: Its multi-stage, human-verified curation pipeline emphasizes visual-first entity discovery and multi-hop query expansion while avoiding shortcut-prone instances and perfect-retrieval bias.The design requires both visual search and textual reasoning for each instance.
- Benchmark and workflow: The proposed multi-round cropped-search workflow iteratively refines visual queries by cropping regions of interest to improve entity localization and reduce retrieval noise.The workflow is presented as a practical response to insufficient visual retrieval capabilities in current MLLMs.
2. Related Work
Vision-DeepResearch systems extend multimodal question answering with web search and multi-hop reasoning, but existing benchmarks do not reliably test realistic visual retrieval. Their main weaknesses are shortcut solutions and idealized search settings.
- Vision-DeepResearch systems: Vision-DeepResearch agents search, read, and synthesize open-web knowledge through multi-step reasoning and cross-modal grounding.They must integrate visual evidence with textual information in complex real-world web environments.
- Prior frameworks: Prior frameworks improve multimodal search through reverse image search, reinforcement learning, or entity-level image cropping.Entity-level cropping reduces background noise and enables more targeted visual retrieval.
- Benchmark limitations: Existing benchmarks cover multimodal factuality, external-knowledge grounding, and dynamic visual reasoning, but many instances remain solvable through language priors or text-only retrieval.These shortcuts reduce the need for genuine visual verification.
- Benchmark limitations: Current evaluations also rely on near-exact whole-image matching and overly direct text queries, failing to reflect realistic visual search and sufficiently challenging textual reasoning.The resulting settings underrepresent noisy retrieval and multi-hop evidence aggregation.
3. A Quantitative Analysis of Existing Vision-DeepResearch Benchmarks
A controlled analysis finds that existing Vision-DeepResearch benchmarks often permit success without meaningful visual search and use unrealistically easy retrieval protocols. These findings motivate VDR-Bench’s emphasis on iterative localization, cross-modal evidence, and challenging text search.
- Finding 1: Existing Benchmarks Do Not Enforce Visual-Search–Centric Reasoning: Across representative benchmarks, text search alone can produce substantial gains comparable to or exceeding image search, revealing textual cue leakage.Answers can often be inferred by cross-validating textual evidence without searching for visual entities.
- Finding 1: Existing Benchmarks Do Not Enforce Visual-Search–Centric Reasoning: Caption-plus-direct-answer settings can also achieve competitive performance by relying on language priors and parametric world knowledge.This indicates that many benchmarks do not enforce a closed visual-evidence loop.
- Finding 2: Overly Idealized Retrieval Evaluation Setting: Single-shot whole-image search frequently improves over direct answering because near-duplicate images provide identifying titles or metadata.This perfect-match bias reduces evaluation to whether a search engine returns an exact or near-exact match.
- Finding 2: Overly Idealized Retrieval Evaluation Setting: Existing text-side multi-hop settings are often overly simplistic and insufficiently challenging for modern text deep-research capabilities.The benchmark protocols therefore under-test multi-hop reasoning and evidence aggregation.
- Implications: VDR-Bench responds by enforcing visual-search–centric reasoning, mitigating perfect-retrieval bias, and evaluating realistic multimodal search conditions.Its design emphasizes multi-scale cropping, entity-level verification, knowledge-graph–based multi-hop reasoning, and cross-modal evidence aggregation.
4. VDR-Bench
VDR-Bench is built through a vision-centric, multi-stage curation pipeline that links cropped visual search to verified entities, multi-hop reasoning, and quality checks. It contains 2,000 multi-hop VQA instances and evaluates both answer correctness and entity-level retrieval.
- Data Curation: VDR-Bench uses a strict vision-centric pipeline that begins with raw images and produces multi-hop VQA instances grounded in visual and textual evidence.The process explicitly annotates retrieved entities and their supporting search trajectories.
- Data Curation: Annotators crop salient regions and search them on the web instead of relying on full-image retrieval.Crops target objects, logos, landmarks, or individuals and generate candidate visual search results.
- Data Curation: Candidate entities are extracted from search-result titles and captions, then checked by an MLLM and human validators to prevent trivial full-image shortcuts.Human validation verifies consistency between crops and results and ensures entity names are not obtainable through full-image search alone.
- Data Curation: Verified visual entities seed VQA pairs that are manually reviewed for recognition, grounding, clarity, answer uniqueness, and dependence on retrieved context.Seed questions cover entities such as landmarks, brands, and object categories.
- Data Curation: Knowledge-graph random walks expand entity recognition into multi-hop questions involving related people, places, years, or organizations.Inference chains originate from visual entities and traverse one or more knowledge-graph nodes.
- Benchmark Composition: 2,000 multi-hop VQA instances span 10 visual domains and vary in visual complexity, entity density, and reasoning depth.The benchmark composition and domain distribution are reported in Figure 1.
- Evaluation Metrics: The benchmark measures answer accuracy and Entity Recall, which evaluates whether searched entities align semantically with a predefined gold entity sequence.Entity Recall uses an LLM judge rather than exact string matching, allowing synonyms and multiple valid search paths.
5. Experiments
Experiments compare models under direct answering, cropped-image plus text search, and multi-turn visual forcing with fixed budgets. Results show that realistic tasks require active search and that iterative visual retrieval improves performance and entity-level recall.
- Experimental Setup: Models are evaluated under Direct Answer, CIS+TS, and CIS+TS+MVF settings using identical interaction constraints and a fixed search budget.The settings isolate contributions from visual search, textual retrieval, and multi-turn visual grounding.
- Results Analysis: Direct-answer scores are relatively low, indicating that VDR-Bench requires active search rather than relying solely on prior knowledge.The result is reported across the benchmark’s model evaluations.
- Results Analysis: 21.2 is the highest CIS+TS score, achieved by Qwen3-VL-235B-A22B-Instruct and exceeding the evaluated closed-source models.The authors describe stronger-prior models as sometimes avoiding search or relying on textual reasoning, calling this lazy search.
- The Correct Paradigm and Challenges: Multi-turn visual forcing guides fine-grained, multi-scale retrieval and deeper reasoning grounded in retrieved visual evidence.Iterative search is intended to help models acquire relevant world knowledge about key entities.
- The Correct Paradigm and Challenges: Answer accuracy and entity-level recall show a strong positive association, and MVF tends to improve both metrics across models.Figure 4 compares CIS+TS with CIS+TS+MVF across model families and search modes.
- The Correct Paradigm and Challenges: Gemini improves from 16.2 to 30.0 with MVC, while the strategy improves vision deep-research performance across different models.The reported comparison is from Table 2.
6. Conclusion
The paper presents VDR-Bench as a more realistic evaluation of visual and textual search, designed to enforce visual evidence and reduce shortcut exploitation. Its experiments emphasize iterative visual search and cross-modal evidence aggregation as important for Vision-DeepResearch systems.
- Conclusion: VDR-Bench addresses text-only or prior-based shortcuts and idealized one-shot image retrieval through multi-scale cropping, entity verification, and knowledge-graph reasoning.The benchmark is intended to provide a realistic testbed for Vision-DeepResearch systems.
- Conclusion: Experiments show that strong performance depends on iterative visual search and cross-modal evidence aggregation.A simple multi-round cropped-search strategy already yields meaningful gains.
- Conclusion: The results provide practical guidance for building more robust multimodal deep-research agents.
A. More Results
Tables 3 and 4 compare model performance on the testmini set across categories using accuracy and entity recall.
- Table 3 reports category-wise model performance using Accuracy on the testmini set.
- The comparisons include Direct Answer versus search-assisted CIS+TS in Table 3, and CIS+TS versus CIS+TS+MVF in Table 4.
- Table 4 reports category-wise model performance using Entity Recall on the testmini set.