PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent

Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai-Tao Zheng, Philip S. Yu, Fei Huang, Jingren Zhou

arXiv:2411.02937v5cs.CL

TL;DR

Existing heuristic mRAGs use rigid retrieval, and common VQA benchmarks do not adequately reflect the complex dynamic retrieval needs of real-world questions. The paper constructs Dyn-VQA and proposes OmniSearch, a self-adaptive planning agent; experiments report that heuristic mRAGs struggle on Dyn-VQA and that OmniSearch is effective.

  • Problem

    Heuristic mRAGs use fixed retrieval processes, and commonly used VQA benchmarks often require only textual knowledge obtainable through standard two-step retrieval.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs Dyn-VQA and proposes OmniSearch, which decomposes complex multimodal questions into sub-question chains and adapts retrieval actions to the solving stage and retrieved content.

    Source: §1 · pp. 2–3

  • Results

    On Dyn-VQA, heuristic mRAGs struggle to provide sufficient, precisely relevant knowledge for dynamic questions, while experiments report OmniSearch’s effectiveness.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Dyn-VQA and OmniSearch provide a benchmark and self-adaptive retrieval approach for studying complex dynamic questions in multimodal RAG.

    Source: §1 · p. 2, §7 · p. 11

    OmniSearch remains substantially below human performance on fast-changing, more-than-two-hop, and external-visual-knowledge questions.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. The code and dataset will be open-sourced at https://github.com/Alibaba-NLP/OmniSearch.

1 INTRODUCTION

Existing heuristic mRAGs and commonly used VQA benchmarks do not adequately address dynamic retrieval needs. The paper introduces Dyn-VQA and OmniSearch, then reports that heuristic methods struggle on the dataset and that OmniSearch is effective.

1 INTRODUCTION

  • mRAG aims to provide MLLMs with comprehensive, accurate, and up-to-date external knowledge and help mitigate hallucination.The paper frames this as the motivation for retrieval-augmented multimodal systems.Source: §1 · p. 1
  • Existing heuristic mRAGs use fixed retrieval processes, while many VQA benchmarks focus on textual knowledge reachable through standard two-step retrieval.The paper argues these datasets do not adequately reflect heuristic methods’ rigidity in acquiring complex knowledge.Source: §1 · pp. 1–2
  • Dyn-VQA contains 1,452 questions spanning rapidly changing answers, multimodal knowledge, and varied multi-hop reasoning.Its questions require flexible retrieval choices across query, tool, and timing.Source: §1 · p. 2
  • Dyn-VQA experiments find that heuristic mRAGs struggle to provide sufficient, precisely relevant knowledge for dynamic questions, while experiments report OmniSearch’s effectiveness.The reported challenge is attributed to heuristic methods’ rigid retrieval processes.Source: §1 · pp. 2–3
  • OmniSearch dynamically decomposes complex multimodal questions into sub-question chains and adjusts retrieval actions using the solving stage and retrieved content.It is designed as a plug-and-play module for arbitrary MLLMs.Source: §1 · p. 3

2 RELATED WORKS

Prior multimodal RAG work retrieves through visual-feature similarity or text derived from images, while much agentic RAG work focuses on text or lacks plug-and-play capability. OmniSearch differs through multi-action, adaptive retrieval across the Internet.

2.1 MULTIMODAL RETRIEVAL AUGMENTED GENERATION

  • Multimodal RAG approaches either retrieve knowledge by visual-feature similarity or convert images into text before retrieving text documents.Examples include CLIP-based image retrieval and object-detection or captioning pipelines followed by dense passage retrieval.Source: §2.1 · p. 3
  • Several agentic RAG studies focus on text, while a causality-enhanced MLLM agent addresses unimodal biases but lacks plug-and-play capability.The passage identifies a distinction from OmniSearch’s intended multimodal, plug-and-play role.Source: §2.1 · p. 3
  • OmniSearch distinguishes itself by planning multiple retrieval actions with diverse tools, adapting actions to retrieved content, and searching across the Internet.The paper contrasts these features with approaches that formulate queries only from input questions and images.Source: §2.1 · p. 3

3 DYN-VQA DATASET

Dyn-VQA is a human-curated benchmark for mRAG systems, built to test complex, dynamic retrieval across question types and modalities. Its construction and analysis emphasize broader knowledge coverage and greater retrieval difficulty than existing knowledge-seeking VQA datasets.

3.1 DYN-VQA CONSTRUCTION

  • Annotators first wrote textual questions using answer-update frequency, external visual knowledge, and reasoning depth, including concepts emerging in the previous six months.English QA instances from FreshQA were also included.Source: §3.1 · p. 4
  • Questions were made multimodal by replacing visual concepts with coreferences and pairing them with relevant images found through Google search.Images from commonly used pre-trained corpora such as Wikipedia and Baidu Encyclopedia were excluded.Source: §3.1 · p. 4
  • Chinese and English questions were translated, manually checked, and filtered when translation was intractable; each question also received a golden query focused on its last hop.The golden query omits visual references and complex intermediate reasoning.Source: §3.1 · p. 4

3.2 DYN-VQA ANALYSIS

  • Dyn-VQA is an mRAG benchmark with about 1.5K questions across nine domains and three types of dynamic-retrieval questions.The types cover rapidly changing answers, multimodal knowledge, and multi-hop questions.Source: §3.2 · p. 4
  • Compared with other knowledge-seeking VQA datasets, Dyn-VQA spans more heterogeneous real-world knowledge and open-ended answers, with human curation and ongoing updates to changing answers.The authors characterize its instances as higher-quality and more difficult, though not necessarily larger in scale.Source: §3.2 · pp. 4–5, Table 1 · p. 4
  • Existing VQA questions typically take fewer than two human searches and do not require text-query image search, while humans achieve their lowest overall accuracy on Dyn-VQA.The authors attribute Dyn-VQA's difficulty to complex external knowledge and retrieval processes that are not fixed.Source: Table 3 · p. 5, §3.2 · p. 5

4 RETRIEVAL BASELINES AND OMNISEARCH

The paper describes heuristic retrieval baselines and introduces OmniSearch, a self-routing framework that plans retrieval actions from question-solving feedback. Its design combines a planning agent, retriever, and sub-question solver in an iterative process.

4.1 BASELINES

  • Single-hop baselines retrieve similar images from the input image or web content from the textual question, testing retrieval across input modalities despite potentially imprecise support.Both use only part of the question's modality as the search query.Source: §4.1 · p. 5
  • Two-hop heuristic mRAG converts image concepts to text and combines that representation with the question for web retrieval, but its retrieval process remains fixed.The text representation comes from either the top retrieved image's caption or an image-caption model.Source: §4.1 · p. 5
  • The estimated upper bound uses a golden last-hop query and selects web or image search according to whether external visual knowledge is needed.The paper treats this simplified setting as one where heuristic mRAG can provide accurate information to MLLMs.Source: §4.1 · p. 5

4.2 OMNISEARCH FRAMEWORK

  • OmniSearch's planner decomposes a complex question into actions and chooses follow-up sub-questions and retrieval queries based on retrieved content or solver feedback.The planner can clarify ambiguity, refine queries, verify responses, or move to the next sub-question or final answer.Source: §4.2 · p. 6
  • The retriever performs web search, text-query image search, or image-query image retrieval, while a pluggable solver summarizes results and returns feedback to the planner.The solver can be an MLLM or the planner itself, and its size can be varied to control GPU cost.Source: §4.2 · p. 6
  • OmniSearch iterates until it deems its knowledge sufficient; GPT-4V is prompted for planning, while Qwen-VL-Chat is trained on retrieval-API conversations and general instruction data.The Qwen-VL-Chat training data uses GPT-4V synthetic data and Infoseek.Source: §4.2 · p. 6
  • Unlike Chain-of-Thought prompting, which stimulates inherent reasoning, OmniSearch uses tools and interacts with the environment through retrieval.The cited discussion frames tool use and environmental response as the distinction between the multimodal agent and CoT.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

Experiments compare heuristic mRAGs and OmniSearch on Dyn-VQA, then analyze solver choices, retrieval content, and computational costs. OmniSearch performs strongly overall, while dynamic question types and retrieval strategy remain important challenges.

5.1 EXPERIMENTAL SETTINGS

  • The experiments use Qwen-VL-Chat, GPT-4V, Qwen-VL-Max, and text-only Qwen-7B-Chat as backbones; F1-Recall measures generated tokens that also occur in the gold answer.The evaluation was conducted up to July 1, 2024.Source: §5.1 · p. 7, §5.2 · p. 7

5.2 MAIN RESULTS

  • OmniSearch (GPT-4V) significantly outperforms state-of-the-art MLLMs with heuristic mRAGs and commercial generative search engines on Dyn-VQA.Qwen-VL-Chat-based OmniSearch also surpasses GPT-4V with two-step heuristic mRAG.Source: Table 4 · p. 7, §5.2 · p. 8
  • The remaining performance gap is concentrated in fast-changing, >2-hop, and external-visual questions, while these retrieval-intensive questions challenge both AI systems and humans.Questions requiring multi-step retrieval or additional visual knowledge underperform other questions in the same classification schema; rapidly changing knowledge is especially intractable.Source: §5.2 · p. 8
  • Image captions benefit two-step heuristic mRAG most on questions needing visual knowledge, but their advantage reverses when additional visual knowledge is unnecessary.The latter questions are primarily 2-hop (74%) and do not require visual knowledge beyond the image concepts.Source: §5.2 · p. 8
  • Commercial generative search engines perform poorly overall, while mRAG narrows the performance gap between text-only Qwen-7B-Chat and Qwen-VL-Chat.The best-performing engine, Gemini, only matches GPT-4V with two-step mRAG; the authors attribute engine failures to weak grounding and multimodal integration.Source: §5.2 · p. 8

5.3 ANALYSIS EXPERIMENTS ON OMNISEARCH

  • Using GPT-4V instead of Qwen-VL-Chat as OmniSearch’s sub-question solver improves performance, and a mixed GPT-4V/GPT-4 invocation strategy adds further gains.The mixed strategy assigns GPT-4V to multimodal contexts and GPT-4 to purely textual contexts.Source: Table 5 · p. 8, §5.3 · p. 9
  • Replacing Qwen-VL-Chat with OmniSearch as the sub-question solver improves question-solving ability, suggesting retrieval-path planning may also enhance understanding and reasoning about retrieval knowledge.The authors describe this as a potential source of cross-task gains.Source: §5.3 · p. 9
  • With Qwen-VL-Chat as solver, OmniSearch’s absolute performance declines by under 4 points (approximately 7.9%) versus GPT-4V, while expenditure is nearly halved.The authors report that the performance-cost relationship is proportional but not linear, and recommend prioritizing a larger retrieval-planning model when resources are constrained.Source: §5.3 · p. 9, Table 6 · p. 9

6 ANALYSIS EXPERIMENTS ON DYN-VQA DATASET

The domain analysis finds that retrieval broadens model performance, but over-retrieval can obscure answers in some cases. Correct-answer overlap is limited, including between the strongest models.

6.1 PERFORMANCE COMPARISON ON DIFFERENT DOMAINS

  • mRAG broadens both Qwen-VL-Chat and GPT-4V coverage across domains, particularly for the smaller Qwen-VL-Chat.In transportation, Qwen-VL-Chat-based OmniSearch can outperform its GPT-4V-based counterpart; the authors attribute this to a long-tail domain with only 10 instances, mostly 2-hop or non-changing questions, where GPT-4V-based OmniSearch may over-retrieve.Source: §6.1 · pp. 9–10, Figure 4 · p. 10

6.2 PREDICTION OVERLAP

  • No Dyn-VQA question was answered correctly by every model, and 31% were answered correctly by none.The two highest-performing models, Qwen-VL-Max and GPT-4V, have pairwise overlap around 60%.Source: §6.2 · p. 10, Figure 5 · p. 10
  • Model strengths differ: 26.87% of InstructBLIP-Vicuna-7B’s correctly answered questions were missed by GPT-4V, supporting ensemble-based approaches as a research direction.The paper notes that stronger overall performance does not mean consistently solving the same hard or easy questions.Source: §6.2 · p. 10

7 CONCLUSION

The paper introduces Dyn-VQA for questions needing retrieval strategies that vary in query, tool, and time, and proposes OmniSearch as a self-adaptive planning agent. Dataset statistics show broad representation across answer-change frequencies, with two domains comprising about half the data.

7 CONCLUSION

  • Dyn-VQA targets dynamic questions that require complex retrieval strategies variable in query, tool, and time, addressing limitations of existing VQA benchmarks.The paper identifies three types of dynamic questions, without detailing them in this conclusion passage.Source: §7 · p. 10
  • OmniSearch is presented as a self-adaptive planning agent for multimodal retrieval, and experiments report its effectiveness while highlighting challenges posed by Dyn-VQA..Source: §7 · p. 11

A.1 STATISTICS

  • Sports and Recreation and Companies and Products account for approximately 50% of Dyn-VQA; fast, slow, and never-changing answers are relatively balanced within domains.The reported distribution has no long tail for answer-change frequencies.Source: §A.1 · p. 15, Figure 6 · p. 15

A.2 DATASET QUALITY

The appendix reports quality checks and dataset-diversity comparisons, describes planned answer maintenance, and situates mRAG among approaches to hallucination mitigation and retrieval from varied knowledge sources.

A.2 DATASET QUALITY

  • Two quality-control annotators re-evaluated answers, domains, update frequencies, reasoning steps, and external-visual-knowledge labels; Fleiss’s Kappa agreements all exceeded 0.8.Instances judged incorrect by both QC annotators were filtered out.Source: §A.2 · p. 15, Table 7 · p. 15

A.3 DATASET DIVERSITY

  • Dyn-VQA has larger pairwise cosine distances than comparison datasets, and its manually curated, open-ended questions yield longer answers than VQAv2 and A-OKVQA.The passage contrasts Dyn-VQA’s diversity with the template-driven question formats of InfoSeek.Source: §A.3 · pp. 15–16, Table 8 · p. 15

A.4 REGULAR UPDATES TO DYN-VQA

  • The authors commit to updating Dyn-VQA answer annotations every three months to maintain timeliness.They plan a semi-automatic process in which LLMs flag potentially outdated answers for human review using retrieved knowledge.Source: §A.4 · p. 16

B.1 MULTIMODAL LARGE LANGUAGE MODELS

  • mRAG is presented as an alternative to knowledge-enhancing MLLM training approaches, whose expensive training costs challenge scalability.The appendix frames mRAG as a way to retrieve knowledge from external sources for hallucination mitigation.Source: §B.1 · p. 16

B.2 MULTIMODAL RETRIEVAL AUGMENTED GENERATION

  • Related work includes REVEAL, which integrates multiple knowledge sources, and PICa, which prompts GPT-3 with image descriptions to retrieve implicit knowledge..Source: §B.2 · p. 16

B.3 VQA DATASETS AS MRAG BENCHMARKS

Existing knowledge-seeking VQA datasets often use fixed-format questions about image properties that standard two-step retrieval can answer, motivating benchmarks with broader knowledge demands.

B.3 VQA DATASETS AS MRAG BENCHMARKS

  • These benchmarks cover constrained knowledge scopes and commonly ask fixed-format, internet-answerable object-property questions, often solvable through standard two-step retrieval.Dyn-VQA instead targets dynamic, multimodal, or more complex multi-hop knowledge retrieval.Source: §B.3 · p. 17

C.1 BASELINES

  • The main experiment includes generative search engines and human participants with at least a bachelor’s degree as baselines.The selected search engines were Bing Chat, PerplexityAI Pro, and Gemini-Advance.Source: §C.1 · p. 17

C.2 OTHER BACKBONE MLLMS FOR HEURISTIC MRAGS

  • The supplementary experiments evaluate heuristic mRAGs with Deepseek-VL-7B-Chat, VisualGLM-6B, Llava-V1.6-Mistral-7B, and mPLUG-Owl2.1 backbones.Source: §C.2 · p. 17

C.3 TRAINING DETAILS

  • OmniSearch (G) uses GPT-4V prompting, while OmniSearch (Q) fine-tunes Qwen-VL-Chat on filtered planning trajectories mixed with general instruction data.The trajectories were synthesized from GPT-4V and raw InfoSeek data; about 40K examples were reduced to 13K.Source: §C.3 · p. 17
  • Training loss is computed on response tokens from each conversation round, excluding input tokens.Source: §C.3 · p. 17

C.4 SUPPLEMENT TO MAIN EXPERIMENTS

Supplementary experiments examine backbone choice, retrieved-content use, retrieval volume, and evaluation metrics, reporting gains alongside differences in how models use retrieval.

C.4 SUPPLEMENT TO MAIN EXPERIMENTS

  • 16.60 average absolute gain across four heuristic mRAG approaches with InstructBLIP-Vicuna-7B was a 223.1% enhancement over that backbone’s baseline.Deepseek-VL-7B-Chat achieved the highest overall performance among the open-source MLLMs evaluated as backbones.Source: §C.4 · p. 18
  • 4.92 to 2.42: performance variance among open-source MLLMs fell with image-caption-based two-step mRAG.The authors suggest mRAG may bolster models with weaker foundational capabilities and note its lower resource dependence and deployment convenience.Source: §C.4 · p. 18
  • 6.78 average improvement came from web-page retrieval, while single-step heuristic mRAGs still benefited original model capabilities despite potentially imprecise retrieval.The authors state that Dyn-VQA’s challenge is not solely image-object recognition.Source: §C.4 · p. 18

D.1 ANALYSIS EXPERIMENTS ON RETRIEVED CONTENT

  • Image captions contributed most to final performance, while search-engine knowledge added a relatively modest benefit because it was often static background information.Using any single part of retrieved content degraded performance compared with using the full content.Source: §D.1 · p. 18
  • As retrieval volume increased, OmniSearch used extensive retrieved content more effectively than GPT-4V, which did not improve continually.The authors report that OmniSearch can filter or disregard potentially harmful noise in voluminous content.Source: Table 11 · p. 20, §D.1 · p. 20
  • For English questions, both models’ performance continued to grow as retrieved content increased; the authors cite backbone capability and Google’s English-language search tendency as possible reasons.They suggest language-tailored or multiple search tools as future directions.Source: §D.1 · p. 20

D.2 CONSISTENCY OF DIFFERENT EVALUATION METRICS

  • F1-Recall trends matched GPT-based and human-based accuracy across models, with all Pearson correlations positive; the accuracy measures were more consistent, while F1-Recall was cheaper and more scalable.The consistency comparison is reported in Table 12.Source: §D.2 · p. 20, Table 12 · p. 20

D.3 SUPPLEMENTARY ANALYSIS OF COMPUTATIONAL COSTS

The supplementary cost analysis relates retrieval complexity to token use and latency, and considers whether smaller models can take on parts of OmniSearch’s computation.

D.3 SUPPLEMENTARY ANALYSIS OF COMPUTATIONAL COSTS

  • More complex questions, including those with fast or slow answer updates, consumed more tokens, primarily because they required more retrieval steps.Table 13 reports token costs by answer-update frequency.Source: §D.3 · p. 21, Table 13 · p. 21
  • Resource-consumption differences across question difficulty were more pronounced for OmniSearch (Q); GPT-4V sometimes planned more than three retrieval steps even on easier questions.Source: §D.3 · p. 21
  • Replacing OmniSearch (G)’s Sub-question Solver with Qwen-VL-Chat left combined token consumption comparable and did not significantly reduce overall performance.The authors interpret this as offloading computational burden to smaller models.Source: §D.3 · p. 21
  • Substituting smaller models for some modules reduced latency; search time was roughly two-thirds of model-inference time.The authors note that latency also depends on network configuration, caching, inference acceleration, and hardware; Table 14 reports average latency.Source: §D.3 · p. 21, Table 14 · p. 21

E.1 MODEL PERFORMANCE ON OTHER VQA DATASETS

OmniSearch gained on prior VQA datasets and showed domain-dependent difficulty, while case studies identify remaining limits in long reasoning chains and fine-grained retrieval.

E.1 MODEL PERFORMANCE ON OTHER VQA DATASETS

  • 30.25 F1-Recall: GPT-4V performed substantially worse on Dyn-VQA than on prior datasets, where its average exceeded 74.The authors relate this discrepancy to the greater prevalence of common-sense questions in traditional datasets.Source: §E.1 · p. 21, Table 15 · p. 22
  • Heuristic mRAG performance was unstable across datasets and impaired GPT-4V on prior benchmarks, especially InfoSeek, where the decline exceeded 10 points.The authors report misleading shallow retrieval for some building, flora, and fauna images; common-sense questions in VQAv2 and A-OKVQA saw a smaller negative impact.Source: §E.1 · pp. 21–22
  • OmniSearch achieved steady gains on each prior VQA dataset, including slight growth on VQAv2 and A-OKVQA.The authors attribute this adaptability in part to its ability to avoid unnecessary retrieval when external knowledge is not beneficial.Source: §E.1 · p. 22

E.2 OMNISEARCH PERFORMANCE ON DIFFERENT DOMAINS

  • In most domains, performance bars followed fast < slow < never, ≤2-hop > >2-hop, and no external visual knowledge > external visual knowledge, indicating greater difficulty for complex retrieval.Transportation was an exception among data-sparse domains, where performance was also relatively poor.Source: Figure 7 · p. 22, §E.2 · p. 22

E.3 CASE STUDY

  • Figure 8 contrasts a successful strategy adjustment after retrieval failure with cases of an abruptly ended reasoning chain and a misdirected visual search.The authors report that 73 of 100 analyzed errors involved partially containing the correct answer but failing to complete retrieval in long context; current OmniSearch also lacks the fine-grained retrieval needed for the described screenshot case.Source: Figure 8 · p. 23, §E.3 · p. 23

Related papers

Loading 2411.02937v5…