PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Multimodal Retrieval-Augmented Generation

Lang Mei, Siyu Mo, Zhihan Yang, Chong Chen

arXiv:2504.08748v1cs.IRcs.AIcs.CLcs.ETcs.LG

TL;DR

MRAG research lacks a consolidated understanding of how to process multimodal data and evaluate multimodal retrieval-augmented systems. This survey organizes the field’s components, datasets, evaluation methods, limitations, challenges, and future directions, offering a structured overview to guide further development.

  • Problem

    Research on effectively processing multimodal text and images with LLMs remains active, while MRAG lacks a consolidated understanding of its key components, datasets, evaluation methods, and limitations.

    Source: §1 · pp. 2–3

  • Method

    The survey systematically analyzes MRAG through its essential technologies, evaluation datasets, evaluation methods and metrics, and limitations.

    Source: §1 · p. 3, §8 · p. 57

  • Results

    The survey identifies current MRAG challenges, including multimodal knowledge integration and the reliability of generated outputs, and proposes future research directions.

    Source: §8 · p. 57

  • Takeaways & Limitations

    The structured overview and forward-looking analysis aim to guide researchers in advancing MRAG.

    Source: §1 · p. 3, §8 · p. 57

    MRAG evaluation lacks a universally accepted capability taxonomy and remains insufficient in complex multimodal reasoning, multi-turn dialogue, creativity, and several domain-specific tasks.

    Source: §6.5 · p. 50

Abstract

from arXiv · show

Multimodal Retrieval-Augmented Generation (MRAG) enhances large language models (LLMs) by integrating multimodal data (text, images, videos) into retrieval and generation processes, overcoming the limitations of text-only Retrieval-Augmented Generation (RAG). While RAG improves response accuracy by incorporating external textual knowledge, MRAG extends this framework to include multimodal retrieval and generation, leveraging contextual information from diverse data types. This approach reduces hallucinations and enhances question-answering systems by grounding responses in factual, multimodal knowledge. Recent studies show MRAG outperforms traditional RAG, especially in scenarios requiring both visual and textual understanding. This survey reviews MRAG's essential components, datasets, evaluation methods, and limitations, providing insights into its construction and improvement. It also identifies challenges and future research directions, highlighting MRAG's potential to revolutionize multimodal information retrieval and generation. By offering a comprehensive perspective, this work encourages further exploration into this promising paradigm.

1 Introduction

LLMs can hallucinate when specialized or current knowledge exceeds their training, while RAG grounds responses in retrieved external text but overlooks multimodal knowledge. This survey examines MRAG’s components, datasets, evaluation, limitations, challenges, and future directions.

Motivation

  • LLMs may produce misleading or fabricated answers for specialized or up-to-date queries because their training data may lack the required knowledge.This hallucination problem motivates methods that ground responses in verified, context-relevant information.Source: §1 · p. 2

Background

  • RAG reduces hallucinations by retrieving relevant external knowledge, chunking documents for fine-grained retrieval, and conditioning generation on the retrieved context.It can also integrate current information during inference, but traditional systems primarily use plain-text knowledge bases and text-pretrained LLMs.Source: §1 · p. 2

Motivation

  • MRAG extends retrieval and generation across text, images, videos, and other modalities, addressing the perceptual limitations of text-only LLMs.Multimodal context can clarify real-world queries by combining textual information with visual or instructional content.Source: §1 · pp. 2–3

Background

  • MRAG research indicates improved question answering and fewer hallucinations than text-modal RAG by retrieving, integrating, and generating knowledge across modalities.Unlike text-modal RAG, MRAG must model cross-modal knowledge relationships and produce answers informed by multimodal queries and retrieved evidence.Source: §1 · p. 3

Contributions

  • The survey systematically reviews MRAG components and technologies, datasets, evaluation methods and metrics, limitations, current challenges, and future research directions.Its structured analysis aims to clarify how MRAG systems can be constructed, evaluated, and improved.Source: §1 · p. 3

Paper Organization

  • The paper proceeds from MRAG development and technical components to datasets, evaluation metrics, technical challenges, future work, and conclusions.The technical discussion includes multimodal retrieval and generation, while evaluation addresses datasets and specialized competency assessments.Source: §1 · pp. 3–4

2 Overview of MRAG

MRAG extends text-only RAG by processing images, audio, video, and text, evolving from caption-based pseudo-MRAG to architectures that preserve multimodal data and support multimodal outputs. MRAG3.0 further broadens scenarios through enhanced parsing, dynamic search planning, and unified understanding-generation capabilities.

2 Overview of MRAG

  • MRAG extends text-only RAG by integrating images, audio, video, and text for applications whose information spans multiple modalities.This evolution enables more complex real-world use cases than plain-text RAG.Source: §2 · p. 4

2.1 MRAG1.0

  • MRAG1.0, or pseudo-MRAG, converts multimodal content into captions and uses document parsing and indexing, retrieval, and generation modules modeled on traditional RAG.Parsing extracts and chunks text, captions non-textual elements, embeds them in a vector database, retrieves top-k items, and integrates them into an LLM prompt; this design risks modality-specific information loss and retrieval bottlenecks.Source: §2.1 · pp. 4–6, Figure 1 · p. 5

2.2 MRAG2.0

  • MRAG2.0 preserves original multimodal data, uses MLLMs for unified captioning and generation, and adds cross-modal retrieval to reduce conversion loss.Its dual caption-based and cross-modal retrieval improves downstream QA for interrelated multimodal data, although multimodal retrieval and diverse input organization remain challenging.Source: §2.2 · pp. 6–7, Figure 2 · p. 6

2.3 MRAG3.0

  • MRAG3.0 retains document screenshots, enables multimodal output, and expands beyond knowledge-base VQA toward unified understanding and generation scenarios.Its architecture introduces enhanced document parsing, end-to-end multimodality, and broader scenario coverage.Source: §2.3 · p. 7, Figure 3 · p. 8

2.3.1 Scenario for MRAG. •

  • MRAG3.0 supports retrieval-augmented answering, direct VQA, multimodal generation, and fusion outputs that combine text, images, or videos in one response.Retrieval augmentation improves generation tasks, while fusion outputs may answer directly with multimodal data, improve accuracy, or enrich user experience.Source: §2.3.1 · pp. 7–9

2.3.2 Modified Modules. •

  • MRAG3.0 upgrades parsing by indexing document screenshots and supports native or augmented multimodal output through generation, retrieval, matching, and insertion.Augmented output identifies insertion positions, retrieves candidate multimodal elements, and inserts the best matches into generated text.Source: §2.3.2 · pp. 9–10, Figure 5 · p. 10

2.3.3 New Modules. •

  • The multimodal search planning module classifies retrieval needs and reformulates queries using textual, visual, and historical retrieval cues to improve relevance while avoiding unnecessary searches.It can choose none, text, or image retrieval, decompose complex queries into atomic sub-queries, and adapt actions across multi-hop iterations.Source: §2.3.3 · pp. 10–11

3 Components & Technologies of MRAG

MRAG comprises five key technical components, covering document parsing and indexing, search planning, retrieval, and generation.

Components & Technologies of MRAG

  • The section introduces MRAG’s key components: multimodal document parsing and indexing, multimodal search planning, multimodal retrieval, and multimodal generation.These components are presented sequentially in sections 3.1–3.4.Source: §3 · p. 11

3.1 Multimodal Document Parsing and Indexing

MRAG document parsing and indexing determines how effectively systems use multimodal knowledge, especially from unstructured and semi-structured documents. Methods range from extractive pipelines to holistic representations, each balancing structure preservation, efficiency, and multimodal coverage.

3.1 Multimodal Document Parsing and Indexing

  • MRAG documents include unstructured, semi-structured, and structured data, with challenges involving lost structure in parsing and accurate natural-language-to-query formulation.Unstructured examples include text, images, videos, and audio; semi-structured examples include PDFs, HTML, XML, and JSON; structured examples include relational databases and knowledge graphs.Source: §3.1 · p. 12
  • Extraction-based methods first extract multimodal information and then structure it for storage, whereas representation-based methods store document content holistically without explicit extraction.The choice depends on task requirements such as scalability or computational efficiency.Source: §3.1 · p. 12

3.1.1 Extraction-based.

  • Extractive parsing evolved from efficient but lossy plain-text extraction to OCR and multimodal extraction, while OCR pipelines remain resource-intensive and vulnerable to cascading errors.Transformer-based systems such as LayoutLM, LayoutLMv2 [413], LayoutLMv3 [133], and DocFormer [12] improve joint modeling of text, layout, and visual information.Source: §3.1.1 · pp. 12–14

3.1.2 Representation-based.

  • Representation-based methods use document screenshots and unified encodings to preserve multimodal context, but page splitting can lose inter-page relationships and multimodal retrieval can lag text-only retrieval.A parallel paradigm combines OCR-based text indexing with screenshot-based multimodal indexing, then fuses textual and visual RAG results through modality integration [342].Source: §3.1.2 · p. 14

3.2 Multimodal Search Planning

MRAG search planning comprises fixed and adaptive strategies for retrieving and integrating multimodal information. Fixed pipelines often impose inflexible retrieval operations, motivating adaptive methods that refine actions according to query context and intermediate results.

3.2 Multimodal Search Planning

  • Multimodal search planning addresses complex queries by integrating information across text and visual modalities through fixed or adaptive retrieval strategies.The two broad approaches are fixed planning and adaptive planning.Source: §3.2 · p. 14

3.2.1 Fixed Planning.

  • Fixed planning uses predetermined pipelines that retrieve from either one modality or multiple modalities, limiting adaptation to diverse multimodal query requirements.Single-modal strategies include text-centric conversion of visual queries, such as Plug-and-Play [357], and image-centric retrieval, such as Wiki-LLaVA [24]; MMSearch [147] uses a rigid multimodal pipeline.Source: §3.2.1 · pp. 14–15
  • Fixed pipelines can retrieve irrelevant or redundant information, incur computational overhead, and fail when external knowledge is unnecessary or the optimal modality depends on query needs.Compulsory image retrieval may mislead models or waste computation, while indiscriminate retrieval adds cost and potential generation noise.Source: §3.2.1 · p. 15

3.2.2 Adaptive Planning.

  • Adaptive planning dynamically decomposes complex multimodal questions, selects retrieval actions, and refines subsequent steps using the problem-solving state and retrieved content.OmniSearch [197] uses sub-question chains and adaptive actions, while CogPlanner [440] iteratively refines queries and supports parallel or sequential retrieval modeling.Source: §3.2.2 · pp. 15–16

3.3 Multimodal Retrieval

This section surveys multimodal retrieval in MRAG, covering retriever architectures and retrieval across single, cross, and unified modalities, alongside reranking and refinement. It examines approaches for semantic alignment, scalability, modality gaps, relevance accuracy, and efficient context use.

3.3.1 RETRIEVER.

  • Retriever architectures are organized into Single/Dual-stream and Generative Structures, encompassing single-modal and cross-modal retrieval.Single-stream fusion captures fine-grained image-text interactions but is computationally costly, whereas dual-stream contrastive alignment is more efficient but has weaker explicit interaction and feature alignment.Source: §3.3.1 · p. 16, Figure 7 · p. 18
  • Text retrieval has evolved from sparse lexical matching to dense semantic retrieval, using PLM-based term weighting or expansion, vector embeddings, approximate-nearest-neighbor indexing, and multi-vector interactions.Sparse methods improve efficiency and cross-domain transfer but remain limited by lexical matching, while dense methods address semantic relationships through two-tower representations and similarity functions.Source: §3.3.1 · pp. 18–19
  • Dense retrieval training relies on negative sampling and pretraining, with hard or ambiguous negatives improving discrimination while false negatives can introduce noise.RocketQA filters likely false negatives, AR2 jointly trains retrievers and rankers, and SimANS samples moderately similar ambiguous negatives that are more informative and less likely to be false negatives.Source: §3.3.1 · pp. 19–20
  • Single-modal retrieval covers text and image search, while cross-modal retrieval matches queries and data across modalities including text, images, audio, and video.Single-modal retrieval supports modality-specific indexing and similarity matching, but cannot capture cross-modal relationships; cross-modal retrieval bridges textual and visual semantics for MRAG.Source: §3.3.1 · pp. 16, 20
  • Cross-modal retrieval methods span CNN/RNN, Transformer, and vision-language-pretraining approaches for text-image, text-video, and text-audio alignment, with video additionally requiring temporal and spatiotemporal modeling.VLP models use large-scale visual-linguistic pretraining, fine-tuning, reranking, and contrastive or token-level alignment to improve cross-modal interaction.Source: §3.3.1 · pp. 20–22
  • Unified-modal retrieval encodes hybrid-modal data into shared spaces, while generative retrieval maps queries to document identifiers whose quality determines memorization and retrieval effectiveness.Unified models address modality imbalance and document visual features; generative retrieval is shifting from static identifiers toward learnable identifiers optimized for document semantics and retrieval.Source: §3.3.1 · pp. 22, 24
  • Retriever methods encode images as visual tokens or identifiers and align queries with multimodal candidates through generative, coarse-to-fine, or two-step search.GeMKR uses object-aware prefix tuning and knowledge-clue generation; GRACE, ACE, and AVG construct image identifiers or vokens for retrieval.Source: §3.3.1 · p. 25

3.3.2 RERANKER.

  • Rerankers score retrieved multimodal documents through fine-tuning or prompting, with prompting improving flexibility but remaining sensitive to prompt quality, positional bias, and context limits.Fine-tuned methods require substantial computation and labeled data, whereas prompting supports point-wise, pair-wise, and list-wise ranking with limited task-specific training.Source: §3.3.2 · pp. 26–27, §3.3.3 · p. 27

3.3.3 REFINER.

  • Refiners condense retrieved context through hard or soft prompts, including text summarization and visual-token compression, improving efficiency while introducing interpretability, robustness, or information-loss trade-offs.Hard methods filter or compress natural-language content, while soft methods use continuous representations; multimodal approaches reduce visual tokens through staged, similarity-based, or lifecycle-aware pruning.Source: §3.3.3 · pp. 27, 28, 30

3.4 Multimodal Generation

Multimodal generation extends large language models to produce coherent content across text, images, audio, and video. Research increasingly supports arbitrary multimodal inputs and outputs, including retrieval-grounded combinations of text with images, tables, and videos.

3.4 Multimodal Generation

  • MLLMs integrate diverse data types to generate rich, coherent, contextually relevant outputs across text, images, audio, and video.The survey classifies MLLMs according to their generative input and output modalities and summarizes related research in Figure 8.Source: §3.4 · pp. 30–31

3.4.1 MODALITY INPUT.

  • Multimodal inputs have progressed from images and videos to unified arbitrary-modal inputs, with systems supporting cross-modal understanding, task planning, and any-to-any generation.NExT-GPT connects an LLM with multimodal adaptors and diffusion decoders, generating text, images, video, and audio while tuning only 1% of certain projection layers.Source: §3.4.1 · pp. 31–32

3.4.2 MODALITY OUTPUT.

  • Multimodal-output systems generate text plus other modalities by identifying insertion positions, retrieving and matching candidates, or directly generating aligned multimodal content.MuRAR extends retrieval-grounded output from images to tables and videos [508], while M^2RAG uses placeholders and chain-of-thought conversion [259], and NExT-GPT directly generates multimodal data [401].Source: §3.4.2 · pp. 32–33

4 Dataset for MRAG

The survey curates existing datasets to evaluate MRAG systems’ real-world multimodal understanding and knowledge-based question answering, organizing them into two complementary evaluation classes.

Dataset Categorization

  • The datasets are divided into Retrieval & Generation-Joint Components and Generation, covering pipeline synergy and contextually accurate output production, respectively.The first class requires external knowledge retrieval followed by accurate response generation, while the second evaluates generation without external retrieval.Source: §4 · p. 33

4.1 Dataset for Retrieval & Generation

MRAG retrieval-and-generation datasets evaluate end-to-end multimodal retrieval and response generation, emphasizing the use of external knowledge to improve answer quality and relevance. The benchmark landscape spans closed-knowledge VQA, cross-modal reasoning, multimodal search, video understanding, and specialized tasks.

Dataset overview

  • MRAG retrieval-and-generation datasets assess systems that retrieve relevant text, images, or videos and dynamically use external knowledge to generate accurate, relevant responses.Figure 9 categorizes these benchmarks, while Table 1 summarizes representative dataset statistics.Source: §4.1 · p. 33, Figure 9 · p. 34, Table 1 · p. 34

Knowledge-based VQA

  • Early benchmarks progress from closed-knowledge retrieval in KB-VQA [375] and FVQA to external-knowledge reasoning in OK-VQA [263], A-OKVQA, and S3VQA [140], though several reduce to simple retrieval.KVQA primarily tests named-entity recognition and Wikipedia retrieval, while A-OKVQA adds rationale annotations and S3VQA [140] requires object detection and web queries.Source: §4.1 · p. 34

Cross-modal reasoning

  • MultiModalQA [345], ManyModalQA [114], and MIMOQA extend evaluation across snippets, tables, images, and multimodal outputs, but template-based or modality-selection designs can limit knowledge aggregation and reasoning.MultiModalQA [345] requires cross-modal knowledge extraction, ManyModalQA [114] emphasizes answer-modality choice, and MIMOQA requires selecting both text and images.Source: §4.1 · pp. 34–35

Multimodal retrieval

  • WebQA [28], ViQuAE, InfoSeek [42], Encyclopedic-VQA [272], and MMSearch [147] evaluate multimodal retrieval or knowledge-based question answering, including named-entity grounding and image-to-image search.WebQA [28] retrieves visual content but produces only textual answers, while MMSearch [147] evaluates MLLMs as multimodal search engines.Source: §4.1 · p. 35

Retrieval-augmented benchmarks

  • MRAG-bench [126] and MRAMG-Bench [435] target vision-centric retrieval-augmented knowledge and answers combining text and images, respectively.MRAG-bench [126] examines when visual knowledge outperforms textual knowledge, whereas MRAMG-Bench [435] uses multimodal corpus data.Source: §4.1 · p. 35

Specialized multimodal tasks

  • VCR, VisualCOMET, KnowIT VQA [97], SOK-Bench, and Ads [138] cover visual commonsense, video-temporal reasoning, and automatic advertisement understanding with rich semantic annotations.KnowIT VQA [97] and SOK-Bench combine visual, textual, temporal, and knowledge-based reasoning, while Ads [138] annotates topics, sentiments, and persuasive reasoning.Source: §4.1 · p. 35

4.2 Dataset for Generation · 4.2.1 Comprehensive.

The Generation category evaluates multimodal models without external retrieval using representative benchmarks organized by evaluation scope, revealing progress alongside persistent reasoning, perception, OCR, and multimodal-understanding limitations. Comprehensive benchmarks increasingly target real-world use, complex reasoning, multi-image and interleaved inputs, and fine-grained visual-text processing.

4.2 Dataset for Generation

  • Generation benchmarks isolate models’ intrinsic knowledge and reasoning without external retrieval, with representative datasets organized in Figure 10 and summarized in Table 2.This framework supports systematic analysis of MRAG generation strengths and limitations across diverse scenarios.Source: §4.2 · p. 35

4.2.1 Comprehensive.

  • VQA v2 [104], VizWiz [112], and NLVR2 [336] established open-ended, accessibility-focused, and multi-image evaluation, but newer benchmarks expose weaknesses in OCR, counting, localization, and fine-grained attributes.LVLM-eHub, MDVP [212], and LAMM report strong commonsense performance but weaker advanced visual capabilities, with fine-tuning mitigating some limitations.Source: §4.2.1 · p. 35
  • MME [29], MMBench, SEED-Bench [180], SEED-Bench-2, and MMT-Bench [431] broaden evaluation across perception, reasoning, generation, and diverse scenarios.SEED-Bench-2 contains 24K QA pairs, while MMT-Bench [431] scales to 31K; performance generally improves with model scale, but spatial localization, visual mathematics, charts, and interleaved image-text understanding remain difficult.Source: §4.2.1 · pp. 35–36
  • Real-world benchmarks show persistent fine-grained perception challenges: GPT-4o outperforms other models, yet human performance remains substantially higher than general models.RealWorldQA, BLINK [86], WV-Bench [245], Visit-Bench [21], V*-Bench, and MME-RealWorld [480] test spatial understanding, visual correspondence, instruction following, high-resolution details, and human preferences.Source: §4.2.1 · p. 36
  • Open-ended and evaluator-based benchmarks address reasoning-process limitations that binary or multiple-choice tests overlook, although LLM-based scoring can be inaccurate.MM-Vet [439] evaluates integrated vision-language abilities through diverse formats, Touchstone [17] tests real-world dialogue, and InfiMM-Eval [113] includes deductive, abductive, and analogical reasoning with intermediate steps.Source: §4.2.1 · pp. 36–37
  • High-resolution inputs improve recognition of small objects, dense text, and fine-grained details, while Chain-of-Thought strategies significantly benefit closed-source but not open-source models.Closed-source models perform strongly in reasoning but struggle with complex localization, structural relations, charts, and visual mathematics.Source: §4.2.1 · p. 37
  • Specialized benchmarks expose remaining gaps in data leakage control, vision-centric perception, visual details, sequential and multi-image reasoning, and interleaved image-text comprehension.MMStar [38], CV-Bench [359], FOCI [101], MMVP, Q-Bench [398], Visual-COT, Mementos [381], MIRB [482], ReMI [162], MuirBench [368], and VEGA [493] target these capabilities; GPT-4V and Gemini 1.5 Pro achieve only modest performance on VEGA [493].Source: §4.2.1 · p. 37
  • OCR benchmarks progress from text recognition to structural and multimodal document understanding, but fine-grained OCR remains weaker than specialized OCR systems and varies by data type.TextVQA, OCR-VQA, WebSRC [41], SEED-Bench-2-Plus [178], OCRBench [233], and VCR [473] cover text, layouts, charts, maps, web pages, and obscured-text reconstruction; knowledge graphs and maps are harder than charts.Source: §4.2.2 · p. 37

4.2.2 Optical Character Recognition (OCR). · 4.2.3 Structural Document.

Structural-document understanding requires models to reason over layouts, spatial relationships, and semantic connections among text, tables, and graphics rather than relying only on visual patterns. The section surveys benchmarks spanning charts, webpages, tables, documents, scientific figures, and visual mathematics, while noting persistent gaps in reasoning and long-context comprehension.

4.2.2 Optical Character Recognition (OCR).

  • Structural documents demand layout, spatial-relation, and semantic reasoning across embedded text, tables, and graphical components.Their information-dense, highly structured nature distinguishes them from natural images, which primarily rely on visual patterns and textures.Source: §4.2.3 · p. 37

4.2.3 Structural Document.

  • Benchmarks such as FigureQA [160], ChartQA [266], ChartX [404], VisualMRC, ChartBench [414], and MMC-Benchmark [217] evaluate chart and visual-document extraction, reasoning, classification, and generation.These datasets range from synthetic scientific figures and chart VQA to multimodal chart understanding with multiple topics, chart types, and tasks.Source: §4.2.3 · pp. 37–38
  • Web2Code [447] and VisualWebBench [221] assess webpage understanding, HTML code translation, and web tasks at website, element, and action levels.The benchmarks address instruction tuning and evaluation for multimodal models operating on webpage structure and code.Source: §4.2.3 · p. 38
  • ComTQA [486], DocVQA [268], InfographicVQA, DocGenome [403], and MP-DocVQA extend evaluation to tables, document images, infographics, scientific papers, and multipage documents.Their tasks include perception, information extraction, layout detection, visual question answering, code generation, and multipage document understanding.Source: §4.2.3 · p. 38
  • CharXiv [387], MMLongBench-Doc [258], SciGraphQA [192], and SciFIBench [313] target challenging scientific charts, long documents, academic graphs, and scientifically grounded figure interpretation.MMLongBench-Doc averages 47.5 pages, while SciGraphQA contains 295K QA dialogues and SciFIBench uses adversarial filtering with human verification.Source: §4.2.3 · p. 38
  • A performance gap remains between proprietary and open-source models, while MLLMs still struggle with reasoning and comprehension of extended multimodal document contexts.Long-context document comprehension remains a critical limitation despite advances in conventional structural-document benchmarks.Source: §4.2.3 · p. 38
  • MathVista [242], Math-Vision [372], OlympiadBench [115], We-Math [300], and MathVerse evaluate visual mathematical reasoning through diverse tasks, fine-grained assessments, subproblem decomposition, and varied visual-textual formulations.MathVista [242] aggregates 28 existing datasets and adds 3 new ones, while MathVerse creates six versions with different proportions of visual and textual content.Source: §4.2.4 · p. 38

4.2.4 Mathematics. · 4.2.5 Video Understanding.

The mathematics discussion highlights persistent MLLM weaknesses in diagram interpretation and underlying reasoning, while the video-understanding discussion surveys increasingly diverse benchmarks and identifies long-context and temporal-perception limitations. It also presents MME-Industry [429] as a domain-expert-validated benchmark spanning more than 21 industrial sectors.

4.2.4 Mathematics.

  • MLLMs struggle with complex diagrams, depend heavily on textual cues, and often solve composite mathematical problems through memorization rather than underlying reasoning.These limitations motivate further development of MLLM capabilities.Source: §4.2.4 · p. 38

4.2.5 Video Understanding.

  • Traditional video-QA benchmarks such as TGIF-QA [141] and ActivityNet-QA focus primarily on human-activity tasks.Their domain-specific scope contrasts with newer benchmarks targeting more complex video understanding.Source: §4.2.5 · p. 38
  • Video-MME [85] covers diverse video domains with multimodal inputs and manual annotations, while MVBench [188] reannotates existing datasets using ChatGPT.These benchmarks broaden evaluation beyond traditional human-activity video-QA tasks.Source: §4.2.5 · p. 38
  • MMBench-Video [77] uses free-form questions for short- to medium-length videos, expanding evaluation of video understanding beyond fixed task formats.Its questions target short and medium video contexts.Source: §4.2.5 · pp. 38–39
  • MLVU [496], LVBench [376], Event-Bench [71], and VNBench [488] emphasize long-video understanding across extended multimodal contexts.VNBench [488] introduces synthetic retrieval and ordering tests by inserting irrelevant images or text into videos.Source: §4.2.5 · p. 39
  • Specialized benchmarks address distinct video settings: EgoSchema [262] evaluates egocentric videos, TempCompass tests fine-grained temporal perception, and MovieChat targets long videos but often reduces tasks to short-video problems.These benchmarks expose the breadth and limitations of current video-evaluation settings.Source: §4.2.5 · p. 39
  • Current MLLMs, particularly open-source models, remain challenged by long-context processing and temporal perception.These shortcomings underscore the need for improved capabilities in both areas.Source: §4.2.5 · p. 39
  • MME-Industry [429] addresses the lack of comprehensive cross-industry evaluation by covering over 21 industrial sectors with expert-annotated and validated test cases.Its scope includes power generation, electronics manufacturing, textile production, steel, and chemical processing, supporting reliable and practically relevant assessment in industrial contexts.Source: §4.2.6 · p. 39

4.2.6 Industry. · 4.2.7 Conversational QA.

The survey highlights benchmarks for evaluating multimodal models in extended conversational interactions and multidisciplinary knowledge, addressing capabilities that single-round QA does not fully capture. These benchmarks vary in interaction complexity, subject coverage, and reasoning demands.

4.2.6 Industry.

  • Multi-round QA benchmarks simulate human-AI interactions with extended contextual histories, better aligning evaluation with real-world conversational scenarios.Current MLLMs are primarily designed for multi-round chatbot interactions, while most benchmarks focus on single-round QA tasks.Source: §4.2.7 · p. 39
  • SparklesDialogue [134] evaluates conversational proficiency across multiple images and dialogue turns using flexible text-image interleaving.Each instance contains two rounds and four images.Source: §4.2.7 · p. 39
  • SciGraphQA [192] constructs multi-turn question-answering conversations from scientific graphs in Arxiv papers, emphasizing complex scientific discourse.Source: §4.2.7 · p. 39
  • ConvBench [225] evaluates perception, reasoning, and creation both within individual rounds and across complete conversations.Its findings indicate that reasoning and creation failures often stem from inadequate fine-grained perception.Source: §4.2.7 · p. 39
  • MMDU [235] tests multi-turn, multi-image conversations with up to 20 images and 27 turns, linking open-source performance gaps largely to limited conversational instruction-tuning data.Source: §4.2.7 · p. 39

4.2.7 Conversational QA.

  • Multidisciplinary-knowledge benchmarks span scientific, grade-level, college-level, and domain-specific evaluation settings.ScienceQA [243] includes lectures and explanations for chain-of-thought evaluation; MMMU covers diverse college-level subjects with interleaved text and images; CMMU [117] and CMMMU [457] target Chinese grade-level and college-level knowledge, while MMMU-Pro [446] strengthens MMMU for advanced evaluation.Source: §4.2.8 · p. 39

4.2.8 Multidisciplinary.

This section surveys a broad MRAG dataset landscape spanning multimodal understanding, optical character recognition, documents, mathematics, video, industry, and conversational question answering. The datasets vary substantially in scale, modality, task coverage, reasoning demands, and annotation structure.

Multimodal understanding

  • Multimodal understanding benchmarks span perception, reasoning, multi-image analysis, real-world scenarios, and broad capability evaluations, with datasets ranging from hundreds to hundreds of thousands of samples.Examples include MME-RealWorld’s 29,429 question-answer pairs across 43 subtasks and five scenarios, MMStar’s six capabilities and 18 dimensions, and LVLM-eHub’s 42 datasets.Source: §4.2.8 · p. 40

Optical character recognition

  • Optical-character-recognition datasets require reading text in natural images, book covers, webpages, documents, scenes, and handwritten mathematical expressions.OCR-VQA contains more than one million question-answer pairs over 207,572 book-cover images, whereas WebSRC supplies questions with HTML, screenshots, and metadata.Source: §4.2.8 · p. 41

Multi-image understanding

  • Multi-image and visual reasoning benchmarks explicitly require reasoning across heterogeneous image sets, spatial relations, visual world knowledge, multi-hop evidence, and high-resolution attributes.ReMI requires reasoning over at least two and up to six images, while MuirBench contains 2,600 questions over 11,264 images and averages 4.3 images per instance.Source: §4.2.8 · pp. 40–41

Specialized and industry datasets

  • The survey also includes domain-specific and capability-focused resources for visual prompting, CLIP-blind patterns, point clouds, scientific literature, and industry applications.LAMM covers nine image and three point-cloud tasks, VEGA combines 50K scientific-literature entries with over 200K question-answer pairs, and MME-Industry spans 21 domains.Source: §4.2.8 · pp. 41–42

Structural documents

  • Document and structural-figure datasets evaluate question answering over charts, tables, webpages, scientific documents, layouts, and cross-page evidence, often combining visual content with text or code.DocGenome annotates 500K scientific documents across 153 disciplines, while MMLongBench-Doc includes single-page, cross-page, and unanswerable questions over lengthy PDFs.Source: §4.2.8 · p. 41

Mathematics

  • Mathematical benchmarks cover visual algebra, geometry, scientific figures, school-level problems, and Olympiad mathematics and physics, with open-ended, multiple-choice, bilingual, and step-by-step formats.MathVista combines 28 existing datasets with three new datasets and 6,141 examples, while MathVerse provides 15K multimodal test samples across three geometry and functions subjects.Source: §4.2.8 · p. 42

Video understanding

  • Video-understanding datasets test temporal reasoning across repetition, actions, state transitions, long videos, events, subtitles, audio, and multiple duration scales.Video-MME contains 2,700 questions from 900 videos with subtitles and audio, spanning six visual domains and videos from 11 seconds to one hour.Source: §4.2.8 · p. 42

Conversational QA

  • Conversational datasets evaluate multi-round multimodal interaction through dialogues involving multiple images, open-vocabulary graph questions, and sequences of perception, reasoning, and creation.SparklesDialogue includes dialogue variants with two or three images, while ConvBench contains 577 image-instruction pairs across 215 tasks.Source: §4.2.8 · p. 42

5 Evaluation Metrics of MRAG

MRAG evaluation spans human, rule-based, and LLM/MLLM-based strategies, each balancing accuracy, scalability, objectivity, and bias differently. Metrics further assess retrieval and generation through lexical, semantic, factual, multimodal, and granularity-sensitive criteria.

5 Evaluation Metrics of MRAG

  • MRAG evaluation covers document parsing, search planning, retrieval, and generation through human, rule-based, and LLM/MLLM-based strategies.These strategies offer distinct trade-offs in reliability, scalability, objectivity, labor, and susceptibility to bias.Source: §5 · p. 43
  • Human evaluation is precise and user-centered, but costly, difficult to scale, and vulnerable to evaluator bias; Bingo [58], M-HalDetect [108], and WV-Arena [245] illustrate its uses.Diverse evaluator pools and cross-validation can improve balance and representativeness, while human evaluation remains stronger for subtle inaccuracies.Source: §5 · p. 43
  • Rule-based evaluation provides objective, reproducible, efficient comparisons, whereas LLM/MLLM-based evaluation reduces human effort but can inherit systematic and model-dependent biases.LLM/MLLM evaluators may be sensitive to response order, and different evaluators can produce inconsistent results for the same task.Source: §5 · pp. 43–44

5.1 Metrics of Retrieval & Generation

  • Rule-based metrics include EM, ROUGE-N, BLEU, MRR, CIDEr, SPICE, BERTScore, and perplexity, while LLM/MLLM metrics assess answer precision, recall, context quality, faithfulness, and hallucination.Rule-based metrics are interpretable and efficient but may miss semantic or contextual nuance; LLM/MLLM metrics capture richer relationships at higher computational cost.Source: §5.1 · p. 44
  • Coarse-grained evaluation offers scalable holistic scoring, while fine-grained evaluation decomposes responses into semantic units for precise diagnosis; combining them balances efficiency and rigor.Coarse methods can overlook subtle errors, whereas fine-grained methods are computationally intensive but support iterative refinement through detailed accuracy, relevance, and alignment analysis.Source: §5.1 · pp. 46–47

6 Challenges of MRAG

MRAG faces interconnected challenges across parsing, planning, retrieval, generation, and evaluation. Progress requires preserving multimodal context, adapting to query and modality diversity, improving cross-modal reasoning, and developing comprehensive real-world benchmarks.

6 Challenges of MRAG

  • MRAG challenges span document parsing and indexing, search planning, retrieval, generation, datasets, and evaluation, with each module affecting system effectiveness and robustness.These components form interconnected parts of the MRAG pipeline, so weaknesses in one module can constrain overall performance.Source: §6 · p. 47

6.1 Document Parsing and Indexing

  • Document parsing must preserve accurate, complete, and inter-page context while balancing original multimodal data with textual representations, increasing pipeline complexity and storage demands.Upstream errors can propagate downstream; textual captions may outperform raw images, while additional representations and components increase processing, storage, and management requirements.Source: §6.1 · pp. 47–48

6.2 Multimodal Search Planning

  • Search planning requires adaptive mechanisms, semantically aligned query reformulation, and comprehensive benchmarks spanning simple retrieval, multi-hop reasoning, creative tasks, efficiency, and scalability.Fixed pipelines poorly accommodate heterogeneous queries; multi-agent coordination is suggested for complex cases, while evaluation must cover adaptability, robustness, alignment, and resource use.Source: §6.2 · p. 49

6.3 Retrieval

  • Multimodal retrieval must align heterogeneous modality-specific features in a shared semantic space while developing fine-grained rerankers and post-retrieval refiners for cross-modal reasoning.Text, images, audio, and video require different feature extraction strategies, and current approaches often fail to preserve semantic richness and inter-modal consistency.Source: §6.3 · p. 49

6.4 Generation

  • Multimodal generation must flexibly integrate arbitrary input combinations and produce coherent, contextually placed, relevant, high-quality, and sufficiently diverse outputs.Systems must align text with images or videos, choose insertion points, retrieve or generate suitable content, and balance diversity against relevance and quality.Source: §6.4 · pp. 49–50

6.5 Dataset & Evaluation

  • MLLM evaluation lacks standardized capability taxonomies and comprehensive coverage of instruction following, complex reasoning, dialogue, creativity, audio, 3D, commercial tasks, efficiency, and agent interaction.Current benchmarks overemphasize objective perception and reasoning questions, while real-world and integrated multimodal skills remain insufficiently assessed.Source: §6.5 · pp. 50–51

7 Future Directions

The paper proposes future MRAG research directions spanning document parsing, search planning, retrieval, generation, and dataset design, targeting accuracy, adaptability, coherence, and evaluation gaps. These recommendations aim to overcome current limitations and support robust MRAG systems in complex real-world scenarios.

7 Future Directions

  • Future MRAG development should address existing limitations across document parsing, search planning, retrieval, generation, and evaluation to unlock its potential in complex real-world scenarios.The recommendations are informed by related research and identified challenges.Source: §7 · p. 51

7.1 Documents Parsing

  • Document parsing should preserve cross-page and intra-document relationships, detect and correct errors, unify modalities, generate domain-specific descriptions, and combine MLLMs’ extraction with LLMs’ textual synthesis.Graph-based representations, hierarchical modeling, cross-referencing, consistency checks, human review, shared embeddings, OCR integration, and hybrid LLM/MLLM architectures are proposed for complex documents.Source: §7.1 · pp. 51–53

7.2 Multimodal Search Planning

  • Multimodal search planning should combine collaborative agents, human feedback, reinforcement learning, semantic query reformulation, cross-modal alignment, explainability, and benchmarks covering diverse queries and multidimensional metrics.The proposed metrics include adaptability, robustness, efficiency, and semantic alignment across heterogeneous real-world modalities and query complexities.Source: §7.2 · pp. 53–55

7.3 Retrieval

  • Retrieval research should develop unified cross-modal representations and attention mechanisms, then use multimodal reranking and knowledge-enhanced refinement to improve semantic alignment and retrieval relevance.These methods target fine-grained interactions among text, images, audio, and video during reranking and refinement.Source: §7.3 · p. 55

7.4 Generation

  • Multimodal generation should support arbitrary modality combinations, maintain cross-modal consistency, intelligently position and retrieve or generate non-textual elements, and balance output diversity with relevance and quality.Shared latent spaces and conditional sampling are proposed to align generated content with textual or multimodal inputs.Source: §7.4 · p. 56

7.5 Dataset & Evaluation

  • Datasets and evaluation should include instruction following, multiturn dialogue, complex multimodal reasoning, creativity, emerging modalities, and metrics for relevance, precision, diversity, alignment, coherence, fluency, factuality, and consistency.Benchmarks should simulate varied real-world queries, multimodal sources, and complexity levels to assess end-to-end MRAG retrieval and generation.Source: §7.5 · pp. 56–57

8 Conclusion

This survey examines MRAG as a framework that integrates text, images, and videos into retrieval and generation, improving response accuracy and relevance while reducing hallucinations. It organizes the field around core technologies, datasets, evaluation, limitations, challenges, and future research directions.

Conclusion

  • MRAG integrates text, images, and videos to retrieve and generate information across modalities, improving response accuracy and relevance while reducing hallucinations.Unlike text-based RAG, MRAG addresses multimodal retrieval and generation challenges.Source: §8 · p. 57
  • The survey systematically analyzes MRAG’s essential components and technologies, datasets, evaluation methods and metrics, and existing limitations.Source: §8 · p. 57
  • It identifies challenges in integrating multimodal knowledge and ensuring the reliability of generated outputs, while proposing future research directions.The survey aims to guide research toward more robust and versatile MRAG systems.Source: §8 · p. 57

Related papers

Loading 2504.08748v1…