PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Multimodal Retrieval-Augmented Generation

Lang Mei, Siyu Mo, Zhihan Yang, Chong Chen

arXiv:2504.08748v1cs.IRcs.AIcs.CLcs.ETcs.LG

TL;DR

Multimodal Retrieval-Augmented Generation addresses the limitations of text-only systems by processing information across text, images, and videos. This survey organizes MRAG research around its components, datasets, evaluation methods, and limitations, while identifying challenges and future directions.

  • Problem

    Text-only LLM and RAG systems remain limited when tasks require multimodal context, motivating research into effectively processing multimodal data for retrieval-augmented generation.

  • Method

    The survey systematically reviews MRAG's components and technologies, datasets, evaluation methods and metrics, limitations, challenges, and future research directions.

  • Results

    MRAG integrates text, images, and videos into retrieval and generation to improve response accuracy and relevance while reducing hallucinations.

  • Takeaways & Limitations

    The structured overview is intended to guide researchers in advancing MRAG and improving the reliability of multimodal retrieval and generation.

  • Takeaways & Limitations

    MRAG evaluation lacks a universally accepted capability taxonomy and has gaps in complex multimodal reasoning, multi-turn dialogue, creativity, and commercially relevant tasks.

Abstract

from arXiv · show

Multimodal Retrieval-Augmented Generation (MRAG) enhances large language models (LLMs) by integrating multimodal data (text, images, videos) into retrieval and generation processes, overcoming the limitations of text-only Retrieval-Augmented Generation (RAG). While RAG improves response accuracy by incorporating external textual knowledge, MRAG extends this framework to include multimodal retrieval and generation, leveraging contextual information from diverse data types. This approach reduces hallucinations and enhances question-answering systems by grounding responses in factual, multimodal knowledge. Recent studies show MRAG outperforms traditional RAG, especially in scenarios requiring both visual and textual understanding. This survey reviews MRAG's essential components, datasets, evaluation methods, and limitations, providing insights into its construction and improvement. It also identifies challenges and future research directions, highlighting MRAG's potential to revolutionize multimodal information retrieval and generation. By offering a comprehensive perspective, this work encourages further exploration into this promising paradigm.

1 Introduction

The survey motivates MRAG as an extension of text-only RAG that incorporates multimodal knowledge to address LLM limitations in specialized, knowledge-intensive, and visually grounded tasks. It reviews MRAG components, datasets, evaluation methods, limitations, challenges, and future directions.

  • Motivation: LLMs can generate fluent, coherent answers but may lack sufficient specialized or up-to-date knowledge for domain-specific and knowledge-intensive tasks.The introduction highlights medicine, law, finance, and other technical fields as settings where precision and current knowledge matter.
  • Motivation: RAG mitigates hallucinations by retrieving external knowledge and combining relevant document chunks with the query during generation.Its two-step process retrieves semantically relevant chunks and supplies them as context for the LLM.
  • Motivation: Text-focused RAG overlooks multimodal sources such as images and videos, although real-world queries often require information from multiple formats.Multimodal data can provide contextual or instructional information alongside text.
  • MRAG: MRAG extends retrieval and generation to multimodal queries, knowledge, and outputs through multimodal large language models.Compared with text-modal RAG, MRAG must handle inputs and retrieved knowledge from different modalities and produce multimodal answers.
  • Survey scope: The survey systematically examines MRAG components and technologies, datasets, evaluation methods and metrics, limitations, challenges, and future research directions.Its stated goal is to clarify how MRAG can be constructed, evaluated, and improved.

2 Overview of MRAG

MRAG evolves from text-oriented pseudo-MRAG toward systems that preserve and retrieve multimodal data directly. Across MRAG1.0–3.0, the architecture progressively reduces information loss and expands from multimodal understanding to end-to-end multimodal generation.

  • MRAG extends text-only RAG by processing diverse modalities, including images, audio, video, and text, for more complex real-world applications.
  • MRAG1.0: MRAG1.0, or pseudo-MRAG, uses document parsing and indexing, retrieval, and generation, converting multimodal content into modality-specific captions stored with textual data.
  • MRAG1.0: Caption-based parsing in MRAG1.0 increases complexity and can lose fine-grained multimodal information, making retrieval and generation sensitive to input quality.
  • MRAG2.0: MRAG2.0 preserves original multimodal data, adds multimodal retrieval and cross-modal search, and uses MLLMs to process multimodal data directly during generation.
  • MRAG2.0: MRAG2.0 still faces challenges from reduced textual-query accuracy, weaker multimodal retrieval than text retrieval, and the diversity of generation inputs.
  • MRAG3.0: MRAG3.0 retains document screenshots, adds Multimodal Search Planning, and produces multimodal outputs, unifying VQA, retrieval-augmented generation, and broader generation scenarios.

3 Components & Technologies of MRAG

The survey organizes MRAG technologies around four components: multimodal document parsing and indexing, multimodal search planning, multimodal retrieval, and multimodal generation.

  • The survey identifies four key MRAG components: multimodal document parsing and indexing, multimodal search planning, multimodal retrieval, and multimodal generation.

3.1 Multimodal Document Parsing and Indexing

Multimodal document parsing and indexing organize unstructured and semi-structured data for MRAG, using extraction-based or representation-based approaches. These methods trade simplicity and efficiency against preservation of multimodal content, document structure, and downstream reliability.

  • Document types: Unstructured data lacks a specific schema, while semi-structured formats retain organizational features that can be lost during parsing.Examples include images, videos, audio, PDFs, HTML, XML, and JSON.
  • Document parsing approaches: MRAG primarily processes unstructured and semi-structured documents through extraction-based or representation-based parsing approaches.The choice depends on task-specific requirements such as scalability or computational efficiency.
  • Extraction-based parsing: Plain-text extraction is simple and efficient but misses multimodal information and can substantially lose document structure.It struggles with complex document formats and cannot extract information such as text embedded in images.
  • Extraction-based parsing: OCR-dependent pipelines hinder parallelization, consume substantial computing resources, and propagate upstream errors into downstream performance.Transformer-based approaches address these issues through parallel computation, global context modeling, and end-to-end image-to-structured-text mapping.
  • Extraction-based parsing: OCR-based parsing uses page segmentation, text recognition, and text parsing to process multimodal documents.The pipeline locates regions, recognizes text, and performs layout-aware processing for downstream requirements.
  • Representation-based parsing: Representation-based methods use document screenshots and unified encodings to preserve global and local information, but page-level splitting can limit full-context modeling.Holistic representations instead segment large documents into passages within MLLM token limits.

3.2 Multimodal Search Planning

Multimodal search planning determines how MRAG systems retrieve and integrate information across modalities. The survey contrasts fixed pipelines with adaptive strategies that adjust retrieval to query needs and intermediate results.

  • Planning overview: Multimodal search planning comprises fixed and adaptive strategies for retrieving and integrating information across modalities.The planning problem concerns how MRAG systems address complex multimodal queries.
  • Fixed planning: Early fixed strategies typically use a single retrieval modality, choosing text-centric or image-centric processing regardless of query variation.Text-centric systems convert visual query content into descriptions, whereas image-centric systems consistently trigger image retrieval.
  • Fixed planning: Compulsory image retrieval can impair MLLM performance when textual information suffices or retrieved images introduce misleading visual contexts.Text-centric and image-centric strategies are therefore suited to different information needs.
  • Fixed planning: Fixed pipelines struggle with diverse queries, introduce redundant information, and impose computational overhead through mandatory retrieval operations.The survey also notes that some queries do not require external knowledge retrieval.
  • Fixed planning: Non-adaptive retrieval ignores evolving context, while overloaded queries combine visual descriptions and questions in ways that can produce ambiguous retrievals.These limitations motivate planning mechanisms that revise retrieval decisions during problem solving.
  • Adaptive planning: Adaptive systems dynamically decompose complex multimodal questions, select retrieval actions, and refine strategies using retrieved content and problem-solving state.OmniSearch uses sub-question chains, while CogPlanner iteratively refines queries and supports parallel and sequential modeling.

3.3 Multimodal Retrieval

Multimodal retrieval in MRAG combines retrievers, rerankers, and refiners to source and optimize relevant information across modalities. The survey organizes retrieval methods by architecture, modality, representation, and training strategy.

  • Component overview: MRAG multimodal retrieval comprises retriever, reranker, and refiner components that jointly improve information quality and relevance for LLMs.The retriever sources documents, while subsequent components optimize their ranking and usability.
  • Retriever architectures: Retriever architectures include single/dual-stream and generative structures, supporting both single-modal and cross-modal retrieval.Single-stream models capture fine-grained interactions but incur higher computational costs and slower inference, whereas dual-stream models separate vision and language processing.
  • Text retrieval: Text retrieval methods are commonly categorized as sparse or dense retrieval, with sparse approaches relying on lexical matching and dense approaches using vector representations.Sparse methods include tf-idf and BM25, while dense methods use embeddings and similarity functions.
  • Dense retrieval: Dense retrieval uses approximate nearest-neighbor search for efficient top-ranked retrieval and typically combines two-tower architectures with negative sampling and pretraining.Representations may be single-vector or multi-vector, with relevance computed through similarity functions such as cosine similarity or inner product.
  • Dense retrieval: Ambiguous negatives with moderate query similarity provide more informative training examples and can enhance dense retrieval performance.They are less likely to be false negatives than highly similar alternatives.
  • Retrieval modalities: Single-modal retrieval covers text and images, while cross-modal retrieval identifies data in one modality using queries from another.Cross-modal retrieval supports information access across text, images, audio, and video.

3.4 Multimodal Generation

Multimodal generation extends language models to integrate diverse inputs and produce outputs across text, images, audio, and video. The survey traces this progression from image-capable models to unified multimodal systems and multimodal-output workflows.

  • Multimodal generation: Multimodal large language models integrate diverse data types to generate rich, coherent, contextually relevant content across text, images, audio, and video.The survey classifies these models according to their generative input and output modalities.
  • Multimodal inputs: Early multimodal generation efforts added images to language-model inputs, enabling visual question answering and related multimodal understanding tasks.BLIP-2 is cited as an example of an efficient image-input pretraining strategy.
  • Multimodal inputs: Later systems incorporated video through visual encoders, temporal representations, and tokenization strategies that support video understanding and generation.Video-LaVIT represents videos using keyframes and temporal motions before adapting them to an LLM.
  • Unified multimodal inputs: Unified-input models handle arbitrary modalities and can use instruction-aware feature extraction to tailor multimodal representations to user instructions.InstructBLIP and InternVL exemplify this development.
  • Multimodal outputs: Multimodal output research remains constrained by limited image-only coverage and scenarios requiring at least one relevant input image.The survey notes that subsequent research addresses these limitations.
  • Multimodal outputs: A common multimodal-output workflow identifies insertion positions, retrieves candidate multimodal data, and selects final content through relevance matching.InternLM-XComposer is cited as producing text-and-image outputs.

4 Dataset for MRAG

The survey evaluates MRAG through curated datasets covering the full retrieval-and-generation pipeline and separate generation capabilities. This organization targets real-world multimodal understanding and knowledge-based question answering.

  • Dataset organization: MRAG datasets are divided into Retrieval & Generation-Joint Components and Generation categories to assess different parts of the pipeline.The joint category evaluates retrieval-generation synergy, while the generation category focuses on generation capabilities.

4.1 Dataset for Retrieval & Generation

Retrieval-and-generation datasets evaluate whether systems can retrieve relevant multimodal knowledge and generate accurate responses. The survey covers benchmarks spanning closed knowledge, external knowledge, multimodal search, multimodal outputs, and visual reasoning.

  • Benchmark purpose: Retrieval-and-generation datasets evaluate end-to-end systems that retrieve text, images, or videos and generate accurate, relevant responses.These benchmarks test the synergistic use of external multimodal knowledge.
  • Knowledge-based datasets: Early benchmarks differ in knowledge and reasoning demands: FVQA uses fixed knowledge graphs, KVQA emphasizes named entities and Wikipedia retrieval, and OK-VQA and A-OKVQA use external knowledge.These datasets vary in how much reasoning is required beyond retrieving known information.
  • Multimodal question answering: ManyModalQA uses snippets, images, and tables while emphasizing answer-modality choice, whereas MIMOQA requires both text and image selections.WebQA performs multi-hop multimodal retrieval but provides textual answers.
  • Multimodal search and MRAG: MMSearch evaluates multimodal search with image-to-image retrieval, while MRAG-bench tests whether vision-centric retrieved knowledge can outperform textual knowledge in specific scenarios.MRAMG-Bench evaluates answers combining text and images from a multimodal corpus.
  • Visual reasoning: VCR and VisualCOMET evaluate visual commonsense reasoning using movie scenes, alongside datasets such as KnowIT VQA and SOK-Bench.These benchmarks broaden evaluation beyond direct knowledge retrieval.

4.2 Dataset for Generation

Generation datasets evaluate multimodal models’ intrinsic capabilities, broad perception and reasoning skills, and performance in practical scenarios. Across benchmarks, models improve with scale but remain challenged by fine-grained perception, OCR, localization, and complex visual reasoning.

  • Generation evaluation: Generation benchmarks isolate models’ intrinsic ability to produce contextually accurate outputs without external retrieval.This separates the generation component from retrieval and supports analysis of foundational language understanding.
  • Foundational benchmarks: 453K annotated QA pairs make VQA v2 an early benchmark for open-ended questions with concise answers.VizWiz adds 8K questions from visually impaired users, while NLVR2 evaluates captions against image pairs.
  • Capability gaps: Recent comprehensive evaluations find that MLLMs excel at commonsense tasks but lag in image classification, OCR, counting, fine-grained attributes, and precise localization.Fine-tuning can mitigate some of these weaknesses.
  • Comprehensive evaluation: Models improve with scale, yet fine-grained perception, spatial localization, chart and visual mathematics comprehension, and interleaved image-text understanding remain difficult.Open-source MLLMs have rapidly progressed and often match or surpass closed-source models.
  • Real-world evaluation: Real-world benchmarks show persistent fine-grained perception difficulties, while artistic style recognition and relative depth perception are comparatively stronger.Human performance still exceeds that of general models, and GPT-4o outperforms other evaluated models.
  • Reasoning evaluation: Open-ended and LLM-based evaluation captures reasoning processes better than binary or multiple-choice tests but faces inaccurate automated scoring.Reasoning benchmarks assess deductive, abductive, and analogical reasoning, while closed-source models remain weak on localization, charts, and visual mathematics.
  • OCR evaluation: OCR benchmarks span ordinary text recognition, web layouts, charts, maps, and obscured text, but fine-grained performance varies substantially by data type.Models generally handle charts better than knowledge graphs and maps, while specialized OCR systems can outperform general models.

4.2.2 Optical Character Recognition (OCR).

Structural-document benchmarks evaluate layout, spatial relationships, semantic connections, and information extraction across charts, web content, and other dense formats. They broaden multimodal assessment beyond natural-image understanding toward document-centered reasoning.

  • Structural documents: Structural documents require models to interpret intricate layouts, spatial relationships, and semantic connections rather than only visual patterns and textures.Their high information density makes them important for practical applications.
  • Chart and figure understanding: FigureQA evaluates relationships between plot elements using synthetic images and scientific-style figures.It is an early benchmark for visual reasoning over structured graphical content.
  • Chart and figure understanding: ChartQA combines chart data extraction with mathematical reasoning, while ChartX covers 22 topics, 18 chart types, and 7 multimodal tasks.VisualMRC targets visual machine reading comprehension, and ChartBench evaluates chart comprehension and data reliability through complex reasoning.
  • Remaining limitations: A performance gap persists between proprietary and open-source models, especially for reasoning and long-context document comprehension.Extended multimodal contexts remain a critical limitation for current MLLMs.
  • Visual mathematics: Visual mathematics benchmarks assess logical, algebraic, and scientific reasoning across diverse visual inputs and increasingly complex tasks.MathVista aggregates 28 existing datasets and introduces 3 new ones, while Math-Vision and OlympiadBench add more demanding evaluations.

4.2.4 Mathematics.

Mathematical and video-oriented evaluations expose weaknesses in multimodal reasoning, complex diagrams, temporal understanding, and long-context comprehension. Industry-focused benchmarking extends this assessment to specialized real-world domains.

  • Mathematics: Current MLLMs often struggle with complex diagrams, rely heavily on textual cues, and may solve composite problems through memorization rather than underlying reasoning.These limitations motivate further development of multimodal reasoning capabilities.
  • Video understanding: Video benchmarks have expanded from domain-specific human-activity QA to diverse, long-video, egocentric, retrieval, ordering, and fine-grained temporal tasks.Video-MME, MVBench, MMBench-Video, MLVU, LVBench, Event-Bench, VNBench, EgoSchema, and TempCompass represent this broader coverage.
  • Industry: A comprehensive industry benchmark was developed because limited cross-sector evaluation constrained understanding of MLLM applicability in specialized real-world scenarios.MME-Industry covers more than 21 sectors, including power generation, electronics manufacturing, textiles, steel, and chemical processing.

4.2.6 Industry.

Multimodal evaluation also targets conversational and multidisciplinary expertise, aligning benchmarks with extended interaction histories and knowledge spanning diverse academic domains.

  • Conversational evaluation: Multi-round QA benchmarks address the mismatch between chatbot interaction patterns and the single-round format used by most evaluations.SparklesDialogue tests multiple images and dialogue turns with flexible text-image interleaving.
  • Multidisciplinary knowledge: ScienceQA and MMMU evaluate multidisciplinary knowledge through scientific explanations, lectures, and questions spanning diverse subjects.ScienceQA supports chain-of-thought evaluation, while MMMU targets more challenging college-level knowledge.

4.2.8 Multidisciplinary.

The survey covers multidisciplinary multimodal generation datasets spanning visual question answering, charts, documents, mathematics, videos, dialogues, and multi-image reasoning. These datasets vary in modality, task design, scale, and annotation format.

  • Multidisciplinary: Generation benchmarks span visual question answering, chart comprehension, document understanding, mathematics, video question answering, dialogue, and multi-image reasoning.Examples include NLVR2 and VizWiz, ChartBench and ChartX, DocVQA and MP-DocVQA, MathVista, ActivityNet-QA, SparklesDialogue, and MuirBench.
  • Multidisciplinary: Multi-image and video datasets require reasoning over temporal sequences, heterogeneous images, or multiple images per instance.MuirBench contains 2,600 multiple-choice questions over 11,264 images, averaging 4.3 images per instance; ActivityNet-QA uses 58,000 video question-answer pairs.
  • Multidisciplinary: Document and scientific-figure benchmarks test evidence retrieval across pages, visual interpretation, and hallucination detection.One document benchmark includes single-page, cross-page, and unanswerable questions, while SciFIBench targets scientific figure interpretation.
  • Multidisciplinary: Dialogue and reasoning datasets extend generation evaluation to multi-turn image conversations and graph-based scientific question answering.SparklesDialogue datasets contain image-grounded dialogues, while SciGraphQA uses scientific-paper graphs for open-vocabulary multi-turn questions.

5 Evaluation Metrics of MRAG

MRAG evaluation combines component-level assessment with output-quality metrics using human, rule-based, and LLM/MLLM-based strategies. The survey describes metrics for exact matching, semantic similarity, ranking, caption quality, retrieval grounding, and answer faithfulness.

  • Evaluation Strategies: MRAG systems are evaluated across document parsing, search planning, retrieval, and generation using human, rule-based, and LLM/MLLM-based strategies.These strategies differ in directness, interpretability, computational cost, and susceptibility to evaluator-model limitations.
  • Evaluation Strategies: Human evaluation is treated as the gold standard, while LLM/MLLM evaluation reduces human effort but can vary across evaluator models.LLM/MLLM evaluators may show presentation-order bias and produce inconsistent outcomes because of their own capabilities and limitations.
  • Rule-based Metrics: Rule-based metrics include Exact Match, ROUGE-N, BLEU, Mean Reciprocal Rank, CIDEr, and SPICE.These metrics respectively assess exact agreement, n-gram recall, generated-text similarity, first-relevant-result rank, caption-reference agreement, and semantic scene-graph similarity.
  • LLM/MLLM-based Metrics: BERTScore measures semantic similarity with contextual embeddings, unlike BLEU and ROUGE, which rely more directly on lexical overlap.Perplexity measures next-word prediction confidence, with lower values indicating greater confidence and accuracy in predictions.
  • Grounding Metrics: Grounding metrics assess whether generated answers and retrieved contexts capture or support information in reference responses.Answer precision, ground-truth recall, retrieved-context precision, and retrieved-context recall target complementary aspects of factual support and retrieval coverage.

6 Challenges of MRAG

MRAG challenges span the full pipeline, from preserving accurate multimodal document structure to planning searches, aligning heterogeneous modalities, generating coherent outputs, and evaluating real-world capabilities. The survey emphasizes that current systems remain constrained by data, adaptability, integration, and benchmark coverage.

  • Document Parsing and Indexing: Upstream parsing errors or omissions can propagate downstream, while long-document systems still struggle to preserve inter-page relationships.Balancing original multimodal inputs with textual descriptions also increases processing, storage, and management complexity.
  • Multimodal Search Planning: Fixed search pipelines do not adequately adapt to heterogeneous queries with differing complexity or computational constraints.The survey identifies multi-hop reasoning and creative problem-solving as cases that may benefit from adaptive or multi-agent planning.
  • Retrieval: Cross-modal retrieval is difficult because text, images, audio, and video have structurally different representations and require specialized feature extraction.Aligning these features into a shared space while preserving semantic relationships remains challenging.
  • Generation: Multimodal generation must integrate diverse inputs and maintain coherence, placement, contextual relevance, and an appropriate balance between output diversity and quality.These requirements apply to outputs such as text-image or text-video combinations.
  • Dataset and Evaluation: Current evaluation lacks a universal capability taxonomy and has gaps in instruction following, complex reasoning, multi-turn dialogue, creativity, commercial tasks, audio, and 3D representations.Existing benchmarks also focus heavily on perception and reasoning through objective questions, leaving creative evaluation largely unexplored.
  • Dataset and Evaluation: Because MLLMs have limited business applications, current evaluations emphasize foundational capabilities rather than real-world performance.The survey calls for task-specific evaluation that also considers computational cost, inference speed, and agent-based interaction.

7 Future Directions

The survey proposes future directions centered on more reliable multimodal document parsing, unified representations, adaptive search planning, query refinement, and broader evaluation. These directions target the documented challenges of preserving context, aligning modalities, adapting to query complexity, and measuring practical performance.

  • Document Parsing and Indexing: Future document parsing should preserve inter-page and intra-document relationships through graph-based representations and hierarchical document modeling.These approaches are intended to maintain dependencies among sections, tables, figures, and other document elements.
  • Document Parsing and Indexing: Error detection and correction should validate extracted content against source documents and check consistency across multimodal data.LLMs and MLLMs are proposed as components of this validation process.
  • Document Parsing and Indexing: Unified multimodal representations should encode text, images, and tables into shared structures or embedding spaces for context-aware analysis.The survey also recommends stronger captioning and description generation for non-textual elements.
  • Document Parsing and Indexing: Hybrid LLM-MLLM architectures can divide multimodal feature extraction and textual generation between models specialized for those functions.MLLMs process raw multimodal inputs, while LLMs generate coherent textual outputs.
  • Multimodal Search Planning: Future search planning should combine multi-agent collaboration, hierarchical planning, human feedback, and reinforcement learning for adaptive decision-making.These mechanisms target multi-hop, creative, ambiguous, and cross-modal queries while considering efficiency and retrieval accuracy.
  • Multimodal Search Planning: Query reformulation should preserve user intent across modalities through contextual understanding, cross-modal alignment, interactive refinement, and explainable transformations.User feedback can help resolve ambiguity and improve contextual relevance during search.

8 Conclusion

The survey presents MRAG as a framework for integrating text, images, and videos into retrieval and generation to improve response quality and reduce hallucinations. It organizes the field around core technologies, datasets, evaluation, limitations, challenges, and future directions.

  • MRAG integrates text, images, and videos across retrieval and generation, unlike text-based RAG systems.
  • This multimodal approach aims to improve response accuracy and relevance while reducing hallucinations.
  • The survey systematically examines MRAG components and technologies, datasets, evaluation methods and metrics, and existing limitations.
  • It identifies challenges in integrating multimodal knowledge and ensuring the reliability of generated outputs.
  • The survey proposes future research directions intended to guide development of more robust and versatile MRAG systems.
Loading 2504.08748v1…