Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

Retrieval-Augmented Generation

• RAG (Lewis et al., 2020) retrieves external knowledge for a generator, improving factual accuracy and reducing hallucinations while grounding responses in verifiable sources.Its retriever-generator pipeline identifies relevant passages and supplies them as context; planning-guided, agentic, and feedback-driven approaches further develop these components.Source: §1 · p. 1

Multimodal RAG

• Multimodal RAG incorporates sources such as images, audio, and structured data to enrich contextual grounding and generation, while raising challenges in modality selection, fusion, and cross-modal relevance.It leverages cross-modal cues to improve reasoning.Source: §1 · p. 2

Multimodal RAG Formulation

• The formulation encodes documents into a shared semantic space, retrieves those exceeding modality-specific relevance thresholds, and generates a response conditioned on the query and retrieved context.It assumes one modality per document for clarity, while noting that mixed-modality documents may be decomposed or processed with universal encoders.Source: §1 · p. 2

Related Works

• Unlike prior RAG surveys, it offers an innovation-driven multimodal taxonomy and reviews over 100 recent papers, addressing trends and open challenges (Zhao et al., 2023a).The survey notes that Zhao et al. (2023a) categorized multimodal RAG by application and modality.Source: §1 · p. 2

Contributions

• The survey reviews multimodal RAG across task formulation, datasets, benchmarks, applications, retrieval, fusion, augmentation, generation, training strategies, loss functions, and agent frameworks.It also identifies research gaps and offers directions for future work.Source: §1 · p. 3
• The survey provides open-access datasets, benchmarks, and implementation details to facilitate future research.It also proposes a structured taxonomy of state-of-the-art models and emerging methodological trends.Source: §1 · p. 3

2 Datasets, Evaluation, and Applications

• The survey reviews datasets and benchmarks for tasks including multimodal summarization, visual QA, and video understanding.Full details are provided in Appendix (§B) and Tables 1 and 2.Source: §2 · p. 3
• Multimodal RAG applications span healthcare, software engineering, fashion, entertainment, and emerging fields.Tasks and applications are overviewed in Appendix (§E) and Figure 3.Source: §2 · p. 3
• Evaluation requires metrics for retrieval performance, generation quality, and modality alignment.Evaluation methods, metrics, definitions, and formulations are provided in Appendix (§C).Source: §2 · p. 3

3.1 Retrieval Strategy

• Contrastive alignment and hard-negative mining improve cross-modal discrimination, while approximate and adaptive search methods seek efficient top-k retrieval without sacrificing recall.Hybrid sparse–dense retrieval, distributed MIPS, and learned indexes offer additional ways to balance retrieval quality, latency, and storage.Source: §3.1 · p. 3
• Modality-aware retrieval spans text, images, video, audio, and documents, using specialized representations and processing for each evidence type.Examples include token-level text matching, composed-image and temporally aware video retrieval, raw-audio embeddings, and OCR-free document-page retrieval with layout reasoning.Source: §3.1 · pp. 3–5
• Re-ranking and filtering improve candidate quality through optimized example selection, multimodal relevance scoring, and removal of irrelevant or confusing evidence.Methods include statistical calibration, cross-modal and visual similarity, hierarchical confidence scoring, hard-negative mining, consensus filtering, and dynamic modality filtering.Source: §3.1 · p. 5

3.2 Fusion Mechanisms

• Fusion methods align or consolidate modalities using shared embedding spaces, score fusion, cross-attention, unified representations, and learned projections.Approaches include attention-based interaction and hierarchical or iterative fusion; some trade-offs include increased inference complexity, reduced modality-specific granularity, or extra reasoning passes.Source: §3.2 · pp. 6–7

3.3 Augmentation Techniques

• Augmentation refines retrieved context and adapts or iterates retrieval to better match query complexity, modality, granularity, and task needs.Methods enrich context with entities or auxiliary multimodal information, route queries to suitable corpora, vary retrieval hops, filter irrelevant results, and refine results using feedback from previous rounds.Source: §3.3 · p. 7

3.4 Generation Techniques

• Generation methods use retrieved examples, structured reasoning, instruction tuning, and evidence attribution to support task-specific multimodal responses.The surveyed approaches include multimodal in-context example selection, chain- or tree-based reasoning with evidence verification, retrieval-aware fine-tuning, and source citation or screenshot highlighting; attribution can fail when evidence spans sections or modalities.Source: §3.4 · pp. 7–8
• Agentic generation applies autonomous or semiautonomous systems to interaction, specialized applications, and complex multimodal reasoning.Examples include mobile navigation, personalized conversation, industrial and clinical tasks, scientific exploration, hierarchical multi-agent collaboration, and adaptive planning.Source: §3.4 · p. 8

3.5 Training Strategies

• Training combines paired-data pretraining and task-specific fine-tuning with contrastive, alignment, regularization, and modality-specific losses.The surveyed strategies also use hard or adversarial negatives, synthetic positive pairs, and dense-to-sparse objectives to improve discrimination, robustness, generalization, and retrieval precision.Source: §3.5 · pp. 8–9

Generalization, Explainability, and Robustness

• Systems face domain and modality biases, coarse cross-modal attribution, degraded performance on low-quality or outdated sources, and adversarial or poisoning risks.MM-PoisonRAG (Ha et al., 2025) and Poisoned-MRAG (Liu et al., 2025b) show that a few adversarial knowledge injections can hijack cross-modal retrieval and derail generation; multimodal robustness remains open.Source: §4 · p. 9

Reasoning, Alignment, and Retrieval Enhancement

• Stronger compositional reasoning, modality alignment, and entity-aware retrieval are needed, while knowledge graphs remain underexplored and retrieval suffers from position, redundancy, and content biases.A unified embedding space could enable direct multimodal search without intermediary models, but mapping multimodal knowledge into such a space remains an open challenge.Source: §4 · p. 9

Agent-Based and Self-Guided Systems

• Future agent-based systems should iteratively refine retrieval, reasoning, and generation through interactive feedback and self-guided decisions rather than static pipelines.Reinforcement learning and end-to-end human-aligned feedback could assess retrieval need and relevance and choose modalities; systems should support any-to-any tasks and real-world sources such as sensors, advancing embodied applications.Source: §4 · p. 9

• The review categorizes advances in retrieval, multimodal fusion, augmentation, generation, training strategies, and agents.Source: §5 · p. 9
• It examines task-specific applications, datasets, benchmarks, and evaluation methods while highlighting open challenges.Source: §5 · p. 9
• Future research directions include improving cross-modal reasoning and retrieval, developing agent-based interactive systems, and advancing unified multimodal embedding spaces.Source: §5 · p. 9

6 Limitations

• The survey does not compare model performance because task definitions, metrics, and implementation details vary across studies, while running the models requires substantial computational resources.Source: §6 · p. 10
• Descriptions of individual methodologies are necessarily concise due to space constraints.Source: §6 · p. 10
• Despite curating studies from major venues and arXiv, the survey may overlook emerging or domain-specific research, with emphasis on recent advancements.Source: §6 · p. 10
• Open questions include optimizing fusion across diverse modalities and addressing scalability; the authors plan to monitor developments and update the survey and repository.As new paradigms emerge, the survey's taxonomy and conclusions will evolve.Source: §6 · p. 10

7 Ethical Statement

• The survey analyzes publicly available literature, uses no personal data or user interactions, and introduces no new models, datasets, or experiments, so it presents no direct risks.Only a very small number of the studies, datasets, and benchmarks reviewed require institutional access.Source: §7 · p. 10
• Multimodal RAG raises risks of bias, misinformation, privacy violations, and attribution or copyright problems; the survey calls for careful curation, bias mitigation, and transparent evaluation.Hallucinations or misinformation may occur when retrieval fails or relies on unreliable sources.Source: §7 · p. 10

A Taxonomy

• The taxonomy groups retrieval strategies into efficient search, multimodal and modality-centric retrieval, and re-ranking through example selection, relevance scoring, and filtering.It includes maximum inner product search variants and text-, vision-, audio-, video-, and document-centric models.Source: A Taxonomy · p. 26
• Fusion mechanisms include score fusion and alignment, attention-based cross-modal interaction, and unified frameworks or projections that consolidate multimodal inputs.Examples include CLIP score fusion, prototype-based embeddings, cross-attention and co-attention transformers, hierarchical fusion, and dense-to-sparse projections.Source: A Taxonomy · p. 26
• Augmentation techniques enrich context and support adaptive, iterative retrieval.Source: A Taxonomy · p. 26
• Generation methods span in-context learning, reasoning, instruction tuning, source attribution, and agentic frameworks.Source: A Taxonomy · p. 26
• Training strategies are characterized by approaches to alignment and robustness.Source: A Taxonomy · p. 26
• The application-domain taxonomy covers healthcare and medicine, software engineering, fashion and e-commerce, entertainment and social computing, and emerging applications.The classification is presented as an overview and framework for more detailed analyses.Source: A Taxonomy · p. 26

Dataset and Benchmark

• Datasets range from image-text, video-text, and medical collections to knowledge-intensive QA, with unimodal datasets often combined to construct multimodal contexts.Examples include MS-COCO (Lin et al., 2014), ActivityNet (Caba Heilbron et al., 2015), MIMIC-CXR (Johnson et al., 2019), and MultimodalQA; audio-language datasets and 3D collections such as ShapeNet (Chang et al., 2015) extend modality coverage.Source: B Dataset and Benchmark · pp. 27–28
• Benchmarks assess visual reasoning, external-knowledge integration, dynamic retrieval, and multimodal generation across diverse task settings.M2RAG (Ma et al., 2024d) combines fine-grained text-modal and multimodal metrics; other evaluations cover captioning, question answering, fact verification, image reranking, document understanding, and multi-hop reasoning.Source: B Dataset and Benchmark · p. 27

Comparative Analysis of Datasets

• Comparative Analysis of Datasets: large web-crawled datasets support broad pretraining but may be noisy, while curated or domain-specific collections prioritize annotation quality for fine-tuning and specialized evaluation.Explicit multimodal datasets offer stronger modality alignment than aggregating unimodal datasets; audio-language datasets address audio integration, while 3D remains underrepresented despite its relevance to spatial reasoning.Source: B Dataset and Benchmark · pp. 27–28

Limitations of Existing Datasets and Benchmarks

• Limitations of Existing Datasets and Benchmarks: persistent gaps include bias, noisy annotations, limited out-of-domain coverage, weak real-world complexity, scarce adversarial tests, and limited any-to-any modality support.OHRBench (Zhang et al., 2024d) illustrates how OCR errors can cascade through RAG; current evaluations often separate retrieval from generation, motivating holistic measures of retrieval accuracy, context relevance, and output quality.Source: B Dataset and Benchmark · pp. 28–29

Retrieval Evaluation

• Retrieval evaluation uses accuracy, precision, recall, F1, Top-K Accuracy, Recall@K, and MRR, with Recall@K preferred over standard recall.MRR measures the rank of the first relevant result; Adjali et al. (2024) and Nguyen et al. (2024) use it.Source: C Evaluation and Metrics · p. 29

Modality Evaluation

• Text evaluation includes Exact Match, BLEU, ROUGE, and METEOR; ROUGE-N measures n-gram overlap, while ROUGE-L measures the longest common subsequence.MultiRAGen uses Multilingual ROUGE in multilingual settings.Source: C Evaluation and Metrics · pp. 29–30
• Image-caption and semantic-alignment evaluation uses CIDEr, SPICE, SPIDEr, BERTScore, and CLIP Score to assess caption quality, semantics, fluency, and image-text similarity.CIDEr uses TF-IDF and cosine similarity; SPIDEr combines CIDEr and SPICE.Source: C Evaluation and Metrics · p. 30
• Image quality metrics include FID, KID, and Inception Score, while audio evaluation uses human ratings for sound quality and text relevance plus Fréchet Audio Distance.FID compares image-feature distributions, and FAD is an audio-specific variant of FID.Source: C Evaluation and Metrics · p. 30

Efficiency and Domain-Specific Metrics

• System efficiency is measured with FLOPs and execution, response, and retrieval time per query; domain-specific metrics include geodesic distance and Clinical Relevance.Geodesic distance is used for geographical accuracy, and Clinical Relevance for medical applications.Source: C Evaluation and Metrics · p. 30

D.1 Robustness and Noise Management

• MORE (Cui et al., 2024) injects irrelevant retrieval results during training to improve focus on relevant inputs.Its approach targets noisy retrieval inputs, which the survey identifies as important to maintaining model performance.Source: §D.1 · p. 30
• AlzheimerRAG (Lahiri and Hu, 2024) uses progressive knowledge distillation to reduce noise while preserving multimodal alignment, while RAGTrans uses hypergraph-based aggregation to refine representations and propagate relevant information.Source: §D.1 · p. 30
• RA-BLIP (Ding et al., 2024b) uses ASKG to filter relevant knowledge through LLM capabilities and a denoising-enhanced loss, without fine-tuning.The approach achieves strong baseline performance while reducing computational overhead by minimizing trainable parameters.Source: §D.1 · pp. 30–31
• RagVL (Chen et al., 2024e) builds noise resilience with hard negatives at the data level and Gaussian noise plus loss reweighting at the token level.Source: §D.1 · p. 31
• RA-CM3 (Yasunaga et al., 2023) applies Query Dropout by randomly removing query tokens during retrieval as regularization to improve generator performance and generalization.Source: §D.1 · p. 31

D.2 Loss Function

• The loss subsection presents InfoNCE for contrastive learning, GAN discriminator and generator losses, and Triplet Loss for bringing similar embeddings closer than dissimilar ones.For InfoNCE, zi and zj are a positive pair and τ is the temperature parameter; Triplet Loss uses an anchor, positive, and negative samples.Source: §D.2 · p. 31

E Applications and Relevant Tasks

• Multimodal RAG grounds image captioning, text-to-image synthesis, QA, fact verification, and multimodal summarization in retrieved knowledge, helping mitigate hallucinations.Applications also include zero-shot image–text retrieval and chain-of-thought reasoning for complex inference; AI assistants support visual search, document understanding, and multimodal reasoning.Source: E Applications and Relevant Tasks · p. 31
• MMED-RAG (Xia et al., 2024a), RULE (Xia et al., 2024b), RA-RRG (Choi et al., 2025), and FactMM-RAG (Sun et al., 2024b) apply retrieval to medical QA and radiology reports.Approaches align radiology images with patient data, retrieve similar cases or ontology-based biomarker correlations, and link visual features to phrases extracted from report corpora; RA-RRG generates reports without fine-tuning to reduce hallucinations.Source: E Applications and Relevant Tasks · p. 31
• AsthmaBot (Bahaj and Ghogho, 2024) enables multilingual semantic searches, Realm (Zhu et al., 2024c) fuses patient data for risk assessment, and Su et al. (2024a) advances federated clinical-data privacy.These applications extend multimodal RAG to patient support, predictive frameworks, and privacy-preserving clinical data integration.Source: E Applications and Relevant Tasks · p. 32
• DocPrompting (Zhou et al., 2023), RACE (Shi et al., 2022), CEDAR, and REDCODER (Parvez et al., 2021) use retrieval for code completion, commit messages, few-shot prompting, and summarization.Retrieval draws on API specifications, debugging patterns, repository history, and open-source code to support context-aware generation and preserve programming conventions.Source: E Applications and Relevant Tasks · p. 32
• UniFashion (Zhao et al., 2024), Dang (2024), Fashion-RAG (Sanguigni et al., 2025), and LLM4DESIGN (Chen et al., 2024d) apply cross-modal retrieval to discovery, editing, and design automation.Fashion-RAG retrieves garments matching textual descriptions and integrates their attributes into diffusion-model image generation; LLM4DESIGN retrieves architectural compliance constraints and environmental-impact assessments.Source: E Applications and Relevant Tasks · p. 32
• SoccerRAG (Strand et al., 2024), MMRA, RAG-Driver (Yuan et al., 2024), ENWAR (Nazar et al., 2024), Riedler and Langer (2024), and Img2Loc (Zhou et al., 2024e) extend applications to entertainment and emerging domains.Examples include linking match footage with player statistics, predicting content virality, retrieving traffic scenarios during navigation, fusing network sensors, retrieving maintenance schematics, and correlating landmarks for geolocalization.Source: E Applications and Relevant Tasks · p. 32

F Additional Future Directions

• Long-context costs, inefficient retrieval and fusion, limited personalization, privacy risks, and inadequate evaluation data remain key challenges for multimodal RAG.Video frame sampling and image-rich multi-page documents create computational and memory bottlenecks; fixed frame extraction rates may miss relevant content, motivating adaptive selection. Evaluation is constrained by the lack of complex-reasoning datasets and multimodal adversarial examples.Source: F Additional Future Directions · p. 32