Source-linked AI summary
Retrieving Multimodal Information for Augmented Generation: A Survey
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
TL;DR
Generative models remain limited by hallucinations, reasoning difficulties, and insufficient interpretability, motivating access to external multimodal knowledge. This survey organizes and analyzes multimodal RAG across image, code, structured knowledge, audio, and video, finding broad applications while identifying unresolved synthesis and attribution challenges.
Problem
Generative models face hallucinations, reasoning difficulties, and limited interpretability, while relevant world knowledge exists in diverse modalities beyond text.
Method
The survey categorizes and reviews multimodal retrieval-augmented generation methods across image, code, structured knowledge, audio, and video.
Results
The reviewed methods support applications including knowledge-intensive image generation, code-based reasoning, and multimodal generation with external retrieval.
Takeaways & Limitations
Multimodal RAG offers a promising framework for improving factuality, reasoning, interpretability, and robustness and motivates future work on multimodal knowledge indexes and retrieval-pretraining combinations.
Takeaways & Limitations
RAG can face an attribution-fluency tradeoff because retrieved-knowledge constraints may affect output quality.
Abstract
from arXiv · showhide
As Large Language Models (LLMs) become popular, there emerged an important trend of using multimodality to augment the LLMs' generation ability, which enables LLMs to better interact with the world. However, there lacks a unified perception of at which stage and how to incorporate different modalities. In this survey, we review methods that assist and augment generative models by retrieving multimodal knowledge, whose formats range from images, codes, tables, graphs, to audio. Such methods offer a promising solution to important concerns such as factuality, reasoning, interpretability, and robustness. By providing an in-depth review, this survey is expected to provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs.
1 Introduction
Multimodal retrieval-augmented generation addresses generative models’ hallucinations, reasoning, interpretability, and robustness challenges by incorporating knowledge from diverse formats. This survey organizes the emerging field by modality, analyzes 146 papers, and identifies connections, applications, and future directions.
- Multimodal retrieval-augmented generation addresses factuality, reasoning, interpretability, and robustness challenges by retrieving knowledge from diverse modalities.
- The survey groups research by image, code, structured knowledge, audio, and video modalities and systematically analyzes 146 relevant papers.
- The authors aim to encourage adapting existing techniques and advancing multimodal retrieval methods for large language models.
- The survey establishes multimodal RAG as an important emerging method group and contextualizes shared connections and challenges across modalities.
2 Definitions and Background
The paper defines multimodal learning as unified representation learning across non-language modalities and describes RAG as retrieval followed by generation guided by retrieved knowledge. RAG helps LLMs interact with external information while addressing hallucination and costly model updating, but multimodal systems still face data and semantic-generation challenges.
- Multimodal Learning: Multimodal learning seeks unified representations that extract complementary information across image, code, structured knowledge, audio, and video.
- Multimodal Learning: Multimodal generative models face challenges in accessing large multimodal datasets and producing semantically meaningful outputs.
- Retrieval-Augmented Generation: RAG retrieves contextually relevant information and then guides generation using the retrieved knowledge.
- Retrieval-Augmented Generation: RAG helps LLMs interact with external information instead of relying only on costly-to-update internal parameters that can produce hallucinations.
- Retrieval-Augmented Generation: Textual retrieval dominates existing RAG applications across translation, dialogue, summarization, and knowledge-intensive generation.
3 Multimodal Retrieval-Augmented Generation
The survey reviews retrieval-augmented generation across image, code, structured knowledge, audio, and video, covering modality-specific retrieval, synthesis procedures, tasks, and challenges. Across these modalities, retrieval supplies external information for generation, reasoning, coding, and data augmentation, while representation alignment and correct synthesis remain open problems.
- Multimodal RAG methods are organized by modality because each modality has distinct retrieval procedures, synthesis mechanisms, target tasks, and challenges.
- Image: Retrieving image-text pairs improves knowledge-intensive image generation and enables multimodal in-context learning.
- Code: Code retrieval supports code generation, summarization, completion, and program repair by supplying relevant snippets, documentation, examples, and fixes.
- Code: Code-generating LLMs can produce executable intermediate reasoning steps whose results are obtained by external interpreters for logical and arithmetic tasks.
- Structured Knowledge: Structured-knowledge RAG still requires more efficient retrieval and reliable decisions about which information to augment in generated text.
- Audio: Audio retrieval addresses scarce audio-text data by retrieving captions, audio snippets, or features for captioning and speech translation.
4 Future Directions
Future work should extend multimodal retrieval augmentation through reasoning, scalable multimodal indexing, and retrieval-aware pretraining. The survey identifies concrete benefits alongside unresolved challenges in unified indexing and retrieval supervision.
- Retrieval Augmented Multimodal Reasoning: Multimodal retrieval can mitigate image-understanding difficulties in multimodal reasoning by supplying informative image-text pairs.Existing two-stage rationale-generation and answer-inference methods may struggle with images such as maps.
- Building a Multimodal Knowledge Index: A multimodal knowledge index should support compact storage, dynamic updates, accurate search, and scalable retrieval speed.Locality-sensitive hashing is identified as one mechanism for addressing scaling and robustness concerns.
- Building a Multimodal Knowledge Index: Unifying multimodal knowledge representations remains a long-term challenge because few studies build multimodal indexes jointly for downstream generation.Dense representations are widely studied for text and images, but simultaneous multimodal indexing remains underexplored.
- Pretraining with Multimodal Retrieval: Retrieval during pretraining could improve models’ interaction with retrieval tools and support more grounded, current, and out-of-domain information use.The survey cites multimodal retrieval augmentation during training as enabling coherent long-form generation with interleaved text and images.
- Pretraining with Multimodal Retrieval: Retrieval-aware pretraining still requires suitable datasets labeled with retrieval API calls, with human annotation costly and automatic labeling potentially biased.The survey presents neuro-symbolic nearest-neighbor methods as one possible alternative.
5 Conclusions
The survey organizes research on retrieving multimodal information to augment generative models across image, code, structured knowledge, speech, and video. It aims to provide a comprehensive overview and motivate work on multimodal reasoning, knowledge indexing, and retrieval-aware pretraining.
- The survey categorizes multimodal retrieval augmentation across image, code, structured knowledge, speech, and video.
- It presents the field as an emerging overview while highlighting multimodal reasoning, multimodal knowledge indexing, and retrieval with pretraining as future research directions.
Limitations
Retrieval-augmented generation faces an attribution–fluency tradeoff in which retrieved-knowledge constraints can affect output quality.
- Retrieved-knowledge constraints can create an attribution–fluency tradeoff that affects output quality.
A.1 Search Criteria and Results
The survey searches ACL Anthology and Google Scholar with modality-specific retrieval and generation or grounding keywords, then manually filters results. The resulting analysis reports the number of papers studied and a temporal trend in multimodal RAG research.
- ACL Anthology searches require “retriev” plus “generat” or “ground,” with additional modality-specific keywords for each modality.
- Google Scholar searches add “language models” and manually filter the top three pages of returned results.
- The survey reports analyzed-paper counts after manual filtering and presents their distribution in Table 1.
- A trend analysis tracks how the number of multimodal retrieval-augmented generation papers changes over time.