PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Generation for AI-Generated Content: A Survey

Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, Bin Cui

arXiv:2402.19473v6cs.CV

TL;DR

AIGC still faces outdated and long-tail knowledge, data leakage, and cost challenges, motivating Retrieval-Augmented Generation. This survey organizes RAG foundations by retriever-generator augmentation, reviews enhancements and multimodal applications, and discusses benchmarks, limitations, and future directions. It concludes that RAG has broad cross-modal principles but remains constrained by retrieval noise, overhead, interaction design, and lengthy contexts.

  • Problem

    AIGC faces outdated knowledge, limited long-tail knowledge, private-data leakage risks, and high generation costs, while prior RAG surveys lack comprehensive coverage.

  • Method

    The paper systematically surveys RAG foundations, enhancement methods, applications across modalities and tasks, benchmarks, limitations, and future directions.

  • Results

    The survey distills common RAG foundation abstractions across retrievers, generators, modalities, and tasks, while organizing practical enhancements and applications.

  • Takeaways & Limitations

    RAG applications can be viewed as adaptations of shared foundation principles despite variation in retrievers and generators across modalities and tasks.

  • Takeaways & Limitations

    RAG remains limited by noisy retrieval, non-negligible latency and storage overhead, retriever-generator mismatch, and lengthy contexts.

Abstract

from arXiv · show

Advancements in model algorithms, the growth of foundational models, and access to high-quality datasets have propelled the evolution of Artificial Intelligence Generated Content (AIGC). Despite its notable successes, AIGC still faces hurdles such as updating knowledge, handling long-tail data, mitigating data leakage, and managing high training and inference costs. Retrieval-Augmented Generation (RAG) has recently emerged as a paradigm to address such challenges. In particular, RAG introduces the information retrieval process, which enhances the generation process by retrieving relevant objects from available data stores, leading to higher accuracy and better robustness. In this paper, we comprehensively review existing efforts that integrate RAG technique into AIGC scenarios. We first classify RAG foundations according to how the retriever augments the generator, distilling the fundamental abstractions of the augmentation methodologies for various retrievers and generators. This unified perspective encompasses all RAG scenarios, illuminating advancements and pivotal technologies that help with potential future progress. We also summarize additional enhancements methods for RAG, facilitating effective engineering and implementation of RAG systems. Then from another view, we survey on practical applications of RAG across different modalities and tasks, offering valuable references for researchers and practitioners. Furthermore, we introduce the benchmarks for RAG, discuss the limitations of current RAG systems, and suggest potential directions for future research. Github: https://github.com/PKU-DAIR/RAG-Survey.

I. INTRODUCTION

AIGC has advanced rapidly but still faces outdated knowledge, long-tail coverage, data leakage, and cost challenges. This survey presents a systematic RAG overview spanning foundations, enhancements, applications, benchmarks, limitations, and future directions.

  • I. INTRODUCTION: AIGC has progressed through advances in algorithms, foundation models, and high-quality datasets across multiple modalities.Examples include LLMs for text and code, diffusion models for images, and generative models for videos.
  • I. INTRODUCTION: RAG addresses key AIGC challenges by using retrievable knowledge as flexible, updatable non-parametric memory.Retrieved knowledge can accommodate long-tail information, encode confidential data, and sometimes lower generation costs.
  • I. INTRODUCTION: A typical RAG pipeline retrieves relevant information for a query, augments generation with the results, and produces multimodal outcomes.Retrieved results may enter as generator inputs, latent representations, logits, or influences across the entire pipeline.
  • I. INTRODUCTION: The survey fills a gap left by prior work that often focuses on one foundation, limited scenarios, or text-centered applications.It argues that a broader overview is needed to cover foundations, enhancements, and applications across domains.
  • I. INTRODUCTION: The paper surveys RAG foundations, enhancement techniques, multimodal applications, limitations, benchmarks, and future research directions.Its contributions include distilling abstractions for different retrievers and generators and reviewing techniques for more effective systems.

C. Related Work

Prior surveys cover selected RAG foundations, enhancements, or modalities, leaving no comprehensive synthesis. The paper introduces RAG architecture, generators, and retrievers as groundwork for its broader review.

  • C. Related Work: Existing RAG surveys commonly emphasize text-related tasks, query-based RAG, or retrieval-based language models.Several works provide only basic overviews, specific applications, or enhancement strategies for limited scenarios.
  • C. Related Work: Other surveys address multimodal applications but omit RAG foundations or cover only part of the non-text literature.The authors therefore identify a need for coverage spanning foundations, enhancements, and cross-domain applicability.
  • C. Related Work: The general RAG architecture contains a retriever and generator: retrieval finds relevant information, and generation produces the required content.The query and retrieved results reach the generator through an augmentation methodology before output is produced.
  • C. Related Work: Typical generators discussed include Transformer models, LSTMs, diffusion models, and GANs for different generation scenarios.The paper describes their use across text, image, video, molecule, audio, and related tasks.

C. Retriever

RAG retrievers locate relevant objects using sparse, dense, or alternative retrieval methods, while foundations classify how retrieval augments generation. These paradigms support applications across modalities and tasks.

  • C. Retriever: Retrieval searches a key-value resource for the top-k keys most similar to a query, using sparse, dense, or alternative similarity methods.The retrieved paired values become relevant objects for downstream generation.
  • C. Retriever: Sparse retrievers use term-matching statistics such as TF-IDF, query likelihood, and BM25 with inverted indices.Query terms retrieve candidate documents that are ranked by statistical scores.
  • C. Retriever: Dense retrievers encode queries and keys as embedding vectors and use Approximate Nearest Neighbor indices for efficient search.Training commonly uses contrastive learning, while inference uses structures such as trees, hashing, and neighbor graphs.
  • C. Retriever: Alternative retrievers include edit-distance or AST matching, knowledge-graph k-hop search, and Named Entity Recognition.These methods retrieve objects without relying exclusively on dense or sparse representations.
  • C. Retriever: RAG foundations are classified into four classes according to how retrieved results augment the generator.The survey introduces foundational paradigms before discussing methods that improve their effectiveness.
  • C. Retriever: Query-based RAG merges retrieved content with the user query at the generator input, whereas latent representation-based RAG integrates retrieved objects inside the model.FiD encodes retrieved paragraphs separately before joint decoding, while RETRO uses chunked cross-attention for sub-query tokens.
  • C. Retriever: These RAG foundations appear in text, code, science, images, knowledge tasks, and other applications through modality-specific retrievers and generators.Examples include FiD use in code and AI-for-Science, cross-attention in image generation, and entity- or mention-level knowledge encoding.

B. RAG Enhancements

RAG enhancements target the input, retriever, generator, result, or entire pipeline, using techniques that refine queries, retrieval quality, generation, outputs, and retrieval decisions.

  • Input Enhancement: Query transformation modifies the input query to improve retrieval, while pseudo-document generation supplies richer retrieval queries for more accurate results.Query2doc and HyDE generate pseudo documents from the original query; TOC and RQ-RAG decompose ambiguous questions into clearer sub-queries.
  • Input Enhancement: Data augmentation improves downstream retrieval or generation by adding information or modifying training examples, such as renamed code and inserted dead code.Make-An-Audio addresses audio data sparsity, while ReACC augments code data for retrieval-model pretraining.
  • Retriever Enhancement: Retriever enhancements include recursive retrieval, chunk optimization, fine-tuning, hybrid retrieval, reranking, filtering, and multi-embedding approaches.Recursive retrieval performs multiple searches; chunk optimization balances fine-grained matching with richer returned context; hybrid retrieval combines sparse and dense methods.
  • Retriever Enhancement: Reranking reorders retrieved content for greater diversity and better results, while filtering removes extraneous material or uses metadata to select documents.Re2G reduces information loss from vector compression, AceCoder removes redundant programs, and FILCO isolates supporting content for the generator.
  • Generator Enhancement: Generator enhancement uses prompt engineering and compression to improve outputs and reduce irrelevant context, including methods addressing the “Lost in the Middle” phenomenon.Prompt methods include compression, Stepback, Active Prompt, and Chain of Thought; LLMLingua compresses query length to accelerate inference.
  • Result and Pipeline Enhancement: Result enhancement rewrites generated content for downstream needs, while adaptive retrieval decides whether external retrieval is necessary.SARGAM refines code outputs with a Transformer and classifiers; Self-RAG, Judgment Prompting, and SKR use model-based retrieval decisions.

IV. APPLICATIONS

The survey examines RAG applications across different modalities and tasks, connecting those applications to the taxonomy of RAG foundations and enhancements.

  • Applications: RAG applications are surveyed across various modalities and tasks, with their utilization demonstrated in Table I.The application survey is structured to echo the paper’s taxonomy of RAG foundations and enhancements.

A. RAG for Text

RAG is applied across text tasks including question answering, fact verification, commonsense reasoning, conversation, translation, event extraction, and summarization.

  • Question Answering: Question-answering systems retrieve relevant snippets, knowledge-graph subgraphs, or multiple documents and combine retrieved information with generation.FiD and REALM synthesize responses generated from top-k snippets; MultiHop-RAG aggregates information from distinct documents.
  • Fact Verification: Fact verification uses multilingual, search-augmented, or end-to-end optimized retrieval to provide evidence and improve knowledge-intensive verification.CONCRETE targets underrepresented languages, while Stochastic RAG addresses non-differentiable top-k retrieval through stochastic sampling.
  • Commonsense Reasoning: Commonsense reasoning uses knowledge graphs to model relationships among concepts and support more coherent generated sentences.KG-BART applies graph attention, while CONFLICTINGQA studies how textual features affect handling of controversial questions.
  • Human-Machine Conversation: Conversation systems use knowledge graphs, retrieved and distilled examples, cross-lingual knowledge, and retrieval-based post-processing to improve dialogue.CEG verifies generated answers through retrieval, while cross-lingual methods address scarce non-English conversation datasets.
  • Neural Machine Translation: Neural machine translation uses monolingual or multilingual retrieval and jointly trains retrieval and generation to improve translation precision.kNN-MT performs token-level retrieval, whereas TRIME reduces the discrepancy between training and inference.
  • Event Extraction and Summarization: Text RAG is also applied to event extraction and summarization, including extractive and abstractive settings with retrieval for richer context or non-English generation.RAMKG supports non-English keyphrase generation, while Unlimiformer addresses input-length constraints in abstractive summarization.

B. RAG for Code

RAG for code combines retrieval and generation across code generation, summarization, completion, repair, semantic parsing, and other code-related tasks.

  • Code Generation: Code generation commonly uses query-based RAG to prompt models with similar examples, APIs, documentation, imports, or global functions.Other approaches use logits from retrieved AST subtrees, hidden-state search, or online and offline tool retrieval during decoding.
  • Code Summarization: Code summarization retrieves similar code or changes using lexical, syntactic, semantic, or edit-distance methods, then fuses retrieved representations for decoding.Systems use separate encoders, LSTMs, graph neural networks, or fusion networks to combine code and retrieved context.
  • Code Completion: Code completion mainly uses query-based RAG, including sparse-dense, iterative, and generated-content-guided retrieval strategies.RepoCoder augments retrieval with previously generated code, while De-Hallucinator retrieves API references from initial generations.
  • Code Completion: Latent-representation and logit-based RAG fuse retrieved code or retrieval probabilities with generation for code completion.Retrieve-and-edit, RepoFusion, and EDITAS use multiple encoders; kNM-LM combines retrieval and generation logits through Bayes inference.
  • Automatic Program Repair: Automatic program repair uses hybrid retrieval to construct prompts from similar errors, buggy code, fixes, and rationale, followed by generation or refinement.RTLFixer iteratively retrieves Verilog errors and solutions while combining reasoning and action planning.
  • Semantic Parsing and Other Tasks: Text-to-SQL and other semantic-parsing systems use query-based RAG to retrieve similar utterances, SQL, and multilingual examples before constrained generation.Other code-related tasks use retrieved examples or programs as intermediate reasoning and refinement artifacts.

C. RAG for Knowledge

RAG for knowledge tasks retrieves structured information such as entities, relations, triplets, tables, and documents to support question answering and dialogue generation. Query-based RAG is prominent, while latent-representation methods and varied retrieval sources provide alternative designs.

  • Knowledge Base Question Answering: Query-based RAG is the mainstream approach for knowledge-base question answering.Systems retrieve entities, relations, logical forms, facts, or triplets and use them to construct generator inputs.
  • Knowledge Base Question Answering: Knowledge-base QA systems retrieve and combine entities, schemas, paths, tables, logical forms, or graph facts before generating answers.Examples include mention detection, dense retrieval, re-ranking, entity disambiguation, and prompt construction.
  • Knowledge-augmented Open-domain Question Answering: Knowledge-augmented open-domain QA commonly uses fusion-in-decoder RAG with triplet documents, KG-re-ranked documents, or question-passage graphs.Other systems retrieve from multiple sources, generate retrieval code, or iteratively retrieve and re-rank evidence.
  • Table for Question Answering: Tables support question answering through fusion-in-decoder readers, re-ranking, table-to-text conversion, and prompts combining retrieved tables with queries or passages.Systems retrieve from heterogeneous sources including knowledge bases, tables, texts, databases, and knowledge graphs.
  • Others: Other knowledge applications retrieve facts, subgraphs, nodes, edges, or dialogue prototypes and fuse them through hidden states, logits, embeddings, or prompts.These methods support dialogue generation, query suggestion, triplet completion, and graph-based generation.

D. RAG for Image

RAG supports image generation, captioning, visual question answering, grounded dialogue, and multimodal translation by incorporating retrieved images, image-text pairs, captions, or visual knowledge.

  • Image Generation: Retrieval helps image generators produce high-quality outputs for rare or unseen subjects while reducing parameter count and computational expense.RetrieveGAN selects image patches differentiably, whereas IC-GAN conditions its generator and discriminator on training instances.
  • Image Generation: Diffusion-based image RAG conditions generation on retrieved image neighbors, CLIP embeddings, labels, prompts, or image-text pairs.Re-imagen uses interleaved guidance to balance prompt alignment with retrieval conditions.
  • Image Captioning: Retrieval-augmented image captioning synthesizes descriptions from retrieved captions used as memory, discriminator references, attention inputs, or in-context examples.CRSR refines retrieved captions by filtering misleading details and emphasizing visually salient content.
  • Others: Other image-related systems retrieve textual descriptions, captions, visual experiences, or external knowledge for VQA, visually grounded dialogue, and multimodal translation.PICa prompts GPT-3 with image descriptions, while RA-VQA uses differentiable retrieval for end-to-end answer generation.

E. RAG for Video

Video RAG retrieves textual, visual, audio, temporal, and expert information to support captioning, question answering, dialogue, prediction, autonomous driving, and text-to-video generation.

  • Video Captioning: Video captioning systems retrieve background documents, relevant sentences, videos, and multimodal data to guide descriptive generation.CARE combines frame, audio, and retrieved text to provide global and local semantic guidance.
  • Video QA&Dialogue: Video QA and dialogue systems retrieve long-term visual-textual information, semantically similar texts, relevant moments, visual concepts, cues, entities, and actions.These retrieved signals support responses aligned with video content and spatio-temporal-aware generation.
  • Others: Other video RAG systems create temporal-aware LLM prompts for captioning, question answering, and future event prediction.RAG-Driver grounds multimodal language models in retrieved expert demonstrations to produce driving action explanations.
  • Others: Animate-A-Story divides text-to-video generation into plot-based video augmentation and video-diffusion generation conditioned on text and video inputs.

F. RAG for Audio

RAG applications extend across audio generation and captioning, 3D asset and motion generation, molecular design, biomedical informatics, and mathematical reasoning.

  • Audio Generation: Audio generation systems retrieve caption-audio pairs or audios from text prompts to augment text-to-audio diffusion-model training and generation.Re-AudioLDM uses dense CLAP retrieval, while Make-An-Audio constructs pseudo prompts from retrieved audios.
  • Audio Captioning: Audio captioning systems use dense retrievers to obtain captions from audio, then incorporate them into prompts or attention modules.RECAP uses CLAP, while another system uses VGGish for retrieval.
  • Text-to-3D: Retrieval augments 3D generation by supplying motion entities, similar motions, or 3D assets to diffusion and variational score-distillation models.AMD combines a prompt-conditioned motion diffusion branch with a branch using anatomically decomposed prompts and retrieved motions.
  • Drug Discovery: Molecular-generation systems retrieve exemplar molecules or curated ligand references and fuse or interact with them to guide molecule and ligand synthesis.RetMol adds lightweight retrieval to a pretrained encoder-decoder, while PromptDiff uses retrieval-augmented 3D molecular diffusion.
  • Biomedical Informatics Enhancement: Biomedical RAG retrieves domain-specific database or literature information for protein prediction, medical responses, clinical reports, pharmaceutical compliance, and biomedical text generation.Approaches include prompt augmentation, chunked-cross attention, hierarchical retrieval, dual-track retrieval, reinforcement learning, and re-ranking.
  • Math Applications: Mathematical RAG retrieves relevant premises from mathematical libraries or integrates textbooks to support theorem proving and school-level question answering.

V. BENCHMARK

RAG benchmarks evaluate robustness, faithfulness, relevance, task coverage, and domain-specific performance, while current systems face inherent limitations involving noise, overhead, interaction design, complexity, and context length.

  • Benchmarks assess RAG along dimensions including noise robustness, negative rejection, information integration, and counterfactual robustness.
  • RAGAS, ARES, and TruLens evaluate faithfulness, answer relevance, and context relevance using a separate evaluator LLM.
  • CRUD-RAG covers create, read, update, and delete tasks through text continuation, question answering, hallucination correction, and open-domain multi-document summarization.
  • Limitations: RAG systems inherently face retrieval noise, which can introduce irrelevant or misleading content, although its effect on generation quality remains unclear.
  • Limitations: Retrieval and interaction add latency, storage, and access costs, especially with recursive or iterative methods and larger retrieval sources.
  • Limitations: Misaligned retriever and generator objectives, increased tuning complexity, and lengthened contexts complicate practical RAG deployment.

B. Potential Future Directions

The survey identifies future RAG directions spanning augmentation design, flexible pipelines, broader and more efficient applications, dynamic knowledge, combination with complementary techniques, and long-context generation.

  • Novel Design of Augmentation Methodologies: More advanced retriever–generator augmentation foundations could further improve RAG generation results.The survey emphasizes that the practical augmentation process significantly affects final outcomes because retrievers and generators have distinct objectives.
  • Flexible RAG Pipelines: Recursive, adaptive, and iterative pipelines could combine retrieval sources, retrievers, generators, and subsystems for complex tasks.The survey highlights precise tuning and engineering as important for these flexible pipelines.
  • Broader Applications: Domain-specific RAG techniques are needed because some generative tasks remain unexplored and many existing applications overlook domain characteristics.The survey presents domain-aware design as a route toward broader applications.
  • Efficient Deployment and Processing: Efficient deployment remains challenging because retrieval overhead and increasing retriever–generator complexity require further system-level optimization.Plug-and-play solutions exist for query-based RAG with LLMs, but not broadly across other foundations or generation tasks.
  • Incorporating Long-tail and Real-time Knowledge: Future systems should incorporate continuously updated, long-tail, flexible, and personalized information rather than relying only on generators’ training data.The survey notes that knowledge updating and expansion remain relatively underexplored.
  • Combined with Other Techniques: Combining RAG with fine-tuning, reinforcement learning, chain-of-thought, and agent-based generation remains an early-stage research direction.The survey characterizes RAG as orthogonal to these techniques and calls for algorithm designs that exploit their combination.
  • Combined with Other Techniques: Long-context generation may complement rather than replace RAG, whose flexibility supports dynamic, up-to-date, and long-tail knowledge.The survey explicitly argues that long-context models do not eliminate RAG’s role in managing dynamic information.
  • Survey Scope: The survey organizes RAG foundations, enhancements, applications, benchmarks, limitations, and future directions within AIGC.Its stated scope covers augmentation interactions, component- and pipeline-level improvements, and practical use across modalities and tasks.
Loading 2402.19473v6…