PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Generation for Natural Language Processing: A Survey

Shangyu Wu, Ying Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei-Wei Kuo, Nan Guan, Chun Jason Xue

arXiv:2407.13193v4cs.CL

TL;DR

LLMs face hallucination, costly knowledge updating, and limited domain expertise. This survey systematically reviews RAG components, fusion methods, applications, evaluation, training, and deployment. It organizes retrieval fusion into four categories and identifies benchmark limitations and future directions including security and graph-based retrieval.

  • Problem

    LLMs can hallucinate, require costly retraining or fine-tuning for knowledge updates, and lack domain-specific expertise.

  • Method

    The paper conducts a systematic survey of RAG components, retrieval fusions, NLP applications, evaluation, training paradigms, deployment, and emerging directions.

  • Results

    The survey presents a four-part retrieval-fusion taxonomy—query-based, logits-based, latent, and parametric—and compares methods across accessibility, efficiency, and use cases.

  • Takeaways & Limitations

    RAG is presented as a practical framework for augmenting LLM generation with external knowledge across NLP applications and deployment settings.

  • Takeaways & Limitations

    RAG evaluation remains constrained by static Wikipedia-centric benchmarks and limited fine-grained diagnostic supervision.

Abstract

from arXiv · show

Large language models (LLMs) have achieved strong empirical performance in various fields, benefiting from their huge amount of parameters that store knowledge. However, LLMs still suffer from several key issues, such as hallucination problems, knowledge update issues, and lacking domain-specific expertise. The appearance of retrieval-augmented generation (RAG), which leverages an external knowledge base to augment LLMs, mitigates these limitations. This paper presents a systematic review of RAG techniques for natural language processing (NLP), with a focus on retrievers and retrieval fusions. We introduce a novel taxonomy of retrieval fusions, such as query-based, logits-based, latent, and parametric fusion, and provide structured comparisons across accessibility, efficiency, and use cases. The paper further examines RAG applications across diverse NLP tasks, discusses evaluation methodologies and benchmark limitations, and analyzes training paradigms with and without knowledge base updates. Finally, we explore industrial deployment considerations and identify emerging challenges and future directions, including security, efficiency, and graph-based retrieval.

1 Introduction

LLMs face hallucination, costly knowledge updates, and limited domain expertise, motivating RAG. This survey organizes RAG techniques, applications, evaluation, training, deployment, and emerging directions.

  • Motivation: LLMs can generate fluent but factually incorrect responses, require costly retraining or fine-tuning for knowledge updates, and often lack domain-specific expertise.Building domain-specific LLMs also demands substantial data curation and model adaptation.
  • Motivation: RAG augments LLMs with an external knowledge base to provide relevant information at inference time.The survey describes this as a response to hallucination, knowledge staleness, and domain-expertise limitations.
  • Contributions: The survey proposes a taxonomy of query-based, logits-based, latent, and parametric retrieval fusion, with structured comparisons across practical dimensions.The comparisons include accessibility, efficiency, implementation complexity, and use cases.
  • Scope: It reviews RAG applications across NLP tasks, evaluation methodologies, benchmarks, training with or without knowledge-base updates, and deployment considerations.The paper also identifies security, privacy, and GraphRAG among emerging directions.

2 Overview

An RAG system retrieves relevant knowledge, fuses it with the query or model representations, and generates an answer. Its main components are the retriever, generator, and retrieval-fusion mechanism.

  • System components: RAG retrieves top-k knowledge chunks from an external vectorized document database for a given query.Documents are divided into chunks and transformed into vector embeddings before retrieval.
  • Retriever: The retriever uses an encoder, approximate-nearest-neighbor indexing, and a vector database, balancing retrieval efficiency against retrieval quality.Efficiency concerns search speed, while quality concerns the relevance of retrieved information.
  • Retrieval fusion: Retrieval fusion augments generation through query-based, logits-based, latent, or parametric mechanisms.Parametric fusion selects and integrates relevant LoRA modules without modifying the original model parameters.
  • Workflow: The workflow retrieves relevant information, fuses it with inputs or intermediate states, and generates predictions from the query and retrievals.Generators include proprietary, open-weight, and retrieval-aware model categories.

3 Retriever

Using a retriever involves building it and then querying it, covering preparation of the retrieval system and execution of searches.

  • Retriever stages: The retriever workflow has two stages: building the retriever and querying the retriever.The paper presents these stages as the organizing structure for the retriever discussion.

3.1 Building the Retriever

Building a retriever involves chunking and encoding corpus text, constructing a vector database, and optimizing approximate-nearest-neighbor search. Design choices trade retrieval efficiency against semantic fidelity and depend on the RAG scenario.

  • Construction pipeline: Building the retriever involves chunking the corpus, encoding chunks, constructing an ANN index, and storing key-value data in a vector database.The vector database stores embeddings as keys and domain-specific knowledge as values.
  • Chunking: Chunking aims to create semantically independent units, with size depending on task, encoder, and query preferences.The paper states that no single chunk size is optimal across RAG scenarios.
  • Chunking: Chunking methods include fixed-length, semantic, and content-based splitting.Semantic chunking can use sentence or newline boundaries, while fixed-length chunking uses a length hyperparameter.
  • Encoding: Encoding converts text chunks into embeddings that support similarity search based on content relevance rather than keyword matching.Sparse methods include one-hot, BoW, TF-IDF, and BM25; dense methods include neural encoders such as BERT variants.
  • Encoding: Sparse encoding is efficient but may fail to capture deeper semantic meanings.This is presented as a limitation of sparse representations rather than of encoding in general.
  • Indexing: Vector-database indexing targets efficient ANN search through similarity metrics, dimension reduction, and advanced indexing techniques.Dimension reduction improves efficiency but can harm semantic representations and lose information.
  • Indexing: IVFPQ combines coarse data clustering with fine-grained product quantization to reduce the search space and compress vectors.The method is described as an efficient and scalable ANN indexing framework.

3.2 Querying the Retriever

Querying a pre-built retriever encodes the query, searches an indexed vector database for nearest neighbors, fetches their values, and applies task-specific post-processing such as reranking.

  • Encoding and ANN search: ANN search uses the encoder aligned with the pre-built embedding space and searches indexed vectors for similar data.The search returns corresponding values from the vector database.
  • Index search: Index search returns the top-k nearest-neighbor identifiers by comparing query embeddings with indexed clusters and reordering merged candidates.The IVFPQ example first selects candidate clusters, searches within them, then merges and reorders candidates.
  • Value retrieval: Retrieved values are fetched from the vector database using the nearest-neighbor key identifiers.The vector database maps nearest keys to their corresponding values.
  • Post-processing: Post-processing refines initial retrievals for task objectives, including reranking that reorders information using task-specific criteria.Reranking addresses the gap between task-agnostic retrieval metrics and downstream objectives.
  • Retriever querying: Retriever querying comprises query encoding, approximate nearest-neighbor search, and fetching retrieved knowledge for fusion.These steps depend on encoder, index, and vector-database APIs.

4 Retrieval Fusions

The survey organizes retrieval fusion into query-based, logits-based, latent, and parametric approaches, then compares their access requirements, efficiency, and deployment trade-offs.

  • Taxonomy: Retrieval fusion has four types: query-based, logits-based, latent, and parametric fusion.They differ in where retrieved information enters the generation process.
  • Query-based fusion: Query-based fusion combines retrieved text or features with the query before generation.Text concatenation suits black-box LLM APIs, while feature concatenation encodes retrievals before combining them with input features.
  • Query-based fusion: Query-based text concatenation can truncate inputs because adding many retrievals exceeds maximum sequence length, making prompt design important.Feature concatenation instead merges encoded retrievals and input features before decoding.
  • Logits-based fusion: Logits-based fusion incorporates retrieval-derived logits through ensemble or calibration methods to enhance or calibrate predictions.Ensemble fusion combines retrieval and output logits, while calibration dynamically determines a fusion parameter.
  • Latent fusion: Latent fusion integrates retrieved representations into generator hidden states through attention-based or weighted-addition mechanisms.Attention-based methods use cross-attention, whereas weighted addition combines retrieved representations into hidden states.
  • Latent fusion: RETRO uses cross-attention to integrate retrievals and matches major-model performance with a 2 trillion token database and 25 times fewer parameters.The survey presents this as evidence for retrieval-enhanced scaling and efficiency.
  • Parametric fusion: Parametric fusion injects document-specific lightweight representations, such as LoRA updates, into generator parameters rather than hidden states.Documents are converted offline into parameter-efficient knowledge modules.
  • Deployment trade-offs: Only query-based text and logits-based fusion support black-box LLMs; latent and parametric methods require white-box access and usually fine-tuning or pre-training.The access distinction follows whether a method needs architectural modification or parameter injection.

5 Generators

RAG generators must ingest retrieved information, use it rather than relying solely on parametric memory, and ideally ground outputs with verifiable attribution. Their architecture and training shape information utilization, grounding quality, and system-level efficiency.

  • Generator requirements: RAG generators map queries and retrieved information to output sequences while handling longer, noisier contexts than standalone LLMs.They should use retrieved information instead of relying on parametric memory and ideally provide verifiable attribution.
  • Information capacity and context: Query-based fusion requires large context windows and efficient attention or KV-cache use, but long context alone does not guarantee robust information use.Document-level marginalization, retrieval filtering, and mixed-quality retrieval training can reduce over-reliance on individual passages.
  • Deployment constraints: Proprietary API models generally constrain RAG to prompt-time fusion because their architectures cannot be modified for cross-attention or retrieval-aware pretraining.Prompt formatting and multi-call orchestration remain available optimization strategies.
  • Performance implications: Cross-attention or separate information encoding tends to use retrieved information more systematically than raw concatenation.Architecture also affects scaling with top-k, long-context reliability, and grounding behavior.
  • Selection guidelines: Generator selection should match deployment constraints and goals such as controllable grounding, citations, domain compliance, and inaccessible generator weights.Overall performance depends on how generator architecture and training interface with retrieval, not only on general LLM strength.

6 NLP Tasks

The survey reviews RAG techniques across language modeling, machine translation, summarization, question answering, information extraction, and text classification. Across these tasks, retrieval can help, but its benefits depend on retrieval quality, task-compatible evidence, and appropriate fusion.

  • Language modeling: RAG language modeling modifies generators or retrieval inputs to incorporate retrieved knowledge during next-token prediction.Some approaches add cross-attention modules to Transformer blocks so similar prefixes and retrieved continuations inform predictions.
  • Language modeling: Language-modeling performance is highly sensitive to retrieval quality because perplexity evaluates every token and retrieval noise can be amplified.Failure modes include copying mismatched neighbors and shifting probability mass toward incorrect tokens.
  • Machine translation: RAG for machine translation retrieves similar translations or examples for input concatenation or output-logit fusion.Retrieved examples must support adequacy, fluency, terminology consistency, and target-side stylistic compatibility.
  • Machine translation: Machine-translation retrieval can cause terminology drift, entity or number errors, and reduced adequacy when examples are mismatched or over-copied.Sentence-level retrieval may not guarantee document-level consistency for domain terms and named entities.
  • Text summarization: RAG summarization uses retrieved summaries or documents through input concatenation, intermediate cross-attention, or output-layer fusion.The central challenge is balancing compression and grounding under limited context while preserving abstraction rather than encouraging copying.
  • Question answering: Open-domain QA commonly retrieves knowledge or question-answer demonstrations before generators produce answers from the retrieved context.For reference-document QA, retrievers select relevant documents that generators read before answering.
  • Information extraction: Information extraction requires structured, label-compatible retrievals because simple concatenation does not fully convey relational or type information.Mismatched schemas can cause injection failure, schema drift, incorrect span boundaries, and erroneous role assignments.
  • Text classification: Text-classification retrieval can mislead predictions when similar documents have different labels, especially under domain shift.Nearest-neighbor confusion and demonstration ordering can produce systematic bias or prediction instability.

7 RAG Evaluation and Benchmark

RAG evaluation must assess both retrieval and generation, while also measuring their integrated behavior. Existing benchmarks remain limited in diagnostic attribution, operational realism, modality coverage, and temporal freshness.

  • Evaluation Framework: RAG evaluation must separate retrieval quality from generation faithfulness while assessing end-to-end system efficacy.Retrieval metrics assess context relevance, while generation metrics assess groundedness, relevance, and correctness.
  • Evaluation Framework: Evaluation also covers robustness to noise, insufficient evidence, multi-source integration, and counterfactual information.These dimensions extend evaluation beyond basic answer quality.
  • Evaluation Challenges: Faithfulness and relevance require application-specific trade-offs: legal and medical tasks prioritize source fidelity, whereas creative tasks may favor usefulness.The preferred balance depends on the task’s requirements.
  • Evaluation Challenges: Retriever–generator error attribution is difficult because failures may arise from retrieval or interpretation, often requiring costly fine-grained annotations.Without attribution, targeted retrieval, grounding, or decoding interventions are harder to design.
  • Evaluation Challenges: Lexical-overlap metrics often correlate weakly with human judgment for open-ended and long-form RAG, motivating model-based diagnostic evaluation.LLM-as-a-judge approaches have shown better alignment with human preferences on open-ended tasks.
  • Benchmark Limitations: Wikipedia-centric benchmarks weakly represent production RAG over dynamic, proprietary, and domain-specific knowledge bases.Domain-focused datasets broaden coverage, but benchmark diversity remains limited.
  • Benchmark Limitations: Current benchmarks underrepresent fine-grained error attribution, efficiency and cost, multimodal retrieval, and temporal knowledge freshness.These gaps limit reliable system comparison and principled iteration.
  • Future Directions: The field is shifting toward multidimensional, module-aware evaluation emphasizing grounding, robustness, and diagnostic usefulness.Future suites should incorporate realistic serving constraints, multimodal and structured retrieval, and temporal freshness.

8 RAG Training and KB Update

RAG training is organized around whether the knowledge base remains fixed or is updated. Training may target the retriever, generator, or both, with joint optimization improving coordination but introducing computational and stability challenges.

  • Training Paradigms: RAG training distinguishes fixed-knowledge-base settings from settings that update the knowledge base before model training.The two classes differ in whether knowledge-base contents change during training.
  • Training Without KB Updates: Without knowledge-base updates, training can target the retriever, generator, or both to adapt short-term generator memory.The retriever, generator, and joint cases are the main configurations.
  • Training Without KB Updates: Retriever training updates the encoder and generally requires rebuilding indexes because vector-database embeddings change.Training goals include better semantic or domain-specific representations and faster encoding.
  • Joint Training: Joint training faces computational costs, retrieval collapse, feedback-loop instability, and sparse gradients from discrete retrieval.Large-scale retrieval and marginalization over latent documents are central sources of complexity.
  • Joint Training: Joint retriever–generator training can improve coordination and contextual understanding through end-to-end optimization.REALM, RAG, and Atlas treat retrieval and generation as jointly optimized components, with different index-refresh strategies.
  • Training With KB Updates: With knowledge-base updates, the process first updates embeddings, values, or the corpus, then optionally trains the retriever and generator.New values require in-place updates, while new corpus entries require insertion and index rebuilding or updating.

9 Applications

RAG applications span agent memory, up-to-date tool use, development frameworks, and production deployment. The survey emphasizes modular frameworks and the practical constraints of quality, scale, latency, cost, security, and evaluation.

  • Agent Applications: LLM agents use RAG to retrieve relevant items from external memory and integrate them into generation for understanding and decision-making.The external memory functions as a knowledge base for the agent.
  • Agent Applications: Tools and governed enterprise interfaces let agents retrieve fresh information for tasks requiring up-to-date knowledge while preserving permissions and compliance.Examples include market and policy updates and enterprise search connectors.
  • Frameworks and Libraries: LangChain and LlamaIndex pioneered modular composition of retrieval, augmentation, and generation components, while later frameworks target production and programmable optimization.The ecosystem spans different architectural preferences and language stacks.
  • Frameworks and Libraries: These tools shift RAG development from bespoke implementations toward standardized, reusable, and often language-agnostic paradigms.
  • Industrial Deployment: Production RAG must balance retrieval quality, scalability, token and context budgets, latency, security, governance, and evaluation.The survey treats these as multidimensional deployment constraints.
  • Industrial Deployment: Large-scale deployments rely on efficient vector indexing and approximate nearest-neighbor search, including HNSW and GPU-accelerated similarity search.Industry systems also provide operational features such as endpoints, governance, and budget policies.

10 Discussion and Future Directions

The survey identifies security, privacy, retrieval quality, and semantic drift as major RAG challenges. It highlights that external knowledge bases create new attack surfaces and that iterative retrieval can depart from the original information need.

  • Security and Privacy: RAG improves factuality through external grounding but adds the retrieval database as a trust boundary and attack surface.Knowledge-base manipulation or contamination can steer downstream generation.
  • Security and Privacy: Embeddings can leak or permit recovery of original sentences, challenging the assumption that vector representations are inherently privacy-safe.The survey also notes risks in multi-tenant vector databases.
  • Security and Privacy: Adaptive queries can induce RAG systems to reveal private or proprietary sentences originating in the knowledge base.Black-box extraction frameworks target knowledge-derived information through the retrieval-generation pipeline.
  • Security and Privacy: Knowledge poisoning and prompt injection can target the knowledge base, retriever, or retrieval-generation interaction.Defenses span ingestion-time provenance and access control, retrieval-time anomaly detection, and generation-time verification.
  • Retrieval Quality: Imperfect retrieval can degrade generation quality and even induce hallucinations when relevant passages are missing, contradictory, or noisy.Retrieval quality depends on factors including key representation and embedding models.
  • Retrieval Quality: Multi-hop and iterative retrieval can improve coverage but risk semantic drift as errors or off-topic tangents compound across rounds.Each retrieval step depends on the previous step’s output, allowing deviation from the original information need.

10.3 RAG Efficiency

RAG efficiency is constrained by retrieval volume, computation, and cost, with bottlenecks varying by database size. The survey also presents retrieval as complementary to long-context LLMs: retrieval supplies precision while long context supplies breadth.

  • Efficiency Challenges: Reducing retrieved data can affect quality, whereas adding computing and memory resources increases cost.These are two straightforward ways to improve efficiency without new algorithms.
  • Retriever Efficiency: Retriever efficiency consists of encoding, approximate nearest-neighbor search, and knowledge-base data-fetching time.The bottleneck differs with database size, so joint optimization is unnecessary.
  • Retriever Efficiency: For larger databases, index search and data fetching become the major bottlenecks because search covers more data and fetching incurs I/O overhead.Efficient ANN algorithms and system-level optimizations are therefore central.
  • Fusion Efficiency: Query-based fusion can impose substantial inference overhead through long sequence lengths, motivating methods that reduce integration computations.Fid-light and ReFusion are cited as examples.
  • RAG and Long Context: RAG does not fully replace long-context LLMs because it remains advantageous for numerical reasoning, smaller models, precision, scattered facts, resource efficiency, and cost.Long-context systems can obscure key facts in verbose contexts and incur higher computational overhead as input length grows.
  • RAG and Long Context: Retrieval can complement long-context LLMs by selecting relevant information as a context compressor, reducing computation and mitigating the lost-in-the-middle issue.The survey characterizes long context as providing breadth and retrieval as providing precision and verifiability.

10.6 RAG Training

RAG training must address joint optimization without knowledge-base updates and representation alignment when the knowledge base changes. The survey also discusses cross-modality retrieval and graph-based retrieval as directions for richer representations and structured access.

  • Training Without Updates: Without knowledge-base updates, RAG training must jointly optimize retriever and generator parameters through multi-objective losses, efficient tuning, or other strategies.
  • Training With Updates: With knowledge-base updates, retrieval representations must remain aligned with generator representations while update operations add time cost.Asynchronous updating can reduce update frequency while maintaining alignment.
  • Cross-Modality Retrieval: Adding images, videos, or audio to text can provide richer context and clarify meanings that are difficult to convey through text alone.Cross-modality information can improve representation quality and support more natural interaction.
  • Graph-Based Retrieval: Graph-based retrieval transforms documents into entity-relation representations and retrieves over graph neighborhoods or communities.GraphRAG constructs an entity knowledge graph and pre-generates summaries for related-entity communities.

11 Conclusion

The survey synthesizes RAG for NLP, covering its components, applications, evaluation, training, deployment, and future directions.

  • The survey reviews RAG’s retriever, generator, and retrieval-fusion components.
  • It proposes a taxonomy of fusion methods with comparisons across accessibility, efficiency, and use cases.
  • The survey examines RAG applications across NLP tasks and discusses evaluation methodologies and benchmark limitations.
  • It analyzes training paradigms with and without knowledge-base updates.
  • It identifies security and graph-based retrieval among emerging challenges and future directions.
Loading 2407.13193v4…