PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Retrieval-Augmented Text Generation for Large Language Models

Yizheng Huang, Jimmy Huang

arXiv:2404.10981v2cs.IRcs.AIcs.CL

TL;DR

LLMs need mechanisms to incorporate current external information and reduce plausible but inaccurate responses despite static training data. This survey organizes RAG from an information-retrieval perspective, reviews its evolution, techniques, and evaluation, and identifies challenges and future directions. It provides a unified four-phase framework and concludes that retrieval quality and systematic evaluation are central to advancing accurate retrieval and generation.

  • Problem

    LLMs cannot readily incorporate post-training information and can generate convincing but inaccurate responses, motivating a need for current external data.

  • Method

    The survey develops an information-retrieval-oriented framework covering RAG’s pre-retrieval, retrieval, post-retrieval, and generation phases, techniques, and evaluation methods.

  • Results

    The survey finds that RAG research predominantly uses external data, iterative multi-hop retrieval, dense retrieval, query manipulation, and retrieval-phase optimization, while customization receives comparatively less attention.

  • Takeaways & Limitations

    The framework consolidates RAG concepts and supports further research on accurate retrieval and generation, especially in textual applications.

  • Takeaways & Limitations

    Retrieved irrelevant or misleading documents can introduce noise that causes hallucinations or unreliable answers, although some conditions may make irrelevant documents beneficial.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) merges retrieval methods with deep learning advancements to address the static limitations of large language models (LLMs) by enabling the dynamic integration of up-to-date external information. This methodology, focusing primarily on the text domain, provides a cost-effective solution to the generation of plausible but possibly incorrect responses by LLMs, thereby enhancing the accuracy and reliability of their outputs through the use of real-world data. As RAG grows in complexity and incorporates multiple concepts that can influence its performance, this paper organizes the RAG paradigm into four categories: pre-retrieval, retrieval, post-retrieval, and generation, offering a detailed perspective from the retrieval viewpoint. It outlines RAG's evolution and discusses the field's progression through the analysis of significant studies. Additionally, the paper introduces evaluation methods for RAG, addressing the challenges faced and proposing future research directions. By offering an organized framework and categorization, the study aims to consolidate existing research on RAG, clarify its technological underpinnings, and highlight its potential to broaden the adaptability and applications of LLMs.

1 Introduction

LLMs face limitations from static training data, including weak specialization, difficulty staying current, and hallucinations. The survey presents RAG as a structured response and organizes prior work to clarify its mechanisms and development.

  • Motivation: LLMs struggle with specialized tasks, rapidly changing information, and convincing but inaccurate responses because their training data is limited and static.These issues can restrict effective use across domains and mislead users.
  • Motivation: RAG supplements an LLM with externally retrieved data to produce more accurate and current outputs beyond the model’s original training scope.The introduction describes retrieval as a response to both outdated knowledge and hallucinations.
  • Survey scope: The survey addresses a literature gap involving fragmented research and inconsistent terminology surrounding RAG mechanisms and subsequent progress.It aims to provide a more comprehensive analysis of the field.
  • Survey scope: The paper organizes RAG methods into pre-retrieval, retrieval, post-retrieval, and generation phases and examines related evaluation methods, challenges, and future directions.The survey also reviews representative studies and retrievers and generators used in RAG research.

2 RAG Framework

RAG evolved from supplementing language models with retrieved information to supporting repeated interactions between retrieval and generation. Its central motivation is to provide current external information more cost-effectively than repeated model retraining or fine-tuning.

  • RAG evolution: RAG supplements an LLM’s training data with current information retrieved from external sources to address inaccurate responses caused by outdated knowledge.The retrieval model dynamically supplies information during generation.
  • RAG evolution: RAG is presented as a more cost-effective alternative to extensive training and fine-tuning for incorporating fresh information.Traditional retrieval methods can provide dynamic access to external data without rebuilding the model.
  • RAG evolution: RAG has evolved from providing supplementary information to enabling multiple retrieval-generation interactions that iteratively refine retrieved content and output quality.The survey describes this as a progression toward repeated retrieval and generation rounds.

2.1 Basic RAG Workflow

The basic RAG workflow indexes external sources, retrieves and ranks documents for a query, and passes the retrieved information with the query to an LLM for generation. Indexing and semantic retrieval methods help align the search process with relevant information.

  • Workflow: RAG begins by indexing external sources, then uses a retriever to find relevant information for a specific query.The index provides the searchable basis for retrieval.
  • Workflow: A generator combines the retrieved information with the query to produce the desired output.The generation stage uses retrieved context rather than relying only on the query.
  • Indexing: Indexing preprocesses and segments text to support focused searches that identify passages containing relevant keywords.Preparation can include normalization and organizing text into sentences or paragraphs.
  • Retrieval: Modern retrievers use pretrained language models to capture query semantics, improving ranking beyond term-frequency-based methods such as BM25.Semantic representations help account for synonyms and phrase structure.

2.2 RAG Paradigm

The survey frames RAG as a four-phase paradigm: pre-retrieval, retrieval, post-retrieval, and generation. These phases prepare data and queries, select and refine documents, and integrate retrieved information into tailored responses.

  • 2.2 RAG Paradigm: RAG research is organized into pre-retrieval, retrieval, post-retrieval, and generation phases from an information-retrieval perspective.Both single-hop and iterative multi-hop retrieval follow this four-phase structure.
  • 2.2.1 Pre-Retrieval: Pre-retrieval prepares indexes, queries, and data through indexing, query manipulation, and data modification.Query manipulation can reformulate, expand, or normalize queries, while data modification removes redundancy or adds metadata.
  • 2.2.2 Retrieval: Retrieval searches indexed data and ranks candidate documents according to their relevance to the user’s query.The phase combines search with initial prioritization of matching documents.
  • 2.2.3 Post-Retrieval: Post-retrieval re-ranks and filters retrieved documents to improve the document set used for final generation.Re-ranking reassesses candidates, while filtering removes documents below quality or relevance criteria.
  • 2.2.4 Generation: Generation enhances retrieved information by merging it with the query, verifying content, and adapting responses to user context and preferences.The described enhancement process emphasizes coherence, clarity, relevance, and stylistic quality.

3 Pre-Retrieval

Pre-retrieval prepares queries and documents before search by organizing indexes, manipulating queries, and modifying available information. The section emphasizes dense-vector indexing and query or document enrichment as ways to improve semantic retrieval.

  • Indexing: Dense-vector indexing addresses the inability of traditional indexes to retrieve semantically related documents lacking exact query terms.Embeddings capture semantic meaning, while approximate nearest neighbor search accelerates retrieval with a slight reduction in accuracy.
  • Indexing: Graph indexes speed retrieval by limiting distance computations to a local subgraph.The graph structure narrows the nodes examined during retrieval.
  • Indexing: Product quantization accelerates large-scale search by segmenting vectors and quantizing each part.PQ reduces the time spent calculating vector distances rather than primarily reducing the number of vectors examined.
  • Indexing: Locality-sensitive hashing places similar vectors into the same or nearby buckets to retrieve approximate neighbors efficiently.Hashing reduces the dimensionality of the search problem but may introduce trade-offs.
  • Query Manipulation: Query manipulation refines user queries to address ambiguity, vocabulary mismatch, and user-intent differences.The survey discusses query expansion, query reformulation, and prompt-based rewriting as distinct approaches.
  • Document Modification: Document modification combines internal augmentation with external enrichment to improve the context available for retrieval and generation.Internal methods use existing document or model information, whereas external methods add facts or knowledge from outside sources.

4 Retrieval

Retrieval in RAG searches and ranks information using strategies that range from linear single-pass workflows to iterative, recursive, conditional, and adaptive procedures. These strategies increasingly support multi-hop retrieval and dynamic interaction between retrieval and generation.

  • Search and Ranking: Dynamic retrieval methods connect search with generation by using reasoning, confidence, or retrieved-document assessments to guide subsequent actions.IRCOT interleaves retrieval with reasoning, FLARE triggers retrieval for low-confidence tokens, and retrieval evaluators can act on confidence thresholds.
  • Retrieval Strategies: RAG retrieval commonly uses multi-hop searches rather than only single-hop retrieval, with multiple retrieval-generation interactions.The survey distinguishes single-hop searches from approaches that retrieve repeatedly as generation proceeds.
  • Retrieval Strategies: Basic retrieval follows a linear sequence through pre-retrieval, retrieval, post-retrieval, and generation without iterative or conditional modifications.Atlas and REPLUG exemplify this straightforward strategy.
  • Retrieval Strategies: Iterative retrieval performs multiple retrieval steps informed by previous results and can interleave retrieval with chain-of-thought reasoning.IRCOT is presented as an example for multi-step problem-solving scenarios.
  • Retrieval Strategies: Recursive retrieval creates a hierarchy or tree of retrievals by breaking complex queries into simpler sub-queries.This strategy is suited to hierarchical or layered information.
  • Retrieval Strategies: Conditional retrieval applies predefined or dynamically determined rules so retrieval satisfies specific constraints or criteria.The survey identifies compliance checking, rule-based recommendation, and context-sensitive retrieval as relevant uses.
  • Retrieval Strategies: Adaptive retrieval changes its strategy on the fly according to the query context or information retrieved so far.AAR is cited as an example that adjusts retrieval based on LLM preferences.

5 Post-Retrieval

Post-retrieval improves the retrieved set before generation through filtering and re-ranking. Filtering removes irrelevant material, while re-ranking orders retained documents by relevance or usefulness.

  • Re-ranking: Unsupervised re-rankers rank documents without labeled training data, including zero-shot language-model scoring of BM25 results.In-Context RALM uses an off-the-shelf model to select among top-k retrieved documents.
  • Re-ranking: Supervised re-rankers fine-tune models on ranking datasets using query-document scoring, ranking as generation, or prompt-based relevance estimation.The survey gives BERT, T5, and RankLLaMA as examples of these designs.
  • Re-ranking: Data augmentation for re-ranking generates additional training examples, such as pseudo-relevance labels, to diversify supervision.DKS-RAC is cited as an approach that incorporates additional knowledge into retrieval improvement.
  • Filtering and Re-ranking: Filtering removes irrelevant or low-quality documents, whereas re-ranking orders the remaining documents by relevance or utility.The two processes reduce the set or prioritize its contents for later generation.
  • Filtering: Filtering methods include model reflection, selective summarization, token-level attention filtering, self-consistency, and sentence-level relevance measures.Examples include Self-RAG, RECOMP, FiD-TF, CoK, and FILCO.

6 Generation

Generation-stage enhancement integrates retrieved information with queries, ensembles, feedback loops, and personalization. These approaches aim to keep outputs aligned with user intent while enriching, refining, or adapting the generated response.

  • Enhancement Methods: Query-based enhancement combines the original query with retrieved documents so generation remains aligned with user intent.RETRO injects relevant retrieved chunks through chunked cross-attention during generation.
  • Enhancement Methods: Ensemble enhancement synthesizes multiple sources to reconcile conflicting details and combine diverse perspectives.The approach avoids relying on a single retrieved source.
  • Enhancement Methods: Feedback enhancement iteratively evaluates and adjusts draft responses using mechanisms such as self-reflection or predefined criteria.The survey states that these loops can produce outputs that become more accurate, coherent, and contextually grounded.
  • Customization: Customization adapts generated content through content alignment or contextual adaptation to user preferences, profiles, and needs.The survey distinguishes tailoring knowledge from tailoring responses to audience or situational context.
  • Customization: LAPDOG aligns persona profiles with external stories and dialogue history to enrich context for personalized generation.Its story retriever expands limited persona profiles with relevant narratives.
  • Customization: PersonaRAG and ERAGent adapt responses using dynamic user information, rewritten questions, filtered knowledge, and stated preferences.These systems incorporate session behavior, feedback, or user-specific profiles during response generation.
  • Customization: ROPG selects retrieval strategies before and after generation according to input context and user preferences.Candidate strategies include recency-based, keyword, and semantic retrieval.

7 Evaluation in RAG

RAG evaluation spans downstream answer quality, retrieval relevance, generated-text quality, and robustness to misinformation. Beyond conventional metrics, eRAG evaluates retrieved documents individually and shows stronger correlation with downstream RAG performance.

  • Downstream evaluation: Exact Match and F1 are widely used across dialogue-oriented RAG evaluations and datasets including TriviaQA, HotpotQA, FEVER, NQ, WoW, and T-REX.These metrics benchmark retrieval and generation components in knowledge-intensive tasks.
  • Evaluation dimensions: RAG evaluation considers generated-text quality, retrieved-document relevance, and resilience to misinformation rather than downstream task performance alone.Proposed frameworks assess noise robustness, negative prompting, information integration, and counterfactual robustness.
  • Retrieval-based evaluation: Traditional retrieval metrics include MAP, Precision, Reciprocal Rank, and NDCG for measuring how well retrieved documents match a query.In RAG, retrieval evaluation additionally considers how retrieved information contributes to generated answers.
  • Retrieval-based evaluation: eRAG feeds each retrieved document separately to the LLM with the query, scores the generated output using downstream metrics, and aggregates document scores with ranking metrics such as MAP.This document-level procedure evaluates how individual retrieved items support the final task output.
  • Retrieval-based evaluation: eRAG correlates more strongly with downstream RAG performance than human annotations or provenance labels, regardless of the retrieval model or document count.The framework therefore treats the LLM as the primary consumer and evaluator of retrieved results.
  • Generation-based evaluation: BLEU, ROUGE-L, EM, and F1 assess linguistic quality, reference overlap, complete correctness, and the balance between precision and recall.Specialized criteria such as perplexity, entropy, Misleading Rate, and Mistake Reappearance Rate address dialogue naturalness and misinformation.

8 Comparisons of RAG

The survey finds that RAG research emphasizes retrieval, with performance shaped by corpus-retriever alignment and retrieval quality. Generator scale alone does not consistently determine gains, while better retrieval can make smaller models competitive.

  • Comprehensive Summary: Most surveyed RAG studies emphasize optimizing retrieval, especially through external data, multiple-hop search, dense retrieval, and query manipulation.The survey identifies comparatively little customization at the generation stage.
  • Retriever and Generator: Retriever and generator selections jointly affect RAG systems, but traditional retrievers such as BM25 remain common while T5 remains a prevalent generator.The survey notes that IR-focused LLMs are relatively scarce among retrievers.
  • Retriever and Generator: Retriever choice should match the target corpus: domain-specific models improve specialized retrieval, while general-purpose models perform better on matching general-domain data.Combining retrievers does not always improve results; excluding SPECTER from RRF-2 on Wikipedia outperformed RRF-4.
  • Retriever and Generator: Higher retrieval quality significantly improves LLM performance across models, with SPLADEv3 reranked by DeBERTa-v3 achieving the best results across datasets and metrics.The eRAG analysis compares BM25, RetroMAE, and SPLADEv3 on NQ.
  • Retriever and Generator: Retrieval gains do not follow a straightforward model-size relationship: LLaMA2-7B with retrieval outperforms LLaMA2-70B in a closed-book setting.The eRAG results also indicate that T5-small versus T5-base does not significantly alter the correlation between retrieval effectiveness and downstream performance.

9 Challenges and Future Directions

RAG systems face challenges in retrieval quality, efficiency, and multimodal integration as workflows and applications become more complex. Future directions emphasize noise-aware retrieval, scalable processing, modular optimization, and stronger cross-modal capabilities.

  • Retrieval Quality: Retrieval quality is fundamental, but noise, irrelevant documents, and fragmented information can compromise generated content.
  • Retrieval Quality: Irrelevant documents can cause hallucinations, although their inclusion may improve accuracy under certain conditions.This finding challenges conventional retrieval strategies and motivates approaches that strategically integrate noise.
  • Retrieval Quality: HyDE improves retrieval for poorly expressed or context-limited queries by generating pseudo-documents, but increases computational cost.
  • Retrieval Quality: Information integration remains difficult when documents are fragmented or conflicting, while LLM-dependent post-retrieval methods impose significant costs.Entity-level retrieval, re-ranking, and knowledge distillation are identified as ways to improve cohesion and scalability.
  • System Efficiency: Multi-step workflows and large document collections create latency and computational bottlenecks across classification, retrieval, re-ranking, and generation.Lightweight, hybrid, differentiable, and pruning-based methods are discussed as potential efficiency improvements.
  • System Efficiency: Modular optimization of chunking, embeddings, re-ranking, and hybrid search can improve throughput while balancing retrieval precision and speed.
  • Multimodal RAG: Multimodal RAG must align diverse data types and generate coherent responses across text, images, audio, and potentially video.Recent systems show potential in visual question answering, image captioning, and text-to-audio generation, but scaling remains a future focus.

10 Conclusions

The survey presents a comprehensive retrieval-oriented framework for understanding RAG, organizing methods and evaluation approaches while identifying areas for improvement and future research. It aims to support further work on accurate information retrieval and generation, especially in textual applications.

  • The paper provides a comprehensive framework that categorizes RAG methods and analyzes core technologies and evaluation methods.
  • The survey identifies areas for improvement and future research, particularly for advancing accurate retrieval and generation in textual applications.
Loading 2404.10981v2…