A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models

RAG foundations

• RAG retrieves relevant documents from external databases and uses them as context for generation, often with minimal or no additional training [54].The survey describes retrieval as providing faithful, timely external knowledge for generation.Source: §1 · p. 1

Motivation

• Legal-query hallucination rates of 69% to 88% illustrate LLMs’ domain-knowledge limitations, alongside hallucinations and the high computational cost of updating models [21].These limitations are especially notable in fields such as medicine and law.Source: §1 · p. 2

RAG for LLMs

• For out-of-scope queries, external retrieved knowledge can improve LLM responses; examples include scientific-literature QA, molecular discovery [77], and reduced conversational hallucinations.The survey illustrates this with a dialog system that answers correctly after retrieving and integrating relevant external knowledge.Source: §1 · p. 2, Figure 1 · p. 2

Survey scope

• The survey organizes RA-LLM research around retrieval, generation, and augmentation; training techniques; applications; and challenges and future directions.It also reviews the necessity and application frequency of retrieval in RAG.Source: §1 · p. 2
• Unlike related surveys on multimodal RAG [193], RAG for AIGC [192], and RAG for LLMs [41], this survey emphasizes RA-LLM architectures, training paradigms, and application tasks.Source: §1 · p. 2

2.1 Large Language Models (LLMs)

• LLMs are pretrained at billion-parameter scale, adapted to downstream tasks, and broadly categorized as encoder-only, decoder-only, or encoder-decoder models with distinct context-processing and generation roles.BERT exemplifies bidirectional encoding, GPT left-to-right generation, and T5 sequence-to-sequence processing.Source: §2.1 · pp. 2–3

2.2 Prompt Learning

• Prompt learning guides downstream tasks through designed inputs as an alternative to extensive LLM fine-tuning; early methods used manually crafted templates [110].Source: §2.2.1 · p. 3

2.2.1 Prompting Engineering.

• Prompt engineering uses cloze prompts for BERT [20] [110] and prefixes for GPT; soft prompts learn continuous embeddings, but prompts can lack effectiveness guarantees or domain knowledge.Prefix-Tuning trains prepended prefix embeddings, allowing prompts that are not real text.Source: §2.2.1 · p. 3

2.2.2 In-Context Learning (ICL).

• ICL uses in-prompt demonstrations to adapt pretrained LLMs without fine-tuning, but demo quality and missing knowledge can undermine accuracy, motivating RAG [135].GPT-3 demonstrated few-shot capability with carefully selected examples.Source: §2.2.2 · pp. 3–4

• The framework includes retrieval, generation, and augmentation.Source: §3 · p. 4
• A mechanism determines whether retrieval is needed.Source: §3 · p. 4
• The section introduces important techniques involved in each process.Source: §3 · p. 4

3.1.1 Retriever Type.

• Sparse retrieval uses term-based matching and is simple but untrainable, while dense retrieval embeds queries and documents and is generally trainable and more adaptable.Sparse methods such as BM25 depend heavily on database and query quality and are limited to similarity-based criteria; dense systems include bi-encoder designs and retrievers using generation-model components.Source: §3.1.1 · p. 4
• For open-domain QA with InstructGPT, untuned Contriever performs comparably to BM25, while both underperform DPR fine-tuned on target datasets.General-purpose one-encoder retrievers are pre-trained on unaligned documents for transferability; the comparison also indicates the value of target-task fine-tuning.Source: §3.1.1 · p. 4

3.1.2 Retrieval Granularity.

• Chunk retrieval is mainstream because chunks provide compact, relatively complete information with less redundancy, while token retrieval suits rare patterns or out-of-domain data but costs more storage.Document, passage, token, and entity-level units trade off retrieval effectiveness, database space, and search cost; entity or mention retrieval suits entity-centric tasks and uses less space than token retrieval.Source: §3.1.2 · p. 5

3.1.3 Pre-retrieval and Post-retrieval Enhancement.

• Pre-retrieval methods improve queries through LLM-generated pseudo-documents, hypothetical document embeddings, or query rewriting; Rewrite-Retrieve-Read outperforms naive RAG or generation on tested QA datasets.Query2doc expands queries to disambiguate them, while HyDE embeds hypothetical documents as queries; Rewrite-Retrieve-Read uses either a frozen LLM or trainable rewriter, with varied performance across datasets.Source: §3.1.3 · pp. 5–6
• Query augmentation combines the original query with preliminary outputs, and is reported to outperform other query-enhancement strategies by retrieving relevant knowledge collectively during answer generation.The combined query can improve lexical and semantic overlap between supporting documents and the question, clarifying query-output relationships.Source: §3.1.3 · p. 6
• Post-retrieval methods rerank, adapt, filter, or compress retrieved content, while iterative retrieval-and-reasoning can fail when low-quality IR or LLM outputs hinder one another.PRCA adapts a lightweight module using generator rewards, R2G reranks results, and BlendFilter combines query blending with knowledge filtering; RECOMP summarizes documents and FiD compresses encoded passages to address input-length or latency constraints.Source: §3.1.3 · p. 6

3.1.4 Database.

• Closed-source databases store similarity-search keys and retrieved values, whereas Internet search avoids index maintenance, accesses up-to-date and broader knowledge, and is effective for zero-shot knowledge-intensive tasks.Database sources vary by domain, including Wikipedia and task-specific collections; Internet search is used for knowledge augmentation, fact-checking, and agent enhancement, with LLMs serving as readers of search results.Source: §3.1.4 · pp. 6–7

3.2 Generation

• Generator design depends on downstream tasks; decoder-only and encoder-decoder structures dominate text generation, while closed-source foundation models have made black-box generation mainstream.The survey reviews parameter-accessible (white-box) and parameter-inaccessible (black-box) generators.Source: §3.2 · p. 7
• Encoder-decoder generators connect input and target tokens with cross-attention, whereas decoder-only models concatenate them; both types incorporate retrieved data to improve generation accuracy and relevance.Examples include BART in RAG and Re2G, and T5 in FID [54] and EMDR2.Source: §3.2.1 · p. 7

3.2.1 Parameter-Accessible Generators (White-box).

• White-box generators allow parameter optimization and can be trained to adapt to different retrieval and augmentation approaches for better generation performance.Source: §3.2.1 · p. 8
• Black-box generators expose only query-in and response-out operations, so RA-LLMs focus on augmenting prompts with knowledge, guidance, or examples rather than changing model parameters.Rubin et al. train a prompt retriever using language-model-labeled data to provide in-context examples; Xu et al. compress retrieved documents to reduce computation and ease the burden of identifying relevant information in long documents.Source: §3.2.2 · p. 8

3.3.1 Input-Layer Integration.

• Input-layer integration combines retrieved documents with the query, while prompt retrieval can supply demonstrations or prompts to guide LLMs in context.Concatenating all documents can exceed input-length limits; processing documents independently, as in FiD [54], supports more contexts, while In-Context RALM removes tokens from the beginning of the combined input. Prompt retrieval can also help induce reasoning [162]; Cheng et al. [16] learn a prompt retriever from input-prompt pairs labeled by a frozen LLM.Source: §3.3.1 · p. 8
• Output-layer integration combines retrieval and generation results, but its simple linear form can limit reasoning about retrieved text.kNN-LM [62] interpolates language-model and neighbor-induced next-token distributions; a gating network can post-process retrieved data. REFEED [183] refines an initial answer using retrieved information, while COMBO [190] pairs generated and retrieved passages for Fusion-in-Decoder processing.Source: §3.3.2 · p. 8

3.3.2 Output-Layer Integration.

• Intermediate-layer integration inserts modules such as RETRO’s Chunked Cross Attention [6] to incorporate retrieved representations into generator layers.It can efficiently use many retrieved text chunks, including chunks that are difficult to process through input-layer integration, but requires deep access to the generation model and is generally infeasible with inference-API LLMs [98].Source: §3.3.3 · p. 8

3.4 Retrieval Augmentation Necessity and Frequency

• LLMs may double their hallucination rate on non-relevant retrieved passages compared with relevant ones, so RA-LLMs should retrieve only when necessary [147].Indiscriminate augmentation can override knowledge the LLM already possesses and produce incorrect responses [99]; RAG is therefore not a universal solution [75].Source: §3.4 · p. 9
• Necessity methods use preliminary answers, reasoning, special tokens, confidence thresholds, self-knowledge, or proxy-generated heuristic answers to guide retrieval [5] [57] [146].Traditional RAG approaches also assess generation-logit confidence [47] [56] [59]; SlimPLM detects missing knowledge and can support query rewriting [146].Source: §3.4 · p. 9
• Fixed retrieval frequency can be one-time, every-n-token, or every-token, with one-time retrieval suited to needs that are obvious to LLMs [57].One-time retrieval may not support an entire long-form output when token dependencies matter, motivating in-generation retrieval such as every-n-token methods [6] or every-token retrieval [62].Source: §3.4 · p. 9
• More frequent retrieval can improve performance but raises computing cost, making retrieval frequency a trade-off between effectiveness and efficiency.Retrieval frequency determines reliance on retrieved results and affects both model effectiveness and efficiency.Source: §3.4 · p. 9

4 RA-LLMS TRAINING

• Training-free methods insert retrieved text into prompts at inference, avoiding extra training, but may use retrieved knowledge suboptimally because retrievers and generators are not optimized for downstream tasks.Training-based methods fine-tune these components to help LLMs adapt to and integrate retrieved information.Source: §4 · p. 9
• Training-based methods use independent training of each component, sequential training that freezes one component while tuning the other, or joint training of retriever and generator.Figure 5 illustrates these training strategies alongside training-free approaches.Source: §4 · p. 9, Figure 5 · p. 10

4.1 Training-free

• Because frequent fine-tuning is costly, training-free retrieval lets LLMs acquire external knowledge without additional training and has improved performance on knowledge-intensive tasks such as open-domain question answering.The cited passage notes the time and computational resources required to update knowledge stored in model parameters.Source: §4.1 · p. 9
• Training-free methods divide by how retrieved information is used: prompt engineering integrates it into the original prompt, while retrieval-guided token generation uses it to calibrate token generation.The training-free approach categories are illustrated in Figure 5.Source: §4.1 · pp. 9–10, Figure 5 · p. 10

4.1.1 Prompt Engineering-based Methods.

• Prompt-engineering methods combine retrieved texts with the original prompt as context; In-Context RALM keeps LLM parameters unchanged and places retrieved documents before the prompt.These methods refine prompts because LLM generation depends heavily on the input query.Source: §4.1.1 · p. 10
• IRCoT interleaves chain-of-thought generation with retrieval for subsequent reasoning, while GENREAD generates contextual documents from a query before answering from the context and question.IRCoT retrieves more relevant information for later reasoning steps than standard retrieval using only the question as the query.Source: §4.1.1 · p. 10
• SKR [159] selectively calls a retriever after assessing whether internal knowledge can answer a question, while TOC recursively disambiguates ambiguous questions and aggregates them into long-form answers.TOC first retrieves relevant knowledge, then builds a tree of disambiguated questions.Source: §4.1.1 · p. 10

4.1.2 Retrieval-Guided Token Generation Methods.

• Retrieval-guided token methods calibrate model outputs with retrieved information: KNN-KMs [62] interpolates a neighbor distribution with the model distribution, while Rest [49] retrieves tokens for speculative decoding.KNN-KMs retrieves the k most relevant contexts by query distance; Rest replaces the parametric draft model with a non-parametric datastore and retrieves tokens based on current context.Source: §4.1.2 · p. 10

4.2 Independent Training

• Independent training keeps retriever and LLM training separate, enabling offline use of off-the-shelf models without extra training costs.Training the two modules independently involves no interaction between them during training.Source: §4.2 · p. 11, §4.3 · p. 11
• LLMs commonly use negative log-likelihood, while sparse retrievers rank via task-specific metrics and dense retrievers encode queries and documents for inner-product retrieval.DPR [61] uses independent BERT [25] encoders and contrastive learning; CoG trains prefix and phrase encoders and recasts generation as copy-and-paste operations.Source: §4.2 · p. 11

4.3 Sequential Training

• Sequential training fixes one pretrained module while training the other, in Retriever First or LLMs First order, so the trainable module benefits from the fixed one.Existing models such as BERT [25], CLIP, and T5 can be used directly as the fixed module.Source: §4.3 · p. 11

4.3.1 Retriever First.

• Retriever-first methods fix a pretrained retriever and train the LLM to use retrieved knowledge, including RETRO [6], RALMs, and ITER-RTGEN.ITER-RTGEN uses adaptive hybrid retrieval and fine-tunes T5 on target labels and inputs combined with retrieved demonstrations. SMALLCAP uses CLIP for image-text retrieval, trains cross-attention, and generates captions with GPT-2.Source: §4.3.1 · p. 11
• LLM-first approaches use trained readers or LLMs to supervise retriever learning, as DKRR [53] and AAR do with generated signals.RA-DIT first fine-tunes LLMs to use retrieved knowledge, then trains the retriever to align with them; UPRISE [16] uses a frozen LLM to guide prompt-retriever fine-tuning for zero-shot unseen tasks.Source: §4.3.2 · p. 11

4.4 Joint Training

• Joint training optimizes retriever and generator end-to-end; RAG uses negative log-likelihood, while REALM uses MIPS and asynchronously refreshes its search index.MIPS locates relevant documents using embeddings and a search index; asynchronous updates refresh the index every several hundred training steps to avoid re-indexing all documents each time.Source: §4.4 · p. 11

5 APPLICATIONS

• Figure 6 groups RA-LLM applications into NLP tasks, downstream tasks, and domain-specific applications, including QA, chatbots, fact verification, recommendations, software engineering, science, and finance.Source: Figure 6 · p. 12, §5.1 · p. 12

5.1.1 QA Systems.

• RA-LLMs improve QA by retrieving and synthesizing external knowledge; REALM retrieves during training and inference, while Fusion-in-Decoder [54] fuses retrieved passages with questions.Borgeaud et al. [6] report that answer quality may depend more on retrieval output.Source: §5.1.1 · p. 12

5.1.2 ChatBot.

• RA-LLM chatbots retrieve external knowledge from static databases, internet search, and dynamic search engines to support context-rich conversation [43],,.They aim for coherent, natural conversation and use retrieval to augment interactions [60].Source: §5.1.2 · p. 12
• For fact verification, Self-RAG [5] assesses retrieved information’s helpfulness and reliability, improving verification accuracy; earlier work includes retrieval-augmented tasks and few-shot Atlas [55].Source: §5.1.3 · p. 12

5.2.1 Recommendations.

• Recommendation systems use retrieval and generation for personalized, contextually relevant suggestions [163]; CoRAL [163] uses reinforcement learning to align retrieved collaborative and semantic information.Examples retrieve knowledge from movie or book datasets and reviews to enrich item information.Source: §5.2.1 · pp. 12–13

5.2 Downstream Tasks

• In software engineering, RA-LLMs support code generation, program repair, code summarization, tabular data processing, and Text-to-SQL.Parvez et al. retrieve top-ranked code or summaries from a codebase and combine them with the input.Source: §5.2.2 · p. 13

5.3 Domain-specific Applications

• In science, RA-LLMs support molecule generation and property prediction [77], protein representation and generation [144], and clinical QA.Methods retrieve molecule structures and biomedical entities [161]; Liu et al. retrieve textual knowledge for molecular property prediction, and RSA retrieves similar protein sequences.Source: §5.3.1 · p. 13

5.3.1 AI for Science.

• In finance, RA-LLMs retrieve external news and social-media information for sentiment analysis, and financial-report knowledge for QA [85] [178].Yepes et al. [178] use structure-based rather than paragraph-based document chunking to improve output quality.Source: §5.3.2 · p. 13

Trustworthy RA-LLMs

• Manipulation may cause harmful unreliable decisions [23] [200], including in safety-critical scenarios [11] [13] [32] [38] [88], while private databases risk leakage [186]; trustworthy systems should ensure robustness, fairness, explainability, and privacy.Source: §6 · p. 13

Multi-Lingual RA-LLMs

• Multilingual retrieval and generation can draw on diverse linguistic sources, support cross-cultural knowledge sharing, and let speakers of less prevalent languages use English and Chinese corpora for downstream tasks [58].Source: §6 · p. 13

Multi-modal RA-LLMs

• Multimodal RAG adds images, video, and audio to text; visual cues can improve generation precision [51] [199] across healthcare [199], drug discovery, and molecular analysis.Source: §6 · p. 13

Quality of External Knowledge

• Wikipedia's articles vary in reliability and accuracy [61] [199], so filtering low-quality or unreliable external knowledge is needed to prevent misleading generation and improve output reliability.Source: §6 · p. 14

7 CONCLUSION

• RA-LLMs leverage retrieved external knowledge to help address LLM hallucination and outdated internal knowledge.The survey situates this work amid increasing efforts to provide LLMs with current auxiliary information and teach them to use retrieved knowledge.Source: §7 · p. 14
• The survey systematically reviews RA-LLMs across architecture, training strategy, and application areas.It aims to give researchers an in-depth understanding of the field.Source: §7 · p. 14
• The survey discusses current RA-LLM limitations and potential future research directions because the field remains at an early stage.Source: §7 · p. 14