Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

1 Introduction

• Parametric language models resist memory revision, lack straightforward prediction insight, and can hallucinate [38]; retrieval-based memory allows knowledge revision and inspection.REALM and ORQA [31] had explored hybrid memory only for open-domain extractive question answering; RAG extends it to seq2seq models.Source: §1 · pp. 1–2
• RAG jointly fine-tunes DPR [26] and BART [32], combining a dense Wikipedia index with a seq2seq generator conditioned on retrieved documents and input.Maximum Inner Product Search retrieves top-K documents, whose predictions are marginalized either per output sequence or per token. Both memory components and the access mechanism are pre-trained, enabling knowledge access before additional training.Source: §1 · p. 2, Figure 1 · p. 2
• RAG achieves state-of-the-art results on Natural Questions, WebQuestions, and CuratedTrec, outperforming specialized-pretraining approaches on TriviaQA [24].Despite these tasks being extractive, unconstrained generation outperforms previous extractive approaches.Source: §1 · p. 2
• RAG generates more factual, specific, and diverse responses than BART in experiments on MS-MARCO [1] and Jeopardy question generation.These experiments extend the evaluation beyond extractive question answering to knowledge-intensive generation.Source: §1 · p. 2
• 4.3% is the reported gap to state-of-the-art pipeline models on FEVER fact verification, where those pipelines use strong retrieval supervision.RAG achieves results within this gap.Source: §1 · p. 2
• Replacing RAG’s non-parametric memory updates the model’s knowledge as the world changes.The paper demonstrates knowledge updating through memory replacement.Source: §1 · p. 2

2.1 Models

• RAG-Sequence marginalizes complete-sequence probabilities over top-K retrieved documents, using the same latent document to condition every token in a sequence.The retriever supplies the documents, and the generator computes an output-sequence probability for each before marginalization.Source: §2.1 · p. 3
• RAG-Token marginalizes document-conditioned next-token distributions at each generation step, allowing an answer to draw content from multiple retrieved documents.For sequence classification with a one-token target, RAG-Token and RAG-Sequence are equivalent.Source: §2.1 · p. 3

2.2 Retriever: DPR

• DPR [26] retrieves top-K passages using BERTBASE [8] query and document encoders, with the document index serving as non-parametric memory.The bi-encoder is initialized from DPR trained to retrieve answer-containing documents for TriviaQA [24] and Natural Questions. Retrieval uses Maximum Inner Product Search, approximately solvable in sub-linear time [23].Source: §2.2 · p. 3

2.3 Generator: BART

• 400M parameters make up BART-large [32], RAG's parametric memory, which generates from concatenated input and retrieved passage text.BART was pre-trained using a denoising objective and multiple noising functions; other encoder-decoder generators could also be used.Source: §2.3 · p. 3

2.4 Training

• RAG jointly fine-tunes the query encoder and generator without document supervision, minimizing negative marginal log-likelihood with Adam [28] while fixing the document encoder and index.Keeping the document encoder fixed avoids costly periodic index updates.Source: §2.4 · pp. 3–4

2.5 Decoding

• RAG-Sequence uses document-specific beam searches; Thorough Decoding scores missing document–hypothesis pairs, whereas Fast Decoding approximates their generator probabilities as zero.Thorough Decoding weights hypothesis probabilities by retrieval probabilities and sums across documents, potentially requiring many forward passes for long outputs. RAG-Token instead behaves as a standard autoregressive seq2seq generator.Source: §2.5 · p. 4

3 Experiments

• 21M disjoint 100-word Wikipedia chunks form the shared non-parametric knowledge source, using the December 2018 dump following Lee et al. [31] and Karpukhin et al. [26].Document-encoder embeddings populate one FAISS [23] maximum-inner-product-search index with a Hierarchical Navigable Small World approximation [37]. Training retrieves the top k documents with k ∈ {5, 10}; development data determines test-time k.Source: §3 · p. 4

3.1 Open-domain Question Answering

• Open-domain QA evaluates Exact Match (EM), comparing RAG against extractive retrieval [5] [7] [31] [26] and parametric-only Closed-Book QA [52].Answer negative log-likelihood trains RAG on Natural Questions (NQ), TriviaQA (TQA) [24], WebQuestions (WQ), and CuratedTrec (CT). Following DPR [26], CT and WQ initialize from NQ RAG; evaluation uses prior splits [31] [26] and additionally tests TQA Wiki for comparison with T5 [52].Source: §3.1 · p. 4

3.2 Abstractive Question Answering

• MSMARCO NLG v2.1 becomes an open-domain abstractive QA test by using only questions and answers, excluding its supplied gold passages.Some reference answers require the gold passages, so excluding them is expected to lower performance. Some questions also exceed Wikipedia’s coverage, requiring RAG to rely on parametric knowledge for reasonable responses.Source: §3.2 · pp. 4–5

3.3 Jeopardy Question Generation

• Jeopardy generation tests whether RAG can produce precise, factual questions conditioned on answer entities, with BART comparisons assessing generation quality, factuality, and specificity.SearchQA [10] supplies 100K training, 14K development, and 27K test examples; evaluation uses SQuAD-tuned, entity-weighted Q-BLEU-1 [42], which correlates better with human judgments than standard metrics. Pairwise human evaluations [34] assess corroboration by trusted sources and input–output mutual dependence [33], allowing evaluators to prefer RAG, BART, both, or neither.Source: §3.3 · p. 5

3.4 Fact Verification

• FEVER tests RAG’s evidence retrieval and classification of claims as supported, refuted, or unverifiable from Wikipedia, without retrieved-evidence supervision.Class labels become single output tokens, trained directly from claim–class pairs. Label accuracy evaluates standard three-way classification and the two-way supports/refutes variant studied in Thorne and Vlachos [57].Source: §3.4 · p. 5

4.1 Open-domain Question Answering

• RAG sets state-of-the-art results across all four open-domain QA tasks, with TriviaQA’s claim limited to the T5-comparable split.Generation exploits clues without verbatim answers, reaching 11.8% NQ accuracy when no retrieved document contains the answer, versus 0% for extraction. RAG uses DPR-initialized retrieval supervised on Natural Questions and TriviaQA, but achieves these results without a reranker, extractive reader, or specialized salient-span-masking pretraining.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 Abstractive Question Answering

• 2.6 Bleu points and 2.6 Rouge-L points separate RAG-Sequence from BART in its favor on Open MS-MARCO NLG.RAG approaches systems given gold passages despite questions that require those passages or cannot be answered from Wikipedia alone. Qualitative comparisons find fewer hallucinations and more factually correct text than BART.Source: Table 2 · p. 6, §4.2 · p. 6

4.4 Fact Verification

• 4.3% or less separates RAG from state-of-the-art FEVER three-way classifiers, without their intermediate retrieval supervision.On two-way classification, RAG retrieves its own evidence yet achieves accuracy within 2.7% of Thorne and Vlachos [57], who use gold evidence. A gold-evidence article appears as the top retrieved document in 71% of cases and within the top 10 in 90%.Source: Table 2 · p. 6, §4.4 · pp. 6–7

4.3 Jeopardy Question Generation

• 42.7% of human comparisons favored RAG-Token’s factuality versus 7.1% favoring BART; evaluators also found RAG generations more specific.Across 452 generation pairs, both models were factual in another 17%; RAG-Token outperformed RAG-Sequence, and both exceeded BART on Q-BLEU-1. A document-posterior example suggests RAG-Token can combine information across documents, with retrieved clues guiding title generation that parametric memory can complete.Source: Table 2 · p. 6, §4.3 · p. 6, Figure 2 · p. 7

4.5 Additional Results

• RAG-Sequence generates more diverse text than RAG-Token, and both are significantly more diverse than BART without diversity-promoting decoding.Diversity is measured using distinct-to-total n-gram ratios, with Table 5 reporting tri-grams.Source: §4.5 · p. 7, Table 5 · p. 8
• Learned retrieval improves every task over a frozen retriever, while differentiable retrieval outperforms fixed BM25 on every task except FEVER.The BM25 replacement uses retrieval scores as p(z|x) logits; its FEVER advantage may reflect entity-centric claims. Differentiable retrieval is especially important for open-domain QA.Source: §4.5 · p. 7, Table 6 · p. 8
• 70% and 68% accuracy on 2016 and 2018 world leaders, respectively, demonstrate knowledge updates by replacing RAG’s index.The experiment queries the NQ RAG model about 82 leaders whose officeholders changed between the December 2016 and December 2018 Wikipedia indices. Mismatched indices yield 12% accuracy for 2016 leaders using the 2018 index and 4% for 2018 leaders using the 2016 index.Source: §4.5 · pp. 7–8
• 10 retrieved documents maximize RAG-Token’s open-domain QA performance, whereas RAG-Sequence improves monotonically as more documents are retrieved.Training with 5 versus 10 documents produces no significant performance difference, while test-time document counts affect performance and runtime. On MS-MARCO, retrieving more documents increases RAG-Token’s Rouge-L at the expense of Bleu-1, with less pronounced effects for RAG-Sequence.Source: Figure 3 · p. 8, §4.5 · p. 8

Single-Task Retrieval

• RAG unifies previously task-specific retrieval successes within one architecture capable of strong performance across several NLP tasks.Earlier applications include open-domain question answering [5], fact checking, translation, and language modeling [19] [27].Source: §5 · p. 8

General-Purpose Architectures for NLP

• RAG augments pre-trained generative language models with learned retrieval, extending retrieval-free general-purpose architectures such as BART [32] and T5 [51] [52].BART [32] and T5 [51] [52] use pre-trained encoder-decoder models with bidirectional attention for discriminative and generative tasks.Source: §5 · p. 9

Learned Retrieval

• RAG extends task-specific learned retrieval to multiple tasks using a single fine-tunable architecture.Prior downstream retrieval optimization uses search [46], reinforcement learning, or latent-variable approaches [31]; RAG also uses a latent-variable approach.Source: §5 · p. 9

Memory-based Architectures

• RAG’s raw-text document memory is human-readable and human-writable, supporting interpretability and dynamic updates through edits to the document index.Concurrent work [14] retrieves trained entity embeddings instead; earlier dialogue systems retrieve raw text using TF-IDF rather than end-to-end learned retrieval [9].Source: §5 · p. 9

Retrieve-and-Edit approaches

• RAG combines latent retrieval with aggregation of evidence documents, rather than emphasizing light edits to a retrieved training input-output pair.Retrieve-and-edit methods have succeeded in Machine Translation and Semantic Parsing [21]; applying RAG in these settings remains prospective.Source: §5 · p. 9

6 Discussion

• RAG models achieve state-of-the-art results on open-domain question answering.The hybrid generation models access both parametric and non-parametric memory.Source: §6 · p. 9
• People prefer RAG’s generation over purely parametric BART, judging it more factual and specific.Source: §6 · p. 9
• The retrieval index can be hot-swapped to update the model without retraining.Source: §6 · p. 9
• An investigation of the learned retrieval component validates its effectiveness.Source: §6 · p. 9
• Future work could jointly pre-train both components from scratch and investigate how parametric and non-parametric memories interact and can be combined effectively.The authors suggest a BART-like denoising objective or another objective, and see promise for applying the approach across a wide variety of NLP tasks.Source: §6 · p. 9

Broader Impact

• Grounding RAG in factual knowledge such as Wikipedia reduces hallucinations, produces more factual generations, and offers greater control and interpretability.Source: Broader Impact · p. 10
• Potential beneficial applications include answering open-domain medical questions using a medical index and helping people work more effectively.Source: Broader Impact · p. 10
• Wikipedia and other external knowledge sources are unlikely ever to be entirely factual or free of bias.Source: Broader Impact · p. 10
• RAG shares misuse concerns raised for GPT-2, including abusive or misleading content, impersonation, and automated spam/phishing.The authors suggest these concerns may apply to a lesser extent and propose using AI systems to combat misleading content and automated spam/phishing.Source: Broader Impact · p. 10
• Advanced language models may automate various jobs in the coming decades [16].Source: Broader Impact · p. 10

• The excerpt names knowledge-intensive NLP tasks but provides no appendix methods, findings, or supporting details.Source: Appendices for Retrieval-Augmented… · p. 17

A Implementation Details

• 15 retrieved documents support RAG-Token testing on open-domain QA, compared with 50 for RAG-Sequence.RAG-Sequence uses Thorough Decoding because answers are generally short.Source: A Implementation Details · p. 17
• Ten retrieved documents support both RAG variants on Open-MSMarco and Jeopardy question generation.A BART-large model is also trained as a baseline.Source: A Implementation Details · p. 17
• Four beams are used for Open-MSMarco and Jeopardy generation with RAG-Sequence Fast Decoding, whereas QA uses greedy decoding.Beam search did not improve QA results, and Thorough Decoding did not improve generation performance.Source: A Implementation Details · p. 17

B Human Evaluation

• Model assignments to sentences A and B were randomized for each example to avoid screen-position bias.Source: B Human Evaluation · p. 17
• Two annotators’ judgments were removed after poor performance on gold sentences used to assess annotator accuracy.Source: B Human Evaluation · p. 17
• Annotators were encouraged to research topics online and could access detailed instructions and worked examples through the evaluation interface.Source: B Human Evaluation · p. 17, Figure 4 · p. 17

C Training setup Details

• 36GB of CPU memory suffices for the compressed Wikipedia index, down from approximately 100GB, while CPU-based FAISS search is sufficiently fast.Training uses Fairseq [45], mixed precision [40], and eight 32GB NVIDIA V100 GPUs, although training and inference can run on one GPU. The open-source HuggingFace Transformers implementation achieves equivalent performance with a cleaner, easier-to-use implementation.Source: C Training setup Details · p. 17

D Further Details on Open-Domain QA

• Open-domain QA training uses separate answer annotations, retrieval-filtered TriviaQA targets, and regex-derived CuratedTrec supervision.Training each Natural Questions and WebQuestions answer separately slightly improves accuracy; TriviaQA targets must occur within the top 1000 retrieved documents. CuratedTrec uses the most frequent regex-matching answer in the top 1000 documents, falling back to enumerating regex permutations with nondeterministic symbols replaced by whitespace.Source: D Further Details on Open-Domain QA · p. 18
• TriviaQA performance is much higher on the official Wikipedia test set than on the conventional open-domain Web development split.The authors attribute this difference to Wikipedia-test questions being easier to answer from Wikipedia. They report both DPR-style splits [26] and the official Wikipedia test set used by Roberts et al. [52] and Févry et al. [14] for fair comparison.Source: D Further Details on Open-Domain QA · p. 18

E Further Details on FEVER

• FEVER classification regenerates the claim, classifies from its final hidden state, and marginalizes predictions across documents, following [32].The evaluated labels are Supported, Refuted, and Not Enough Info. Evidence-sentence extraction is left for future work because FEVER uses a different Wikipedia dump.Source: E Further Details on FEVER · p. 18

F Null Document Probabilities

• Null-document variants, similar to REALM, did not improve performance and were omitted.The variants add an empty document and marginalize over k + 1 predictions, learning its logit through an embedding, static bias, or neural network. On Open MS-MARCO, questions unlikely to benefit from retrieval instead trigger a recurring document set, suggesting an explicit null mechanism may be unnecessary.Source: F Null Document Probabilities · p. 18

G Parameters

• 44.5 EM on Natural Questions is achieved by RAG-Sequence, compared with 28.9 EM for T5-large (770M parameters) [52].RAG combines 110M-parameter DPR query and document encoders with 406M-parameter BART-large, leaving the document encoder untrained. Its non-parametric index contains 21M 728-dimensional vectors, totaling 15.3B values that can be stored at 8-bit floating-point precision.Source: G Parameters · pp. 18–19

H Retrieval Collapse

• Retrieval sometimes collapses to input-independent documents, after which the generator ignores retrieval and performs equivalently to BART.Preliminary story-generation experiments [11] exhibited this behavior; proposed causes include weaker factual requirements or longer targets producing less informative retriever gradients. Perez et al. [46] also observed spurious retrieval when optimizing retrieval for downstream performance.Source: H Retrieval Collapse · p. 19

I Number of instances per dataset

• Table 7 reports training, development, and test instance counts, with a hidden subset used for evaluation where marked.The supplied caption does not provide the numerical counts.Source: I Number of instances per dataset · p. 19, Table 7 · p. 19