PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela

arXiv:2005.11401v4cs.CLcs.LG

TL;DR

Knowledge-intensive NLP requires models that can access, inspect, and update external knowledge beyond parametric memory alone. This paper introduces retrieval-augmented generation, combining seq2seq generation with a retriever over Wikipedia, and achieves state-of-the-art open-domain QA while producing more factual, specific, and diverse language than BART.

  • Problem

    Existing hybrid-memory models had only been explored for open-domain extractive question answering, leaving their use with general-purpose seq2seq generation underexplored.

  • Method

    RAG combines a pre-trained seq2seq generator with a neural retriever over a dense Wikipedia index in an end-to-end probabilistic model.

  • Results

    RAG achieves state-of-the-art results on three open-domain QA tasks and generates more factual, specific, and diverse language than BART.

  • Takeaways & Limitations

    RAG supports strong knowledge-intensive generation while allowing its external memory to be inspected and updated without retraining.

  • Takeaways & Limitations

    Some MS-MARCO questions cannot be answered from Wikipedia alone and require gold passages or parametric knowledge for reasonable responses.

Abstract

from arXiv · show

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

1 Introduction

RAG combines pre-trained parametric generation with an external non-parametric Wikipedia memory to address limitations of implicit knowledge in language models. The approach achieves state-of-the-art results on three open-domain QA tasks and strongly outperforms specialized methods on TriviaQA.

  • Motivation: Parametric language models encode substantial knowledge [51] [52] but cannot easily revise it, explain predictions, or avoid hallucinations [38].
  • Approach: The model uses maximum inner product search to retrieve top-K documents and marginalizes seq2seq predictions over documents treated as a latent variable.
  • Problem: Prior hybrid-memory systems focused on open-domain extractive question answering, whereas this work extends parametric and non-parametric memory to seq2seq generation.
  • Approach: RAG combines a pre-trained seq2seq transformer with a dense Wikipedia index accessed by a pre-trained neural retriever in an end-to-end probabilistic model.
  • Results: RAG achieves state-of-the-art results on Natural Questions, WebQuestions, and CuratedTrec, while strongly outperforming specialized pre-training approaches on TriviaQA [24].

2 Methods

RAG combines a neural retriever over text passages with a seq2seq generator, treating retrieved documents as latent variables during end-to-end training. It supports sequence-level or token-level document marginalization, with distinct decoding procedures for the two variants.

  • Architecture: RAG retrieves top-K passages with pη(z|x) and conditions the generator pθ(yi|x, z, y1:i−1) on each passage when producing the target sequence.The retriever and generator jointly provide non-parametric and parametric context for generation.
  • RAG variants: RAG-Sequence marginalizes documents as one latent variable for the entire output, whereas RAG-Token can select a different latent document for each target token.Token-level marginalization allows answers to combine content from several retrieved documents.
  • Model components: The retriever uses DPR’s BERTBASE bi-encoder and approximate maximum inner product search, while the generator uses 400M-parameter BART-large with input and retrieved passage concatenation.The document encoder builds the dense index, and the query encoder represents the input query.
  • Training: Training minimizes negative marginal log-likelihood without document-retrieval labels, updating the query encoder and BART generator while keeping the document encoder and index fixed.Fixing the document encoder avoids periodically rebuilding the document index during training.
  • Inference: At test time, RAG-Sequence requires document-specific beam searches and additional scoring because its likelihood is not a conventional per-token likelihood.RAG-Token instead has an autoregressive transition probability, so the two variants require different approximations to arg max_y p(y|x).

3 Experiments

The experiments evaluate RAG across diverse knowledge-intensive tasks using a shared Wikipedia-based non-parametric memory. They cover extractive and abstractive QA, question generation, and evidence-based claim verification, while testing limitations arising from incomplete Wikipedia coverage and unavailable gold passages.

  • Experimental setup: All experiments use the December 2018 Wikipedia dump, split into 21M disjoint 100-word documents and indexed with FAISS using document-encoder embeddings.The same Wikipedia-derived non-parametric knowledge source supports the experiments.
  • Open-domain question answering: For open-domain QA, RAG directly minimizes answer negative log-likelihood and is compared with extractive retrieval-based QA and closed-book approaches.The setup treats questions and answers as input-output text pairs, contrasting RAG’s generation with extractive and parametric-only paradigms.
  • MSMARCO natural language generation: MSMARCO NLG is treated as open-domain abstractive QA using only questions and answers, without supplied gold passages.Performance is expected to be lower for questions requiring unavailable passage-specific information, while parametric knowledge can help when Wikipedia lacks the answer.
  • Open-domain question generation: The question-generation evaluation introduces demanding Jeopardy-question generation, using SearchQA splits, Q-BLEU-1, and human assessments of factuality and specificity.Jeopardy questions require inferring an entity from a fact, and the evaluation compares RAG with a trained BART baseline.
  • FEVER claim verification: FEVER tests RAG on Wikipedia evidence retrieval followed by entailment reasoning to classify claims as supported, refuted, or unverifiable.The task combines retrieval with challenging reasoning over evidence and therefore probes more than document lookup alone.

4 Results

RAG achieves state-of-the-art performance across open-domain QA and remains competitive on generation and classification tasks. Its retrieved non-parametric memory also improves factuality, diversity, retrieval effectiveness, and test-time knowledge updating.

  • Open-Domain QA: RAG achieves 11.8% accuracy on NQ questions whose answers appear in none of the retrieved documents, whereas an extractive model would score 0%.Generation can use documents containing clues without stating the answer verbatim.
  • Open-Domain QA: RAG sets a new state of the art on all four open-domain QA tasks, combining closed-book generation flexibility with open-book retrieval performance.This requires no specialized salient-span-masking pre-training.
  • Generation and Classification: RAG-Sequence beats BART on Open MS-MARCO NLG by 2.6 Bleu points and 2.6 Rouge-L points, while RAG-Token is more factual than BART in human evaluation.BART was more factual in 7.1% of cases, whereas RAG was more factual in 42.7%.
  • Generation and Classification: RAG classification is within 4.3% of state-of-the-art FEVER 3-way systems and within 2.7% of a gold-evidence RoBERTa model for 2-way classification.RAG receives only the claim and retrieves its own evidence in the 2-way comparison.
  • Retrieval Ablations and Index Hot-Swapping: Learned differentiable retrieval improves every task except that BM25 performs best on FEVER, while replacing the index updates RAG’s world knowledge at test time.Using matched indices, RAG reaches 70% for 2016 leaders and 68% for 2018 leaders; mismatched indices yield 12% and 4%, respectively.
  • Generation Diversity: RAG generations are more diverse than BART without diversity-promoting decoding, with RAG-Sequence more diverse than RAG-Token.Both RAG variants are significantly more diverse than BART.

5 Related Work

Related work spans task-specific retrieval, general-purpose pretrained architectures, learned retrieval, memory-based models, and retrieve-and-edit methods. This work positions itself as unifying retrieval successes across individual tasks while using raw-text external memory and aggregating content rather than lightly editing retrieved examples.

  • Single-Task Retrieval: Retrieval has improved performance across open-domain question answering [5], fact checking, dialogue [41] [9], translation, and other individual NLP tasks.The paper frames its contribution as unifying these task-specific successes.
  • General-Purpose Architectures for NLP: Pretrained language models achieve strong performance across classification, discriminative, and generative tasks without retrieval, including GLUE after fine-tuning [8] and GPT-2.
  • Learned Retrieval: Learned retrieval methods use pretrained neural language models [44] [26] and task-specific optimization through search [46], reinforcement learning, or latent variables [31].
  • Memory-based Architectures: The document index functions as large external memory analogous to memory networks [64] [55], while related systems retrieve entity or fact embeddings [14] [15] rather than raw text.The paper identifies raw text as a key feature of its memory.
  • Retrieve-and-Edit approaches: Retrieve-and-edit approaches retrieve similar training input-output pairs for editing and have succeeded in Machine Translation and Semantic Parsing [21].The paper contrasts its approach with lightly editing one retrieved item, emphasizing aggregation of content from several items.

6 Discussion

The work presents RAG as a hybrid of parametric and non-parametric memory, achieving state-of-the-art open-domain QA results. RAG generations were preferred over purely parametric BART for being more factual and specific, while retrieval analysis supported the learned retriever and index hot-swapping enabled model updates.

  • RAG models achieve state-of-the-art results on open-domain question answering.
  • People prefer RAG generations over purely parametric BART because they are more factual and specific.
  • RAG combines parametric and non-parametric memory for hybrid generation.
  • An investigation of the learned retrieval component validates its effectiveness.
  • The retrieval index can be hot-swapped to update the model.

Broader Impact

RAG’s grounding in Wikipedia can make generations more factual while improving control and interpretability, but external knowledge sources may contain bias and enable harmful uses similar to GPT-2.

  • Grounding RAG in real factual knowledge from Wikipedia can reduce hallucinations and improve factuality, control, and interpretability.
  • RAG could provide societal benefits in applications such as answering open-domain medical questions using a medical knowledge index.
  • External knowledge sources may remain biased or factually imperfect, while RAG could support abuse, misleading content, impersonation, or spam and phishing automation, as with GPT-2.

Appendices for Retrieval-Augmented Generation for

This appendix section accompanies the paper “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.”

  • The appendix is associated with the paper titled “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.”
  • The paper focuses on knowledge-intensive NLP tasks, as specified in its title.

A Implementation Details

The implementation uses task-specific retrieval depths and decoding strategies: QA uses 15 or 50 documents depending on the RAG variant, while question generation uses ten documents for both variants.

  • A Implementation Details: For open-domain QA, RAG-Token uses 15 retrieved documents, whereas RAG-Sequence uses 50 and Thorough Decoding.Greedy decoding is used for QA because beam search did not improve results.
  • A Implementation Details: For Open-MSMarco and Jeopardy question generation, both RAG-Token and RAG-Sequence use ten retrieved documents.
  • A Implementation Details: The implementation also specifies training settings for question generation, including an incompletely reported BART-large configuration.

B Human Evaluation

Human evaluation used a purpose-built factuality annotation interface with detailed guidance and quality checks. Model labels were randomized across sentence positions, while annotators could research topics and were evaluated using gold sentences.

  • B Human Evaluation: The factuality evaluation interface provided detailed instructions and a worked example through a dedicated guide.Annotators could access the full instructions tab and view the tool guide from the interface.
  • B Human Evaluation: Model assignments to sentences A and B were randomized for each example to avoid screen-position bias.
  • B Human Evaluation: Annotators were encouraged to research topics online, and gold sentences were included to assess annotation accuracy.Two annotators performed poorly on the gold examples, and their annotations were not retained.

C Training setup Details … I Number of instances per dataset

The appendix details RAG’s training, task-specific preprocessing, parameterization, and dataset sizes, while documenting null-document and retrieval-collapse investigations. It also shows that RAG achieves strong open-domain QA with substantially fewer trainable parameters than comparable closed-book models.

  • C Training setup Details: RAG and BART baselines use Fairseq, mixed-precision training, and distributed training across 8 or 32GB NVIDIA V100 GPUs.Training and inference can run on one GPU; FAISS search runs sufficiently fast on CPU, requiring approximately 100GB of CPU memory for Wikipedia’s index vectors.
  • D Further Details on Open-Domain QA: Training separately on multiple question-answer annotation pairs slightly improves RAG accuracy for Natural Questions and WebQuestions.The approach uses each (q, a) pair separately rather than treating annotations only as document-matching alternatives.
  • D Further Details on Open-Domain QA: CuratedTrec supervision is derived by retrieving the top 1000 documents and selecting the answer most frequently matching each regular-expression annotation.When no match is found, the method falls back to a heuristic that generates possible answer permutations.
  • D Further Details on Open-Domain QA: Experiments use the DPR dataset splits for open-domain QA; TriviaQA uses its public Web Development split rather than the official Wikipedia test set used by Roberts et al. [52].These splits are described as consistent with common open-domain QA practice.
  • E Further Details on FEVER: For FEVER classification, RAG regenerates the claim, classifies from the final hidden state, and marginalizes probabilities across documents.The main-paper task is three-way classification into Supported, Refuted, or Not Enough Info.
  • F Null Document Probabilities: Experiments tested a null document alongside retrieved documents, modeling its logit with a learned embedding or static bias before marginalization.The mechanism was intended to handle inputs for which no useful information could be retrieved.
  • H Retrieval Collapse: Retrieval collapse caused the retriever to select the same documents regardless of input, after which the generator ignored retrieval and matched BART performance.This occurred in preliminary experiments on tasks such as story generation and may reflect weaker factual requirements or longer target sequences; Table 7 reports dataset instance counts.
Loading 2005.11401v4…