PaperlayerView exact revision on arXiv

Source-linked AI summary

Generate rather than Retrieve: Large Language Models are Strong Context Generators

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, Meng Jiang

arXiv:2209.10063v3cs.CLcs.AI

TL;DR

Knowledge-intensive tasks typically retrieve external documents before answering, but retrieval uses fixed evidence chunks and requires an external corpus. GenRead instead generates contextual documents with a large language model, then reads them to answer; clustering-based prompts diversify those documents. Across open-domain QA, fact checking, and dialogue, GenRead matches or outperforms retrieve-then-read systems without external retrieval, while its knowledge is harder to update and may hallucinate.

  • Problem

    Knowledge-intensive tasks require substantial world or domain knowledge, while common retrieve-then-read systems depend on external corpora and fixed retrieved documents.

  • Method

    GenRead prompts a large language model to generate contextual documents for a question, reads them for the answer, and uses clustering-based prompts to produce diverse perspectives.

  • Results

    Across three knowledge-intensive tasks, GenRead matches or outperforms retrieve-then-read methods without retrieving external documents; clustering-based prompting improves EM by +2.2 on average over sampling.

  • Takeaways & Limitations

    Generated documents can contain correct answers more often than top retrieved documents and can improve performance over directly generating answers without new external information.

  • Takeaways & Limitations

    GenRead is limited in updating knowledge and adapting to new domains, and generated documents may hallucinate or preserve stale information and biases.

Abstract

from arXiv · show

Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer. Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, resulting in the generated documents that cover different perspectives, leading to better recall over acceptable answers. We conduct extensive experiments on three different knowledge-intensive tasks, including open-domain QA, fact checking, and dialogue system. Notably, GenRead achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly outperforming the state-of-the-art retrieve-then-read pipeline DPR-FiD by +4.0 and +3.9, without retrieving any documents from any external knowledge source. Lastly, we demonstrate the model performance can be further improved by combining retrieval and generation. Our code and generated documents can be found at https://github.com/wyu97/GenRead.

1 INTRODUCTION

The paper replaces retrieval with large language model generation of contextual documents, then reads those documents to answer knowledge-intensive tasks. Clustering-based prompts generate diverse perspectives, and experiments show strong performance without external retrieval.

  • Generated contextual documents contain correct answers more often than top retrieved documents and outperform directly generated answers without adding external information.The paper attributes this to question-specific generation and better use of knowledge stored in model parameters.
  • Clustering-based prompting samples question-document demonstrations from diverse clusters to generate documents covering different perspectives.The method is intended to improve recall as more documents are generated.
  • The generate-then-read pipeline generates contextual documents first and uses a reader to predict the final answer.Readers can be large zero-shot models or smaller models fine-tuned with generated documents.
  • GenRead replaces retrieving documents from Wikipedia or Google with prompting a large language model to generate relevant contextual documents.
  • GenRead matches or outperforms retrieve-then-read methods across three knowledge-intensive tasks without retrieving external documents.The experiments include open-domain QA, fact checking, and open-domain dialogue.

2 RELATED WORK

Related work centers on retrieve-then-read systems, language-model-based document identifiers, and prompting methods that elicit relevant knowledge from language models.

  • Retrieve-then-read systems retrieve relevant documents from a large evidence corpus before a reader predicts the answer.
  • Recent work generates document identifier strings, such as entity names or distinctive n-grams, as intermediate targets for retrieval.
  • Other studies prompt large language models to generate relevant contexts or intermediate text for knowledge-intensive tasks.

3 PROPOSED METHOD

The proposed method generates contextual documents and then reads them to answer questions, with clustering-based prompts designed to increase knowledge coverage and diversity. It supports zero-shot and supervised settings.

  • Generate-then-read: GENREAD generates contextual documents for a query and then uses a reader to predict the final answer.The reader may be a large zero-shot model or a smaller FiD model fine-tuned on generated documents.
  • Zero-shot setting: In zero-shot QA, directly predicting answers from questions can leave world knowledge unexploited, whereas GENREAD introduces generated documents as an auxiliary variable.
  • Clustering-based prompts: Figure 1 depicts distinct question-document pairs from embedding clusters prompting diverse document generation before answer reading.
  • Zero-shot setting: The zero-shot pipeline first prompts a large language model to generate a document and then reads it with the question to produce the final answer.Greedy decoding was used throughout the zero-shot experiments for simplicity and reproducibility.
  • Supervised setting: Under supervision, repeated generation from one prompt produces similar documents and low knowledge coverage, motivating diverse human prompts and clustering-based prompts.Nucleus sampling diversifies generation somewhat, but generated knowledge remains highly repetitive.
  • Clustering-based prompts: The clustering method clusters question-document representations into K classes, samples n pairs from each cluster, and uses each cluster’s pairs as in-context demonstrations.Enumerating the cluster-specific demonstrations yields K generated documents biased toward different perspectives.

4 EXPERIMENTS

Experiments across open-domain QA, fact checking, and dialogue evaluate GENREAD against retrieval-based and language-model baselines. Results show strong performance from generated contextual documents, especially with clustering-based prompts and combinations of generated and retrieved documents.

  • Experimental Setup: Experiments cover open-domain QA, fact checking, and open-domain dialogue using datasets including NQ, TriviaQA, WebQ, FEVER, FM2, and WoW.The evaluation uses exact match for open-domain QA and includes both zero-shot and supervised settings.
  • Zero-Shot Results: GENREAD improves the EM score by +6.9 on three open-domain QA benchmarks compared with original InstructGPT without retrieving contextual documents.The generator and reader use the same InstructGPT parameters, and no new data is introduced.
  • Zero-Shot Results: GENREAD achieves on-par performance with zero-shot retrieve-then-read models on NQ and FM2 and outperforms them on all other benchmarks.The comparison includes retrieval models and Google search used to supply contextual documents.
  • Retrieval and Generation Results: For fewer than 10 documents, GENREAD significantly outperforms DPR and Google search, while clustering-based prompts perform best among GENREAD variants.The clustering approach addresses reduced recall from duplicate information in generated documents.
  • Supervised Open-Domain QA: Clustering-based prompts improve EM by +2.2 on average over sampling methods and produce state-of-the-art supervised QA performance when used with generated documents.GENREAD also outperforms Google search on all benchmarks, while performance can be lower than DPR on datasets containing many time-dependent questions.
  • Other Knowledge-Intensive Tasks: Under supervision, GENREAD performs on par on fact checking and better on dialogue-system tasks, supporting large language models as knowledge generators.The paper attributes weaker fact-checking performance than dense retrieval to the task’s smaller semantic gap between statements and contextual documents.
  • Complementarity of Documents: Merging retrieved and generated documents achieves state-of-the-art performance, improving +5.7 on average over DPR alone and +4.4 over the large language model alone across three QA benchmarks.Generated documents can complement retrieved documents because correct answers occur more frequently in generated text.
  • Readability Analysis: When both document types contain the correct answer, FiD produces more correct answers from generated documents than from retrieved documents.The paper illustrates this readability difference with a generated explanation that makes the answer “Olympia” easier to infer.

5 EPILOGUE

The paper presents generate-then-read as a replacement for dense retrieval in knowledge-intensive tasks, using generated contextual documents before answer prediction. It reports strong results without external document retrieval while identifying limitations in knowledge updating, domain adaptation, and hallucination.

  • Generate-then-read prompts a large language model to generate contextual documents and then reads them to infer the final answer.
  • The approach is limited in updating its knowledge state and adapting quickly to new domains because adding knowledge would likely require retraining.
  • Generated documents may contain hallucinations that produce incorrect predictions.

ETHICS STATEMENT

The paper notes that generating contextual documents from model parameters can reproduce biases, stale information, and limited diversity from training data. It also identifies uncertainty about whether conclusions generalize across models and calls for further work on bias and fabricated facts.

  • Generated contextual documents may further reproduce existing biases because they rely on knowledge stored in language-model parameters.
  • The approach may produce stale information from outdated training documents and less diverse outputs that favor common entities and terms.
  • The experiments cover only three large language models, so conclusions may not hold for models trained with different data or objectives.
  • Future work includes exploring bias and harm and aligning language models to generate less biased content and fewer fabricated facts.

A.1 DATASETS AND SPLITS

The paper evaluates knowledge-intensive tasks using established datasets for open-domain QA, fact checking, and dialogue. It describes their sources, answer or evidence formats, and the dataset splits used for evaluation.

  • TriviaQA contains trivia questions originally scraped from trivia and quiz-league websites.
  • WebQuestions consists of questions selected using Google Suggest, with entity answers from Freebase.
  • Natural Questions were mined from real Google search queries, with answers identified as spans in Wikipedia articles by human annotators.
  • FEVER requires retrieving external-corpus evidence to determine whether a statement is supported or refuted, while Fool Me Twice contains entailed or refuted claims grounded in Wikipedia evidence.
  • Wizard of Wikipedia trains open-domain dialogue agents, requiring one speaker to ground utterances in a specific Wikipedia knowledge sentence.
  • The study uses KILT train, development, and test splits for FEVER and WoW, official splits for FM2, and established open-domain QA splits, with hidden KILT test labels evaluated online.

A.2 IMPLEMENTATION DETAILS

The implementation uses FiD readers built on T5 backbones and reports hyperparameter settings and validation sources for the evaluated tasks. Baseline implementations rely on existing repositories.

  • FiD uses T5-770M and T5-3B backbone models with AdamW, 2,000 warm-up steps, dropout probability 0.1, and weight decay 0.01.
  • The experiments use one A100 for T5-770M and eight A100 GPUs for T5-3B, with total batch size 16 in both configurations.
  • Other baseline methods are implemented using publicly available repositories.
  • Hyperparameter settings and validation performance are reported for open-domain QA, fact checking, and dialogue system experiments, including GENREAD FiD-large and FiD-extra-large variants.

A.3 REPRODUCIBILITY VIA OPEN SOURCE LARGE LANGUAGE MODELS

Open-source generators were evaluated as alternatives to InstructGPT for producing contextual documents. OPT underperformed InstructGPT but remained comparable to DPR, while Codex achieved the best TriviaQA and WebQ performance.

  • OPT performed worse than InstructGPT but still achieved comparable performance with DPR.
  • OpenAI Codex achieved the best performance on both TriviaQA and WebQ.
  • Table 8 reports exact match scores for DPR and open-source generators including OPT and Codex.

A.4 SCALING WITH NUMBER OF LARGE LANGUAGE MODEL PARAMETERS

GENREAD performance improves as the InstructGPT generator grows from 150M to 175B parameters. Only the largest generator outperforms DPR-FiD, indicating that this capability emerges with scaling.

  • TriviaQA and WebQ performance continually improve as InstructGPT generator parameters increase.The evaluated sizes are Ada-150M, Babbage-1.3B, Curie-6.7B, and Davinci-175B.
  • Only the largest InstructGPT model enables GENREAD to outperform DPR-FiD.
  • Figure 4 compares performance across Ada-150M, Babbage-1.3B, Curie-6.7B, and Davinci-175B generators.

A.5 ADDITIONAL NUMBERS FOR TABLES IN THE MAIN PAPER

The additional evaluations cover retrieval, prompt variants, model costs, and combining retrieved with generated documents. The reported cost comparison places the DPR–InstructGPT crossover at approximately 2,473 questions.

  • Table 9 reports zero-shot open-domain QA without external retrieval, with GENREAD improving InstructGPT’s EM score by +6.9 on average.
  • Tables 9 and 10 provide additional QA and retrieval numbers, while Table 11 evaluates document-sampling strategies.
  • Merging DPR-retrieved and language-model-generated documents achieves significantly better performance than using DPR documents alone.
  • GENREAD’s retrieval evaluation reports Recall@K for baselines and multiple variants, with some values overlapping Figure 2.
  • The cost equations are YDPR-cost = 3.2e10X + 2.3e17 and YGPT3-cost = 9.6e13X, crossing at X ≈2473.

A.7 ERROR ANALYSIS AND CASE STUDIES ON THE NQ DATASET

The NQ analysis identifies annotation problems and separates GENREAD’s remaining prediction errors into retrieval and reading failures. Additional materials examine prompt choices, temporal and incomplete-answer issues, and hallucinations.

  • The NQ analysis highlights time-dependent answers and Wikipedia-version differences as dataset issues affecting evaluation.
  • Among 100 NQ examples, 29 contain data-collection or annotation mistakes, including temporal and incomplete-answer issues.The case-study table reports 49 correct predictions, corresponding to EM = 49%.
  • GENREAD produced correct answers for 49 questions; its incorrect predictions included retrieval errors and reading errors.Retrieval errors lacked a generated document containing the answer, whereas reading errors had an answer-containing document that the reader failed to use.
  • Incomplete-answer labels favor DPR-FiD because NQ answers are spans identified from Wikipedia passages.
  • Hallucinated generated documents can contradict facts and world knowledge, producing wrong predictions.
Loading 2209.10063v3…