PaperlayerView exact revision on arXiv

Source-linked AI summary

Recitation-Augmented Language Models

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, Denny Zhou

arXiv:2210.01296v2cs.CLcs.AI

TL;DR

Few-shot task prompts may not align with causal language-model pretraining, motivating a way to draw factual knowledge from model memory without external retrieval. RECITE recites relevant passages before answering, and the authors report improved closed-book QA performance across three datasets and four models.

  • Problem

    Few-shot task prompts may not resemble causal language-model pretraining, which can hinder models from reciting factual knowledge from memory.

    Source: §1 · p. 2

  • Method

    RECITE has a language model recite relevant information from its own weights before generating the final answer.

    Source: §1 · p. 2

  • Results

    Recite-and-answer improves closed-book QA performance across Natural Questions, TriviaQA, and HotpotQA with PaLM, UL2, OPT, and Codex.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that recite-and-answer is effective across three closed-book QA datasets and that diversifying recitation can be beneficial.

    Source: §5 · p. 9

    Updating time-sensitive knowledge in a pure LLM-based method requires training or fine-tuning on new corpus data, which can be costly.

    Source: §5 · p. 9

Abstract

from arXiv · show

We propose a new paradigm to help Large Language Models (LLMs) generate more accurate factual knowledge without retrieving from an external corpus, called RECITation-augmented gEneration (RECITE). Different from retrieval-augmented language models that retrieve relevant documents before generating the outputs, given an input, RECITE first recites one or several relevant passages from LLMs' own memory via sampling, and then produces the final answers. We show that RECITE is a powerful paradigm for knowledge-intensive NLP tasks. Specifically, we show that by utilizing recitation as the intermediate step, a recite-and-answer scheme can achieve new state-of-the-art performance in various closed-book question answering (CBQA) tasks. In experiments, we verify the effectiveness of \method~on four pre-trained models (PaLM, UL2, OPT, and Codex) and three CBQA tasks (Natural Questions, TriviaQA, and HotpotQA). Our code is available at "https://github.com/Edward-Sun/RECITE".

1 INTRODUCTION

RECITE has LLMs recite relevant information from their own memory before generating an answer, without retrieving from an external corpus. The paper reports improved performance across several few-shot closed-book question-answering tasks and models.

1 INTRODUCTION

  • RECITE is evaluated as a closed-book alternative to direct generation and open-book retrieval-augmented generation.Figure 1 illustrates these three approaches to open-domain question answering.Source: Figure 1 · p. 1
  • RECITE first recites relevant information from model weights, then generates the final output as a separate task-execution step.The two stages are knowledge-recitation and task execution.Source: §1 · p. 2
  • Few-shot task prompts may not resemble causal language-model pretraining, which can hinder models from reciting knowledge from memory.The paper motivates recitation as an intermediate step that better resembles the pretraining objective.Source: §1 · p. 2
  • Recite-and-answer improves performance across Natural Questions, TriviaQA, and HotpotQA with PaLM, UL2, OPT, and Codex.The experiments cover single-hop, trivia, and multi-hop closed-book QA.Source: §1 · p. 2

2 RELATED WORK

The related work situates RECITE among open- and closed-book question answering, in-context learning, rationale-based methods, and studies of memorization. It distinguishes reciting knowledge from model weights from both external retrieval and generating reasoning rationales.

2.1 OPEN-DOMAIN QUESTION ANSWERING

  • Open-domain QA generates answers to arbitrary context-free questions; closed-book systems rely on knowledge stored in model parameters, and few-shot evaluation ideally limits train-test overlap.Earlier fine-tuning results have been attributed mainly to training-set question memorization, while smaller models still struggle with closed-book questions.Source: §2.1 · pp. 2–3

2.2 IN-CONTEXT FEW-SHOT LEARNING

  • In-context learning uses a few prompted exemplars without gradient updates, while RECITE argues that better prompting can retrieve underestimated knowledge from model weights.This contrasts with Atlas, which uses retrieval augmentation and few-shot fine-tuning.Source: §2.2 · p. 3

2.3 RATIONALE-AUGMENTED REASONING

  • Unlike rationale-based reasoning methods, RECITE separates answering into reciting relevant knowledge stored in model parameters and then answering the question.Prior work generates step-by-step solutions or rationales before a final answer.Source: §2.3 · p. 3

2.4 MEMORIZATION IN LARGE LANGUAGE MODELS

  • Memorization studies examine exact-prompt continuation, whereas RECITE relies on fuzzy memorization: prompts need not match training data exactly but can still improve memorization accuracy.Prior findings associate memorization with model capacity, example duplication, and prompt context length.Source: §2.4 · p. 3

3 LEARNING TO RECITE FOR CLOSED-BOOK QUESTION ANSWERING

The method recites evidence before answering few-shot closed-book questions, with extensions for multiple paths and multi-hop questions. For more diverse recitations, it samples passage hints, generates passages from them, and fine-tunes models on synthetic question-hint-passage data.

3 LEARNING TO RECITE FOR CLOSED-BOOK QUESTION ANSWERING

  • RECITE uses an evidence-recitation module to produce relevant passages and a question-answering module to answer from them in the few-shot setting.For Natural Questions, the method assumes access to top-retrieved Wikipedia pages to generate synthetic question-recitation data for fine-tuning.Source: §3 · p. 4

3.1 PROMPT-BASED RECITE-AND-ANSWER FOR QUESTION-ANSWERING

  • Prompt-based recite-and-answer generates a relevant passage first, then conditions few-shot question answering on that recitation.For self-consistency, it samples several recitations, greedily decodes answers, and selects by plurality or majority vote.Source: §3.1 · pp. 4–5
  • For multi-hop questions, multiple sequential recitations cover different topics, and later passages can use information from earlier ones.The scheme can also sample multiple recitation paths and majority-vote the final answers.Source: §3.1 · p. 5

3.2 PASSAGE HINT-BASED DIVERSIFIED RECITATION WITH FINE-TUNING

  • Passage-hint diversification addresses the trade-off between avoiding factually wrong sampled recitations and retaining enough diversity.The method samples hints, deduplicates them, greedily generates their passages, then aggregates diverse recitations as context for answering.Source: §3.2 · pp. 5–6
  • Because pretrained models may not learn question-to-passage-hint mappings through few-shot prompting, the method fine-tunes on synthetic question-hint-passage pairs.It generates questions for sampled Wikipedia passages and trains the model to predict each passage hint and content.Source: §3.2 · p. 6

4 EXPERIMENTS

Across the evaluated closed-book question-answering tasks, recite-and-answer improves performance on Natural Questions and TriviaQA and outperforms chain-of-thought prompting on multi-hop HotpotQA. The experiments also examine diversified recitation, self-consistency, robustness to exemplars, comparisons with retrieval, and error patterns.

4.2.1 PROMPT-BASED RESULTS

  • Recite-and-answer significantly improves CBQA on Natural Questions and TriviaQA across models, and outperforms chain-of-thought prompting on multi-hop HotpotQA.Table 1 covers the three evaluated tasks; the reported improvement pattern is more consistent across models on Natural Questions and larger for smaller models on TriviaQA.Source: Table 1 · p. 7, §4.2.1 · p. 7

4.1.1 EVALUATION DATASETS

  • Evaluations use four language models; Natural Questions is tested at 5 and 64 shots, TriviaQA at 5 shots, and HotpotQA as a 4-shot multi-hop task.TriviaQA and HotpotQA evaluation uses the first 1,024 examples; normalized-answer Exact Match and F1 are reported.Source: §4.1.1 · pp. 6–7, §4.1.2 · p. 7, §4.2 · p. 7

4.2.2 RESULTS OF PASSAGE HINT-BASED DIVERSIFIED RECITATION

  • Diversified recitation further improves fine-tuned PaLM performance on Natural Questions.Table 2 compares standard prompting, self-consistency sampling, and diversified recitation for PaLM-62B.Source: §4.2.2 · p. 7, Table 2 · p. 8

4.3.1 ON THE NUMBER OF SELF-CONSISTENCY PATHS

  • On TriviaQA with OPT-30B and UL2-20B, sampling more recitation passages tends to improve performance, while variability decreases with more self-consistency paths.Each path-count result is averaged over five randomly selected subsets; the analysis samples up to 20 recitation passages.Source: §4.3.1 · p. 8, Figure 4 · p. 8

4.3.2 ON THE ROBUSTNESS OF FEW-SHOT EXEMPLARS

  • With UL2 on TriviaQA, recite-and-answer retains small performance deviations across exemplar seeds and self-consistency path counts, while its gains over standard prompting remain significant.The analysis uses five random seeds with randomly sampled and shuffled 5-shot exemplars.Source: §4.3.2 · p. 8

4.3.3 RECITATION V.S. RETRIEVAL V.S. GROUND-TRUTH

  • BM25 remains a strong NQ retrieval baseline, but recite-and-answer is competitive without an external corpus; Codex tends to benefit more from its own recitation than BM25 context.The comparison uses the first 1,024 NQ validation examples and contrasts retrieval, recitation, and ground-truth context passages.Source: §4.3.3 · pp. 8–9, Table 3 · p. 9

4.3.4 ERROR ANALYSIS

  • On TriviaQA, most errors are attributed to missing recitation or answer aggregation; per-path analysis finds 7%–10% of questions with correct recitations but wrong answers and around 12% answered correctly despite incorrect recitations.The question-level categories also include correct answers produced on a path but lost in majority voting.Source: §4.3.4 · p. 9, Table 4 · p. 9, Table 5 · p. 9

5 CONCLUSION & DISCUSSION

The paper concludes that recite-and-answer is effective for closed-book question answering and that diversifying recitations can be beneficial. It also identifies costly updating of time-sensitive knowledge as a limitation.

5 CONCLUSION & DISCUSSION

  • The approach shows promising results across three large language models and three closed-book QA datasets.The conclusion describes these results as demonstrating the effectiveness of recite-and-answer.Source: §5 · p. 9
  • RECITE first recites a specific knowledge source to augment model memory, and diversifying recitations can help when multiple sources could answer a question.Source: §5 · p. 9
  • Updating time-sensitive knowledge requires training or fine-tuning the model on a new corpus, which can be costly.Source: §5 · p. 9

ETHICS STATEMENT

The ethics discussion notes a possible bias trade-off, while the appendices specify dataset-specific recitation examples and prompt-separation conventions.

ETHICS STATEMENT

  • Because RECITE generates intermediate steps with the LLM itself, it may further enhance biases in model weights compared with RAG using an external trustworthy corpus.Source: ETHICS STATEMENT · p. 10

A ILLUSTRATIONS OF PROMPTS AND LANGUAGE MODEL OUTPUTS

B PRINCIPLES OF PROMPT DESIGNS

  • Prompt exemplars are separated with three newlines and their components with two; UL2 uses semicolons instead because its vocabulary does not encode newline.Source: B PRINCIPLES OF PROMPT DESIGNS · p. 15

C DETAILS OF PASSAGE HINT-BASED FINE-TUNING

The passage-hint fine-tuning setup builds synthetic question–hint–passage pairs from Wikipedia passages and trains PaLM on the constructed corpus.

C DETAILS OF PASSAGE HINT-BASED FINE-TUNING

D PRE-TRAINED LANGUAGE MODELS

The section describes the paper’s PaLM, UL2, OPT, and Codex configurations and gives the scope of two benchmark figures.

D PRE-TRAINED LANGUAGE MODELS

E ANALYSIS OF THE NUMBER OF EXAMPLES IN FEW-SHOT LEARNING

On Natural Questions, recite-and-answer consistently improves over standard prompting across the analyzed shot counts, with its largest improvement in the 1-shot setting.

F ANALYSIS OF MODEL SIZES ON INSTRUCTION-FINETUNED LANGUAGE

Recite-and-answer consistently improves standard prompting on TriviaQA with FLAN-T5, with the largest improvement in the xxl (11B) setting.

G LANGUAGE MODEL HYPERPARAMETERS

The paper reports the hyperparameters used for its language models in Table 7, while noting that Codex's details are unknown.

Loading 2210.01296v2…