PaperlayerView exact revision on arXiv

Source-linked AI summary

Context Generation Improves Open Domain Question Answering

Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, Bryan Catanzaro

arXiv:2210.06349v2cs.CLcs.AI

TL;DR

Closed-book QA methods do not fully exploit pretrained LMs’ parameterized knowledge and have lagged open-book methods. CGAP generates relevant context before prompting the same LM to answer, and results on three benchmarks show it outperforms previous closed-book methods while performing on par with open-book methods.

  • Problem

    Closed-book QA methods have not fully exploited pretrained LMs’ parameterized knowledge and have been less accurate than open-book methods.

    Source: §1 · p. 1

  • Method

    CGAP prompts a pretrained LM to generate question-relevant context, then prompts the same LM to answer using that context and the question, with majority voting over multiple contexts.

    Source: §1 · p. 2

  • Results

    Across three QA benchmarks, CGAP significantly outperforms previous closed-book methods and is on par with open-book methods; exact matching is 68.6% vs. 55.3% and 68.6% vs. 68.0%, respectively.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    CGAP uses pretrained-LM knowledge without external knowledge sources, finetuning, or extra learnable parameters.

    Source: §7 · p. 9

    Accuracy depends on LM size; the highest reported results use a 530B model available only via API, and generated contexts may hallucinate.

    Source: §8 · p. 9

Abstract

from arXiv · show

Closed-book question answering (QA) requires a model to directly answer an open-domain question without access to any external knowledge. Prior work on closed-book QA either directly finetunes or prompts a pretrained language model (LM) to leverage the stored knowledge. However, they do not fully exploit the parameterized knowledge. To address this issue, we propose a two-stage, closed-book QA framework which employs a coarse-to-fine approach to extract relevant knowledge and answer a question. Our approach first generates a related context for a given question by prompting a pretrained LM. We then prompt the same LM for answer prediction using the generated context and the question. Additionally, to eliminate failure caused by context uncertainty, we marginalize over generated contexts. Experimental results on three QA benchmarks show that our method significantly outperforms previous closed-book QA methods (e.g. exact matching 68.6% vs. 55.3%), and is on par with open-book methods that exploit external knowledge sources (e.g. 68.6% vs. 68.0%). Our method is able to better exploit the stored knowledge in pretrained LMs without adding extra learnable parameters or needing finetuning, and paves the way for hybrid models that integrate pretrained LMs with external knowledge.

1 Introduction

Closed-book QA methods have not fully exploited pretrained LMs’ parameterized knowledge, and have lagged open-book methods. CGAP generates question-relevant context before answer prediction, improving closed-book results and matching open-book methods on reported comparisons.

1 Introduction

  • Prior closed-book methods either finetune pretrained LMs or prompt them directly, while open-book methods retrieve contexts before answer extraction.The authors hypothesize that closed-book methods’ accuracy gap reflects insufficiently sophisticated use of parameterized knowledge.Source: §1 · p. 2
  • CGAP uses a coarse-to-fine, two-stage pipeline: prompt a pretrained LM to generate context, then prompt the same LM to predict an answer from the context and question.It does not finetune the LM; multiple contexts are generated in parallel and majority voting selects the final answer.Source: §1 · p. 2
  • Exact matching is 68.6% vs. 55.3% for previous closed-book methods, and 68.6% vs. 68.0% for open-book methods.The authors report results across three open-domain QA benchmarks.Source: §1 · p. 2
  • 11.7% to 28.5%: generated context improves standard few-shot closed-book QA, while the gap widens from 357M 28.5% vs. 530B 68.6%.The authors identify this as the first use of generated context from large pretrained LMs for open-domain QA.Source: §1 · p. 2
  • 36.3% to 45.7%: generating multiple contexts and batching them mitigates answer-prediction errors from context variability.Source: §1 · p. 2

2 Methodology

CGAP prompts a pretrained language model to generate question-relevant contexts, then uses those contexts with the question to predict answers. It generates multiple contexts and uses majority voting to reduce sensitivity to any one context.

2 Methodology

  • CGAP first generates question-relevant contexts with a pretrained LM, then predicts answers using each context alongside the question.It generates multiple contexts and selects the final answer by majority vote.Source: §2 · p. 2, Figure 2 · p. 4

2.1 Context Generation

  • DPR retrieves similar question-context examples from the support repository for few-shot context-generation prompts; these examples are prompts, not external knowledge containing the answer.The retrieved samples are used to condition the LM when generating context.Source: §2.1 · p. 3

2.2 Answer Prediction

  • For answer prediction, the same LM receives the question, a generated context, and selected question-context-answer examples, producing one answer per context.The selected examples follow the same criteria as those used for context generation.Source: §2.2 · pp. 3–4

2.3 Context Marginalization

  • CGAP approximates marginalization over unknown contexts by generating multiple contexts and choosing the most-voted answer.As the number of contexts grows, the majority-vote answer converges to the answer produced by marginalizing over all possible contexts.Source: §2.3 · p. 4

3 Experimental Setup

The experiments use three open-domain QA benchmarks and compare CGAP with closed-book and open-book baselines. They evaluate exact match and test several model sizes and prompting configurations.

3.1 Datasets

  • The evaluation uses Natural Questions, TriviaQA, and WebQuestions with the same train, validation, and test splits as prior work.NQ questions come from Google searches, TQA from trivia and quiz-league websites, and WQ from Google Suggest.Source: §3.1 · p. 4

3.2 Baselines

  • Closed-book baselines include standard few-shot prompting and fine-tuned models, while an open-book few-shot prompting method conditions answers on Google Search API documents.The fine-tuned baselines include T5 and BART variants.Source: §3.2 · p. 5

3.3 State-of-the-art Open-book QA Models

  • Open-book comparisons include FiD, which retrieves 100 Wikipedia passages with DPR, and RAG, an end-to-end retrieval-augmented generation model.FiD encodes passages independently and combines the encoder outputs before answer generation.Source: §3.3 · p. 5

3.4 Implementation Details

  • Experiments use decoder-only LMs with 357M, 1.3B, or 530B parameters at both stages, with top-p sampling for contexts and greedy decoding for answers.The prompt configuration uses 10 samples, constrained by maximum sequence length.Source: §3.4 · p. 5

3.5 Evaluation

  • Performance is evaluated using Exact Match after lowercasing predictions and references and stripping articles, punctuation, and duplicate whitespace.The evaluation follows prior open-domain QA work.Source: §3.5 · pp. 5–6

4 Results and Ablation Studies

CGAP outperforms closed-book baselines, while ablations show gains from context generation and marginalization across model sizes and context counts.

4.1 Main Results

  • CGAP outperforms existing closed-book methods, especially on NQ and TriviaQA, and beats the 530B few-shot baseline by at least 13.3 EM points on all three datasets.It also achieves the highest score on TriviaQA and comes close to open-book RAG on NQ and WebQuestions, while using 8 generated contexts versus FiD’s 100 retrieved passages.Source: §4.1 · p. 6

4.2 Ablation Studies

  • The authors omit GPT-3 API results from their comparison because those results differ from Brown et al.’s reported results.They provide details in Appendix A.Source: §4.2 · p. 6

4.2.1 Context Generation

  • With 357M models for both stages, context generation raises accuracy by 19.00% on NQ, 16.87% on TQA, and 15.26% on WQ; accuracy rises further with a larger generation model.The accuracy-gain curves are steeper with a larger answer-prediction model across all three datasets.Source: §4.2.1 · p. 6

4.2.2 Context Marginalization

  • At k=8, context marginalization improves accuracy on all three datasets across settings, with a 12.8% average EM-score gain at 530B.The ablation compares marginalization on and off across language-model sizes.Source: §4.2.2 · p. 7, Table 2 · p. 7
  • For 357M models in both stages, accuracy improves as more contexts are sampled, then plateaus as the approximation approaches marginalization over all possible contexts.Figure 4 reports the ablation on k, the number of contexts used for marginalization.Source: §4.2.2 · p. 7, Figure 4 · p. 7

5 Analysis

The analysis compares generated and retrieved contexts for answer prediction, including whether marginalizing over generated contexts improves on retrieval alternatives.

5.1 Retrieved vs. Generated Context

  • At 357M and 1.3B, top-1 retrieval gives slightly higher EM than generated context; at 530B, generated context scores higher on all three datasets.The comparison uses top-1 retrieved context versus few-shot generated context in closed-book QA.Source: §5.1 · p. 7, Table 4 · p. 8

5.2 Multiple Retrievals vs. Context Marginalization

  • Top-1 retrieved context also performs better than majority voting over multiple retrieved contexts on all three datasets.The analysis motivates the comparison by noting that generated context may contain hallucinations.Source: §5.2 · p. 8
  • Marginalization over generated contexts consistently outperforms top-1 retrieval and exceeds golden context with a 530B answer model; similar trends appear at 357M and 1.3B.The table compares generated-context marginalization, multiple retrievals, and golden context.Source: Table 5 · p. 8, §5.2 · p. 8

6 Related Works

Related work distinguishes QA systems by whether they retrieve external evidence, use stored model knowledge, or augment few-shot prompting.

Related Work

  • Open-book QA methods index an external corpus and retrieve documents to extract answer spans.The paper describes retrieve-and-read as a typical open-book approach.Source: §6 · p. 8
  • Closed-book approaches use pretrained language models without external context, either by fine-tuning on QA pairs or prompting with few-shot examples.The cited examples include T5 and BART fine-tuning and GPT-2 and GPT-3 prompting.Source: §6 · p. 8
  • Other work augments few-shot prompting with web-search information or uses chain-of-thought prompting for reasoning tasks.The paper cites Google Search API results and prompted chains of short sentences as examples.Source: §6 · p. 9

7 Conclusion

CGAP uses two-stage prompting with pretrained language models for open-domain QA without external knowledge, fine-tuning, or extra learnable parameters. Across three QA benchmarks, it outperforms prior closed-book methods, matches open-book methods, and benefits from larger models.

7 Conclusion

  • On three QA benchmarks, CGAP significantly outperforms previous closed-book QA methods and is on par with open-book methods.The conclusion reports results up to 530B parameters and says larger models boost accuracy by large margins.Source: §7 · p. 9
  • CGAP generates context and then predicts an answer by two-stage prompting with large pretrained language models.The framework does not use external knowledge sources, fine-tuning, or extra learnable parameters.Source: §7 · p. 9
  • The method is evaluated on three QA benchmarks and demonstrated with models up to 530B parameters.Source: §7 · p. 9

8 Limitations

CGAP's accuracy depends on language-model size; the reported highest-accuracy results use a 530B pretrained model available only through an API. Generated contexts may also contain hallucinated content.

8 Limitations

  • The highest accuracy reported in Table 1 uses a 530B pretrained language model accessible only via API.Source: §8 · p. 9
  • CGAP accuracy is affected by the size of the language models it uses.Source: §8 · p. 9
  • Generated contexts may contain hallucinated content.Source: §8 · p. 9

A Standard Few-shot Prompting of GPT-3

The appendix describes reproductions of GPT-3 prompting baselines and compares their results with the paper's standard few-shot baseline. The reproduced results did not match Brown et al.'s reported results, while the authors found their baseline's configuration on par with queried API results.

A Standard Few-shot Prompting of GPT-3

B Examples

The appendix provides example predictions and prompt construction, including an eight-context example that illustrates how context quality relates to answer correctness under marginalization.

B Examples

  • Tables 9 and 10 show predictions for NQ and WebQuestions using standard prompting, top-1 retrieved-context prompting, CGAP without marginalization, and CGAP, all with 530B models.The examples include three samples from the NQ and WebQuestions test sets.Source: B Examples · p. 11, Table 9 · p. 14, Table 10 · p. 14
  • In an eight-context example, inaccurate or irrelevant contexts yielded wrong answers, while relevant and factual contexts helped produce a correct answer through marginalization.Source: B Examples · p. 12, Table 11 · p. 15
  • Tables 6, 7, and 8 report standard zero-shot, one-shot, and few-shot GPT-3 prompting results for open-domain QA.Source: Table 6 · p. 13, Table 7 · p. 13, Table 8 · p. 13
  • Table 12 illustrates constructing Prompt(Q) from eight selected question-context samples for a TriviaQA question.Source: Table 12 · p. 16

Related papers

Loading 2210.06349v2…