Source-linked AI summary
Generation-Augmented Retrieval for Open-domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Weizhu Chen
TL;DR
Open-domain QA needs effective retrieval, but sparse methods miss semantically related passages and prior query reformulation approaches may require external resources or costly supervision. GAR generates heuristically discovered contexts from a pretrained language model and appends them to questions for retrieval. Across Natural Questions and TriviaQA, GAR with BM25 matches or outperforms dense retrieval, fusion with DPR improves results, and GAR delivers state-of-the-art extractive QA performance.
Problem
Sparse retrieval is efficient but misses relevant passages without lexical overlap, while prior generation-based reformulation methods may require task-specific data, external resources, or inefficient reinforcement learning.
Method
GAR expands questions with heuristically discovered contexts generated by a pretrained language model, using targets such as answers, answer-containing sentences, and passage titles.
Results
GAR with BM25 achieves comparable or better performance than dense retrieval, fusion with DPR improves results, and GAR achieves state-of-the-art extractive OpenQA performance on NQ and TriviaQA.
Takeaways & Limitations
Generated contexts enrich query semantics, while combining sparse GAR with dense DPR exploits complementary representations for better retrieval and QA.
Takeaways & Limitations
Future work identifies computational-cost reduction, joint generation-retrieval optimization, and broader hyperparameter tuning as open improvement opportunities.
Abstract
from arXiv · showhide
We propose Generation-Augmented Retrieval (GAR) for answering open-domain questions, which augments a query through text generation of heuristically discovered relevant contexts without external resources as supervision. We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. Moreover, as sparse and dense representations are often complementary, GAR can be easily combined with DPR to achieve even better performance. GAR achieves state-of-the-art performance on Natural Questions and TriviaQA datasets under the extractive QA setup when equipped with an extractive reader, and consistently outperforms other retrieval methods when the same generative reader is used.
1 Introduction
Open-domain QA depends on retrieval over large document collections, but sparse methods miss semantic matches and prior generation-based reformulation methods often require task-specific resources or costly supervision. GAR expands queries with generated relevant contexts, enabling sparse retrieval to match or exceed dense methods and improve end-to-end QA.
- Open-domain QA retrieves a small document subset before a reader extracts or generates an answer, because examining entire collections is infeasible.
- Sparse retrieval is efficient but cannot reliably retrieve relevant passages without lexical overlap, whereas dense retrieval captures semantic similarity beyond lexical matching.
- Prior generation-based query reformulation methods require task-specific data, external resources, or inefficient reinforcement-learning training.
- GAR expands rather than rewrites queries with heuristically discovered contexts generated by a pretrained language model, without external resources or downstream RL feedback.
- GAR with BM25 achieves comparable or better retrieval performance than dense methods, while fused GAR and DPR results consistently improve performance further.
- GAR achieves EM=41.8 (43.8 for GAR+) on NQ with an extractive reader and EM=38.1 (45.3 for GAR+) with a generative reader, while Trivia scores are 62.7 and 62.2 respectively.
2 Related Work
GAR extends conventional query expansion by generating relevant contexts from pretrained language models, retaining sparse retrieval’s simplicity while improving its effectiveness for OpenQA.
- GAR resembles pseudo-relevance-feedback query expansion because both add relevant contexts without external supervision.
- Unlike conventional expansion, GAR draws knowledge from pretrained language models and uses freely accessible in-domain targets with standard seq2seq learning.
- GAR enriches question-only sparse retrieval with relevant contexts, helping BM25 achieve comparable or better performance than dense methods while remaining simple and efficient.
3 Generation-Augmented Retrieval
GAR generates diverse relevant contexts, appends them to questions, and retrieves with BM25; separate result fusion improves coverage while reducing the impact of noisy generations.
- GAR augments questions with heuristically discovered contexts to retrieve more relevant passages in both quantity and quality.
- Fusing retrieval results from multiple generation targets consistently improves retrieval accuracy.
- Generation targets include answers, sentences containing answers, and titles of passages containing answers.
- Generated answers can aid retrieval even when partially or entirely incorrect, provided they remain relevant to passages containing the correct answers.
- Although hallucinated contexts may introduce retrieval noise, experiments show overall gains in retrieval and QA performance, with fusion further alleviating distraction.
- GAR appends generated contexts to the original question because using generated answers alone can retrieve irrelevant or false-positive passages.
- BM25 demonstrates that sparse GAR can match or exceed state-of-the-art dense methods while remaining lightweight and efficient.
4 OpenQA with GAR
GAR is evaluated with extractive and generative readers using retrieval outputs under reader designs aligned with established baselines. The extractive reader follows DPR, while the generative reader maps questions and retrieved passages to answers.
- GAR is evaluated end-to-end with both extractive and generative readers, using major-baseline reader designs for fair comparison.
- 4.1 Extractive Reader: The extractive reader largely follows DPR’s design and selects an answer span using passage- and span-relevance scores.
- 4.2 Generative Reader: The generative reader uses a seq2seq framework whose input concatenates the question with top-retrieved passages and whose output is the answer.
- 4.2 Generative Reader: The generative implementation uses BART-large and concatenates inputs up to a 1,024-token limit, averaging 7.8 passages.
5 Experiment Setup
The experiments use open-domain versions of Natural Questions and TriviaQA, evaluating retrieval and reader performance with established metrics. GAR is compared mainly with BM25, DPR, and RM3, using Anserini for retrieval and BART-large for query-context generation.
- 5.1 Datasets: Experiments use the open-domain versions of Natural Questions and TriviaQA, with dataset statistics reported by split, question and answer length, and answers per question.The experiments evaluate two popular QA benchmarks.
- 5.2 Evaluation Metrics: Top-k retrieval accuracy measures whether retrieved passages contain an answer span, while Exact Match measures exact agreement with a normalized ground-truth answer.Top-k retrieval accuracy upper-bounds the questions answerable by an extractive reader.
- 5.3 Compared Methods: Passage retrieval compares BM25 and DPR as representative sparse and dense OpenQA methods, respectively.The study also compares query expansion with the unsupervised RM3 baseline.
- 5.3 Compared Methods: RM3 provides a fair unsupervised query-expansion comparison because it does not require external resources.Other recent query-reformulation methods may require task-specific data, external supervision, or inefficient training.
- 5.3 Compared Methods: Anserini supplies BM25 and GAR retrieval with default parameters, while BART-large generates GAR query contexts.RM3 uses grid search for its query-expansion settings.
6 Experiment Results
GAR improves retrieval and end-to-end open-domain QA across Natural Questions and TriviaQA, while remaining lightweight and benefiting from diverse contexts and combination with DPR.
- 6.1 Query Context Generation: GAR’s generated contexts are meaningful and increase lexical overlap with ground-truth passages over the original query.On NQ, original-query ROUGE-1/2/L F1 is 6.00/2.36/5.01, compared with 13.21/6.99/10.27 for sentence-augmented queries.
- 6.1 Query Context Generation: Different generated contexts are complementary: fusing their retrieved passages consistently improves retrieval, even when individual contexts perform similarly.Case studies report that correct information in one context can offset wrong answers in another.
- 6.3 Passage Reading with GAR: Extractive GAR achieves state-of-the-art end-to-end performance on both NQ and Trivia while being more lightweight and computationally efficient.Generative GAR performs less well on NQ but outperforms most generative methods on Trivia; GAR+ improves over both DPR-based baselines.
- 6.3 Passage Reading with GAR: GAR improves model generalizability on questions without training-set overlap, with GAR+ outperforming DPR extractively and RAG generatively in the No Overlap category.GAR+ also achieves the best EM in the Answer Overlap Only category.
- 6.3 Passage Reading with GAR: GAR requires 3–6 hours to train on one GPU and about 30 minutes to build Wikipedia indices, compared with substantially higher reported costs for DPR and REALM.Its main overhead is generating contexts and retrieving with longer queries.
7 Conclusion
The paper proposes Generation-Augmented Retrieval and shows that generated contexts enrich query semantics and improve retrieval accuracy. GAR performs similarly to or better than dense-representation methods, combines with dense retrieval for stronger results, and achieves strong end-to-end OpenQA performance.
- GAR generates relevant contexts with pretrained language models without external supervision to augment queries for retrieval.The approach enriches query semantics through generated contexts rather than rewriting the original query.
- GAR with sparse representations performs similarly to or better than state-of-the-art methods using dense representations of original queries.
- Combining GAR with dense representations produces even better results than using GAR alone.
- GAR achieves state-of-the-art end-to-end performance for extractive OpenQA and competitive performance under the generative setup.
8 Future Extensions
Future work focuses on reducing generation cost, improving context generation and retrieval fusion, and jointly optimizing the generator and retriever. The paper also identifies applications beyond OpenQA in other text-matching tasks.
- Multi-task learning could reduce the computational cost of query-context generation.
- Future context-generation work could test whether different contexts mutually enhance one another and sample multiple contexts instead of using greedy decoding.
- Retrieval could benefit from fusion methods using both passage rankings and scores, as well as joint or iterative generator-retriever optimization.
- GAR may extend to text-matching tasks such as conversation utterance selection and information retrieval.For supervised tasks, the default generation target is always available.
A More Analysis of Retrieval Performance
GAR improves sparse retrieval and matches or exceeds DPR at sufficiently large retrieval depths on NQ and Trivia. Its gains are especially relevant where lexical matching and question paraphrasing limit BM25.
- GAR performs comparably or better than DPR when k ≥100 on NQ and k ≥5 on Trivia.
- GAR improves BM25 and achieves comparable or better top-k retrieval accuracy than DPR on the NQ test set when k ≥100.
- GAR achieves better top-k retrieval accuracy than DPR on the Trivia test set when k ≥5.
- Question-answer overlap analysis: GAR helps BM25 bridge its question-paraphrasing gap and can outperform DPR in the Question Overlap category.
- Question-answer overlap analysis: GAR consistently improves over BM25 across other question-answer overlap categories, while GAR+ also outperforms DPR.