Source-linked AI summary
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Gautier Izacard, Edouard Grave
TL;DR
The paper asks whether generative open-domain QA can reduce reliance on billion-parameter internal knowledge by using retrieved evidence. It retrieves passages and generates answers with Fusion-in-Decoder, achieving state-of-the-art results on Natural Questions and TriviaQA. Performance improves substantially as more passages are retrieved, supporting generative models’ ability to combine multi-passage evidence.
Problem
Generative open-domain QA can be competitive without external knowledge, but requires billions of parameters because information must be stored in model weights, making training and querying expensive.
Method
The method retrieves passages with sparse or dense representations and uses a sequence-to-sequence model that independently encodes passages before jointly fusing them in the decoder.
Results
The approach achieves state-of-the-art results on Natural Questions and TriviaQA, with performance improving as the number of retrieved passages increases.
Takeaways & Limitations
The results provide evidence that generative models are effective at aggregating and combining evidence from multiple passages.
Abstract
from arXiv · showhide
Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge. While promising, this approach requires to use models with billions of parameters, which are expensive to train and query. In this paper, we investigate how much these models can benefit from retrieving text passages, potentially containing evidence. We obtain state-of-the-art results on the Natural Questions and TriviaQA open benchmarks. Interestingly, we observe that the performance of this method significantly improves when increasing the number of retrieved passages. This is evidence that generative models are good at aggregating and combining evidence from multiple passages.
1 Introduction
The paper investigates whether generative open-domain question answering can benefit from retrieved external passages, addressing the cost of storing all knowledge in billion-parameter model weights. It proposes retrieving passages before generation and reports state-of-the-art results while observing gains from more passages.
- Generative open-domain QA can be competitive without external knowledge, but storing information in model weights requires billions of parameters and makes training and querying expensive.
- Performance keeps improving as the number of retrieved passages increases, providing evidence that generative models combine evidence from multiple passages effectively.
- The proposed approach first retrieves supporting passages using sparse or dense representations, then generates an answer conditioned on the question and retrieved text.
- The method sets new state-of-the-art results on the TriviaQA and NaturalQuestions benchmarks.
2 Related work
Related work covers extractive retrieval-based QA, advances in sparse and dense passage retrieval, and generative models for tasks requiring abstractive answers. It positions this paper at the intersection of retrieval and generation for open-domain QA.
- Earlier extractive QA systems retrieved support documents and predicted answer spans from them.
- Prior work proposed global normalization, hard expectation-maximization, and confidence-coverage methods for settings with noisy supervision or evidence across paragraphs.
- Extractive approaches use sparse TF/IDF or dense embeddings for retrieval and contextualized representations such as ELMo or BERT for answer extraction.
- Dense-representation retrieval with approximate nearest neighbors became competitive with traditional retrieval approaches.
- Generative QA was previously studied mainly for datasets whose answers do not correspond to spans in support documents, while large pretrained generative models were also applied without external knowledge.
3 Method
Fusion-in-Decoder retrieves support passages and processes them with a sequence-to-sequence model. Passages are encoded independently for scalability, while the decoder jointly attends to their representations to fuse evidence.
- Retrieval: The method retrieves support passages with either BM25 or DPR before processing them with a sequence-to-sequence model.
- Retrieval: BM25 ranks bag-of-words passages using term and inverse document frequencies, whereas DPR ranks dense passage and question representations by dot product.
- Reading: The generative model independently encodes each retrieved passage and its title together with the question, then generates the answer.
- Reading: Fusion-in-Decoder performs evidence fusion only in the decoder by attending over the concatenated representations of all retrieved passages.
- Design rationale: Independent passage encoding makes computation grow linearly rather than quadratically with the number of passages, while joint decoder processing supports evidence aggregation.
4 Experiments
The experiments evaluate Fusion-in-Decoder on open-domain question answering benchmarks and examine how retrieval and training passage counts affect performance. The method improves over existing approaches, benefits from more retrieved passages, and can reduce training cost through limited fine-tuning.
- Experimental setup: Fusion-in-Decoder is evaluated on NaturalQuestions, TriviaQA, and SQuAD using exact match on validation or development sets.The experiments use dataset-specific retrieval, 100 retrieved passages by default, and greedy decoding.
- Comparison to state-of-the-art: The evaluation compares Fusion-in-Decoder with existing open-domain question-answering approaches on benchmark performance.The comparison includes generative and extractive systems.
- Comparison to state-of-the-art: 44.1% accuracy is obtained on NaturalQuestions with a 770M-parameter model plus BM25-retrieved Wikipedia, versus 36.6% for closed-book T5 with 11B parameters.The paper states that both approaches use roughly the same amount of memory to store information.
- Scaling with number of passages: 6% improvement on TriviaQA and 3.5% improvement on NaturalQuestions result when retrieved passages increase from 10 to 100.Most extractive models are reported to peak around 10 to 20 passages.
- Impact of the number of training passages: Reducing the number of training passages decreases accuracy, but fine-tuning with 100 passages for 1000 steps reaches 46.0 EM on NaturalQuestions using 147 GPU hours.The reported alternative uses substantially fewer resources than training on 100 passages for the full procedure, which requires 425 GPU hours.
5 Conclusion
The paper presents a simple open-domain question-answering approach that retrieves support passages before processing them with a generative model. It reports competitiveness with existing methods and scaling with the number of retrieved passages, while identifying efficiency and end-to-end retrieval integration as future work.
- The approach retrieves support passages before processing them with a generative model and is competitive with existing methods.The conclusion characterizes the approach as simple and scalable with the number of retrieved passages.
- Future work targets greater efficiency for many support passages and end-to-end learning that integrates retrieval into the model.The paper also plans to learn the whole system end-to-end.