Source-linked AI summary
End-to-End Open-Domain Question Answering with BERTserini
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, Jimmy Lin
TL;DR
The paper addresses open-domain question answering over large document collections, unlike systems restricted to small text inputs. It combines Anserini retrieval with a BERT reader, achieving large improvements over previous systems and high answer-span accuracy after SQuAD fine-tuning.
Problem
Open-domain QA systems must identify answers from a large corpus, whereas many QA and reading-comprehension models operate on relatively small text inputs.
Method
BERTserini combines the Anserini IR toolkit with a BERT reader that processes retrieved text segments and combines retrieval and reader scores.
Results
BERTserini achieves large improvements over previous systems, and fine-tuning pretrained BERT with SQuAD achieves high accuracy in identifying answer spans.
Takeaways & Limitations
A simple two-stage pipeline integrating retrieval and BERT can achieve strong end-to-end open-domain QA performance.
Takeaways & Limitations
The largest identified improvement opportunity is score aggregation, where weighted interpolation of BERT and Anserini scores appears insufficient; error analysis also relies on SQuAD ground truth.
Abstract
from arXiv · showhide
We demonstrate an end-to-end question answering system that integrates BERT with the open-source Anserini information retrieval toolkit. In contrast to most question answering and reading comprehension models today, which operate over small amounts of input text, our system integrates best practices from IR with a BERT-based reader to identify answers from a large corpus of Wikipedia articles in an end-to-end fashion. We report large improvements over previous results on a standard benchmark test collection, showing that fine-tuning pretrained BERT with SQuAD is sufficient to achieve high accuracy in identifying answer spans.
1 Introduction
BERTserini combines BERT with Anserini to perform end-to-end open-domain question answering over a large Wikipedia corpus. On a standard benchmark, it substantially improves over previous results while using a simple pretrained-BERT design.
- BERTserini integrates BERT with the open-source Anserini information retrieval toolkit to build an end-to-end open-domain QA system.
- Unlike models operating on small text inputs, BERTserini identifies answers directly from a large corpus of Wikipedia articles.
- Large improvements over previous work were observed on a standard benchmark test collection.
- Fine-tuning pretrained BERT with SQuAD was sufficient to achieve high accuracy in identifying answer spans.
- The architecture’s simplicity is presented as a major feature, and BERTserini was deployed as an interactive chatbot.
2 Background and Related Work
Earlier QA systems emphasized progressively finer-grained retrieval and answer extraction, while many modern benchmarks largely assume candidate sentences are already supplied. End-to-end QA restores large-corpus retrieval by combining term-based retrieval with neural reading.
- Traditional QA pipelines retrieve documents, rank passages, and extract answer spans from increasingly finer-grained text segments.
- Many popular QA benchmarks are answer-selection tasks in which systems choose from candidate sentences rather than retrieve from a large corpus.
- Some QA formulations effectively eschew retrieval because they provide a single document from which to extract answers.
- End-to-end QA begins with a large document corpus and therefore requires term-based retrieval to restrict the text processed by current models.
- BERTserini combines pretrained BERT with Anserini, an IR toolkit built on Lucene and designed to codify current IR best practices.
3 System Architecture
BERTserini uses a two-module architecture: Anserini retrieves text segments from Wikipedia, and a BERT reader identifies answer spans. The system supports multiple retrieval granularities and combines retrieval and reader scores.
- System Architecture: The architecture consists of an Anserini retriever that selects answer-containing text segments and a BERT reader that identifies an answer span.
- System Architecture: The system uses the same December 2016 Wikipedia corpus as prior work, comprising 5.08M articles.
- 3.1 Anserini Retriever: A single-stage retriever directly identifies segments for the reader, while indexing experiments vary the retrieval unit.
- 3.1 Anserini Retriever: Article indexing treats each of the 5.08M Wikipedia articles as one retrieval unit.
- 3.1 Anserini Retriever: Paragraph and sentence indexing treat 29.5M paragraphs and 79.5M sentences, respectively, as retrieval units.
- 3.1 Anserini Retriever: At inference time, Anserini retrieves k segments using the question as a bag-of-words query and BM25 as the ranking function.
- 3.2 BERT Reader: The BERT reader is applied according to segment granularity, selects the best text span, and produces a score.
4 Experimental Results
Experiments evaluate BERTserini across retrieval granularities and varying numbers of retrieved paragraphs, showing that paragraph retrieval provides a strong effectiveness setting while reader scoring limits gains from larger candidate pools.
- The evaluation uses exact match, token-level F1, and retrieval recall on SQuAD development questions, following Chen et al.’s methodology.
- BERTserini compares article retrieval at k = 5, paragraph retrieval at k = 29, and sentence retrieval at k = 78 while considering approximately equal text amounts.Paragraphs average 2.7 sentences, while articles average 5.8 paragraphs.
- Paragraph retrieval substantially outperforms article retrieval and sentence retrieval because articles add distractors while sentences often lack sufficient answer context.
- As k increases, recall continues rising toward k = 100, whereas top exact match saturates around k = 10 and top-k exact match saturates later.
- At k = 100, Anserini retrieves at least one relevant paragraph around 86% of the time, indicating retrieval is not the current implementation’s main bottleneck.
- The largest improvement opportunity is score aggregation: BERT identifies some correct answers but fails to rank them highest, suggesting weighted BERT–Anserini interpolation is insufficient.
5 Demonstration
BERTserini is deployed through Slackbot and RSVP.ai’s platform using a shared backend, with paragraph retrieval and sentence-level answer presentation. The demonstration chooses k = 10 to balance response cost and quality.
- BERTserini is available as both a Slackbot and an RSVP.ai intelligent-platform service, while both interfaces use the same backend.
- The interface indexes paragraphs but returns only the sentence containing the BERT-identified answer, with the answer span highlighted.
- The demonstration sets paragraph retrieval to k = 10 because it represents a good cost/quality tradeoff despite not maximizing accuracy.
- Average processing time is 0.5s per question for Anserini retrieval on CPU and 0.18s for BERT processing on GPU.
6 Conclusion
The paper introduces BERTserini as an end-to-end open-domain QA system combining BERT with Anserini. Its results improve over previous systems, while error analysis identifies retrieval, extraction, and aggregation as areas for further work.
- BERTserini integrates BERT and the Anserini IR toolkit in a simple two-stage end-to-end open-domain QA pipeline.
- The system achieves large improvements over previous systems.
- Error analysis identifies retrieval, answer extraction, and answer aggregation as ongoing improvement areas, alongside planned expansion of multilingual capabilities.