Source-linked AI summary
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering
Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, Bing Xiang
TL;DR
Prior open-domain QA systems treat passages for the same question independently, which can make answer scores incomparable across passages. The paper introduces multi-passage BERT with global normalization and passage ranking, and reports the strongest results across four standard benchmarks.
Problem
Prior systems process passages for the same question as independent training instances, potentially producing incomparable answer scores across passages.
Method
Multi-passage BERT globally normalizes answer scores across passages for each question and uses a passage ranker to select high-quality passages.
Results
Multi-passage BERT outperforms all state-of-the-art models across four standard benchmarks, including 21.4% EM improvement over non-BERT models on OpenSQuAD.
Takeaways & Limitations
The approach benefits from using more passages, 100-word sliding-window passages, and passage ranking for open-domain QA.
Takeaways & Limitations
The OpenSQuAD test set has more annotated answers per question than the development set, making its results significantly better than the development results.
Abstract
from arXiv · showhide
BERT model has been successfully applied to open-domain QA tasks. However, previous work trains BERT by viewing passages corresponding to the same question as independent training instances, which may cause incomparable scores for answers from different passages. To tackle this issue, we propose a multi-passage BERT model to globally normalize answer scores across all passages of the same question, and this change enables our QA model find better answers by utilizing more passages. In addition, we find that splitting articles into passages with the length of 100 words by sliding window improves performance by 4%. By leveraging a passage ranker to select high-quality passages, multi-passage BERT gains additional 2%. Experiments on four standard benchmarks showed that our multi-passage BERT outperforms all state-of-the-art models on all benchmarks. In particular, on the OpenSQuAD dataset, our model gains 21.4% EM and 21.5% $F_1$ over all non-BERT models, and 5.8% EM and 6.5% $F_1$ over BERT-based models.
1 Introduction
Open-domain QA must locate answers across massive article collections, unlike reading comprehension, which supplies a passage. Prior BERT-based systems process passages independently, motivating global score normalization and improved passage handling.
- Open-domain QA finds answers in massive article collections, whereas reading comprehension retrieves answers from a given passage.
- Prior systems retrieve multiple passages but train and evaluate each passage independently before selecting the highest-scoring span.
- Independent passage processing may produce incomparable answer scores across passages, motivating global normalization for passages belonging to one question.
- 4% improvement comes from splitting articles into 100-word passages with a sliding window.
- 2% additional improvement comes from using a BERT-based passage ranker, while experiments report gains across four standard benchmarks.
2 Model
The model extends BERT-RC from independently normalized passages to globally normalized answer scores across passages for the same question. A passage ranker further selects and weights high-quality passages before answer selection.
- BERT-RC encodes each question-passage sequence and independently predicts start and end probabilities for answer spans.
- Multi-passage BERT globally normalizes word-position scores across all passages corresponding to the same question.
- All passages are processed independently until normalization, after which one softmax covers word positions from every passage.
- A BERT-based passage ranker reranks retrieved passages and selects high-quality passages for multi-passage BERT.
- The final answer-span score multiplies the passage score by the predicted start and end probabilities.
3 Experiments
Experiments evaluate the approach on four open-domain QA benchmarks and analyze passage granularity, ranking, global normalization, and inter-sentence matching. Results show benefits from sliding windows, passage ranking, and global normalization, with multi-passage BERT leading across benchmarks.
- Experiments: Experiments cover OpenSQuAD, TriviaQA, and two other open-domain QA datasets, using EM and F1 as evaluation metrics.
- Model Analysis: 4.7% EM and 4.1% F1 improvements result from overlapping 100-word passages with a 50-word stride.
- Model Analysis: The passage ranker reranks 100 retrieved passages and supplies the top 30 passages to multi-passage BERT.
- Model Analysis: Increasing passages significantly reduces BERT-RC performance without passage scores, while multi-passage BERT improves initially and then flattens after more than 10 passages.
- Comparison with State-of-the-art Models: Multi-passage BERT outperforms all state-of-the-art models across all four benchmarks and consistently exceeds the matched BERT-RC model.
- Comparison with State-of-the-art Models: On OpenSQuAD, improvements are 21.4% EM and 21.5% F1 over non-BERT models, and 5.8% EM and 6.5% F1 over BERT-based models.
4 Conclusion
The paper proposes multi-passage BERT with two performance-enhancing techniques and reports that it outperforms state-of-the-art models on four standard benchmarks. It also identifies future work on inter-correlation among passages and notes a difference between its OpenSQuAD test and development settings.
- Multi-passage BERT globally normalizes answer scores across passages corresponding to the same question.
- 100-word sliding-window passages and a passage ranker selecting high-quality passages are identified as effective performance improvements.
- OpenSQuAD test results are higher than development results because the test questions contain more annotated answers and the development questions contain one gold-standard answer.
- With these techniques, multi-passage BERT outperforms all state-of-the-art models on four standard benchmarks.
- Future work will consider inter-correlation among passages for open-domain question answering.