Source-linked AI summary
Passage Re-ranking with BERT
Rodrigo Nogueira, Kyunghyun Cho
TL;DR
Passage ranking had limited large-scale datasets, constraining neural ranking models’ success relative to classical methods. The paper adapts BERT for passage re-ranking using MS MARCO and reports state-of-the-art results on both MS MARCO and TREC-CAR.
Problem
Limited large-scale passage-ranking datasets hindered neural ranking models’ success relative to classical information-retrieval techniques.
Method
The paper re-purposes BERT as the computationally intensive re-ranker that scores candidate passages for a query.
Results
The BERT-based models surpass previous state-of-the-art models by a large margin on both TREC-CAR and MS MARCO passage re-ranking.
Takeaways & Limitations
The experiments establish BERT passage re-ranking as state of the art on two tasks, with code publicly available for reproduction.
Takeaways & Limitations
TREC-CAR evaluation uses automatic relevance annotations because manual test-set annotations cover only the top five retrieved passages.
Abstract
from arXiv · showhide
Recently, neural models pretrained on a language modeling task, such as ELMo (Peters et al., 2017), OpenAI GPT (Radford et al., 2018), and BERT (Devlin et al., 2018), have achieved impressive results on various natural language processing tasks such as question-answering and natural language inference. In this paper, we describe a simple re-implementation of BERT for query-based passage re-ranking. Our system is the state of the art on the TREC-CAR dataset and the top entry in the leaderboard of the MS MARCO passage retrieval task, outperforming the previous state of the art by 27% (relative) in MRR@10. The code to reproduce our results is available at https://github.com/nyu-dl/dl4marco-bert
1 INTRODUCTION
Passage ranking lacked the large datasets and resources that had supported progress in related reading-comprehension tasks. MS MARCO and BERT provide these ingredients, enabling a BERT-based passage re-ranker.
- Passage-ranking neural models had limited success partly because few large passage-ranking datasets were available.
- MS MARCO supplies one million real-user queries with human-annotated relevant passages for passage ranking.
- The paper re-purposes BERT as a passage re-ranker and reports state-of-the-art results on the MS MARCO passage re-ranking task.
2 PASSAGE RE-RANKING WITH BERT
The system uses BERT to score candidate passages for a query and ranks them by predicted relevance. It fine-tunes a pretrained model with cross-entropy over relevant and non-relevant passages retrieved by BM25.
- Passage re-ranking is the second stage of a question-answering pipeline, scoring and ordering initially retrieved candidate documents.
- BERT receives the query as sentence A and the passage as sentence B, using a [CLS]-based binary classifier to predict relevance.
- The final passage list is ranked by independently computed relevance probabilities.
- The model starts from pretrained BERT and is fine-tuned with cross-entropy over relevant and non-relevant passages.
3 EXPERIMENTS
Experiments evaluate BERT-based passage re-ranking on MS MARCO and TREC-CAR under BM25-retrieved candidate settings. The models achieve large-margin gains over prior state of the art, despite using only fractions of available training data.
- Datasets: The models are trained and evaluated on the MS MARCO and TREC-CAR passage-ranking datasets.
- MS MARCO: MS MARCO provides approximately 400M training query-passage tuples, while development queries are paired with BM25’s top 1,000 passages.
- TREC-CAR: TREC-CAR uses Wikipedia section-title queries, section paragraphs as relevant passages, and approximately 2.3M training queries across four predefined folds.
- Evaluation: TREC-CAR evaluation uses automatic relevance annotations because manual annotations cover only the top five submitted passages.
- Evaluation: The TREC-CAR setup avoids Wikipedia test-data leakage by pretraining the BERT re-ranker only on the training portion of Wikipedia.
- Results: The BERT-based models surpass previous state-of-the-art models by a large margin on both passage-ranking tasks despite training on fractions of available data.
- Results: 1.4 MRR@10 points: BERTLARGE trained on 100k question-passage pairs exceeds previous state-of-the-art IR-NET on MS MARCO.
4 CONCLUSION
The paper presents a simple BERT adaptation for passage re-ranking that reaches state-of-the-art performance on TREC-CAR and MS MARCO, with reproducible code released publicly.
- The BERT passage re-ranker becomes state of the art on both TREC-CAR and MS MARCO.
- The authors make code for reproducing the experiments publicly available.