Source-linked AI summary
Multi-Stage Document Ranking with BERT
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, Jimmy Lin
TL;DR
Document ranking needs richer neural models without unacceptable inference cost. This paper combines pointwise monoBERT and pairwise duoBERT in a multi-stage pipeline, achieving results at or comparable to the state of the art on MS MARCO and TREC CAR while characterizing latency–effectiveness tradeoffs.
Problem
The paper addresses how to apply richer BERT-based ranking models while controlling the inference latency of document ranking.
Method
The authors arrange pointwise monoBERT and pairwise duoBERT as successive stages in a multi-stage ranking architecture.
Results
On MS MARCO and TREC CAR, results are either at or comparable to the state of the art, with latency–effectiveness tradeoffs characterized across configurations.
Takeaways & Limitations
Multi-stage ranking makes richer and computationally intensive neural ranking models more practical by controlling candidate-set sizes and inference latency.
Takeaways & Limitations
Held-out MS MARCO figures are unavailable for all conditions because they require formal submissions, so only selected promising conditions were submitted.
Abstract
from arXiv · showhide
The advent of deep neural networks pre-trained via language modeling tasks has spurred a number of successful applications in natural language processing. This work explores one such popular model, BERT, in the context of document ranking. We propose two variants, called monoBERT and duoBERT, that formulate the ranking problem as pointwise and pairwise classification, respectively. These two models are arranged in a multi-stage ranking architecture to form an end-to-end search system. One major advantage of this design is the ability to trade off quality against latency by controlling the admission of candidates into each pipeline stage, and by doing so, we are able to find operating points that offer a good balance between these two competing metrics. On two large-scale datasets, MS MARCO and TREC CAR, experiments show that our model produces results that are either at or comparable to the state of the art. Ablation studies show the contributions of each component and characterize the latency/quality tradeoff space.
1 Introduction
The paper adapts BERT to document ranking through monoBERT and duoBERT, integrating them into a multi-stage pipeline. Experiments on MS MARCO and TREC CAR reach or approach state-of-the-art results while exposing latency–effectiveness tradeoffs.
- monoBERT ranks individual query–document pairs with pointwise binary classification, while duoBERT compares candidate-document pairs with pairwise classification.
- The pipeline balances candidate-set size against model complexity so richer BERT models can improve ranking without incurring their full inference latency.
- The architecture combines monoBERT and duoBERT into an end-to-end multi-stage ranking system.
- On MS MARCO and TREC CAR, the models achieve results that are either at or comparable to the state of the art.
- The study reports significant contributions from both monoBERT and duoBERT and characterizes the latency–effectiveness tradeoff of each model.
2 Background and Related Work
Multi-stage ranking repeatedly reranks shrinking candidate sets, connecting neural ranking with progressive refinement. The paper frames BERT-based ranking as a richer re-ranking stage whose quality can be traded against inference latency.
- Document ranking seeks a query-dependent ordering of k documents that maximizes metrics such as MAP or MRR.
- Multi-stage ranking begins with retrieval and successively reranks candidates passed from earlier stages until producing the final output.
- Shrinking candidate sets make increasingly richer models practical by balancing model complexity against search latency.
- The paper connects BERT-based models with multi-stage ranking to trade off result quality against inference latency.
- Neural ranking models commonly rerank candidates retrieved by bag-of-words queries, providing a starting point for BERT-based multi-stage ranking.
3 Multi-Stage Ranking with BERT
The proposed architecture uses BM25 retrieval, monoBERT pointwise reranking, and duoBERT pairwise reranking in successive stages. Candidate truncation and pairwise-score aggregation control computational cost while producing the final ranking.
- Each stage receives a ranked candidate list, returns a smaller ranked list, and passes it to the next stage, with k_n ≤ k_n−1.
- The instantiated design has three stages: H0 retrieval, H1 monoBERT reranking, and H2 duoBERT reranking.
- H0 retrieval: H0 uses BM25 over an inverted index to retrieve top-k0 candidates, prioritizing recall because later BERT stages can discard non-relevant documents.
- H1: monoBERT: monoBERT independently assigns each candidate a relevance probability and retains the top-k1 candidates for the next stage.
- H2: duoBERT: duoBERT estimates the probability that one candidate is more relevant than another, computing pairwise scores over the H1 output.
- H2: duoBERT: At inference, pairwise scores are aggregated into one score per document using SUM, BINARY, MIN, MAX, or SAMPLE before final reranking.
4 Experimental Setup
The experiments use large-scale MS MARCO and TREC CAR data to train and evaluate the BERT rerankers, with target-corpus pre-training and candidate-count variations. The setup also reflects sparse judgments and safeguards against Wikipedia test-data leakage.
- The experiments are enabled by BERT pre-training and newly available large-scale retrieval datasets.
- MS MARCO: MS MARCO passage ranking uses 8.8M passages from 3.6M web documents and evaluates with MRR@10.
- MS MARCO: MS MARCO training includes approximately 500k relevant query–document pairs and 400M non-relevant pairs, while relevance judgments are sparse.
- Training: The models receive target-corpus pre-training before fine-tuning, including an additional MS MARCO pre-training phase.
- Inference: The base inference configuration uses top-k0 = 1000 candidates for monoBERT and top-k1 = 50 candidates for duoBERT, alongside ablations varying these settings.
- TREC CAR: For TREC CAR, the authors restrict BERT pre-training to the training portion of Wikipedia to avoid exposing test documents during pre-training.
5 Results
The experiments show that monoBERT and duoBERT improve ranking effectiveness on MS MARCO and TREC CAR, while multi-stage configurations expose explicit quality–latency tradeoffs. Carefully selecting candidate cutoffs and aggregation methods yields near-maximum effectiveness with substantially fewer inferences than the most expensive settings.
- Main results: 17.5 points: Applying monoBERT to Anserini’s top 1000 candidates improves MS MARCO effectiveness over the BM25 ranking.The reported gain is relative to the Anserini BM25 baseline with k0 = 1000.
- Model comparisons: 0.5 points: duoBERTSUM improves over monoBERT alone on the held-out MS MARCO evaluation set, using 50 candidates and 50 × 49 additional BERT inferences.The improvement comes with increased latency; MAX aggregation instead makes monoBERT results worse, while BINARY is slightly better than SUM on development data.
- Model comparisons: 0.8 MRR@10 points: Target-corpus pre-training further improves the monoBERT + duoBERTSUM + TCP configuration on MS MARCO.The paper relates this result to improvements from target-corpus rather than out-of-domain pre-training.
- Main results: 19.5 MAP points: monoBERT improves over the BM25 baseline on TREC CAR, while duoBERTSUM or duoBERTBINARY adds another 2.1 points.The paper describes this as the best-known result on the dataset.
- Latency–effectiveness tradeoffs: More than half the gain achieved at k0 = 1000 is obtained with around one-fifth as many inferences, but larger k0 values show diminishing returns on both datasets.The multi-stage experiments also show that jointly varying k0 and k1 changes the tradeoff space: the most expensive configuration has much higher effectiveness than the least expensive one.
- Latency–effectiveness tradeoffs: k1 = 20 provides a good operating point: BINARY on MS MARCO and SUM on TREC CAR reach close to maximum effectiveness with only a 40% latency increase over monoBERT.By contrast, k1 = 50 with SUM or BINARY more than doubles the number of inferences over monoBERT alone.
6 Future Work and Conclusions
Neural ranking systems must reduce large corpora to manageable candidate sets, motivating multi-stage architectures that combine efficient retrieval with richer later-stage models. The paper identifies practical deployment and longer-document handling as important directions for these BERT-based pipelines.
- Large-corpus search makes inference over every document impractical, so neural models rely on bag-of-words retrieval to reduce the working set.
- Multi-stage ranking balances model complexity and latency by applying increasingly richer models to progressively smaller candidate sets.
- Representational-learning approaches remain exploratory because it is unclear whether simple similarity computations capture the complex notion of relevance.
- Jointly tuning pipeline stages and sharing earlier-stage scoring signals are proposed ways to improve end-to-end effectiveness.
- Current BERT-based models handle documents only a few sentences long, motivating evaluation of longer-document models without truncation.
- The authors conclude that multi-stage ranking architectures support practical deployment of complex, computationally intensive neural models.