Source-linked AI summary
Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline
Luyu Gao, Zhuyun Dai, Jamie Callan
TL;DR
Existing BERT rerankers do not fully exploit improved candidate rankings because stronger retrievers produce harder negatives and confounding matches. The paper introduces Localized Contrastive Estimation, which samples negatives from the target retriever and trains with a contrastive objective. LCE significantly outperforms vanilla training with the same language model and supports stronger multi-stage retrieval pipelines.
Problem
Popular BERT rerankers cannot fully exploit improved initial rankings, where top candidates contain harder negatives and confounding characteristics.
Method
LCE localizes negative sampling to the target retriever’s top results and uses contrastive loss to penalize matching with confounding characteristics.
Results
LCE significantly outperforms the vanilla method trained with the same language model, with gains increasing as the initial retriever becomes stronger.
Takeaways & Limitations
Better learning methods can improve existing deep language models without requiring a more advanced language model, while localized negatives and multiple contrastive negatives are both critical.
Abstract
from arXiv · showhide
Pre-trained deep language models~(LM) have advanced the state-of-the-art of text retrieval. Rerankers fine-tuned from deep LM estimates candidate relevance based on rich contextualized matching signals. Meanwhile, deep LMs can also be leveraged to improve search index, building retrievers with better recall. One would expect a straightforward combination of both in a pipeline to have additive performance gain. In this paper, we discover otherwise and that popular reranker cannot fully exploit the improved retrieval result. We, therefore, propose a Localized Contrastive Estimation (LCE) for training rerankers and demonstrate it significantly improves deep two-stage models.
1 Introduction
Modern retrieval pipelines combine efficient first-stage retrievers with contextual neural rerankers, but stronger candidate lists can make false positives harder to distinguish. The paper introduces LCE to train rerankers for these improved candidate pools.
- Problem: Stronger initial rankings do not guarantee better final rankings because false positives may share confounding characteristics with relevant documents.The reranker must handle the top portion of retriever results without relying on those confounding features.
- Approach: Localized Contrastive Estimation (LCE) samples negative documents from the target retriever’s top results and uses a contrastive loss.The loss penalizes signals generated from confounding characteristics, preventing reranker collapse.
- Result: LCE achieves significantly higher accuracy than the vanilla method with the same BERT model and without training or inference overhead.The experiments use the MSMARCO document ranking dataset.
2 Background
Multi-stage retrieval separates fast full-corpus retrieval from slower, more accurate reranking. Although BERT rerankers improve retrieval, the paper questions whether binary-classification training fully exploits harder candidates produced by deep retrievers.
- Pipeline Background: Pipelined retrieval balances efficiency and effectiveness by using fast models for corpus-wide search and slower models for top-candidate refinement.BM25 retrieves efficiently, while BERT provides more accurate reranking.
- Retriever Improvements: Heuristic retrievers rely on exact-match signals and document statistics, while deep language models can improve index term weights or add probable query terms.These changes aim to improve initial retrieval quality.
- Reranker Training: Binary-classification BERT rerankers strongly improve earlier methods, but their sufficiency is questioned for deep retrievers that produce harder negatives.The concern is whether this training paradigm realizes BERT’s full potential in such pipelines.
- Contrastive Learning: Contrastive objectives incorporate negatives directly and train models to assign higher probability to positives, but few prior works applied them to deep LM rerankers.The negative instances help prevent model collapse.
3 Methodologies
The methodology replaces independent binary classification with LCE, which combines target-retriever-localized negatives and contrastive learning. This focuses training on difficult top-ranked candidates while discouraging reliance on confounding matches.
- Vanilla Method: The vanilla method independently trains BERT on query-document pairs using binary cross entropy with positive or negative labels.The scoring function uses BERT’s [CLS] representation projected by a vector.
- Motivation: Vanilla training treats reranking as general binary classification even though top retriever results contain confounding signatures.The reranker is expected to handle these candidates and avoid collapsing onto confounding matches.
- Localized Negatives: LCE samples non-relevant documents from the target retriever’s top m results to form query-specific negative training sets.The training set is rebuilt for the specific target retriever, and the sampled documents are grouped with one relevant positive.
- Contrastive Loss: LCE applies a contrastive loss to groups containing one relevant document and sampled negatives, conditioning loss and gradients on retrieved negatives.This design helps prevent collapse onto simple confounding matchings.
- LCE Design: Compared with standard NCE, LCE localizes negatives using the target retriever and focuses learning on top-ranked candidates rather than randomly sampled noisy negatives.The method combines localized sampling with an expressive multi-negative objective.
4 Experiment Methodologies
The experiments use MSMARCO document ranking with multiple first-stage retrievers and BERT-based rerankers. Candidate lists are limited to the top 100 documents for reranking.
- Dataset: MSMARCO contains 3 million documents, and models are trained on 0.37M training pairs with the dev set used for analysis.Each document has title, URL, and body fields averaging around 900 words.
- Initial Retrievers: The study evaluates Indri, un-tuned BM25, tuned BM25, and HDCT as initial retrievers.HDCT improves document indices using BERT-estimated term weights, and each retriever supplies top-100 candidate lists.
- Implementation: Rerankers receive concatenated document title, URL, and the first 512 tokens of the body, with negatives sampled from the target retriever’s top 100 documents.Training uses BERT implementations with mixed precision and two epochs.
5 Document Ranking Performance
On MSMARCO document-ranking queries, LCE improves BERT reranking across first-stage retrievers and achieves strong leaderboard performance, including a first-place ensemble result.
- LCE significantly improves reranking performance across all evaluated first-stage retrievers on the development set.
- LCE gains increase as the first-stage retriever becomes stronger, indicating better handling of harder negatives and more complex candidate-list matching.
- The HDCT+LCE pipeline outperformed the vanilla baseline by a large margin on leaderboard queries.
- An ensemble of LCE-trained BERT, RoBERTa, and ELECTRA rerankers over HDCT top-100 candidates achieved first place and state-of-the-art performance.
6 Analysis
The analysis examines LCE sample size and negative localization, finding benefits from multiple negatives and from matching training negatives to the target retriever.
- Effect of LCE Sample Size: Increasing LCE sample size from 2 to 4 produces a big improvement, while larger sizes generate additional gains.Size 2 uses one positive and one negative; size 4 uses three negatives.
- Influence of Negative Localization: Negative localization benefits both LCE and vanilla reranking methods.
- Influence of Negative Localization: Vanilla reranker performance drops severely when training negatives come from a weaker retriever than the test retriever.
- Influence of Negative Localization: LCE-trained rerankers also benefit from localization and retain adaptability when the test retriever differs from the training retriever.
7 Conclusion
The paper addresses the inability of previous BERT rerankers to fully exploit improved initial rankings with Localized Contrastive Estimation. Experiments show that LCE outperforms the vanilla method, while its success depends on localized negatives and multiple contrastive negatives.
- LCE targets improved initial retrieval by localizing training negatives to the target retriever and penalizing confounding matching characteristics with contrastive loss.
- LCE significantly outperforms the vanilla reranking method when using the same language model.
- Negative localization and an expressive loss with multiple contrastive negatives are both critical to LCE’s success.
- The results indicate that better learning methods can improve existing deep language models without adopting a more advanced model.