Source-linked AI summary
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, Haifeng Wang
TL;DR
RocketQA addresses challenges in training dual-encoder retrievers for open-domain question answering, including training–inference discrepancy, unlabeled positives, and limited data. It combines three training strategies and significantly outperforms previous state-of-the-art models on MSMARCO and Natural Questions while improving end-to-end QA performance.
Problem
Dense passage retrieval is important for open-domain QA, but effectively training dual-encoders is difficult because sparse retrieval can suffer term mismatch and training faces major challenges.
Method
RocketQA combines cross-batch negatives, denoised hard negatives, and cross-encoder-labeled data augmentation to train dual-encoder retrievers.
Results
RocketQA significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions and improves end-to-end QA performance.
Takeaways & Limitations
The three optimization strategies are effective for improving dense passage retrieval, with the retriever supporting improved end-to-end QA performance.
Takeaways & Limitations
The training objective uses a small number of negative passages because considering many negatives is computationally infeasible.
Abstract
from arXiv · showhide
In open-domain question answering, dense passage retrieval has become a new paradigm to retrieve relevant passages for finding answers. Typically, the dual-encoder architecture is adopted to learn dense representations of questions and passages for semantic matching. However, it is difficult to effectively train a dual-encoder due to the challenges including the discrepancy between training and inference, the existence of unlabeled positives and limited training data. To address these challenges, we propose an optimized training approach, called RocketQA, to improving dense passage retrieval. We make three major technical contributions in RocketQA, namely cross-batch negatives, denoised hard negatives and data augmentation. The experiment results show that RocketQA significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions. We also conduct extensive experiments to examine the effectiveness of the three strategies in RocketQA. Besides, we demonstrate that the performance of end-to-end QA can be improved based on our RocketQA retriever.
1 Introduction
Open-domain QA depends heavily on passage retrieval, but dual-encoder training faces inference mismatch, unlabeled positives, and insufficient coverage from available data. RocketQA addresses these challenges with three training strategies and reports improvements across retrieval benchmarks and end-to-end QA.
- Motivation: Traditional term-based retrievers suffer from term mismatch, while dense dual-encoders learn question and passage representations for semantic matching.The dual-encoder separately encodes questions and passages, then computes similarity between their dense representations.
- Challenges: Dual-encoder training differs from inference because training uses small candidate sets, whereas inference searches millions of passages.Limited device memory constrains the number of negatives considered during training.
- Challenges: MSMARCO averages 1.1 annotated positive passages per question despite an 8.8M-passage collection, leaving many relevant passages unlabeled.The paper identifies limited training data and incomplete annotation as central obstacles for effective retriever training.
- RocketQA: RocketQA introduces cross-batch negatives, denoised hard negatives, and cross-encoder-based data augmentation to improve dense passage retrieval.The strategies respectively increase available negatives, remove false negatives from hard-negative candidates, and generate pseudo labels for unlabeled data.
- Results: RocketQA significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions.The paper also reports that all three strategies improve dense passage retrieval and that the RocketQA retriever improves end-to-end QA.
2 Related Work
Related work contrasts term-based retrieval with dense representations and distinguishes efficient dual-encoders from more effective but impractical cross-encoder rerankers. RocketQA follows the labeled-data fine-tuning line of dense retrieval research.
- Passage retrieval for open-domain QA: Traditional passage retrievers use term-based methods such as TF-IDF and BM25, while newer approaches improve them through document expansion, question expansion, or term-weight estimation.These methods target the limited representation capabilities of traditional retrievers.
- Passage retrieval for open-domain QA: Dense passage retrieval represents questions and documents as dense vectors, with prior work spanning self-supervised pre-training and fine-tuning on labeled data.RocketQA follows the fine-tuning approach, which the passage characterizes as better-performing with less cost.
- Passage re-ranking for open-domain QA: Cross-encoder rerankers yield substantial improvements but are impractical for scoring every passage in a corpus against a question.Their computational cost motivates distinguishing first-stage retrieval from passage reranking.
3 Approach
RocketQA trains a dual-encoder retriever through cross-batch negatives, denoised hard negatives, and cross-encoder-based data augmentation. Its pipeline progressively improves retrieval while addressing training–inference discrepancy, unreliable negatives, and limited labeled data.
- Dual-Encoder Architecture: The dual-encoder separately embeds questions and passages, precomputes passage embeddings, and retrieves passages by nearest-vector similarity.Passage embeddings are indexed for efficient query-time retrieval, while question embeddings are computed for each input question.
- Dual-Encoder Architecture: Training optimizes higher similarity for question–positive passage pairs than for question–negative pairs, but practical losses use far fewer negatives than the full collection.The number and quality of negatives affect retrieval performance, while considering all collection passages is computationally infeasible.
- Cross-Batch Negatives: Cross-batch negatives share passage embeddings across GPUs, expanding each question’s negative set from in-batch examples to approximately A×B−1 negatives.Here, A is the number of GPUs and B is the number of questions per mini-batch; the strategy can also be accumulated on a single GPU with additional training time.
- Denoised Hard Negatives: Denoised hard-negative sampling uses a trained cross-encoder to retain top-retrieved passages predicted as negatives with high confidence.This filters likely false negatives from hard-negative candidates while exploiting the cross-encoder’s stronger semantic interaction and robustness.
- Data Augmentation: Data augmentation uses a cross-encoder to assign high-confidence positive and negative labels to passages for unlabeled questions, then trains the dual-encoder on this pseudo-labeled data.The cross-encoder acts as teacher and the dual-encoder as student; the passage collection is reused with a new collection of questions.
4.1 Experimental Setup
The experiments use MSMARCO Passage Ranking and Natural Questions, evaluate retrieval and answer extraction with standard metrics, and train RocketQA components with large-scale distributed settings and augmented questions.
- Datasets: Experiments use MSMARCO Passage Ranking and Natural Questions, with approximately 8.8 million and more than 21 million passages, respectively.MSMARCO contains questions sampled from Bing search logs, while the reused NQ version contains selected factoid questions and Wikipedia passages.
- Evaluation Metrics: Passage retrieval is evaluated with MRR and top-k recall, while answer extraction is evaluated with exact match after string normalization.MRR averages reciprocal rank across questions; top-k recall measures whether retrieved passages contain answers.
- Implementation Details: The dual-encoder uses ERNIE 2.0 base, the cross-encoder uses ERNIE 2.0 large, and training runs on up to eight NVIDIA Tesla V100 GPUs.The implementation uses PaddlePaddle and distributed training support for large batches.
- Implementation Details: RocketQA uses cross-batch negatives, cross-encoder-based denoising and augmentation, and distributed all-gather to expose passage representations across GPUs.The cross-encoder selects passages with scores below 0.1 as negatives and above 0.9 as positives, with manually evaluated accuracy higher than 90%.
- Implementation Details: Dual-encoders use batch sizes of 512 × 8 on MSMARCO and 512 × 2 on NQ, while questions and passages have maximal lengths of 32 and 128.The experiments also collect 1.7 million unlabeled questions from Yahoo! Answers, ORCAS and MRQA for augmentation.
4.2 Experimental Results
RocketQA outperforms the evaluated baselines on MSMARCO and NQ, while ablations show benefits from more negatives, denoised hard negatives and data augmentation; the retriever also improves end-to-end QA.
- Dense Passage Retrieval: RocketQA significantly outperforms all considered baselines on both MSMARCO and NQ passage retrieval.Dense retrievers are overall better than sparse retrievers in the reported comparison.
- Cross-Batch Negatives: Increasing random negatives improves performance until a point, after which optimization becomes harder with limited training data.The experiments identify a balance between batch size and the number of negatives.
- Denoised Hard Negatives: Hard negatives without denoising substantially reduce retriever performance, whereas denoised hard negatives improve it.Manual examination found about 70% of sampled top-retrieved passages that lacked labels were actually positive or highly relevant.
- Data Augmentation: Increasing the size of augmented data further improves performance without explicitly relying on manually labeled data.A cross-encoder generates pseudo training data for improving the dual-encoder.
- Passage Reading with RocketQA: A RocketQA retriever leads to better end-to-end QA performance when paired with an extractive reader on NQ.The reader is adapted to RocketQA’s retrieval distribution by training with its retrieval results.
5 Conclusions
RocketQA presents an optimized dense-retrieval training approach built from three strategies and reports effectiveness in retrieval and end-to-end QA.
- Conclusions: RocketQA combines cross-batch negatives, denoised hard negatives and data augmentation to improve dense passage retrieval.The conclusion identifies these as the approach’s three major technical contributions.
- Conclusions: Extensive experiments show the effectiveness of the three optimization strategies, and the RocketQA retriever improves end-to-end QA performance.The conclusion links the reported gains to both retrieval experiments and downstream QA.
6 Ethical Considerations
Dense passage retrieval is motivated by term mismatch in question answering and is described as having potential to empower people to find information.
- Ethical Considerations: Term mismatch between questions and answers creates barriers to accurately finding information, motivating dense passage retrieval for semantic matching.The passage frames dense retrieval as relevant to informational question-answering queries.
A.1 The Effects of Pre-trained LMs
RocketQA’s experiments compare pretrained language models used by dense retrieval systems and examine replacing ERNIEbase with BERTbase. Using BERTbase in RocketQA causes a slight performance decrease.
- A.1 The Effects of Pre-trained LMs: DPR uses BERTbase, ANCE uses RoBERTabase, ME-BERT uses BERTlarge, and RocketQA mainly uses ERNIEbase.These pretrained language models are compared in Table 6.
- A.1 The Effects of Pre-trained LMs: Replacing ERNIEbase with BERTbase in RocketQA slightly decreases performance.The replacement is applied to RocketQA’s first step.
- A.1 The Effects of Pre-trained LMs: Table 6 reports the effects of pretrained language models and copies reported numbers from the original papers where available.Blank entries indicate numbers that were not reported.
A.2 The Effectiveness of The Three Training Strategies on NQ
On Natural Questions, the experiments examine RocketQA’s three proposed training strategies. All three strategies are effective, with findings similar to those on MSMARCO.
- A.2 The Effectiveness of The Three Training Strategies on NQ: The NQ experiments evaluate the effectiveness of RocketQA’s three proposed training strategies.The results are presented in Table 7.
- A.2 The Effectiveness of The Three Training Strategies on NQ: All three RocketQA training strategies are effective on the NQ dataset.This finding is reported from Table 7.
- A.2 The Effectiveness of The Three Training Strategies on NQ: The NQ findings are similar to the results obtained on MSMARCO.The passage characterizes the cross-dataset pattern without reporting numerical values.