PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense Passage Retrieval for Open-Domain Question Answering

Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih

arXiv:2004.04906v3cs.CL

TL;DR

Open-domain QA needs better passage retrieval because retrieval errors degrade the downstream reader. This paper trains a dual-encoder dense retriever from question-passage pairs, outperforming BM25 across most datasets and improving end-to-end QA results.

  • Problem

    Retrieval errors substantially degrade open-domain QA, motivating better selection of passages for downstream reading.

  • Method

    DPR fine-tunes BERT-based dual encoders on question-passage pairs to maximize inner products between questions and relevant passages.

  • Results

    DPR consistently outperforms BM25 on four of five retrieval datasets and improves previous state-of-the-art end-to-end QA results on four of five datasets.

  • Takeaways & Limitations

    Dense retrieval can outperform and potentially replace sparse retrieval while yielding new state-of-the-art results on multiple open-domain QA benchmarks.

  • Takeaways & Limitations

    DPR performs worse on SQuAD, where question-passage lexical overlap and a small, biased collection favor BM25.

Abstract

from arXiv · show

Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system largely by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks.

1 Introduction

Open-domain QA can be framed as retrieving a small set of candidate passages before applying a reader, but traditional TF-IDF/BM25 retrieval may miss semantic matches. This paper studies whether BERT-based dual encoders trained only on question–passage pairs can provide a stronger dense retriever and improve end-to-end QA.

  • Problem setting: Open-domain QA uses a two-stage framework in which a retriever selects candidate passages before a machine reading-comprehension model answers the question.The retriever selects a small subset of passages, some containing the answer.
  • Motivation: TF-IDF/BM25 efficiently matches keywords with inverted indexes, whereas dense representations provide complementary latent semantic encodings.Sparse methods represent questions and contexts as high-dimensional vectors, while dense encodings can capture semantic relationships.
  • Motivation: Dense retrieval can match paraphrases such as “bad guy” and “villain” and can be adapted through learned, task-specific embedding functions.The Lord of the Rings example illustrates how dense retrieval can fetch a context that term-based matching may miss.
  • Prior limitations: ORQA’s inverse cloze pretraining is computationally intensive, may use poor question surrogates, and does not fine-tune the context encoder on question–answer pairs.These limitations can make the resulting representations suboptimal.
  • Contributions: Fine-tuning BERT question and passage encoders in a dual-encoder architecture on relatively few question–passage pairs is sufficient to greatly outperform BM25 without additional pretraining.The paper develops the training scheme through ablation studies and reports that proper setup is sufficient for the improvement.
  • Contributions: Higher retrieval precision translates into higher end-to-end QA accuracy when a modern reader model is applied.This relationship is explicitly verified in the paper’s open-domain QA setting.

2 Background

Open-domain extractive QA answers factoid questions from spans in a large, diverse corpus. Because such corpora may contain millions to billions of documents, systems retrieve a small candidate set before answer extraction, and retrievers are evaluated by top-k retrieval accuracy.

  • Problem setting: Open-domain QA requires answering factoid questions using a large corpus covering diverse topics.Examples include identifying who voiced Meg on Family Guy or where the 8th Dalai Lama was born.
  • Problem setting: In extractive QA, the answer must be a span appearing in one or more corpus passages.Documents are split into equal-length passages that serve as retrieval units.
  • Retrieval motivation: Open-domain QA systems need an efficient retriever to select a small set of relevant texts before a reader extracts the answer.Corpus size can range from millions of documents, such as Wikipedia, to billions, such as the Web.
  • Retriever evaluation: A retriever maps a question and corpus to a filter set CF containing k ≪ |C| texts, evaluated by the fraction of questions whose answers appear in CF.For fixed k, this measure is called top-k retrieval accuracy.

3 Dense Passage Retriever (DPR)

DPR indexes passages as low-dimensional dense vectors and retrieves a small set of candidates by comparing them with question embeddings. It uses independent BERT encoders, offline FAISS indexing, dot-product similarity, and in-batch negatives to train the retrieval space.

  • Retriever design: DPR indexes M passages in a low-dimensional continuous space to efficiently retrieve the top k passages relevant to each question.The collection can be very large, while k is usually small, such as 20–100.
  • Encoders: DPR uses two independent base uncased BERT networks, taking each [CLS] representation as a 768-dimensional encoder output.The passage encoder maps passages to vectors for indexing, while the question encoder maps input questions to vectors for retrieval.
  • Retriever design: At run-time, DPR encodes the question and retrieves the k passages whose indexed vectors are closest using dot-product similarity.Passage representations are precomputed so retrieval remains efficient.
  • Inference: During inference, DPR applies the passage encoder offline and indexes embeddings with FAISS, then retrieves the top k passages closest to the question embedding.FAISS is designed for efficient similarity search and clustering of dense vectors and can be applied to billions of vectors.
  • Training: In a batch of size B, DPR forms a B × B similarity matrix and reuses computation to train on B2 question–passage pairs.Each question’s paired passage is positive, while the other B −1 passages in the batch are negatives.

4 Experimental Setup

The experiments use a processed English Wikipedia corpus split into 100-word retrieval passages and evaluate DPR on five QA datasets using established splits and dataset-specific positive-passage selection.

  • Wikipedia corpus: The retrieval corpus is the Dec. 20, 2018 English Wikipedia dump, cleaned with DrQA preprocessing to remove tables, infoboxes, lists, and disambiguation pages.Articles are converted into clean text portions before passage construction.
  • Passage construction: 21,015,324 passages result from splitting each article into disjoint 100-word blocks, which serve as the basic retrieval units.Each passage is also prepended with its source article title and an [SEP] token.
  • QA datasets: Five QA datasets use the same training, development, and testing splitting method as previous work: Natural Questions, TriviaQA, WebQuestions, CuratedTREC, and SQuAD v1.1.The datasets span end-to-end QA, open-domain questions, and reading comprehension settings.
  • Positive passage selection: For TREC, WebQuestions, and TriviaQA, the highest-ranked BM25 passage containing the answer is selected as positive; questions without such a passage in the top 100 are discarded.For SQuAD and Natural Questions, gold passages are matched and replaced to account for differences between the original passages and the candidate pool.

5 Experiments: Passage Retrieval

DPR generally outperforms BM25 on passage retrieval across five QA datasets, while differing qualitatively in the evidence it retrieves and offering much higher query throughput. Its performance depends on training choices, with in-batch negatives and limited data already producing strong results, but dense indexing is substantially more expensive to build.

  • Retrieval performance: 78.4% vs. 59.1% top-20 accuracy on Natural Questions shows DPR’s especially large advantage over BM25 when retrieving few passages.DPR performs consistently better than BM25 on all datasets except SQuAD.
  • Dataset effects: DPR’s lower SQuAD performance is attributed to question–passage lexical overlap and a biased collection from only 500+ Wikipedia articles.These properties give BM25 an advantage and make the SQuAD distribution unusually biased.
  • Sample efficiency: 1,000 training examples are sufficient for DPR to outperform BM25 on Natural Questions development retrieval.This demonstrates strong sample efficiency when using a general pretrained language model.
  • Training schemes: A single BM25 hard negative substantially improves in-batch-negative training, while adding two hard negatives provides no further benefit.In-batch negatives also substantially improve results compared with comparable standard training using seven gold negatives.
  • Impact of gold passages: 1 point lower top-k retrieval accuracy results when Natural Questions uses distantly supervised passages instead of gold passages.Distant supervision selects the highest-ranked BM25 passage containing the answer.
  • Runtime efficiency: 995.0 questions per second versus 23.7 questions per second per CPU thread demonstrates DPR’s query-speed advantage, but dense indexing takes 8.5 hours compared with roughly 30 minutes for Lucene.Computing embeddings for 21-million passages takes roughly 8.8 hours on 8 GPUs.

6 Experiments: Question Answering

The end-to-end QA system combines retrieval with a neural reader that reranks retrieved passages and extracts answer spans. DPR generally improves final QA accuracy over BM25, while multi-dataset training helps smaller benchmarks and isolated retriever-reader training achieves 39.8 EM on Natural Questions.

  • End-to-end QA system: The neural reader reranks up to 100 retrieved passages using cross-attention, selects the best passage, and extracts its highest-scoring answer span.Cross-attention is used only on a small candidate set because it is more expressive but nondecomposable for large-corpus retrieval.
  • End-to-end results: DPR-retrieved passages produce more correct answers than BM25-retrieved passages on every evaluated dataset except SQuAD.Final QA results are measured by exact match after minor normalization.
  • End-to-end results: Multi-dataset training performs comparably to single-dataset training on NQ and TriviaQA but has a clear advantage on smaller WQ and TREC.For WQ and TREC in the Multi setting, the reader is fine-tuned from the NQ-trained reader.
  • Training ablation: 39.8 EM is obtained by jointly training the retriever and reader on Natural Questions, below the paper’s isolated-training strategy.The passage characterizes isolated training as simpler and effective at leveraging available supervision.
  • Inference efficiency: 20ms is the approximate reader latency when DPR processes up to 100 passages in one batch on a single 32GB GPU.The latency remains almost identical to the single-passage case, though throughput comparisons with ORQA are harder because ORQA uses longer passages.

7 Related Work

Open-domain QA traditionally relies on sparse retrieval methods such as TF-IDF and BM25, while dense retrieval builds on a longer history of learned vector representations. Prior QA approaches use iterative question reformulation or answer-vector retrieval, whereas this work emphasizes a simple, effective model without additional pretraining or complex joint training.

  • Sparse retrieval: TF-IDF and BM25 are standard sparse retrieval methods broadly applied across QA tasks.Passage retrieval reduces the search space for answer extraction and identifies support contexts for verification.
  • Augmented retrieval: External structured information, including knowledge graphs and Wikipedia hyperlinks, has been used to augment text-based retrieval.
  • Dense representations: Dense retrieval has roots in Latent Semantic Analysis and increasingly uses discriminatively trained encoders for query-document matching across multiple retrieval applications.Applications include cross-lingual document retrieval, ad relevance prediction, Web search, and entity retrieval.
  • Dense QA retrieval: Prior dense open-domain QA methods iteratively reformulate question vectors or directly retrieve encoded candidate answer phrases instead of passages.Das et al. (2019) use iterative passage retrieval, while Seo et al. (2019) retrieve answer phrases directly.
  • Contribution: The proposed model offers a simple, effective solution with stronger empirical performance without additional pretraining or complex joint training schemes.
  • Subsequent work: Subsequent work improved DPR retrieval by discovering hard negatives across training iterations, and other studies used DPR as a module in broader systems.Xiong et al. (2020a) construct different training examples each iteration, while later work further integrates DPR.

8 Conclusion

The work demonstrates that dense retrieval can outperform and potentially replace traditional sparse retrieval in open-domain question answering. It also shows that a simple dual-encoder can work well, but successful training depends on critical ingredients.

  • Dense retrieval can outperform and potentially replace the traditional sparse retrieval component in open-domain question answering.
  • A simple dual-encoder approach can be made to work surprisingly well for dense retrieval.
  • Successful dense-retriever training requires several critical ingredients identified through empirical analysis and ablation studies.

A Distant Supervision · B Alternative Similarity Functions & Triplet Loss

The analysis examines distant supervision for selecting positive passages and compares alternative DPR similarity functions and triplet loss under controlled settings. It also notes that analysis-experiment baselines slightly exceed earlier Table 3 results because of improved hyperparameters.

  • A Distant Supervision: For final DPR training on Natural Questions, passages best matching the gold context serve as positive passages.
  • A Distant Supervision: Distant supervision uses answer-containing passages as positives when datasets provide only question–answer pairs.The comparison tests whether this setting causes significant performance degradation.
  • A Distant Supervision: The distant-supervision procedure combines each question and answer, runs Lucene-BM25, and selects the top answer-containing passage as positive.
  • B Alternative Similarity Functions & Triplet Loss: Beyond dot product (DP) and softmax negative log-likelihood (NLL), the experiments evaluate Euclidean distance (L2) and triplet loss.
  • B Alternative Similarity Functions & Triplet Loss: The triplet-loss margin is set to 1, while L2 scores are negated before softmax and similarity signs are changed for triplet-loss application.
  • B Alternative Similarity Functions & Triplet Loss: All alternative-similarity and triplet-loss experiments use the same hyperparameters.
  • B Alternative Similarity Functions & Triplet Loss: Analysis-experiment baseline retrieval accuracy in Tables 5 and 6 is slightly better than in Table 3 because of improved hyperparameter settings.The improved setting is documented in the code release.

C Qualitative Analysis

DPR and BM25 retrieve qualitatively different passages: BM25 favors selective lexical matches, while DPR captures semantic relationships but can miss rare salient phrases. Examples show DPR succeeding without lexical overlap and BM25 succeeding when an exact phrase is critical.

  • Retriever Differences: BM25 is sensitive to selective keywords and phrases, whereas DPR captures semantic relationships but may miss rarely occurring salient phrases.The contrast reflects BM25’s limited handling of lexical variation and semantics and DPR’s limited capacity for rare phrases.
  • Retriever Differences: DPR retrieves the correct answer by matching “body of water” with semantic neighbors such as sea and channel despite no lexical overlap.This example illustrates DPR’s strength in semantic representation.
  • Retriever Differences: BM25 outperforms DPR when the salient phrase “Thoros of Myr” is critical, because DPR cannot capture it.The example illustrates DPR’s weakness on rare, highly selective phrases.

D Joint Training of Retriever and Reader

The joint-training scheme fixes the passage encoder while updating only the question encoder through the combined retriever–reader loss, enabling efficient retrieval without passage reindexing. It follows ORQA-style supervision but achieves the same 39.8 exact match on NQ dev as the regular pipeline, with no improvement.

  • Training scheme: The passage encoder is fixed, while only the question encoder receives backpropagation from the combined retriever–reader loss.This design avoids updating passage representations during joint training.
  • Training scheme: The fixed passage encoder allows use of the HNSW-based FAISS index for low-latency retrieval without reindexing passages during model updates.The index remains usable because passage representations do not change.
  • Training scheme: The loss largely follows ORQA by combining log probabilities of retriever-selected positive passages with reader-selected correct spans and passages.The supplied passage describes supervision from both retriever and reader outputs.
  • End-to-end results: 39.8 exact match on NQ dev is achieved by joint training, matching the regular retriever/reader pipeline rather than improving its end-to-end QA results.The question encoder starts from a DPR model trained on NQ, and the reader starts from BERT-base.
Loading 2004.04906v3…