PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Dense Representations of Phrases at Scale

Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, Danqi Chen

arXiv:2012.12624v3cs.CL

TL;DR

Phrase retrieval offers an alternative to on-demand document reading, but prior systems rely on sparse representations and lag behind retriever-reader models. DensePhrases learns dense phrase representations with reading-comprehension supervision, negative sampling, and query-side fine-tuning; across five QA datasets it improves on previous phrase retrieval models and matches state-of-the-art retriever-readers.

  • Problem

    Phrase retrieval must find answers among billions of phrases, yet existing approaches rely heavily on sparse representations and underperform retriever-reader models (Seo et al., 2019; Lee et al., 2020).

    Source: §1 · p. 1

  • Method

    DensePhrases learns phrase and question representations from reading-comprehension supervision, uses batch-negative techniques, and fine-tunes the question encoder for transfer.

    Source: §1 · p. 2, §10 · p. 9

  • Results

    15%–25% absolute improvement over previous phrase retrieval models on most of five open-domain QA datasets, while matching state-of-the-art retriever-reader performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DensePhrases can serve as a pre-indexed phrase-level knowledge base for slot filling without rebuilding phrase storage.

    Source: §1 · p. 2

    SQuAD is drawn from a small number of popular Wikipedia articles, so models trained only on it may be biased toward those topics.

    Source: Ethical Considerations · p. 10

Abstract

from arXiv · show

Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on sparse representations and still underperform retriever-reader approaches. In this work, we show for the first time that we can learn dense representations of phrases alone that achieve much stronger performance in open-domain QA. We present an effective method to learn phrase representations from the supervision of reading comprehension tasks, coupled with novel negative sampling methods. We also propose a query-side fine-tuning strategy, which can support transfer learning and reduce the discrepancy between training and inference. On five popular open-domain QA datasets, our model DensePhrases improves over previous phrase retrieval models by 15%-25% absolute accuracy and matches the performance of state-of-the-art retriever-reader models. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot filling tasks, showing the promise of utilizing DensePhrases as a dense knowledge base for downstream tasks.

1 Introduction

DensePhrases targets the scale and performance challenges of phrase retrieval by learning dense phrase representations with reading-comprehension supervision and new sampling and fine-tuning strategies. It improves over earlier phrase retrieval models on open-domain QA and supports slot filling with pre-indexed phrase representations.

1 Introduction

  • Phrase retrieval must distinguish answers among billions of phrases, while existing approaches rely heavily on sparse representations and underperform retriever-reader models.The paper identifies phrase representations' lower expressiveness than cross-attention models as an additional challenge.Source: §1 · p. 1
  • Training uses reading-comprehension supervision, data augmentation, distillation, pre-batch negatives, and query-side fine-tuning for transfer without rebuilding phrase representations.Pre-batch negatives reuse preceding mini-batches to increase negative samples without retaining their computational graphs.Source: §1 · pp. 1–2
  • 15%–25% absolute improvement over previous phrase retrieval models on most open-domain QA datasets, with performance matching state-of-the-art retriever-reader models.The reported comparison uses open-domain test-set accuracy.Source: Table 1 · p. 2, §1 · p. 2
  • State-of-the-art performance on two slot-filling tasks uses less than 5% of the training data and reuses pre-indexed phrase representations.Only the question encoder is fine-tuned; phrase storage is not rebuilt.Source: §1 · p. 2

2 Background

  • Dense phrase and question representations support maximum inner product search, whereas retriever-reader pipelines can propagate retrieval errors and require slow reader inference.Phrase retrieval searches independently encoded representations without processing passages with an on-demand reader.Source: §1 · p. 1, §2 · p. 3

3 DensePhrases

DensePhrases learns fully dense phrase representations and retrieves answers through maximum inner product search (MIPS). It pre-indexes phrases offline and then fine-tunes the question encoder using corpus-level retrievals.

3.1 Overview

  • DensePhrases learns phrase and question encoders, pre-indexes corpus phrases, then fine-tunes the question encoder using top-k retrieved phrases.The approach is intended to reduce the training–inference discrepancy and support transfer learning to new domains.Source: §3.1 · p. 3

3.2 Base Architecture

  • Each phrase representation concatenates its contextualized start and end token vectors, so storage needs only word vectors rather than every phrase.The paper says this representation requires at least an order of magnitude less storage than indexing all phrases.Source: §3.2 · p. 3
  • The question encoder uses separate start and end encoders, whose [CLS] representations are concatenated to match the phrase representation.The encoders are initialized from pretrained language models and fine-tuned for the task.Source: §3.2 · pp. 3–4

4 Learning Phrase Representations

DensePhrases learns phrase representations from reading-comprehension supervision, then uses in-batch and pre-batch negatives to discriminate phrases at scale. Data augmentation, distillation, and joint optimization contribute to the training objective.

4 Learning Phrase Representations

  • The section’s overall goal is to learn dense phrase representations from reading-comprehension supervision and distinguish phrases at full Wikipedia scale.The method combines single-passage learning with two negative-sampling strategies.Source: §4 · p. 4

4.1 Single-passage Training

  • Single-passage training predicts gold-answer start and end positions without cross-attention, while distillation transfers cross-attention model distributions and generated QA pairs augment supervision.Generated pairs are retained only when a cross-attention reading-comprehension model predicts them correctly.Source: §4.1 · pp. 4–5

4.2 In-batch Negatives

  • In-batch negatives treat other examples’ gold phrases as negatives, while attempts to use non-gold phrases from other passages yielded no meaningful improvement.The method scores each example’s question against gold start and end representations in the same minibatch.Source: §4.2 · p. 5

4.3 Pre-batch Negatives

  • Pre-batch negatives cache phrase representations from the preceding C minibatches, adding B × C negatives alongside B − 1 in-batch negatives.The cache is a FIFO queue, and cached negatives do not retain a computational graph, making memory use more manageable.Source: §4.3 · p. 6

4.4 Training Objective

  • The training objective jointly optimizes all three losses on annotated reading-comprehension examples and generated questions, using λ1 = 1, λ2 = 2, and λ3 = 4.The authors report these loss weights work well in practice.Source: §4.4 · p. 6

5 Indexing and Search

The indexing and search pipeline encodes corpus paragraphs into word vectors, stores them in a phrase dump, and uses maximum inner product search to retrieve answer spans.

5 Indexing and Search

  • Indexing encodes paragraphs with the phrase encoder and stacks token representations into a phrase dump; the process takes hundreds of GPU hours and has a large disk footprint.A MIPS index is built over the dump using faiss.Source: §5 · p. 6
  • For a question, MIPS retrieves top-k start and end positions separately, then a constrained search selects a span no longer than L words.The search enforces 1 ≤ i ≤ j < i + L ≤ |W(D)|.Source: §5 · p. 6

6 Query-side Fine-tuning

Query-side fine-tuning updates the question encoder against a prebuilt phrase dump, optimizing retrieval of the gold answer from the top retrieved phrases.

6 Query-side Fine-tuning

  • The method updates only the question encoder and optimizes the marginal likelihood of the gold answer among retrieved top-k phrases.The phrase dump supports efficient maximum inner product search.Source: §6 · p. 6
  • The objective uses the top 100 retrieved phrases for each question in all experiments.The score function ranks phrases in the retrieved set.Source: §6 · p. 6
  • Query-side fine-tuning reduces the training–inference discrepancy and can improve final performance by filling gaps from training on only a small portion of the full passage collection.Source: §6 · p. 6

7 Experiments

Experiments evaluate DensePhrases on reading comprehension, open-domain QA, and slot filling, using phrase indexes learned from reading-comprehension data and query-side fine-tuning for target tasks.

7.1 Setup

  • Open-domain QA evaluation covers Natural Questions, WebQuestions, CuratedTREC, TriviaQA, and SQuAD; SQuAD and/or Natural Questions provide phrase-index training and query-side fine-tuning data.Source: §7.1 · p. 7
  • Slot filling uses T-REx and zero-shot relation extraction, with the subject and relation forming each query and the object as its answer.The same NQ/SQuAD phrase dump is used, with query-side fine-tuning on randomly sampled 5K or 10K examples.Source: §7.1 · p. 7

7.2 Experiments: Question Answering

  • 78.3 EM on SQuAD, improving on DenSPI by 4.7%, while DensePhrases remains behind cross-attention models.Table 2 reports reading-comprehension results on the SQuAD and Natural Questions development sets.Source: §7.2 · p. 7, Table 2 · p. 7
  • 15%–25% absolute improvement over previous phrase retrieval models on all open-domain QA datasets except SQuAD; DensePhrases is also competitive with retriever-reader models and faster at inference.Table 3 reports exact match on test sets and identifies the training or pre-training data used for retrievers and phrase dumps.Source: §7.2 · p. 7, Table 3 · p. 8

7.3 Experiments: Slot Filling

  • State-of-the-art performance on both slot-filling datasets is achieved using only 5K training examples.The reported KILT metrics account for span-level accuracy and correct retrieval of evidence documents.Source: §7.3 · p. 8, Table 4 · p. 8

8 Analysis

The analysis tests phrase-representation components, negative-sampling choices, and query-side fine-tuning. It reports gains from question generation and distillation, batch negatives, and query-side fine-tuning.

8 Analysis

  • Adding generated questions and cross-attention distillation improves SQuAD performance up to EM = 78.3.Adding generated questions to SpanBERT-QA yields a 0.3% improvement.Source: §8 · p. 8
  • On Natural Questions development data, in-batch and pre-batch negatives substantially improve results; B = 84 helps in-batch, while C = 2 is optimal for pre-batch.The ablation uses gold passages from the development set because full-Wikipedia testing is computationally expensive.Source: §8 · p. 8, Table 6 · p. 9
  • Pre-batch negatives also improve performance with a single gold passage, in an ablation using Natural Questions development gold passages.Source: §8 · p. 8, Table 6 · p. 9
  • Query-side fine-tuning improves unseen TQA, WQ, and TREC by 15% to 20%, and NQ from 32.6% to 40.9% for Cphrase = {NQ}.The authors say it reduces the discrepancy between training and inference.Source: §8 · p. 9

9 Related Work

Prior work explores dense representations across text granularities and phrase representations for tasks including translation and parsing. DensePhrases focuses on QA and other knowledge-intensive tasks, where phrases can be retrieved by maximum inner product search.

9 Related Work

  • Dense representations have been studied for words, sentences, and documents, while prior phrase-representation work includes statistical machine translation and syntactic parsing.Source: §9 · p. 9
  • DensePhrases focuses on phrase retrieval for QA and other knowledge-intensive tasks, using maximum inner product search.Source: §9 · p. 9
  • Dense phrase retrieval can also retrieve sentences or passages when the datastore maps each phrase to its containing text.Source: §9 · p. 9
  • Table 7 reports EM on each test set before and after query-side fine-tuning.Source: Table 7 · p. 9

10 Conclusion

DensePhrases learns retrievable phrase representations at Wikipedia scale for open-domain QA and other knowledge-intensive tasks. The paper also reports slot-filling performance using a small number of training examples, while noting topic-bias and resource-cost concerns.

10 Conclusion

  • DensePhrases achieves strong performance on five open-domain QA datasets and two slot-filling datasets, using only a small number of slot-filling training examples.The conclusion also reports reduced storage footprint and improved latency for open-domain QA.Source: §10 · p. 9

Ethical Considerations

  • Training phrase representations only on SQuAD could bias models toward the topics in its small sample of popular Wikipedia articles.The authors suggest excluding such datasets during training or using an alternative pre-training procedure as possible mitigations.Source: Ethical Considerations · p. 10
  • Using DensePhrases as a knowledge base increases the minimum resource requirement for additional experiments; vector quantization is proposed to reduce the cost.Source: Ethical Considerations · p. 10

A Computational Cost

Indexing the full phrase representations took about 20 hours with limited GPU resources, while storage was reduced from 1,547GB to 320GB. The appendices also describe the benchmark setup and dataset preprocessing.

A Computational Cost

  • Indexing all phrase representations took about 20 hours with 24GB × 4 GPUs, and storage fell from 1,547GB to 320GB.The reduction used removal of sparse representations and a sharing-and-split strategy.Source: A Computational Cost · p. 13

B Server Specifications for Benchmark

  • The speed benchmark measured #Q/sec on 1,000 randomly sampled Natural Questions development questions, using batch size 64 except BERTSerini, ORQA, and REALM, which were limited to batch size 1; implementation settings could affect latency.DPR's #Q/sec includes retrieval and reader execution; its reader batch size was 8 on a 24GB GPU, while the retriever batch size remained 64.Source: B Server Specifications for Benchmark · p. 13

C Data Statistics and Pre-processing

D Hyperparameters

DensePhrases reduces its storage footprint from 1,547GB to 320GB through token filtering and compact dense representations. The appendix also specifies phrase-encoder and query-side fine-tuning settings.

D Hyperparameters

  • Phrase and question encoders use Adam, a 3e-5 learning rate, gradient clipping at 1, batch size 84, and four epochs; pre-batch negatives are used in the last two.The number of preceding batches C is 2.Source: D Hyperparameters · p. 14
  • Query-side fine-tuning uses a 3e-5 learning rate, batch size 12, 10 epochs, and top k of 100, with the best development-set EM model evaluated on test.It uses a single 12GB GPU, compared with a single 24GB GPU for phrase-encoder training; questions are truecased for SpanBERT on Natural Questions and WebQuestions. SQuAD-trained question generation supplies 327,302 SQuAD and 1,126,354 Natural Questions questions, though Natural Questions-trained generation is lower quality due to information-seeking ambiguity.Source: D Hyperparameters · p. 14, E Reducing Storage Footprint · p. 14

E Reducing Storage Footprint

  • Storage falls from 1,547GB for Lee et al. (2020) to 320GB for DensePhrases.The paper attributes the reduction to several storage techniques, in addition to techniques introduced by Seo et al. (2019).Source: E Reducing Storage Footprint · p. 14
  • Filtering removes about 75% of full-Wikipedia tokens, leaving 770M token representations for storage.A linearly transformed filter layer is supervised with binary cross entropy on gold start/end positions; its threshold is tuned on the reading-comprehension development set to maximize filtering without a significant performance drop.Source: E Reducing Storage Footprint · p. 14
  • The architecture uses SpanBERT-base representations of dimension 768, no sparse representations, and int4 scalar quantization for float32 vectors.The quantization is applied during indexing.Source: E Reducing Storage Footprint · p. 14
  • Pure-MIPS inference avoids retaining about 500GB of original start and end vectors, but query-side fine-tuning requires them for reconstruction.The on-disk MIPS index returns top-k scores and indices, not vectors.Source: E Reducing Storage Footprint · p. 14
Loading 2012.12624v3…