PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index

Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur P. Parikh, Ali Farhadi, Hannaneh Hajishirzi

arXiv:1906.05807v2cs.CL

TL;DR

Existing open-domain QA systems are too costly for real-time use because they process long evidence documents for each query. DENSPI instead indexes query-agnostic dense-sparse phrase representations offline and retrieves matching phrases directly. On SQuAD-Open, it achieves higher EM than DrQA while reducing computational cost and accelerating end-to-end inference.

  • Problem

    Existing open-domain QA systems are unsuitable for real-time use because neural models re-encode evidence documents for every query.

  • Method

    DENSPI independently encodes Wikipedia phrases with dense and sparse vectors, indexes them offline, and retrieves the best phrase by nearest-neighbor search.

  • Results

    DENSPI achieves 6.4% higher EM than DrQA while reading words 6000 times faster in controlled conditions and 43 times faster end-to-end in a real setup.

  • Takeaways & Limitations

    Query-agnostic phrase indexing enables real-time open-domain QA over large document collections with CPU-only inference.

  • Takeaways & Limitations

    The query-agnostic constraint leaves a gap of at least 6.1% EM, and experiments assume constrained academic hardware.

Abstract

from arXiv · show

Existing open-domain question answering (QA) models are not suitable for real-time usage because they need to process several long documents on-demand for every input query. In this paper, we introduce the query-agnostic indexable representation of document phrases that can drastically speed up open-domain QA and also allows us to reach long-tail targets. In particular, our dense-sparse phrase encoding effectively captures syntactic, semantic, and lexical information of the phrases and eliminates the pipeline filtering of context documents. Leveraging optimization strategies, our model can be trained in a single 4-GPU server and serve entire Wikipedia (up to 60 billion phrases) under 2TB with CPUs only. Our experiments on SQuAD-Open show that our model is more accurate than DrQA (Chen et al., 2017) with 6000x reduced computational cost, which translates into at least 58x faster end-to-end inference benchmark on CPUs.

1 Introduction

DENSPI replaces query-time document processing with an indexable, query-agnostic phrase representation for real-time open-domain QA. Its dense-sparse design supports web-scale storage and retrieval while substantially reducing computational cost and inference time.

  • Motivation: Existing QA pipelines remain costly because neural reading-comprehension models reprocess evidence documents for every question.This pipeline can require multi-GPU-seconds or tens to hundreds of CPU-seconds per query.
  • Approach: DENSPI indexes query-agnostic phrase representations offline and retrieves the maximum-inner-product phrase after mapping the question into the same space.This removes the need to re-encode documents for each question.
  • Approach: Dense and sparse vectors jointly encode syntactic, semantic, and precise lexical information in each phrase representation.Dense vectors capture local syntactic and semantic cues, while sparse vectors encode lexical information.
  • Scale: Up to 60 billion Wikipedia phrases can be served under 2TB, using indexable representations and approximate nearest-neighbor search.The paper also describes optimization strategies enabling deployment with a 4-GPU, 64GB-memory server.
  • Results: 6000x lower RAM computational cost and at least 58x faster end-to-end inference were reported relative to DrQA on SQuAD-Open.The end-to-end benchmark includes disk access time.

2 Related Work

Related work approaches open-domain QA through knowledge bases, document retrieval, phrase-level neural answering, approximate similarity search, generative models, and memory networks. DENSPI connects these traditions through indexed phrase representations and large-scale similarity search.

  • Open-domain QA: Knowledge-base QA can achieve high precision but remains limited to the ontology of the underlying knowledge graph.Document retrieval instead addresses questions from broader web text collections.
  • Open-domain QA: Phrase-level QA typically retrieves 5–10 relevant documents before a deep neural model extracts the answer.Much prior work focuses on reducing retriever error in this pipelined setting.
  • Approximate similarity search: Approximate similarity search seeks sublinear-time nearest-neighbor retrieval, using methods such as hashing, clustering, and graph-based search.Inner-product search and sparse retrieval require different strategies depending on vector structure.
  • Generative QA: Generative QA provides a surface-form decoder but cannot easily enumerate a compact list of all semantically valid questions for a document.Phrase indexing instead maps document phrases into the question representation space.
  • Memory networks: A phrase index can be viewed as external memory whose key is a phrase vector and whose value is the corresponding answer span.This perspective relates indexed phrase retrieval to memory-network architectures.

3 Overview

DENSPI replaces document re-encoding with offline phrase indexing and maximum-inner-product retrieval for real-time open-domain QA. Its dense-sparse span representations combine semantic, syntactic, and lexical information while avoiding pipeline document filtering.

  • Real-time open-domain QA requires selecting an answer phrase from a large collection of web documents for each factoid question.
  • Pipeline systems first retrieve documents and then re-encode them with a neural QA model, making inference computationally expensive and unsuitable for low-latency applications.
  • DENSPI encodes Wikipedia phrase spans offline, maps each question into the same space, and retrieves the answer by maximum inner product.
  • Dense vectors capture syntactic and semantic cues, while sparse vectors encode precise lexical information.
  • Independent phrase and question encoding avoids re-encoding documents for every question, though efficient search remains necessary because the index is large.

4 Phrase and Question Embedding

The embedding model represents each phrase with dense and sparse components, using contextualized token representations for semantic structure and tf-idf features for lexical precision. Phrase vectors additionally encode start–end coherency, while questions use a shared but distinct query encoder.

  • Dense model: The dense phrase vector decomposes into start and end vectors plus a coherency scalar between their positions.
  • Dense model: The coherency scalar helps exclude nonconstituent spans that would arise from independently selecting distant start and end positions.
  • Dense model: BERT contextualized token representations provide the components of the dense phrase encoding.
  • Sparse model: Sparse phrase embeddings use 2-gram tf-idf features, normalize them for cosine-like inner products, and add paragraph-level information.
  • Phrase representation: Phrase embeddings combine a dense vector for contextual syntax and semantics with a sparse vector for lexical information.
  • Question embedding: The question encoder shares BERT parameters with phrase encoding but uses a special token for dense query features and tf-idf over the entire question.

5 Training, Indexing & Search

Training and indexing are designed for web-scale QA under constrained hardware. The system uses sampled training conditions, auxiliary losses, negative examples, and storage reductions to make large-scale deployment feasible.

  • Hardware setting: The system targets Wikipedia-scale training, indexing, and search on 4 P40 GPUs, 128 GB RAM, 16 cores, and 2 TB of storage.
  • Training: Dense-model training uses golden paragraphs instead of optimizing directly over all Wikipedia, because full-corpus optimization is computationally prohibitive.
  • Training: The phrase loss models answer-span probability from exponentiated logits, while efficient computation avoids explicitly enumerating every phrase during training.
  • Training: Auxiliary start and end losses reduce the effect of summing over T^2 phrase terms and allow larger gradients.
  • Training: Negative examples from other paragraphs and articles support training a trainable no-answer bias.
  • Indexing: Wikipedia contains about 3 billion tokens, yielding roughly 60 billion phrases of length at most 20 and an initial storage requirement of 240 TB.

1. Pointer

Storing shared start and end vectors separately replaces repeated phrase-vector storage with pointers, substantially reducing the index footprint.

  • Shared start and end vectors reduce the phrase-index footprint from 240 TB to 12 TB by storing vector lists once and pointers per phrase.

2. Filtering

The system filters phrase vectors before storing them by predicting likely answer start and end positions without observing the question. This reduces the memory needed for indexed vectors.

  • A single-layer binary classifier is trained on each start and end vector.Training uses the actual answer as supervision without observing the question.
  • The classifiers identify vector positions unlikely to begin or end an answer phrase.Vectors failing this filtering are not stored.
  • 12 TB to 5 TB is the reported reduction in memory footprint from this filtering.

3. Quantization

The system reduces indexed-vector size through scalar quantization and uses approximate search strategies when direct billion-scale dense-sparse search is unavailable. Three retrieval variants trade search order and reranking behavior against scalability.

  • 3. Quantization: Scalar quantization converts float32 values to int8 using offset and scaling, reducing vector size by one-fourth.
  • 3. Quantization: 1.2 TB is the final memory consumption reported after scalar quantization.
  • 5.3 Search: Direct sparse maximum inner product search was not implemented because no suitable open-source system scaled to billions of dense-sparse vectors.
  • 5.3 Search: Sparse-first search retrieves top-ks documents sparsely, then performs exact phrase search over those documents.
  • 5.3 Search: Dense-first search retrieves top-kd dense vectors and reranks them using the corresponding sparse vectors.
  • 5.3 Search: The hybrid approach independently runs sparse-first and dense-first search, then reranks the combined results.

6 Experiments

Experiments evaluate DENSPI on closed-domain SQuAD and open-domain SQuAD-Open, focusing on accuracy, inference efficiency, and search diversity. DENSPI substantially improves speed over query-dependent baselines, while hybrid search provides strong open-domain accuracy and broader document coverage.

  • SQuAD v1.1 Experiments: 6,000 times faster: DENSPI processes 28.7 million words per second, outperforming DrQA and BERT in controlled CPU-memory benchmarks.The comparison uses words processed per second with documents pre-indexed and all information in memory.
  • SQuAD v1.1 Experiments: 3.3% higher EM: DENSPI outperforms DrQA on SQuAD v1.1, while remaining 9.2% below the current state of the art.The paper calls this remaining difference the decomposability gap.
  • Open-domain Experiments: 6.4% better accuracy and 43 times faster inference: DENSPI-Hybrid outperforms DrQA on SQuAD-Open.DENSPI-Hybrid remains 6.1% EM behind Weaver, which co-encodes the top 25 retrieved documents for each question.
  • Open-domain Experiments: 817 unique documents per query: DENSPI-Hybrid searches a substantially more diverse set than DrQA, which considers 5 documents.The comparison is reported using the #D/Q measure for documents visited per query.
  • Open-domain Experiments: Hybrid search achieves the highest F1 and EM, while SFS has 6.6% higher F1 than DFS at comparable inference speed.The paper notes that the SFS advantage may reflect high word overlap between questions and contexts in SQuAD.
  • Qualitative Analysis: DENSPI errors include confusing ‘1940s’ with ‘1930s’ and emphasizing ‘largest’ over ‘fifth-’ in a question.These examples come from the paper’s qualitative error analysis of open-domain predictions.

7 Conclusion

The model uses query-independent dense-sparse phrase representations for real-time open-domain QA. It is substantially faster than DrQA while achieving higher EM, but a remaining accuracy gap motivates better phrase representations.

  • 6,000 times faster under a controlled environment and 43 times faster in a real setup than DrQA, while achieving 6.4% higher EM.These results are reported on SQuAD-Open.
  • The model captures lexical, semantic, and syntactic information with query-independent indexable phrase representations.The representations combine dense and sparse vectors and are learned independently of the query.
  • At least 6.1% EM remains as a gap attributed to the query-agnostic constraint.The authors identify improved phrase representation design as a direction for closing this gap.

A More Prediction Samples

Table 5 presents additional prediction samples comparing DrQA and DENSPI. Each sample includes a document title, context, and predicted answer.

  • Table 5 compares additional prediction samples from DrQA and DENSPI.Each sample shows the document title, context, and predicted answer.
Loading 1906.05807v2…