PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Retrieval for Weakly Supervised Open Domain Question Answering

Kenton Lee, Ming-Wei Chang, Kristina Toutanova

arXiv:1906.00300v3cs.CL

TL;DR

Open-domain QA lacks reliable gold evidence and commonly depends on blackbox IR, despite QA requiring retrieval beyond ordinary matching. ORQA jointly learns a retriever and reader from question-answer pairs, using ICT pre-training to make latent retrieval from Wikipedia tractable. Across five datasets, BM25 suffices when question writers know the answer, while learned retrieval improves exact match by 6 to 19 points when they seek information.

  • Problem

    Open-domain QA must retrieve evidence from an open corpus, but existing systems rely on blackbox IR candidates and may require strong evidence supervision.

  • Method

    ORQA jointly learns dense retrieval and reading from question-answer pairs, using ICT pre-training to initialize retrieval over the open corpus.

  • Results

    On five open QA datasets, BM25 is state-of-the-art for SQuAD and TriviaQA, while learned retrieval improves exact match by 6 to 19 points over BM25 on Natural Questions, WebQuestions, and CuratedTrec.

  • Takeaways & Limitations

    Learning retrieval is crucial when questions reflect an information need and question writers do not already know the answer.

  • Takeaways & Limitations

    SQuAD is unsuitable for learned retrieval because its questions derive from only 536 documents, creating correlated retrieval targets that violate the IID assumption.

Abstract

from arXiv · show

Recent work on open domain question answering (QA) assumes strong supervision of the supporting evidence and/or assumes a blackbox information retrieval (IR) system to retrieve evidence candidates. We argue that both are suboptimal, since gold evidence is not always available, and QA is fundamentally different from IR. We show for the first time that it is possible to jointly learn the retriever and reader from question-answer string pairs and without any IR system. In this setting, evidence retrieval from all of Wikipedia is treated as a latent variable. Since this is impractical to learn from scratch, we pre-train the retriever with an Inverse Cloze Task. We evaluate on open versions of five QA datasets. On datasets where the questioner already knows the answer, a traditional IR system such as BM25 is sufficient. On datasets where a user is genuinely seeking an answer, we show that learned retrieval is crucial, outperforming BM25 by up to 19 points in exact match.

1 Introduction

Open-domain QA must retrieve evidence from an open corpus, yet existing systems depend on fixed IR candidates and often strong evidence supervision. ORQA instead learns retrieval and reading jointly from question-answer pairs, using ICT pre-training, and shows learned retrieval helps most when questions reflect genuine information needs.

  • Open-domain QA is more realistic for practical applications because evidence must be retrieved from an open corpus rather than supplied as input.
  • Existing approaches rely on blackbox IR systems for candidate generation and, in strongly supervised settings, question-answer-evidence training triples.
  • QA differs from IR because under-specified questions require language understanding beyond lexical and semantic matching.
  • ORQA jointly learns retrieval and reading from question-answer string pairs, treating open-corpus evidence retrieval as a latent variable.
  • Inverse Cloze Task pre-training initializes the retriever before end-to-end fine-tuning on the marginal log-likelihood of correct answers.
  • 6 to 19 points in exact match is the improvement over BM25 on datasets where question writers did not already know the answer.

2 Overview

Open-domain QA scores answer derivations by combining retrieval and reading over a large evidence corpus. Existing pipelined models first restrict this search with a blackbox IR system, then apply a reader to the resulting candidates.

  • 2.1 Task: Open-domain QA maps a question string to an answer string, with evidence source selection treated as a modeling choice.
  • 2.2 Formal Definitions: Models operate on a corpus divided into evidence blocks, and an answer derivation pairs one block with a text span.
  • 2.2 Formal Definitions: A model scores each answer derivation with retrieval and reader components, then outputs the answer string from the highest-scoring derivation.
  • 2.2 Formal Definitions: English Wikipedia contains over 13 million evidence blocks, each with over 2000 possible answer spans, creating a major scaling challenge.
  • 2.3 Existing Pipelined Models: Existing pipelined systems use a blackbox IR system to select a closed candidate set before reading or reranking.
  • 2.3 Existing Pipelined Models: ORQA computes retrieval scores with BERT-based encoder inner products, jointly encodes top evidence with the question, and combines retrieval and reading scores.
  • 2.3 Existing Pipelined Models: Weakly supervised readers can learn from question-answer pairs, but retrieval systems are used to remove spurious ambiguities and produce treated-as-gold derivations.

3 Open-Retrieval Question Answering (ORQA)

ORQA jointly learns retrieval and reading over an open corpus, avoiding dependence on a blackbox IR system. Its main challenge is searching millions of evidence blocks under weak supervision, where spurious answer-containing passages can produce ambiguous learning signals.

  • Model overview: ORQA jointly learns retriever and reader components while remaining capable of retrieving any text in an open corpus.Unlike a blackbox IR pipeline, it is not restricted to a closed candidate set returned by an external retrieval system.
  • Retriever component: The retriever scores questions and evidence blocks using the inner product of their dense vector representations.The representations are projected from BERT outputs into 128-dimensional vectors.
  • Reader component: The reader is a span-based reading-comprehension model that scores answer spans using representations of their endpoints.A multi-layer perceptron scores the concatenated endpoints to model start/end interaction.
  • Inference & Learning Challenges: Spurious answer-containing passages can make weak supervision produce ambiguous learning signals, so effective evidence retrieval is necessary for meaningful training.Table 2 illustrates these ambiguities with irrelevant passages containing the answer string “seven.”
  • Inference & Learning Challenges: Over 13 million evidence blocks make retrieval an enormous search problem, while the correct evidence is latent and cannot be learned with standard teacher forcing.Weak supervision also creates many spuriously ambiguous derivations because irrelevant passages may contain the answer string.
  • Inference & Learning Challenges: Unsupervised retriever pre-training addresses the computational and learning difficulties by supporting pre-encoding of evidence blocks and biasing retrieval toward supportive evidence.The pre-trained retriever enables dynamic top-k retrieval during fine-tuning while reducing spurious ambiguities.

4 Inverse Cloze Task

The Inverse Cloze Task pre-trains retrieval by pairing a sentence treated as a pseudo-question with its surrounding context as pseudo-evidence. This trains semantic inference beyond word matching while retaining lexical matching features when available.

  • Motivation: ICT pre-training gives the retriever an unsupervised task that closely resembles evidence retrieval for question answering.Its purpose is to provide a suitable initialization before weakly supervised downstream learning.
  • Motivation: Useful evidence often discusses entities, events, and relations from a question while adding answer information absent from it.A sentence-context pair provides an unsupervised analogue because the context is semantically relevant and can supply missing information.
  • Inverse Cloze Task: ICT reverses the standard Cloze direction: given a sentence, the model predicts its surrounding context.A random sentence becomes a pseudo-question, and the surrounding text becomes pseudo-evidence.
  • Inverse Cloze Task: The ICT objective selects the true context among candidate evidence blocks in a batch.The batch provides sampled negative evidence blocks for the discriminative retrieval objective.
  • Inverse Cloze Task: ICT requires semantic inference because the pseudo-question may omit explicit mentions of the entity discussed in its context.In the zebra example, the retriever must identify the relevant context without the pseudo-question explicitly saying “Zebras.”
  • Inverse Cloze Task: The sentence is removed from its context in 90% of examples, combining abstract representation learning with lexical word-matching features.The remaining examples preserve the sentence-context overlap that supports lexical retrieval.
  • Assumptions: ICT relies on two assumptions: sentence-based zero-shot retrieval can bootstrap latent-variable learning, and pre-trained evidence-block encoders transfer without further training.Only the question encoder is expected to require downstream fine-tuning.

5 Inference

Inference pre-computes evidence-block representations and retrieves candidates through an indexed maximum-inner-product search. It then applies the expensive reader only to the dynamically changing top-k blocks during training and inference.

  • Indexed retrieval: All evidence-block encodings can be pre-computed and indexed for fast maximum inner product search.This avoids re-encoding the enormous evidence corpus during fine-tuning and supports tools such as Locality Sensitive Hashing.
  • Beam search: Inference retrieves the top-k evidence blocks and computes expensive reader scores only for those candidates.The top-k set changes during training as weakly supervised QA data fine-tunes the question encoder.

6 Learning

The model learns a distribution over answer derivations within retrieved evidence, using marginal likelihood and an early retrieval-only update. In practice, ICT pre-training makes matching derivations available for most examples, while trainable query representations allow retrieval beyond a fixed blackbox system.

  • The model defines a distribution over answer derivations among the top k retrieved evidence blocks, with k = 5.TOP(k) contains the blocks ranked highest by the retrieval score.
  • Training maximizes the marginal log-likelihood of all possibly spurious derivations in the beam that exactly produce the gold answer string.A derivation matches when its answer span maps exactly to the gold answer string.
  • An early update considers c = 5000 evidence blocks and updates only the retrieval score to encourage more aggressive retrieval learning.The retrieval-only update is cheap to compute.
  • The final loss combines the standard derivation update with the early retrieval update.
  • Less than 10% of examples are discarded in practice because ICT pre-training supplies non-trivial initial retrieval.With random initialization, the authors would expect almost all examples to be discarded when no matching answer is found.
  • Fine-tuning the query encoder lets the model potentially retrieve any evidence block, unlike blackbox IR systems whose recall improves only by retrieving more evidence.

7 Experimental Setup

The experiments convert five existing QA or reading-comprehension datasets into open-domain question-answer string pairs and evaluate them against dataset-specific biases. The setup uses an English Wikipedia evidence corpus, standardized development handling, and BERT-based components with specified retrieval and training configurations.

  • Open Domain QA Datasets: The evaluation uses five existing QA or reading-comprehension datasets converted to open formats, with each example containing a question and reference answer strings.The original datasets are not all intended for open-domain QA.
  • Open Domain QA Datasets: Natural Questions is filtered to short-answer questions, discarding evidence documents and answers longer than 5 tokens.Long answers are excluded because they often resemble extractive snippets rather than canonical answers.
  • Open Domain QA Datasets: WebQuestions retains the string representations of Freebase entities rather than their original entity annotations.
  • Open Domain QA Datasets: CuratedTrec contains question-answer pairs from real-query sources whose askers did not observe evidence documents.
  • Open Domain QA Datasets: TriviaQA uses its unfiltered question-answer set while discarding distantly supervised evidence, whereas SQuAD supplies answer spans selected from given Wikipedia paragraphs.
  • Experimental Protocol: Development data are created by randomly holding out 10% of training data when needed, with hidden-test settings also using the original development set for testing.
  • Dataset Biases: The diverse datasets matter because their inherent biases affect open-domain systems with learned retrieval, including tool-related bias in genuine information-seeking datasets.
  • Dataset Biases: Known-answer datasets can contain question hints and artificially large question-evidence lexical overlap, especially in SQuAD.

8 Main Results

Main results show that BM25 remains strong when question askers already know the answer, while learned retrieval is especially valuable for questions reflecting genuine information needs.

  • Baselines: BM25 is a de-facto state-of-the-art unsupervised retrieval method used as a baseline.Its implementation retrieves static evidence because BM25 is not trainable, while the final score combines BM25 and reader scores.
  • Baselines: Dense language-model representations do not readily capture the word matching that makes BM25 powerful.The comparison uses alternate encoders to pre-compute evidence representations and initialize question encoding.
  • Results: 6 to 19 points in exact match separate ICT-pre-trained retrieval from BM25 on Natural Questions, WebQuestions, and CuratedTrec.These datasets contain questions from users seeking information rather than question writers who already know the answer.
  • Results: BM25 is sufficient for SQuAD and TriviaQA, where question askers already know the answer and the retrieval problem resembles traditional IR.A highly compressed 128-dimensional vector cannot precisely represent every word in the evidence as BM25 does.
  • Caveat: SQuAD is unsuitable for learned retrieval because its 100k questions derive from only 536 documents, violating the IID assumption.The resulting correlation among training retrieval targets contributes to a notable development-to-test accuracy drop.

9 Analysis

The analyses compare ORQA with strongly supervised systems, examine ICT masking, and illustrate both its robustness to lexical overlap and its compression limits.

  • Strongly supervised comparison: ORQA’s BM25 baseline retrieves 5 evidence blocks and greatly outperforms 5-document BERTserini, while approaching 29-paragraph BERTserini.BERTserini uses true Wikipedia paragraphs rather than arbitrary blocks, producing more evidence blocks because paragraph lengths vary.
  • ICT masking: Masking the pseudo-query 90% of the time balances learning n-gram overlap with avoiding a task reduced to memorization.The masking-rate analysis is evaluated on the open Natural Questions development set.
  • ICT masking: Almost 10 points in end-to-end performance are lost when the pseudo-query is always masked, because the retriever fails to learn n-gram overlap.Never masking loses 6 points and produces results nearly identical to BM25.
  • Example predictions: ORQA separates semantically distinct text with high lexical overlap more robustly, but 128-dimensional vectors represent extremely specific concepts less precisely.The examples motivate a hybrid approach combining dense and sparse representations.

10 Related Work

Related work frames ORQA against closed-set reranking, weakly supervised semantic parsing, representation learning, and latent retrieval in information extraction.

  • Evidence retrieval: Prior evidence-retrieval methods rerank candidates from a closed set, whereas the authors aim to integrate these approaches with open-corpus retrieval.The cited methods improve retrieval by aggregating across multiple retrieval steps.
  • Weak supervision: ORQA shares weakly supervised semantic parsing challenges: coupled inference and learning, latent derivations, and the need for strong inductive biases.These biases help find positive learning signal while avoiding spurious ambiguities.
  • Representation learning: ICT generalizes skip-gram with coarser granularity, a deep architecture, and in-batch negative sampling.The paper relates ICT to existing representation-learning literature while motivating it as an unsupervised proxy for evidence retrieval.
  • Latent retrieval: Compared with latent retrieval in information extraction, ORQA learns a more expressive retriever using ICT’s strong inductive biases.The comparison concerns consulting external evidence sources with latent retrieval.

11 Conclusion

The paper presents ORQA, which jointly learns retrieval and reading end-to-end from question-answer pairs without an IR system. ICT pre-training enables this approach, and learned retrieval is crucial when questions reflect an information need.

  • Conclusion: ORQA jointly learns the retriever and reader end-to-end using only question-answer pairs and without any IR system.The system is presented as the first open-domain QA model with this combination of properties.
  • Conclusion: ICT pre-training makes end-to-end learning possible for ORQA.The conclusion identifies this pre-training as the enabling component.
  • Conclusion: Learning to retrieve is crucial when question writers do not already know the answer.This conclusion applies to questions that reflect an information need.
Loading 1906.00300v3…