PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Domain Question Answering Goes Conversational via Question Rewriting

Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, Srinivas Chappidi

arXiv:2010.04898v3cs.IRcs.CL

TL;DR

Complex information needs motivate conversational QA that uses prior turns and retrieves answers from multiple web pages. QReCC supplies rewrites and annotations for rewriting, retrieval, and reading comprehension, and its baseline reaches F1 19.10 against a 75.45 extractive upper bound.

  • Problem

    Existing conversational QA benchmarks restrict conversations to one document, whereas open-domain answers may be distributed across several documents.

  • Method

    QReCC provides question rewrites and annotations for question rewriting, passage retrieval, and reading comprehension, with a baseline combining QR and open-domain QA models.

  • Results

    F1 19.10 is the first QReCC end-to-end baseline, compared with a 75.45 extractive upper bound.

  • Takeaways & Limitations

    Question rewriting provides an effective solution for resolving ellipsis and coreference while enabling existing non-conversational QA models in conversational dialogue.

  • Takeaways & Limitations

    QReCC baseline QA remains far from solved, with retrieval losses and a sizeable gap between extractive upper bound and known-context performance.

Abstract

from arXiv · show

We introduce a new dataset for Question Rewriting in Conversational Context (QReCC), which contains 14K conversations with 80K question-answer pairs. The task in QReCC is to find answers to conversational questions within a collection of 10M web pages (split into 54M passages). Answers to questions in the same conversation may be distributed across several web pages. QReCC provides annotations that allow us to train and evaluate individual subtasks of question rewriting, passage retrieval and reading comprehension required for the end-to-end conversational question answering (QA) task. We report the effectiveness of a strong baseline approach that combines the state-of-the-art model for question rewriting, and competitive models for open-domain QA. Our results set the first baseline for the QReCC dataset with F1 of 19.10, compared to the human upper bound of 75.45, indicating the difficulty of the setup and a large room for improvement.

1 Introduction

QReCC extends conversational QA to open-domain settings where answers may span multiple web pages and questions require conversational-context rewriting. It provides annotations and a baseline covering rewriting, retrieval, and answer extraction.

  • QReCC addresses open-domain conversational QA, where complex information needs may require multiple questions and answers may come from several documents.
  • QReCC contains 13.6K conversations averaging six turns, with question-answer pairs, rewrites, and answer provenance links.
  • Each question-answer pair includes a rewrite that makes context-dependent questions self-contained by resolving ellipsis and coreference.
  • The dataset collection combines dialogue annotation with web-page downloading, Common Crawl sampling, preprocessing, and passage splitting.
  • The baseline augments open-domain QA with a question-rewriting component and evaluates end-to-end performance alongside individual subtasks.
  • The paper presents QReCC as a resource for evaluating rewrite quality and advancing end-to-end open-domain conversational QA methods.

2 Related Work

QReCC extends conversational QA beyond single-document and retrieval-only settings by jointly supporting question rewriting, passage retrieval, and reading comprehension over a large web collection. It addresses limitations in earlier datasets concerning training data, answer spans, document scope, and answer form.

  • QReCC extends QuAC, CAsT, and NQ to open-domain conversational QA and supports question rewriting, passage retrieval, and reading comprehension.
  • QReCC differs from OR-QuAC through a larger, more diverse collection of 54M Common Crawl passages versus 11M Wikipedia passages.
  • QReCC answers are human-produced conversational responses that may combine several spans, whereas OR-QuAC uses single answer spans.
  • TREC CAsT 2019 provided conversational passage retrieval but lacked training data and answer spans, limitations QReCC was designed to overcome.
  • Because a concurrent CAsT extension was too small for training, its authors trained on non-conversational MS MARCO data, highlighting QReCC’s benchmark role.

3 Dialogue Collection

QReCC dialogues were collected by adapting questions from existing QA datasets and having annotators create conversational questions, answers, rewrites, and web provenance. The process used self-dialogues and post-hoc review to support data quality.

  • Dialogue seeds came from QuAC, CAsT, and NQ, with NQ increasing and diversifying samples beyond coreference-resolution cases.
  • Annotators answered questions through web search, used rewrites as search inputs, and linked each answer to its source web page.
  • QReCC includes summary statistics describing the constructed dataset.
  • Each dialogue was produced by one annotator, a self-dialogue design intended to avoid challenges observed in simulated dialogues involving annotator pairs.
  • Thirty professional annotators and a project lead conducted the task, followed by five reviewers who corrected examples through consensus.

4 Dialogue Analysis

QReCC contains 13,598 dialogues and 79,952 questions, with rewrites categorized by the intervention needed to resolve contextual dependencies. Replacement is the dominant rewrite type, while insertion cases broaden coverage beyond traditional coreference resolution.

  • Dataset composition: 13,598 dialogues contain 79,952 questions, including 9% without answers.The dataset retains rewrites even when no answer was found on the web.
  • Rewrite types: Four rewrite types distinguish insertion, removal, replacement, and copy operations between an original question and its rewrite.The types are identified by comparing token-set differences between Q and R.
  • Rewrite types: The most frequently replaced tokens are pronouns requiring anaphora resolution.Figure 2 identifies the ten tokens most frequently replaced in QReCC.
  • Rewrite types: 52% of QReCC questions require Replacement, where tokens are both added and removed.For example, a context-dependent pronoun can be replaced with its antecedent.
  • Rewrite types: Targeting rarer rewriting cases increased Insertion examples, enabling evaluation of reconstructing missing context beyond traditional coreference resolution.Insertion adds new tokens to produce a self-contained rewrite.

5 Document Collection

The document collection combines provenance-linked web pages with randomly sampled Common Crawl pages, then extracts and indexes 54M passages for retrieval. Relevant passages are augmented using answer-overlap heuristics.

  • Page collection: The collection includes approximately 14K Wayback Machine pages and 9.9M randomly sampled Common Crawl pages.Wayback pages come from annotator-provided answer provenance links; Common Crawl pages are English pages sampled randomly.
  • Passage construction: The downloaded pages are extracted from HTML and split into passages of at least 220 tokens, yielding 54M indexed passages.Anserini is used to index the segmented collection.
  • Relevance augmentation: Human answers are used to search the passage collection for alternative correct sources.For each returned document, the method identifies the span with the highest token-overlap F1 against the human answer.
  • Relevance augmentation: Documents with answer-span F1 ≥0.8 are treated as relevant, although human verification of this heuristic is deferred.The relevance rule is based on token overlap with the human answer.

6 Question Rewriting Metrics Validation

The study evaluates automatic question-rewriting metrics against human judgments and finds retrieval-based overlap most aligned with human assessment, while USE and ROUGE-1 R remain useful alternatives.

  • Metric comparison: BLEU compares unfavorably with alternative metrics for evaluating question-rewriting quality.The evaluation systematically compares BLEU with metrics used in summarization and translation.
  • Human validation: Question rewrites are judged correct when they capture conversational context and form self-contained questions, without requiring exact wording.Human annotators assign binary correctness labels to model-generated rewrites.
  • Human validation: Pearson correlation between annotators averages 0.94, with mean 0.083, variance 0.076, and a two-tailed-test P-value of 0.0201.These measurements quantify agreement and assessment variation.
  • Metric comparison: R@10 is the best-performing metric relative to human judgment, while USE and ROUGE-1 R are the best metrics independent of QA.R@10 compares overlap in web-search results, whereas USE and ROUGE-1 R measure text-level similarity.
  • Metric selection: The baseline selects an optimal rewriting model using the three strongest metrics: R@10, USE, and ROUGE-1 R.The metric set combines retrieval-based and text-similarity-based evaluation.

7 Baseline Approach

The baseline extends BERTserini with question rewriting before passage retrieval and reading comprehension. It evaluates multiple rewriting architectures and combines retrieval and reader scores to select answers.

  • End-to-end pipeline: The end-to-end baseline has three stages: question rewriting, passage retrieval, and reading comprehension.The rewritten stand-alone question retrieves top-k passages, and the reader extracts an answer span.
  • Question rewriting models: Question rewriting models include a coreference heuristic, pointer-generator architectures, Transformer generators, and Transformer++.Transformer++ combines two language-modeling heads through a parameterized weighted sum.
  • Question rewriting models: All generative models outperform the state-of-the-art AllenAI Coref coreference-resolution model on the rewriting evaluation.The evaluated systems include PointerGenerator and several Transformer-based models.
  • Question rewriting models: Transformer++ shows stable metric values with little variance across dialogue turns for ROUGE-1 R, USE, and R@10.Figure 3 reports these metrics across five turns and compares Transformer++ with the baseline and top encoder-decoder models.
  • End-to-end pipeline: BM25 retrieves the top-100 passages for each rewritten question, after which BERT-Large predicts answer spans or no-answer outcomes.The reader evaluates each retrieved passage against the question.
  • End-to-end pipeline: The final answer is the span with the highest weighted combination of retrieval and reading-comprehension scores.The interpolation parameter µ is tuned on held-out training data, with µ = 0.7 reported as best.

8 Baseline Results

The QReCC baseline combines question rewriting with passage retrieval and reading comprehension, showing strong gains from rewriting but substantial remaining difficulty in end-to-end QA.

  • Baseline setup: Question rewrites serve as inputs to both passage retrieval and reading comprehension in the end-to-end QA pipeline.The approach compares question rewriting with models conditioned directly on conversational context.
  • Question Rewriting Effectiveness: 5-fold cross-validation found stable question-rewriting metrics across conversation turns, with Transformer++ showing little variance.Figure 3 reports ROUGE 1-R, USE, and R@10 from the second turn onward.
  • Question Rewriting Effectiveness: All generative models outperform AllenAI Coref on question rewriting, while copy-and-generate and context-aware architectures improve over simpler alternatives.PointerGenerator outperforms Transformer-only Generator; GECOR, CopyTransformer, and Transformer++ provide further improvements.
  • Question Answering Effectiveness: F1 = 75.45 for the extractive upper bound remains far above the end-to-end baseline, indicating that retrieval and extractive reading comprehension leave substantial room for improvement.The evaluation distinguishes end-to-end QA, known context, and an extractive upper-bound setting.
  • Question Answering Effectiveness: Human rewritten questions more than double the effectiveness of original questions, while Transformer++ approaches the performance of human-level rewriting.The comparison covers both passage retrieval and extractive QA results.

9 Conclusion

QReCC is introduced as a dataset covering the main subtasks of open-domain conversational QA and as a benchmark for rewriting and end-to-end QA.

  • Dataset and task coverage: QReCC covers question rewriting, passage retrieval, and reading comprehension for open-domain conversational QA.It is presented as the first dataset covering all three subtasks relevant to conversational QA.
  • Baseline contribution: The authors establish first end-to-end baseline results by combining an open-domain QA model with a question-rewriting model.The baseline evaluates conversational QA using rewritten questions.
  • Rewrite evaluation: The evaluation compares automatic rewrite metrics and identifies those that best proxy human judgment.The paper also reports that question rewriting resolves ellipsis and co-reference in conversational dialogue.
  • Conclusion: The end-to-end baseline achieves F1 = 19.10 versus an extractive upper bound of 75.45, indicating substantial room for improvement and a need for abstractive techniques.The reported gap concerns conversational QA on QReCC.

A Reproducibility

The reproducibility materials describe dataset construction, web-page collection and passage segmentation, annotation instructions, and training-resource details for the rewriting and reading-comprehension models.

  • Question rewriting training: Transformer++ is initialized from GPT-2 medium and fine-tuned on QReCC question rewrites using model parallelism on 5 Tesla V100 GPUs.Hyperparameter selection and tuning ranges are provided in Table 7.
  • Dataset construction: QReCC collection proceeds in dialogue and document phases, linking annotated answers to downloaded web pages before adding Common Crawl pages.Pages are preprocessed and split into passages after collection.
  • Document collection: 97.8% of unique answer-containing web pages identified by annotators had an associated archived copy on the Wayback Machine.The final collection combines archived pages with random Common Crawl pages.
  • Reading comprehension training: BERT-L is trained with human rewritten questions using answer-containing passages plus negative examples selected through a tuned negative-example ratio.The implementation uses 25 paragraphs per document context and follows a standard HuggingFace-style reader setup.
  • Annotation instructions: Annotation instructions require rewrites to remain close to the original, omit prior-context references and pronouns, and produce concise natural answers.Non-text answers are represented by source URLs, while answers may contain up to 30 words.

C Pitfalls of the Query Rewriting Metrics

Question-rewriting metrics differ in what they capture: text similarity can miss semantic equivalence, while retrieval-based metrics better track human judgment but depend on search behavior and are more costly. QReCC examples also show that answer evaluation is complicated by multiple valid answers.

  • Retrieval-based evaluation: Recall@10 correlates better with human judgment than text-similarity metrics, but requires an API call for every query and depends on the underlying collection and search engine.The authors therefore use text metrics for rapid development comparisons and retrieval metrics for evaluating preselected best models.
  • Metric limitations: ROUGE-1 R treats a missing letter and a missing informative word equivalently, making it a rough estimate of rewrite quality.Its word-counting formulation does not distinguish the importance of missing words.
  • Metric limitations: USE is more sensitive than ROUGE to character-level variations in paraphrased question rewrites.For the examples given, USE scores were 0.96 and 0.91 for closely related rewrites.
  • Answer evaluation: A single QReCC question can have multiple correct answers, so standard exact-answer evaluation may undervalue manually verified outputs.The baseline sometimes selected a different passage or answer span yet produced answers judged correct on inspection.
Loading 2010.04898v3…