Source-linked AI summary
CoQA: A Conversational Question Answering Challenge
Siva Reddy, Danqi Chen, Christopher D. Manning
TL;DR
CoQA addresses the lack of large-scale datasets for conversational questions that depend on dialogue history. It introduces a dataset with free-form answers, rationales, and diverse domains, and finds that the best system reaches 65.4% F1 versus 88.8% for humans. The results highlight challenging conversational phenomena and substantial remaining performance headroom.
Problem
Large-scale reading-comprehension datasets did not contain questions dependent on conversation history, despite the need for machines to answer interconnected conversational questions.
Method
CoQA collects conversational questions with free-form answers and passage rationales across seven domains, then benchmarks conversational and reading-comprehension models.
Results
65.4% F1 is achieved by the best system, compared with 88.8% F1 for humans, a 23.4-point gap.
Takeaways & Limitations
CoQA shows that conversational questions involve challenging phenomena such as coreference and pragmatic reasoning, creating substantial room for improvement.
Takeaways & Limitations
Evaluation based on word overlap remains challenging for abstractive responses because different meanings can be expressed with non-overlapping words.
Abstract
from arXiv · showhide
Humans gather information by engaging in conversations involving a series of interconnected questions and answers. For machines to assist in information gathering, it is therefore essential to enable them to answer conversational questions. We introduce CoQA, a novel dataset for building Conversational Question Answering systems. Our dataset contains 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains. The questions are conversational, and the answers are free-form text with their corresponding evidence highlighted in the passage. We analyze CoQA in depth and show that conversational questions have challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning. We evaluate strong conversational and reading comprehension models on CoQA. The best system obtains an F1 score of 65.4%, which is 23.4 points behind human performance (88.8%), indicating there is ample room for improvement. We launch CoQA as a challenge to the community at http://stanfordnlp.github.io/coqa/
1 Introduction
CoQA introduces a conversational question-answering dataset designed for history-dependent questions, natural free-form answers with rationales, and evaluation across diverse domains. The dataset exposes challenges such as anaphora and pragmatic reasoning, while benchmark results leave substantial room for improvement.
- Questions after the first depend on conversation history, making short questions such as “Who?” difficult for systems relying on lexical similarity.
- CoQA uses free-form answers paired with highlighted span-based rationales, balancing answer naturalness with automatic evaluation.
- The dataset spans seven domains, with five used for in-domain evaluation and two reserved for out-of-domain evaluation.
- CoQA contains 127k conversation turns from 8k conversations, with an average conversation length of 15 turns.
- Almost half of CoQA questions use anaphors, and many require pragmatic reasoning, challenging models that rely on lexical cues alone.
- 65.4% F1 is achieved by the best system versus 88.8% F1 for humans, leaving a 23.4-point gap.
2 Task Definition
CoQA asks a system to answer the next question given a passage and the conversation so far. The task requires resolving history-dependent references, selecting evidence, and producing an answer or “unknown” when appropriate.
- The task provides a passage and conversation history, and requires answering the next question in the conversation.
- Answers are supported by rationales selected as contiguous text spans from the passage, while answer wording may be shorter or edited.
- Answering a question can require the full sequence of previous questions and answers, especially when the current question is underspecified.
- Coreference resolution is necessary because entities of focus can change across turns, as pronouns refer to different people in Q4 and Q5.
- Rationales are optional evidence during collection but are withheld at test time, requiring models to identify evidence themselves.
3 Dataset Collection
CoQA passages span seven domains and are selected for conversational richness, while annotation uses separate questioner and answerer roles plus answer-verification to maintain coherence.
- Annotation: Annotators act as separate questioners and answerers, creating natural dialogue, flagging poor work, and discussing disagreements through a separate chat.The setup is designed to prevent spam and obtain high-agreement data.
- Annotation: 12% of the data comes from a single worker acting as both questioner and answerer after a one-minute wait, and is used only for training.
- Passage Selection: Passages come from children’s stories, literature, exams, news, Wikipedia, Reddit, and science articles across seven domains.
- Passage Selection: Passages are selected for multiple entities, events, and pronominal references, then truncated to roughly 200 words.
- Dataset Splits: Reddit and Science are reserved for out-of-domain evaluation, while in-domain data uses 100 passages each for development and test sets.
- Answer Collection: Additional answer collection asks annotators to predict and verify original answers so follow-up questions remain coherent across the conversation.
4 Dataset Analysis
CoQA differs from SQuAD through shorter, conversational questions, free-form answers, and substantial dependence on conversational context, including coreference and pragmatics.
- Question Characteristics: CoQA question prefixes span multiple question types, whereas nearly half of SQuAD questions are dominated by what.
- Conversational Phenomena: Most CoQA question-prefix sectors contain coreferences such as he, him, she, it, and they, indicating conversational dependence.
- Question Characteristics: 5.5 words is the average CoQA question length, compared with 10.1 words for SQuAD.
- Answer Characteristics: 66.8% of CoQA answers overlap with the passage, while 33.2% do not exactly overlap after ignoring punctuation and case mismatches.
- Linguistic Phenomena: 43.0% of questions are paraphrases, 29.8% have lexical matches, and 27.2% rely on pragmatics without lexical cues.
- Conversational Phenomena: 30.5% of questions are answerable independently, 49.7% contain explicit coreference markers, and 19.8% refer implicitly through phenomena such as ellipsis.
- Answer Characteristics: Among non-overlapping answers, Yes and No comprise 48.5% and 30.3%, while 14.3% are edits improving fluency.
- Conversation Flow: Conversation focus shifts from early to later passage chunks across turns, with transitions usually staying within the same or neighboring chunk.
5 Models
CoQA is modeled through conversational response generation, extractive reading comprehension, and a combined system that predicts evidence before naturalizing answers.
- The task predicts answer a_i from passage p, prior question-answer history, and current question q_i.
- Conversational Models: Conversational models append passage, history, and the current question, then use an attention-based encoder-decoder with copying to generate answers.The resulting Pointer-Generator network can optionally copy words from the passage.
- Reading Comprehension Models: Extractive reading-comprehension models locate passage spans, making them effective learners but unable to answer when the answer does not overlap the passage.
- Reading Comprehension Models: Augmented DrQA adds yes and no tokens so extractive prediction can represent answers that are not passage spans.
- A Combined Model: The combined model uses DrQA to identify answer evidence and PGNet to convert that evidence into a free-form answer.For example, DrQA predicts rationale R5 before PGNet generates answer A5.
6 Evaluation
The evaluation uses macro-average word-overlap F1 and compares conversational, extractive, combined, and human performance across history settings and question types.
- Results and Discussion: 65.1 F1 is achieved by the combined model, competitive with augmented DrQA at 65.4, while outperforming vanilla PGNet and DrQA by 21.0 and 12.5 points.
- Results and Discussion: 88.8 F1 is achieved by humans, leaving the best model 23.4 points behind and indicating substantial room for improvement.
- Error Analysis: Human agreement is lower for answers that do not overlap the passage because word-overlap evaluation does not fully capture answer meaning.The authors identify finding a metric for abstractive responses as an unresolved challenge beyond this work.
- Error Analysis: Models and humans find pragmatic questions harder than lexical matches or paraphrases, while model behavior on coreference questions is inconsistent.
- Importance of Conversation History: Performance gains are small beyond one previous turn, and increasing history size decreases model performance.
- Importance of Conversation History: 19.9 F1 is obtained by humans without history, versus 86.4 F1 with full history; one previous turn raises performance to 79.8 F1.Two previous turns reach 85.3 F1, nearly matching full history.
- Results and Discussion: The combined model provides rationales for every answer and avoids deciding augmented classes beforehand, despite augmented DrQA being 0.3 F1 better on the test set.
7 Related work
CoQA extends conversational question answering over text passages by emphasizing multi-turn dialogue, natural questions, free-form answers, and conversational reasoning phenomena.
- Knowledge Source: CoQA uses text passages as its knowledge source, which supports crowd-sourcing without requiring expertise in structured database schemas.
- Naturalness: The dataset prioritizes natural questions by letting humans ask them, trading the scale and low cost of artificial collection methods for naturalness.
- Naturalness: Because questioners see the passage, CoQA adds measures to increase question-document independence, although complete independence is unattainable.
- Conversational Modeling: In a related text-passage conversation dataset, CoQA's interface shows the passage to both participants, whereas the comparison shows it only to the answerer.The related setup has longer average answers: 15.1 words versus CoQA's 2.7.
- Conversational Modeling: CoQA focuses on multi-turn question answering, unlike related datasets centered on clarification, concept-linked questions, or chit-chat.
- Reasoning: CoQA serves as a testbed for reasoning phenomena in conversation, complementing datasets targeting algebraic, logical, common-sense, and multi-fact reasoning.
- Recent Progress on CoQA: Subsequent CoQA work explored stacking single-turn models along conversational flow and incorporating pretrained language representations such as BERT.
8 Conclusions
The paper introduces CoQA as a large-scale dataset for conversational question answering and argues that it can stimulate research in conversational modeling. It also situates the work alongside advances from pretrained BERT models.
- CoQA is a large-scale dataset for building conversational question answering systems.It includes conversational questions, free-form answers with text-span rationales, and passages from seven diverse domains.
- The authors hope CoQA will stimulate research in conversational modeling for natural human-machine communication.
- Pretrained BERT models released in November 2018 demonstrated large improvements across a wide variety of NLP tasks.
Worker Selection
Workers were screened with a qualification test assessing their understanding of conversational question-answering guidelines, while real conversations allowed judgment and diversity.
- Workers first had to pass a qualification test assessing their understanding of conversational QA guidelines.The guidelines described conversations about passages, examples, and relevant do’s and don’ts.
- 57% of 960 attempted workers succeeded on the qualification test.
- Workers retained complete freedom to judge what was good or bad during the real conversation.The authors report that this helped curate diverse categorizations.
Annotation Interface
Figure 5 presents separate annotation interfaces for the questioner and answerer roles in the CoQA collection process.
- Figure 5 shows an annotation interface for questioners.
- Figure 5 shows an annotation interface for answerers.
- The figure presents both questioner and answerer interfaces together.
Additional Examples
The examples illustrate conversational follow-up questions across time, geography, attributes, and named entities, including affirmative, negative, and unknown answers. They also show how rationales connect answers to passage text.
- Figures: Figures 5 and 6 provide additional views of the annotation interfaces and a conversation exploring questions related to time.
- Entity-focused questions: Entity-focused follow-ups resolve pronouns and connect a person to an organization and its founding year.Rudolph Virchow is linked to the Berlin Society of Anthropology, founded in 1869, while the American Anthropological Association is identified as founded after the Italian society.
- Time-focused questions: A time-focused conversation asks about an organization’s founding years and follows references across successive questions.Examples cover the American Anthropological Association in 1902 and related societies founded in 1865, 1870, and 1871.
- Geography-focused questions: A geography-focused conversation asks about New Jersey’s location, borders, island status, size, population, wealth, and country.The examples use passage rationales for each answer, including “No” for island status and “unknown” for population.
- Answer forms: The examples include answers such as “Yes,” “No,” and “unknown,” demonstrating varied conversational answer forms.Figure 7 specifically highlights a conversation containing “No” and “unknown” as answers.