Source-linked AI summary
QuAC : Question Answering in Context
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer
TL;DR
Information-seeking dialog requires answering questions that may be open-ended, unanswerable, or meaningful only through prior context. QuAC constructs a large teacher-student dataset over hidden Wikipedia sections and evaluates context-aware baselines, whose best performance remains 20 F1 below the human upper bound.
Problem
Information-seeking dialog involves questions that can be context-dependent, elliptical, open-ended, or unanswerable, creating challenges for question answering.
Method
QuAC collects 14K teacher-student QA dialogs over Wikipedia sections, with students asking free-form questions and teachers selecting short evidence spans.
Results
The best reported model performs 20 F1 points below the human upper bound on QuAC.
Takeaways & Limitations
The performance gap is intended to encourage development of machines that participate more effectively in information-seeking dialog.
Takeaways & Limitations
Article-filtering steps bias the dataset toward entertainers.
Abstract
from arXiv · showhide
We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation. We also report results for a number of reference models, including a recently state-of-the-art reading comprehension architecture extended to model dialog context. Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data. Dataset, baseline, and leaderboard available at http://quac.ai.
1 Introduction
QuAC is a large-scale dataset for information-seeking question-answering dialogs over hidden Wikipedia sections. Its dialog-centered design creates contextual, open-ended, and sometimes unanswerable questions, while baseline models remain substantially below human performance.
- QuAC targets information-seeking dialogs whose questions can be highly context-dependent, elliptical, open-ended, or unanswerable.
- QuAC contains 14K crowdsourced QA dialogs comprising 100K total QA pairs.
- Each dialog pairs a student, who asks questions about a hidden Wikipedia section, with a teacher, who answers using spans from the evidence text and dialog feedback.
- The dataset includes dialog phenomena such as references to previous questions and answers and questions that avoid repeating previous information.
- Models must incorporate dialog context to perform well, because question interpretation and answering depend on the preceding interaction.
- The reported baseline performs 20 F1 points below the human upper bound on QuAC, indicating room for future improvement.
2 Dataset collection
QuAC is collected through interactive teacher-student dialogs about Wikipedia sections, with students asking unseen questions and teachers selecting text spans plus dialog acts. The process is designed to elicit natural, diverse, contextual, open-ended, and unanswerable questions.
- Two crowd workers discuss a Wikipedia article section, while the student sees limited context and the teacher sees the full section text.
- Teachers answer each free-text question by selecting a contiguous span from the section text rather than writing free-form text.This restriction makes evaluation simpler and more reliable.
- Teacher dialog acts encode continuation guidance, affirmation, and answerability, including whether to ask a follow-up question and whether an answer exists.
- Students ask new questions after each answer, and dialogs end after twelve answered questions, voluntary termination, or more than two unanswerable questions.
- Article filtering selects popular person-related Wikipedia articles across categories, with the filtering steps biasing the data toward entertainers.
3 Dataset Analysis
QuAC’s dialog-style collection produces open-ended, contextual, and sometimes unanswerable questions. Dialog behavior and answer locations also shift across turns, making context central to the task.
- Question and answer types: About half of questions are non-factoid, unlike the factoid focus of most current QA datasets.
- Question and answer types: 86% of questions are contextual; 44% refer to dialog history and 61% refer to the article subject.
- The role of context: Answers move from the beginning toward the end of the section as more questions are asked, while the middle is covered less frequently.
- The role of context: The next answer is usually in the same or an adjacent text chunk, and most dialogs cover three to six chunks.
- Question behavior across turns: As dialogs progress, yes/no questions become more frequent and questions with multiple possible answers decline.
- Qualitative examples: Qualitative dialogs often progress from general to specific questions, while unanswered questions can lead students to request interesting content or eventually end the dialog.
4 Experimental Setup
The task predicts answer spans and dialog acts from the preceding dialog, supporting materials, and the next question. Evaluation uses filtered train/dev/test splits, word-level F1, and human-equivalence measures with multiple references.
- Task definition: The model predicts answer span indices and dialog acts from prior questions and answers, supporting materials, and the next question.
- Data splits: The train/dev/test split contains 83.5k/7.3k/7.3k question-answer pairs with no sections shared across folds.
- Data splits: Dev and test questions have five reference answers, whereas training questions have one reference answer.
- Data filtering: Evaluation excludes questions with human F1 below 40, removing roughly 10% of the noisiest annotations, while unthresholded F1 is also reported.
- Evaluation metrics: Word-level F1 measures stopword-removed overlap between predicted and reference words, assigning 1 for correctly predicted no-answer cases and 0 otherwise.
- Evaluation metrics: HEQ reports whether system F1 matches or exceeds average human F1 at the question level and across every question in a dialog.
5 Experiments
Experiments show that dialog context is essential for QuAC, but even context-aware neural models remain substantially below human performance. Sanity checks, baselines, upper bounds, and error analysis indicate both the value of contextual modeling and unresolved challenges in answer selection and span prediction.
- Sanity checks: Transition-matrix performance is strongest among sanity checks, reinforcing that dialog context significantly influences the task.
- Upper bounds: 80.8 F1 is the human upper bound, while HEQ results indicate that span-based approaches are needed to achieve average human equivalence.
- Baselines: Simple text-matching baselines perform poorly, showing that QuAC questions have little direct lexical overlap with their answers.
- Baselines: Adding one previous QA pair significantly improves BiDAF++, while context lengths beyond three turns show saturation.The context-aware model reaches human equivalence on 60% of questions and 5% of complete dialogs.
- Error analysis: Human disagreement increases with answer-distance shifts, whereas model behavior differs from human behavior across turn number, answer distance, and span length.
- Error analysis: The context-aware model degrades less as dialog turns increase and performs 6 HEQ-Q higher on questions marked for follow-up.The context-agnostic baseline shows no HEQ-Q difference between follow-up and non-follow-up questions.
6 Related Work
QuAC connects sequential question answering, reading comprehension, and open-domain dialog while emphasizing student-generated information seeking over question decomposition or constrained interaction.
- QuAC builds on span-based reading comprehension while reducing trivial lexical overlap and allowing unanswerable questions.
- Unlike sequential QA systems that decompose one question, QuAC relies on a student's curiosity to generate a sequence of questions against a knowledge source.
- QuAC differs from social, visual, and goal-driven dialog through its information-seeking setting and less constrained language.
7 Conclusion
The paper introduces QuAC as a large-scale Wikipedia information-seeking dialog dataset collected through teacher–student interactions. Its contextual, open-ended, and sometimes unanswerable questions challenge existing baselines, which substantially underperform humans.
- QuAC is a large-scale dataset of information-seeking dialogs over sections from Wikipedia articles.
- Teacher–student collection encourages questions that are highly contextual, open-ended, and sometimes unanswerable from the text.
- Baseline systems that include top performers on existing machine-comprehension datasets significantly underperform humans on QuAC.