Source-linked AI summary
QASC: A Dataset for Question Answering via Sentence Composition
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, Ashish Sabharwal
TL;DR
QASC addresses multi-hop QA where systems must retrieve and compose facts from a large corpus despite questions that do not reveal the needed decomposition. The dataset provides supporting-fact annotations and motivates a two-step retrieval approach, which improves retrieval and QA performance while leaving a substantial gap to humans.
Problem
Multi-hop QA needs datasets that support retrieval and reasoning over multiple facts, but existing resources often lack supporting-fact annotations or make decomposition syntactically obvious.
Method
QASC provides annotated supporting and composed facts, uses them for two-step retrieval, and strengthens reasoning models with additional multiple-choice QA training.
Results
A two-step retrieval approach improves gold-fact recall by 43 points and QA accuracy by 14 points, while BERT models remain 20% below the 93% human baseline.
Takeaways & Limitations
QASC establishes a benchmark for retrieving and composing implicitly decomposed facts, with fact-level annotations enabling progress on both retrieval and QA.
Takeaways & Limitations
Current large pre-trained language models still trail the 93% human baseline by 20% despite massive additional training and retrieved knowledge.
Abstract
from arXiv · showhide
Composing knowledge from multiple pieces of texts is a key challenge in multi-hop question answering. We present a multi-hop reasoning dataset, Question Answering via Sentence Composition(QASC), that requires retrieving facts from a large corpus and composing them to answer a multiple-choice question. QASC is the first dataset to offer two desirable properties: (a) the facts to be composed are annotated in a large corpus, and (b) the decomposition into these facts is not evident from the question itself. The latter makes retrieval challenging as the system must introduce new concepts or relations in order to discover potential decompositions. Further, the reasoning model must then learn to identify valid compositions of these retrieved facts using common-sense reasoning. To help address these challenges, we provide annotation for supporting facts as well as their composition. Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges. We use other multiple-choice datasets as additional training data to strengthen the reasoning model. Our proposed approach improves over current state-of-the-art language models by 11% (absolute). The reasoning and retrieval problems, however, remain unsolved as this model still lags by 20% behind human performance.
1 Introduction
QASC is a 9,980-question dataset designed to test retrieving and composing two facts when the needed decomposition is not evident from the question. It provides fact-level annotations and a two-step retrieval approach, but strong models still trail humans.
- QASC contains 9,980 8-way multiple-choice science questions focused on composing facts.The questions cover elementary and middle school science.
- QASC questions are difficult because syntactic cues do not reveal how to decompose the question relation into the two supporting facts.One supporting fact may have no lexical overlap with the question, further complicating retrieval.
- The dataset annotates two supporting facts and their composed fact for each question, enabling supervised retrieval and reasoning.Questions are human-authored through a multi-step crowdsourcing process.
- The proposed two-step retrieval approach improves gold-fact recall by 43 points and QA accuracy by 14 points.The approach uses question-relevant facts to guide a second retrieval step.
- 20% separates current large pre-trained language models from the 93% human baseline, despite additional QA training and retrieved knowledge.This gap indicates that QASC’s retrieval and reasoning challenges remain unresolved.
2 Comparison With Existing Datasets
QASC differs from prior multi-hop datasets by combining a guaranteed corpus, annotated supporting facts, and questions whose decompositions are not syntactically explicit. It targets relation composition rather than primarily discourse tracking or knowledge-base path decomposition.
- QASC explicitly annotates two supporting facts from a large corpus, unlike datasets lacking answer-support annotations or relying on a small fact book.Its retrieval setting uses one large corpus containing both fact sets.
- QASC focuses on relation composition, whereas single-passage datasets emphasize passage discourse and entity tracking.This distinction separates QASC from MultiRC and related single-passage benchmarks.
- Unlike knowledge-base-derived or unannotated web datasets, QASC requires models to retrieve and compose candidate natural-language facts.Its construction is intended to support both retrieval and reasoning research.
- QASC represents a two-hop question through a question relation rQ, two hidden sub-relations rS and rL, and a bridge concept y.The answer candidate supplies the final unknown while the model must infer the missing relations and bridge.
- Many existing multi-hop questions are syntactically decomposable, but QASC questions do not clearly expose their two sub-relations.A single question relation may admit multiple decompositions, increasing retrieval and reasoning difficulty.
3 Multihop Question Collection
QASC questions are created by composing a seed fact with a worker-retrieved fact from a large corpus, then validated and used to construct challenging multiple-choice items.
- Question generation: Workers start from one seed fact and find a relevant fact in a large corpus to compose into a new fact and multiple-choice question.The seed set contains 928 facts, while the large corpus contains 17M cleaned facts.
- Question generation: Automated checks require an intermediate entity to be dropped when composing the two facts, helping enforce compositional consistency.
- Question validation: Questions are screened by baseline QA systems so at least one distractor is preferred over the correct answer, with an extra incentive when both systems are distracted.
- Baseline QA systems: BERT-MCQ scores answer choices with a linear layer over the [CLS] representation and can append retrieved context to the input.
- Question validation: Validation by five workers reduced the collection to 7,660 questions, while IR and BERT accuracy was 32.25% and 38.73%, respectively.Every retained question has two annotated composable supporting sentences.
- Question validation: The resulting analysis motivates a new baseline model intended to outperform existing models and support adversarial distractor selection for the final dataset.
4 Challenges
QASC makes retrieval and reasoning difficult because its supporting facts and relation decompositions are usually not evident from the question, unlike syntactically decomposable benchmarks.
- Retrieval challenges: The first supporting fact may overlap only with the answer, requiring the model to retrieve it and compose its relation with another fact.
- Retrieval challenges: QASC retrieves relevant facts from one large corpus rather than assuming a separate small book of facts.
- Retrieval challenges: 96% of crowd-sourced questions have at least one annotated fact sharing fewer than three non-stopword tokens with the question-plus-answer query.
- Reasoning challenges: QASC compositions can yield the same relation through different rules, such as located + causes ⇒ causes and causes + causes ⇒ causes.
- Reasoning challenges: In an analysis of 50 questions, only 2 explicitly mentioned both relations needed for the answer, compared with 47 of 50 HotpotQA questions.
- Reasoning challenges: The annotated examples illustrate that relations must be composed even when the decomposition is not evident from the question or composed fact.
5 Question Answering Model
The paper combines two-step retrieval with BERT-based multiple-choice reasoning, using retrieved facts and intermediate concepts to improve multi-hop QA performance.
- Two-step retrieval: The two-step retrieval method first retrieves question-plus-answer facts, then searches for facts connecting newly introduced concepts to concepts still missing from the query.
- Two-step retrieval: The method uses K=20 first-stage facts and L=4 second-stage facts per candidate to promote efficiency and diversity.
- Retrieval results: Two-step retrieval raises top-10 recall of both supporting facts from 2.9% to 44.4%, a 15X improvement over single-step retrieval.
- Retrieval results: When retrieving either supporting fact, two-step retrieval reaches 69.9% recall versus 42.0% for single-step retrieval.
- BERT reasoning: The QA models are BERT systems fine-tuned on other QA datasets and supplied with retrieved sentences as context.
- Question answering results: After dataset fine-tuning, BERT accuracy rises from 63.3% without retrieval context to 78.2% with single-step context and 86.4% with two-step context.
6 Adversarial Choice Generation
QASC expands human-authored questions into challenging 8-way multiple-choice items by selecting adversarial distractors that fool multiple QA models. The resulting dataset is evaluated against original and randomly augmented alternatives.
- Adversarial dataset construction: Human-authored questions remain intact while model-guided adversarial choice generation expands each into an 8-way multiple-choice question.Only the answer choices are augmented, reducing the opportunity to reverse-engineer question construction.
- Adversarial dataset construction: The distractor pool starts with approximately 30 choices per question, including correct answers from other, most dissimilar questions.Filtering removes easy distractors and reduces the chance that a correct answer is added as a distractor.
- Dataset validation: The selection process adds seven adversarial choices so each question contains the correct answer and human-authored valid distractors within an 8-way format.Additional validation removes distractors judged valid and low-quality questions from development and test sets.
- Multi-adversary selection: Two BERT-MCQ models rank distractors by how often they fool the models, with score differences resolving ties.The models are fine-tuned on RACE + SCI and randomized-distractor training questions.
- Difficulty evaluation: Adversarial choices lower BERT scores, whereas random distractors have almost no effect compared with the original 4-way dataset.This comparison tests whether the generated choices create genuinely harder questions.
- Final dataset: The final dataset contains 9,980 questions, each annotated with two supporting facts drawn from a 17M-sentence corpus.Questions are split into train, development, and test folds and presented as shuffled 8-way multiple-choice items.
7 Experiments
Experiments evaluate QASC with non-BERT baselines, BERT models, retrieval variants, and additional fine-tuning. Results show that retrieval helps, two-step retrieval is consistently stronger, and substantial progress remains toward human performance.
- Experimental setup: The experiments include GloVe-based odd-one-out and ESIM Q2Choice baselines alongside BERT models.ESIM Q2Choice uses question-choice entailment scores, with and without ELMo embeddings.
- BERT evaluation: BERT reaches up to 67% accuracy with QASC fine-tuning and two-step retrieval, while whole-word masking plus four relevant MCQ datasets raises accuracy to 73.2%.The latter result remains over 19.8 percentage points below the 93% human baseline.
- Baseline evaluation: OpenBookQA models perform close to the random baseline on QASC, indicating that statistical shortcuts from those systems do not transfer effectively.These models were near state of the art on OpenBookQA but rely largely on correlations between questions and answer choices.
- Retrieval comparison: Two-step retrieval consistently outperforms single-step retrieval, even with a larger corpus.For single-step retrieval, the larger corpus performs better; for two-step retrieval, the smaller corpus is better because larger corpora are noisier.
- Human-performance gap: The best reported BERT models remain about 20% below the 93% human score despite additional fine-tuning.Replacing bert-large-cased with whole-word masking improves the score by 4.7%.
8 Conclusion
QASC targets multi-hop reasoning beyond a single paragraph by requiring retrieval and composition of two annotated facts that are not easily recoverable through syntactic decomposition. Its results establish a difficult benchmark for current models.
- Conclusion: QASC annotates two facts needed to answer each question while making their decomposition difficult to infer from the question itself.Models must retrieve and compose candidate knowledge rather than follow an obvious syntactic split.
- Conclusion: The dataset is produced through crowdsourcing and strengthened with multi-adversary distractor choice selection.These design choices target both retrieval and reasoning difficulty in multi-hop multiple-choice QA.
- Conclusion: State-of-the-art BERT models still leave a large margin to human performance despite extensive fine-tuning and two-step retrieval.The conclusion characterizes QASC as a continuing challenge for the research community.
A.2 Crowdsourcing task architecture
The crowdsourcing workflow evolved from separate tasks into one longer, checked task after early submissions were low quality. Server-side checks and examples were used to improve consistency.
- Initial workflow: Initial independent tasks for question-answer creation and distractor collection produced low-quality submissions.The authors attributed this to insufficient checklists and distractors that were too far removed from the question context.
- Workflow revision: The final workflow combined the two simpler tasks into one longer, more complex task.This change was intended to keep workers aligned across question, fact, and distractor creation.
- Quality control: Server-side quality checks supplied immediate feedback, and examples helped workers produce more consistent submissions.Each stage required specific criteria to be satisfied before workers could proceed.
- Worker recruitment: Workers were recruited through Mechanical Turk using United States location, Master’s qualification, extensive HIT history, and high approval requirements.The work was conducted in batches, with each batch checked before the next began.
- Collection statistics: The process began with 928 seed facts and yielded 11,021 distinct human-generated questions.Workers received US$1.25 per question plus a US$0.25 bonus when both QA systems were distracted.
B Dataset Split
QASC constructs train/dev/test splits by assigning seed facts to folds while reducing cross-fold similarity and preserving target question proportions.
- Split objective: Seed-fact similarity is computed with unnormalized tf-idf overlap, and the split seeks low similarity between facts assigned to different folds.Each seed fact may have a variable number of associated questions.
- Optimization: An Integer Linear Program assigns each seed fact to exactly one fold and minimizes similarity across folds.Edge variables indicate whether pairs of seed facts are placed in different folds.
- Fold sizes: The intended train/dev/test allocation is approximately 78/11/11% of questions, with a 1% slack constraint.The extra development and test questions account for validation filtering.
- Efficiency: Low-similarity edges are ignored to make the optimization efficient, using a tf-idf threshold of 10 in the experiments.This setting was chosen to ensure completion within an hour using GLPK.
C Distractor Options
Distractors are selected from dissimilar questions within the same fold, using source-fact similarity and answer-length constraints rather than surface wording alone.
- Candidate selection: Distractor candidates come from correct answers of same-fold questions whose underlying source facts are most dissimilar to the target question.Source facts are used because surface form does not adequately capture topical similarity.
- Answer constraints: Candidate answers are restricted to within two tokens and 50% of the correct answer’s character length.The restriction prevents answer length from becoming a reliable cue.
- Candidate selection: The procedure takes answer choices from the 300 most dissimilar questions after reverse-sorting by overlap between their two source facts.A BERT baseline is then used to evaluate the distractor candidates.
- Retrieval: Retrieval queries are run over an ElasticSearch index, with retrieval time scaling linearly with search breadth K.L is kept small to promote diversity and prevent one high-scoring fact from dominating results.
- Model configuration: BERT-Large experiments use sequences of up to 184 tokens and a fixed learning rate of 1e-5.Epochs and effective batch sizes are varied in a small sweep, with models selected by validation accuracy.
E Overlap Statistics
The paper reports overlap statistics for crowd-sourced questions and illustrates the worker workflow from finding facts through composing them into questions and answer choices.
- Overlap statistics: Overlap statistics measure token overlap between each fact and the question-plus-answer, using stop-word-filtered stemmed tokens.The first three rows report percentages under overlap thresholds, while the last two report average overlap counts.
- Fact composition: Workers begin with a seed fact, search for a second fact sharing at least one word, and write a conclusion from combining them.The workflow examples show this sequence across the MTurk instructions.
- Crowdsourcing workflow: The workflow includes a prerequisite enabling the composition step only after the preceding search task is completed.This condition is shown in the MTurk instructions.
- Question construction: The composed fact is converted into a question and correct answer option by rearranging words and assigning words to question or answer roles.The instructions describe using blue and green words for the two roles.