Source-linked AI summary
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning
TL;DR
Existing QA datasets provide limited support for multi-hop reasoning and explainable predictions. HotpotQA addresses this with a diverse Wikipedia dataset containing multi-document questions and sentence-level supporting facts, and shows that supporting-fact supervision improves performance while full-wiki QA remains substantially below human performance.
Problem
Existing QA datasets inadequately test multi-hop reasoning and provide only distant supervision, limiting learning of reasoning processes and explainable predictions.
Method
HotpotQA crowdsources diverse Wikipedia questions over multiple supporting documents and supplies sentence-level facts, which models predict jointly with answers.
Results
Supporting facts yield a 10+ F1 improvement over not using them, while full-wiki performance remains substantially below human performance.
Takeaways & Limitations
HotpotQA provides a benchmark for developing QA systems that perform explainable multi-hop reasoning across diverse natural-language documents.
Abstract
from arXiv · showhide
Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.
1 Introduction
Existing QA datasets inadequately test multi-document reasoning, offer limited question diversity, and provide insufficient supervision for learning and explaining reasoning. HOTPOTQA addresses these gaps through a crowdsourced Wikipedia dataset built around natural multi-hop questions and supporting evidence.
- Limitations of prior datasets: Existing datasets often test reasoning within a single paragraph, limiting the need for complex reasoning over multiple paragraphs.SQuAD is cited as an example whose questions are usually answerable by matching nearby sentences in one paragraph.
- Limitations of prior datasets: Knowledge-base-based multi-hop datasets constrain question and answer diversity through the schemas of their underlying knowledge bases.The introduction identifies QAngaroo (Welbl et al., 2018) and COMPLEXWEBQUESTIONS (Talmor and Berant, 2018) as examples.
- Dataset contribution: Unlike prior datasets providing only distant supervision, HOTPOTQA aims to identify the text supporting each answer so models can learn meaningful and explainable reasoning.The dataset design provides strong supervision about which text the answer is derived from.
- Dataset contribution: HOTPOTQA is crowdsourced from Wikipedia with multiple supporting documents, producing natural multi-hop questions without relying on a pre-existing knowledge-base schema.Crowd workers were shown multiple supporting context documents and explicitly asked to create questions requiring reasoning over all of them.
2 Data Collection
The data collection pipeline targets diverse, explainable text-based questions requiring multi-hop reasoning. It uses Wikipedia hyperlinks and curated entity lists to generate bridge-entity and comparison questions, while collecting supporting facts for supervision and evaluation.
- Pipeline design: The pipeline addresses the difficulty of collecting meaningful text-based multi-hop questions by carefully designing candidate paragraph generation rather than presenting workers with arbitrary paragraph sets.Pilot studies found arbitrary paragraph sets were counterproductive because most did not support meaningful multi-hop questions.
- Building a Wikipedia Hyperlink Graph: Wikipedia’s hyperlink graph supplies candidate paragraph pairs whose linked entities can support multi-hop reasoning, with article openings providing information suitable for querying.The corpus is the entire English Wikipedia dump, and hyperlinks often entail relations between disambiguated entities.
- Generating Candidate Paragraph Pairs: Bridge-entity questions require workers to infer an intermediate entity from one paragraph and then find its answer in the linked paragraph.In the Radiohead example, workers identify Thom Yorke as the bridge entity before determining his birthday.
- Comparison Questions: Comparison questions are collected from pairs sampled from 42 manually curated lists of similar entities and require reasoning over both paragraphs.The authors describe text-based comparison questions as novel and note that they often require arithmetic comparisons, such as comparing ages from birth dates.
- Comparison Questions: The comparison subset includes yes/no questions to broaden question diversity while requiring systems to reason over both paragraphs.The collection procedure uses a 0.5 yes/no ratio within comparison questions.
- Collecting Supporting Facts: Crowd workers also identify the sentences that determine each answer, providing supporting facts for attention supervision and explainability evaluation.These facts are intended to represent the evidence necessary to arrive at the answer.
3 Processing and Benchmark Settings
The dataset contains 112,779 valid examples, partitioned to isolate mostly single-hop questions and focus development and testing on hard multi-hop cases. Evaluation uses distractor and full-Wikipedia settings to test supporting-fact retrieval and reasoning under increasing retrieval demands.
- Data Collection: 112,779 valid examples were collected through Amazon Mechanical Turk using the ParlAI interface.The collection process is described in Appendix A.
- Data Splits: The train-easy split contains 18,089 mostly single-hop examples identified by sampling questions from top-contributing turkers.Questions were assigned to train-easy when an overwhelming proportion of sampled questions required reasoning over only one paragraph.
- Data Splits: The remaining hard examples were divided into train-hard, dev, test-distractor, and test-fullwiki, with dev and test restricted to questions current techniques could not answer.The three training splits are combined for training, while distractor and full-Wikipedia settings use different test sets so gold paragraphs remain unknown in full-Wikipedia evaluation.
- Benchmark Settings: The distractor setting mixes 2 gold paragraphs with 8 bigram tf-idf-retrieved Wikipedia distractors, shuffles them, and tests identification of supporting facts amid noise.The question is used as the retrieval query.
- Benchmark Settings: The full-Wikipedia setting requires answering from the first paragraphs of all Wikipedia, testing both relevant-fact location and reasoning.This setting therefore removes the fixed distractor-paragraph retrieval setup.
- Split Analysis: Train-medium contains a similar multi-hop ratio to dev but more Type II questions, at 32.0% versus 15.0%, helping explain its strong model performance.The corresponding multi-hop ratios are 93.3% in train-medium and 92.0% in dev.
4 Dataset Analysis
HotpotQA spans diverse question and answer types, with entity-focused questions and answers predominating. Its questions mostly require multi-hop reasoning across paragraphs, especially chain reasoning through bridge entities, while only a small minority are single-hop or unanswerable.
- Question Types: HotpotQA covers diverse questions about entities, locations, events, dates, numbers, and yes/no comparisons between entities.These question types were identified heuristically and visualized in Figure 2.
- Answer Types: 68% of sampled answers are entities, while 9% concern dates, 8% numbers, and 4% adjectives.The answer-type analysis sampled 100 dataset examples and found a broad range of answer types.
- Multi-hop Reasoning Types: 42% of sampled questions require chain reasoning, identifying a bridge entity before answering the second hop, and most require supporting facts from each paragraph.The reasoning analysis indicates that collected supporting facts are high quality because they connect the paragraphs needed to answer.
- Multi-hop Reasoning Types: In train-medium and train-hard, reasoning types comprise Type I 38%, Type II 29%, Comparison 20%, Other 7%, Type III 2%, single-hop 2%, and unanswerable 2%.A separate sample of 100 examples produced these proportions.
- Multi-hop Reasoning Types: About 6% of sampled questions are answerable from one paragraph, while 2% are judged unanswerable.The multi-hop analysis also reports these as the remaining cases outside the principal reasoning types.
5 Experiments
Experiments show that HotpotQA remains challenging as context expands from distractor paragraphs to the full wiki, while supporting-fact supervision improves QA and enables partially explainable predictions. Ablations, question-type analyses, and human comparisons identify retrieval, modeling, and annotation factors that shape performance.
- Benchmark Results: Full-wiki evaluation substantially lowers performance relative to the distractor setting, making retrieval-based question answering particularly challenging.The model first retrieves up to 5,000 paragraphs, selects 10 final candidates, and evaluates using answer and supporting-fact metrics.
- Explainability: The model achieves 60+ supporting-fact prediction F1 and approximately 40 joint F1, indicating room to improve explainability.Joint metrics require both answer-span and supporting-fact performance, penalizing systems that perform poorly on either task.
- Question Types: Comparison questions have lower distractor-setting F1 than bridge-entity questions, but their full-wiki performance declines only marginally because both entities usually appear in context.Bridge-entity performance drops significantly in the full-wiki setting, linking question-type behavior to retrieval difficulty.
- Ablation Study: Strong supervision over supporting facts improves QA, while self-attention and character-level modeling also contribute notably to distractor-setting performance.Removing supporting-fact supervision decreases performance; incorporating supporting facts yields a 10+ F1 improvement over not using them, while the model’s direct supervision gain is approximately 2 F1 points.
- Human Performance: Crowd workers achieve very high answer and supporting-fact performance, while the baseline reaches parity on supporting facts with gold paragraphs but still trails on answers.With distractors present, the baseline’s gap from crowd workers widens on both tasks; human upper-bound metrics are mostly close to 100%.
6 Related Work
Existing QA datasets span single-document, multi-document, KB-based multi-hop, and free-form answer-generation settings, but each has limitations involving reasoning depth, question diversity, answerability, or evaluation reliability.
- Single-document datasets: SQuAD (Rajpurkar et al., 2016, 2018) typically requires answering from no more than one paragraph sentence, making its questions relatively simple.
- Multi-document datasets: TriviaQA (Joshi et al., 2017) and SearchQA (Dunn et al., 2017) use multiple context documents, but retrieval-based supports do not guarantee reasoning across documents.
- KB-based multi-hop datasets: QAngaroo (Welbl et al., 2018) and COMPLEXWEBQUESTIONS (Talmor and Berant, 2018) test KB-based multi-hop reasoning, but fixed schemas limit diversity and incomplete KBs can make questions answerable from one sentence.
- Free-form answer-generation datasets: MS MARCO uses human-generated free-form answers evaluated with ROUGE-L and BLEU-1, whose reliability is questionable because they correlate poorly with human judgment (Novikova et al., 2017).
7 Conclusions
HOTPOTQA is a large-scale dataset designed to advance explainable, multi-hop reasoning over diverse natural language. It also introduces factoid comparison questions requiring systems to extract and compare entity properties in text.
- 7 Conclusions: HOTPOTQA targets explainable, multi-hop reasoning over diverse natural language at large scale.The dataset is intended to facilitate development of QA systems capable of this reasoning.
- 7 Conclusions: It introduces factoid comparison questions that test extracting and comparing various entity properties in text.
A Data Collection Details … A.3 Crowd Worker Interface
The data collection pipeline combined curated Wikipedia page pairs, token- and sentence-aware preprocessing, incentive structures, and a dialog-based crowd-worker interface. These choices were designed to support intuitive multi-hop question writing, valid answer spans, efficient high-quality annotation, and in-task worker guidance.
- A.1 Data Preprocessing: Wikipedia preprocessing used the October 1, 2017 English dump, WikiExtractor, and Stanford CoreNLP 3.8.0 for sentence and token boundaries.Sentence boundaries supported supporting-fact collection, while token boundaries helped reject answers covering nonsensical partial-word spans.
- A Data Collection Details: Overall, the collection details span page curation, linguistic preprocessing, worker incentives, and interface-level guidance within the data collection process.These components jointly describe how candidate material was prepared and how crowd annotation was controlled and supported.
- A.2 Further Data Collection Details: The pipeline curated 591 WikiProject categories and manually checked sampled graph-connected page pairs for whether they supported intuitive multi-hop questions.For each category, pages were sampled as pairs (a, b), with b in the category, before manual validation.
- A.2 Further Data Collection Details: Crowd workers received bonuses either for producing the most examples every 200 examples or for higher hourly productivity.The two settings were mixed during data collection to encourage efficient, higher-quality annotation.
- A.3 Crowd Worker Interface: The worker interface adapted ParlAI into a system-oriented dialog workflow for collecting question-answer pairs.This gave the researchers more control over Turkers’ input and enabled in-the-loop feedback and helpful hints that sped task completion.
- A.3 Crowd Worker Interface: The data collection interface was implemented as an Amazon Mechanical Turk worker interface.The paper illustrates the interface with a screenshot from the collection process.
B Further Data Analysis · C Full Wiki Setting Details · C.1 The Inverted Index Filtering Strategy
The analysis shows that HotpotQA questions vary substantially in length, reflecting differing complexity and detail. In the full-Wiki setting, an inverted-index filtering strategy retrieves candidate paragraphs efficiently, but missing gold paragraphs make reported retrieval metrics upper bounds.
- B Further Data Analysis: Questions vary greatly in length, indicating different levels of complexity and detail covered in HotpotQA.This complements the dataset’s diversity in question types.
- C.1 The Inverted Index Filtering Strategy: The full-Wiki setting uses an efficient inverted-index-based strategy for preliminary candidate paragraph retrieval with control threshold N = 5000.The strategy is specified in Algorithm 2.
- C.1 The Inverted Index Filtering Strategy: The filter extracts unigram and bigram sets from each question and iteratively increases the n-gram overlap criterion until the candidate count is at most N.Candidate documents are collected from an ngram-to-Wikidoc inverted index.
- C.1 The Inverted Index Filtering Strategy: The procedure returns the filtered candidate set Scand after updating its size during each iteration.The algorithm continues while Ncands exceeds the control threshold.
- C.1 The Inverted Index Filtering Strategy: If a gold paragraph is missing from Scand, its evaluation rank is set to |Scand| + 1, making reported MAP and Mean Rank upper bounds of their true values.This limitation applies when relevant gold paragraphs are not included in the output candidate pool.
C.2 Compare train-medium Split to Hard Ones
In the full-wiki setting, train-medium performs similarly to dev and test on retrieval metrics, indicating comparable difficulty for retrieval-based processing.
- C.2 Compare train-medium Split to Hard Ones: Train-medium has a close retrieval-performance gap to dev and test in the full-wiki setting, suggesting a similar difficulty level.Table 9 compares 1,000 random samples from each split using MAP, mean rank, and answer-containing paragraph rank.
- C.2 Compare train-medium Split to Hard Ones: The full-wiki setting requires a retrieval model as the first processing step for these comparisons.