PaperlayerView exact revision on arXiv

Source-linked AI summary

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning

arXiv:1809.09600v1cs.CL

TL;DR

Existing QA datasets provide limited tests of multi-document reasoning and lack supporting-fact annotations. HotpotQA crowdsources Wikipedia questions requiring reasoning across documents, alongside supporting facts and factoid comparison questions. Models perform substantially below humans across benchmark settings, with full-Wikipedia question answering harder than the distractor setting.

  • Problem

    Existing QA datasets offer limited tests of multi-document reasoning and lack supporting-fact annotations for learning reasoning processes and explaining predictions.

    Source: §1 · pp. 1–2

  • Method

    HotpotQA crowdsources Wikipedia questions requiring reasoning across multiple documents, together with supporting facts and factoid comparisons of entity properties.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Models perform substantially below humans across benchmark settings, with full-Wikipedia question answering harder than the distractor setting.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    Existing neural architectures can learn to answer certain multi-hop question types with sufficient training data.

    Source: §3 · p. 4

    The proposed approach to incorporating supporting-fact supervision is likely suboptimal.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

1 Introduction

HotpotQA introduces crowdsourced Wikipedia questions requiring reasoning across multiple documents without a predefined knowledge-base schema. It provides supporting facts for explainable reasoning and comparison questions that test shared properties of two entities.

1 Introduction

  • Existing QA datasets often permit answers from a single sentence or nearby sentences, limiting their ability to test reasoning across documents.SQuAD (Rajpurkar et al., 2016) uses single-paragraph contexts. TriviaQA (Joshi et al., 2017) and SearchQA (Dunn et al., 2017) retrieve multiple documents, but most questions remain answerable within one paragraph.Source: §1 · pp. 1–2
  • Knowledge-base schemas constrain question and answer diversity in QAngaroo (Welbl et al., 2018) and COMPLEXWEBQUESTIONS (Talmor and Berant, 2018), despite their multi-hop focus.Source: §1 · p. 2
  • HotpotQA asks crowd workers to create questions requiring reasoning across all supplied supporting documents, without imposing a pre-existing knowledge-base schema.The dataset uses Wikipedia articles and a carefully designed collection pipeline to obtain high-quality multi-hop questions.Source: §1 · p. 2
  • Worker-annotated supporting facts provide strong supervision for reasoning and explainable predictions, addressing earlier datasets’ answer-only supervision.Answer-only supervision makes it difficult for models to learn the underlying reasoning process or explain their predictions.Source: §1 · p. 2
  • HotpotQA introduces comparison questions requiring systems to compare two entities’ shared properties, testing language understanding and concepts such as numerical magnitude.Source: §1 · p. 2

2 Data Collection

HotpotQA’s collection pipeline selects Wikipedia paragraph pairs to support diverse questions requiring multi-hop reasoning. It combines bridge-entity and comparison questions with sentence-level supporting facts for supervision and explanation evaluation.

2 Data Collection

  • The collection design targets diverse text-based span-answer questions requiring multi-hop reasoning, rather than questions constrained by knowledge-base relations.Pilot studies found that arbitrary paragraph sets made meaningful multi-hop questions difficult for crowd workers to formulate, motivating deliberate paragraph-pair selection.Source: §2 · p. 2

Building a Wikipedia Hyperlink Graph

  • A directed graph built from hyperlinks in Wikipedia articles’ first paragraphs supplies entity relations for constructing multi-hop contexts.First paragraphs often contain useful queryable information, while hyperlinks connect already disambiguated entities; an edge (a, b) denotes a link from article a’s first paragraph to article b.Source: §2 · p. 2

Generating Candidate Paragraph Pairs

  • Candidate paragraph pairs follow Wikipedia hyperlinks whose targets are manually curated bridge entities connecting the two paragraphs.Restricting bridge entities avoids overly generic targets, such as countries, and highly technical targets that make meaningful questions difficult to write.Source: §2 · p. 3

Comparison Questions

  • 42 manually curated Wikipedia lists supply pairs of similar entities for comparison questions.Two paragraphs are randomly sampled from the same list. The authors present text-based comparison questions as novel; answering them usually requires arithmetic comparisons, such as comparing ages from birth dates.Source: §2 · p. 3
  • Yes/no comparison questions increase diversity and require reasoning over both paragraphs rather than allowing answers from just one entity’s article.For example, asking whether two bands share a country requires checking both, unlike asking which band comes from a specified country.Source: §2 · p. 3

Collecting Supporting Facts

  • Crowd workers identify answer-determining sentences as supporting facts, providing strong supervision and ground truth for evaluating model explanations.Models are intended to output the supporting facts necessary to reach their answers, allowing predicted evidence to be compared with annotated evidence.Source: §2 · p. 3

3 Processing and Benchmark Settings

HotpotQA separates mostly single-hop, model-answerable multi-hop, and hard examples, combining them for training but restricting evaluation to hard examples. Its two benchmarks test answering with noisy candidate paragraphs or locating relevant evidence across Wikipedia.

3 Processing and Benchmark Settings

  • 112,779 valid examples were collected through Amazon Mechanical Turk using ParlAI (Miller et al., 2017), with 18,089 mostly single-hop examples assigned to train-easy.The authors sampled approximately 3–10 questions per top-contributing worker, collectively responsible for over 70% of the data. Workers’ questions entered train-easy when an overwhelming percentage of their sampled questions required only one paragraph.Source: §3 · pp. 3–4
  • 56,814 multi-hop examples, representing 60% of the remaining multi-hop questions, became train-medium after models answered them correctly with high confidence during three-fold cross-validation.Confidence was determined by thresholding model loss.Source: §3 · p. 4
  • Development and test sets contain only hard examples, while the default training set combines train-easy, train-medium, and train-hard.Remaining hard examples are randomly divided into train-hard, dev, test-distractor, and test-fullwiki. The authors report that combining all three training subsets yields the best performance.Source: §3 · p. 4, Table 1 · p. 4
  • The full wiki setting requires answering from the first paragraphs of all Wikipedia articles without identifying the gold paragraphs.Separate test sets prevent gold paragraphs exposed in the distractor setting from leaking into full wiki evaluation.Source: §3 · p. 4, Table 1 · p. 4
  • 8 Wikipedia distractors retrieved using bigram tf-idf are shuffled with 2 gold paragraphs to create the distractor setting.Retrieval uses the question as the query, challenging models to find supporting facts amid noise.Source: §3 · p. 4
  • 93.3% of train-medium questions were multi-hop versus 92.0% in dev, showing that model-answerable examples were not predominantly single-hop.Type II questions were more frequent in train-medium (32.0%) than dev (15.0%), supporting the conclusion that existing architectures can learn some multi-hop subsets given enough training data. Retrieval on train-medium nevertheless remained as difficult as on hard examples when gold paragraphs were not supplied alone.Source: §3 · p. 4

4 Dataset Analysis

HotpotQA covers diverse question and answer types and several multi-hop reasoning patterns; bridge-entity chain reasoning accounts for 42% of sampled dev/test questions. The analysis also identifies single-hop and unanswerable questions.

Question Types

  • Question types span entities, locations, events, dates, numbers, and yes/no comparisons between entities.Heuristic classification selects the first WH-word, copula, or auxiliary verb when found within the first three tokens, otherwise selecting the last question word. Types include up to two following tokens and, when applicable, a preceding common preposition.Source: Figure 2 · p. 5, §4 · p. 5

Answer Types

  • 68% of questions in a 100-example answer-type sample concern entities, while other questions ask about dates, numbers, and adjectives.Dates account for 9%, numbers for 8%, and adjectives for 4%.Source: Table 2 · p. 5, §4 · p. 5

Multi-hop Reasoning Types

  • Other reasoning patterns involve intersecting entity properties, inferring properties through bridge entities, and comparing entities, sometimes using counting or numerical comparisons.Type II identifies entities satisfying multiple properties; Type III infers properties shared with a bridge entity. Some questions require more than two supporting facts.Source: §4 · p. 5
  • Worker-provided supporting facts exactly match the relevant context in all Table 3 examples, supporting the authors’ assessment of high-quality annotations.This observation concerns the examples shown in the table, rather than a dataset-wide annotation audit.Source: §4 · pp. 5–6
  • 42% of 100 sampled dev/test questions require chain reasoning: identifying a bridge entity before answering the second hop.Most questions require supporting facts from each paragraph. Chain questions could be decomposed into consecutive single-hop questions.Source: §4 · p. 5, Table 3 · p. 6
  • Among 100 randomly sampled examples from train-medium and train-hard combined, reasoning types were Type I 38%, Type II 29%, Comparison 20%, Other 7%, Type III 2%, single-hop 2%, and unanswerable 2%.The passage also estimates that about 6% of sampled questions can be answered with one of the two paragraphs, and 2% are unanswerable, without identifying that sample as dev/test.Source: §4 · p. 6

5 Experiments

HotpotQA experiments show that a strongly supervised QA baseline performs substantially worse in full-wiki than distractor evaluation and remains below human performance. Supporting-fact supervision improves answer performance and enables sentence-level evidence prediction.

5.1 Model Architecture and Training

  • The Clark and Gardner (2017) baseline combines character-level modeling, self-attention, and bi-attention with a classifier selecting yes, no, or span answers.When the classifier selects a span answer, decoding searches for the most probable text span.Source: §5.1 · p. 6
  • An end-to-end sentence classifier predicts supporting facts from self-attention representations, jointly optimizing binary cross-entropy and QA objectives with shared low-level representations.The classifier concatenates self-attention outputs at each sentence’s first and last positions, enabling direct evaluation of supporting-fact prediction.Source: §5.1 · p. 7

5.2 Results

  • 5,000,000+ Wikipedia paragraphs are filtered to at most 5,000 candidates, then ten are selected using bigram tf-idf for evaluation with the distractor-trained model.Candidate filtering uses an inverted index. Evaluation reports answer, supporting-fact, and joint EM/F1; joint EM requires both tasks to match exactly, and joint metrics penalize weakness on either task.Source: §5.2 · p. 7
  • Full-wiki performance is substantially below distractor performance, and both settings trail humans, despite strong supporting-fact supervision.60+ supporting fact prediction F1 and ∼40 joint F1 indicate substantial room to improve explainability.Source: §5.2 · p. 7, Table 4 · p. 8
  • Comparison questions have lower distractor-setting F1 than bridge-entity questions, but degrade only marginally in full wiki, where bridge-entity performance drops substantially.Comparison questions usually name both entities, simplifying retrieval; the authors attribute much of the full-wiki deterioration to difficulty retrieving both entities.Source: §5.2 · pp. 7–8, Table 6 · p. 8
  • 10+ F1 improvement over not using supporting facts comes from supporting-facts-only oracle context, versus ∼2 points in F1 from supporting-fact supervision.Self-attention and character-level models also contribute notably, suggesting single-hop QA techniques remain useful. Combining train-easy, train-medium, and train-hard yields the best performance and is the default.Source: Table 7 · p. 8, §5.2 · p. 8

5.3 Establishing Human Performance

  • ∼30% gaps in both EM and F1 separate the baseline from crowd workers on answering and supporting-fact prediction when distractor paragraphs are present.With correct supporting paragraphs supplied, the baseline matches crowd workers on supporting-fact prediction but still falls short on answering.Source: Table 8 · p. 9, §5.3 · p. 9
  • Close to 100% on most metrics, the human upper bound reflects agreement among subsets of annotators, with lower agreement on supporting facts.Human evaluation sampled 1,000 dev/test examples, collected annotations from at least three additional Turkers, and selected references maximizing F1 to reduce ambiguity (Rajpurkar et al., 2016). The upper bound maximizes per-example EM and F1 while evaluating each worker against the others.Source: §5.3 · pp. 8–9, Table 8 · p. 9

6 Related Work

Related work spans single-document, multi-document, KB-based multi-hop, and free-form answer-generation datasets, with limitations in reasoning requirements, question diversity, or evaluation reliability.

Single-document datasets

  • SQuAD (Rajpurkar et al., 2016, 2018) questions are relatively simple because answering usually requires no more than one sentence in a paragraph.Source: §6 · p. 9

Multi-document datasets

  • TriviaQA (Joshi et al., 2017) and SearchQA (Dunn et al., 2017) provide multiple context documents but do not guarantee reasoning across them.Their longer contexts challenge QA systems, but supporting documents are retrieved after question-answer pairs are collected.Source: §6 · p. 9

KB-based multi-hop datasets

  • QAngaroo (Welbl et al., 2018) and COMPLEXWEBQUESTIONS (Talmor and Berant, 2018) test multi-hop reasoning using pre-existing knowledge bases and predefined logic rules.Fixed KB schemas or logical forms constrain question and answer diversity. KB incompleteness can also leave some questions answerable from one sentence.Source: §6 · p. 9

Free-form answer-generation datasets

  • 100k Bing Search user queries underpin MS MARCO, which provides human-generated answers for free-form answer generation.Systems are evaluated using metrics such as ROUGE-L and BLEU-1, whose reliability is questioned because they correlate poorly with human judgement (Novikova et al., 2017).Source: §6 · p. 9

7 Conclusions

HOTPOTQA is a large-scale QA dataset designed to support explainable, multi-hop reasoning over diverse natural language. It also introduces factoid comparison questions testing systems’ ability to extract and compare entity properties in text.

7 Conclusions

  • HOTPOTQA targets the development of QA systems capable of explainable, multi-hop reasoning.Source: §7 · p. 9
  • A new type of factoid comparison questions tests systems’ ability to extract and compare entity properties in text.Source: §7 · p. 9
  • HOTPOTQA provides a large-scale question answering dataset covering diverse natural language.Source: §7 · p. 9

A Data Collection Details … A.3 Crowd Worker Interface

Data collection combined Wikipedia preprocessing, curated paragraph-pair sampling, worker bonuses, and a dialog-based interface that provided feedback and hints.

A.1 Data Preprocessing

  • Sentence boundaries supported supporting-fact collection, while token boundaries checked that worker answers covered entire tokens rather than nonsensical partial words.Word and sentence tokenization used Stanford CoreNLP 3.8.0.Source: §A.1 · p. 11
  • Wikipedia text and hyperlinks were extracted with WikiExtractor from the October 1, 2017 English Wikipedia dump.Source: §A.1 · p. 11

A.2 Further Data Collection Details

  • 591 Wikipedia categories were manually curated by checking sampled paragraph pairs for their suitability for multi-hop questions and selecting categories with high suitability.Categories came from WikiProject popular-page lists; sampled pairs (a, b) came from graph G with b in the category under consideration.Source: §A.2 · p. 11
  • Following Yang et al. (2018), collection mixed bonuses for top contributors every 200 examples with bonuses based on examples produced per hour.The first setting ranked workers by their number of examples; both settings aimed to incentivize higher-quality data and more efficient production.Source: §A.2 · p. 11

A.3 Crowd Worker Interface

  • The worker interface adapted ParlAI (Miller et al., 2017) to collect question-answer pairs through a system-oriented dialog.ParlAI is an open-source project supporting dialog-system development and dialog-based data collection.Source: §A.3 · p. 11
  • The dialog interface enabled greater control over worker input and supplied in-the-loop feedback and helpful hints to facilitate task completion and accelerate collection.Source: §A.3 · p. 11

B Further Data Analysis · C Full Wiki Setting Details · C.1 The Inverted Index Filtering Strategy

HotpotQA questions vary greatly in length, indicating different levels of complexity and detail. Full-wiki retrieval uses inverted-index filtering, and its evaluation assigns missing gold paragraphs a rank just beyond the candidate pool.

B Further Data Analysis

  • HotpotQA questions vary greatly in length, indicating different levels of complexity and detail beyond their diversity in question types.Figure 5 visualizes the question-length distribution.Source: B Further Data Analysis · p. 11

C.1 The Inverted Index Filtering Strategy

  • 5000 is the experimental control threshold N for efficient inverted-index filtering of preliminary paragraph candidates in the full wiki setting.Algorithm 2 specifies the filtering procedure.Source: §C.1 · pp. 11–12
  • The filtering algorithm extracts a unigram-and-bigram set from the question and takes an ngram-to-Wikidoc inverted index as input.It initializes the candidate count to infinity and Cgram to zero.Source: §C.1 · p. 12
  • The inverted-index filter resets an overlap dictionary each iteration and increments document overlap counts before assembling the candidate set.The procedure iterates over the question’s extracted ngrams.Source: §C.1 · p. 12
  • The filtering loop increases Cgram and recomputes the candidate-set size until it no longer exceeds N, then returns the candidate set.The candidate count is set to |Scand| after candidate-set construction.Source: §C.1 · p. 12
  • MAP and Mean Rank are reported as upper bounds because missing gold paragraphs receive rank |Scand| + 1 during evaluation.Gold paragraphs can be absent from the filter’s output candidate pool.Source: §C.1 · p. 12

C.2 Compare train-medium Split to Hard Ones

Train-medium has retrieval performance close to dev and test, suggesting similar difficulty in the full-wiki setting, where retrieval is a necessary first step.

C.2 Compare train-medium Split to Hard Ones

  • Train-medium’s close retrieval performance to dev and test suggests similar difficulty in the full-wiki setting.This comparison concerns the setting where a retrieval model is necessary as the first processing step.Source: Table 9 · p. 12, §C.2 · p. 12
  • 1,000 random samples from each of train-medium, dev, and test underpin Table 9’s full-wiki retrieval comparison.MAP is reported as a percentage.Source: Table 9 · p. 12
  • Mean Rank averages retrieval ranks of both gold paragraphs, while CorAns Rank measures the rank of the gold paragraph containing the answer.Source: Table 9 · p. 12

Related papers

Loading 1809.09600v1…