PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Dataset Design Choices for Multi-hop Reasoning

Jifan Chen, Greg Durrett

arXiv:1904.12106v1cs.CLcs.AI

TL;DR

The paper asks whether datasets designed for multi-hop question answering actually require multi-hop reasoning, and investigates WikiHop, HotpotQA, and their answer formats. Sentence-factored models solve many examples, while multiple-choice versions are vulnerable to no-context baselines, leaving uncertainty about how much multi-hop reasoning high-performing models learn.

  • Problem

    Although WikiHop and HotpotQA are designed to test multi-hop reasoning, it is unclear whether models need that reasoning to perform well on them.

    Source: §1 · p. 1

  • Method

    The paper tests sentence-factored models and no-context baselines on WikiHop and HotpotQA, and compares span-based with multiple-choice supervision.

    Source: §1 · p. 1

  • Results

    More than half of the questions in WikiHop and HotpotQA do not require multi-hop reasoning, and in multiple-choice settings the no-context baseline is only around 2% lower than the two models.

    Source: §1 · p. 1, §4.2 · p. 4

  • Takeaways & Limitations

    The authors conclude that span-based supervision is a harder testbed than multiple choice, but strong sentence-factored performance leaves it unclear whether proposed models learn multi-hop reasoning in any setting.

    Source: §1 · p. 2

    The investigation leaves WorldTree, OpenBookQA, and MultiRC for future work because their answers may not appear directly in passages and may instead be entailed by passage content.

    Source: §5 · p. 5

Abstract

from arXiv · show

Learning multi-hop reasoning has been a key challenge for reading comprehension models, leading to the design of datasets that explicitly focus on it. Ideally, a model should not be able to perform well on a multi-hop question answering task without doing multi-hop reasoning. In this paper, we investigate two recently proposed datasets, WikiHop and HotpotQA. First, we explore sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets. Furthermore, we find spurious correlations in the unmasked version of WikiHop, which make it easy to achieve high performance considering only the questions and answers. Finally, we investigate one key difference between these datasets, namely span-based vs. multiple-choice formulations of the QA task. Multiple-choice versions of both datasets can be easily gamed, and two models we examine only marginally exceed a baseline in this setting. Overall, while these datasets are useful testbeds, high-performing models may not be learning as much multi-hop reasoning as previously thought.

1 Introduction

The paper asks whether WikiHop and HotpotQA truly require multi-hop reasoning, and examines how their QA formats affect evaluation. Sentence-factored models solve many examples, while multiple-choice versions can be vulnerable to question-and-answer correlations.

1 Introduction

  • WikiHop and HotpotQA were designed to test multi-hop reasoning by requiring information gathering across multiple documents.The paper frames its investigation against the possibility that strong QA performance may reflect shallow question-to-passage pattern matching.Source: §1 · p. 1
  • More than half of the questions in WikiHop and HotpotQA do not require multi-hop reasoning, and a no-context baseline performs strongly on WikiHop and modified HotpotQA.The paper concludes that span-based supervision is a harder testbed than multiple choice, but strong sentence-factored performance leaves it unclear whether proposed models learn multi-hop reasoning in either setting.Source: §1 · pp. 1–2
  • HotpotQA uses span-based answers, whereas WikiHop uses multiple choice; a multiple-choice HotpotQA version is vulnerable to a no-context baseline.BiDAF++, trained on span-based HotpotQA and adapted to multiple choice, outperforms the same model trained natively in the multiple-choice setting.Source: §1 · p. 1
  • Even in span-based HotpotQA, sentence-factored models perform strongly enough to raise questions about whether models learn multi-hop reasoning.Source: §1 · p. 2

2 Datasets

WikiHop and HotpotQA are Wikipedia-based datasets designed around questions requiring information from multiple documents, but they differ in answer format and annotations.

WikiHop

  • WikiHop contains 40k+ questions, answers, and passages built from multiple Wikipedia documents, with answers selected from entity candidates.Questions combine a relation query and head entity to identify a tail entity; annotators followed links and were required to use multiple documents.Source: §2 · p. 2

HotpotQA

  • HotpotQA contains 113k English Wikipedia question-answer pairs requiring reasoning over multiple supporting documents, with variable-length span answers.The dataset annotates relevant sentences as supporting facts for use during training.Source: §2 · p. 2

3 Probing Multi-hop Datasets

This section tests whether multi-hop datasets require cross-sentence reasoning, using sentence-factored models and a question-and-answer-only baseline. It finds that many questions can be addressed without consulting other sentences or the document.

3 Probing Multi-hop Datasets

  • The section probes whether the two datasets, despite being designed for multi-hop reasoning, can be solved without it.Source: §3 · p. 2

3.1 Sentence-Factored Model Test

  • More than half of WikiHop and HotpotQA questions have answer locations identifiable without multi-hop reasoning by sentence-factored models.The authors suggest lexical cues between questions and answer sentences may help.Source: Table 1 · p. 2, §3.1 · p. 3
  • Sentence-factored models score each sentence independently and predict which contains the answer, providing a test of whether cross-sentence reasoning is needed.The method may overestimate performance on HotpotQA when multiple plausible answers occur in one sentence and need other context to distinguish them.Source: §3.1 · pp. 2–3

3.2 No Context Baseline

  • The no-context baseline performs well on WikiHop, showing the dataset can be solved reasonably well without consulting its documents.The authors suggest answer-type filtering or other question-answer correlations as possible explanations.Source: §3.2 · p. 3

4 Span-based vs. Multiple-choice

This section compares span-based and multiple-choice supervision by recasting the datasets and evaluating models in both settings. Multiple-choice performance remains close to a no-context baseline, while span-based supervision is less vulnerable to that shortcut but still does not establish that models learn multi-hop reasoning.

4 Span-based vs. Multiple-choice

  • The comparison recasts HotpotQA as ten-choice QA using nine random distractor entities, and recasts WikiHop as span prediction over concatenated documents.Table 3 reports dev-set performance for MC models and Span2MC, which trains with span-based supervision and evaluates in a multiple-choice setting.Source: §4 · pp. 3–4, Table 3 · p. 4

4.1 Systems to Compare

  • The systems include MemNet, which gathers information across document timesteps, and BiDAF++, which combines bidirectional attention flow with self-attention.The authors also adapt MemNet to predict answer spans and adapt BiDAF++ to multiple choice, naming the latter MC-BiDAF++.Source: §4.1 · p. 4

4.2 Results

  • On both datasets, multiple-choice models perform only around 2% above the no-context baseline, while span-based BiDAF++ still underperforms Factored BiDAF.The authors interpret these results as evidence that multiple-choice models exploit question/candidate or question/context correlations, and that span-based models may rely on strong single-sentence reasoning.Source: §4.2 · pp. 4–5, Table 3 · p. 4, Table 4 · p. 4
  • On both datasets’ span-based settings, BiDAF++ substantially outperforms MemNet but remains below Factored BiDAF, leaving multi-hop reasoning unconfirmed.The comparison is reported in Table 4; the authors say the remaining gap could reflect strong single-sentence reasoning.Source: Table 4 · p. 4, §4.2 · p. 5
  • With more HotpotQA-MC options, all models’ performance drops, yet the no-context baseline remains comparable to the two more complex models.The authors conclude that adding options does not strengthen the models’ ability to learn multi-hop reasoning.Source: §4.2 · p. 5, Figure 3 · p. 5
  • Training with span-based supervision and evaluating on multiple choice beats MC-BiDAF++ on HotpotQA and is nearly comparable on WikiHop, even with random span selection.The authors argue this learns at least as much as multiple-choice supervision while avoiding question-candidate correspondence shortcuts.Source: §4.2 · p. 5

5 Discussion and Conclusion

The authors leave several other multi-hop datasets for future investigation because their answers may be entailed by passage content rather than appear directly. They recommend checks for whether multi-hop reasoning is necessary and whether multiple-choice candidates enable strong no-context baselines.

5 Discussion and Conclusion

  • The authors leave WorldTree, OpenBookQA, and MultiRC for future investigation because answers may be entailed by passage content rather than appear directly.Source: §5 · p. 5
  • For multiple-choice supervision, check whether a no-context baseline reveals strong correlations between questions and candidates.Source: §5 · p. 5
  • Test whether multi-hop reasoning is needed by evaluating a sentence-factored model on a proposed multi-hop dataset.Source: §5 · p. 5
  • Prefer span-based supervision to avoid exploiting extra candidate information.Source: §5 · p. 5

Related papers

Loading 1904.12106v1…