PaperlayerView exact revision on arXiv

Source-linked AI summary

MuSiQue: Multihop Questions via Single-hop Question Composition

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal

arXiv:2108.00573v3cs.CLcs.AI

TL;DR

Existing multihop QA datasets can often be solved through shortcuts rather than connecting all supporting facts. This paper introduces a bottom-up construction process for connected multihop questions and finds that MuSiQue is substantially harder and less cheatable than prior datasets.

  • Problem

    Existing multihop QA models can exploit shortcuts and need not connect information across all supporting facts, limiting evaluation of genuine multihop reasoning.

  • Method

    The authors compose connected single-hop questions through bridge entities, then apply stringent filters and controls to construct multihop QA datasets.

  • Results

    MuSiQue has a 3x human-machine gap and is substantially less cheatable via disconnected reasoning than HotpotQA and 2WikiMultihopQA.

  • Takeaways & Limitations

    MuSiQue-Ans is designed to require connected reasoning by reducing shortcuts, minimizing train-test leakage, and including harder distractor contexts.

  • Takeaways & Limitations

    The experiments focus on reading comprehension with fixed context paragraphs, and one model is not evaluated where ground-truth decomposition and intermediate answers are unavailable.

Abstract

from arXiv · show

Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts. Can we create a question answering (QA) dataset that, by construction, \emph{requires} proper multihop reasoning? To this end, we introduce a bottom-up approach that systematically selects composable pairs of single-hop questions that are connected, i.e., where one reasoning step critically relies on information from another. This bottom-up methodology lets us explore a vast space of questions and add stringent filters as well as other mechanisms targeting connected reasoning. It provides fine-grained control over the construction process and the properties of the resulting $k$-hop questions. We use this methodology to create MuSiQue-Ans, a new multihop QA dataset with 25K 2-4 hop questions. Relative to existing datasets, MuSiQue-Ans is more difficult overall (3x increase in human-machine gap), and harder to cheat via disconnected reasoning (e.g., a single-hop model has a 30 point drop in F1). We further add unanswerable contrast questions to produce a more stringent dataset, MuSiQue-Full. We hope our datasets will help the NLP community develop models that perform genuine multihop reasoning.

1 Introduction

Existing multihop QA datasets can be solved through shortcuts rather than connected reasoning, motivating MuSiQue’s bottom-up construction of challenging, less-cheatable questions.

  • The problem: Existing multihop datasets often allow models to find correct answers without connecting information across all supporting facts.Shortcuts can arise from overly specific sub-questions, train-test leakage, and insufficient distractors.
  • The problem: A HotpotQA example appears two-hop but can be answered without resolving the first sub-question because only one candidate fits the final description.A connected alternative includes multiple possible answers, forcing the model to identify the referenced author first.
  • The approach: MuSiQue introduces a bottom-up process that composes carefully selected single-hop questions and filters candidates to enforce connected reasoning.The approach also reduces train-test leakage, adds difficult distractors, and creates unanswerable questions at the sub-question level.
  • The datasets: ∼25K 2-4 hop questions comprise MuSiQue-Ans, which has 3x the human-machine gap and substantially lower disconnected reasoning than prior datasets.MuSiQue-Full extends this with ∼50K contrasting multihop questions incorporating insufficient-context cases.
  • The goal: The authors hope these datasets and the construction methodology will support development of genuine multihop reasoning and decomposition-based models.

2 Related Work

MuSiQue is positioned against multihop, unanswerable-QA, and question-composition work, emphasizing connected reasoning and dataset difficulty as its distinct focus.

  • Multihop QA: MuSiQue is closest to HotpotQA and 2WikiMultihopQA but is designed to be harder and less cheatable than both.HotpotQA was not constructed around composition difficulty, while 2WikiMultihopQA uses a limited set of hand-authored compositional rules.
  • Unanswerable QA: Prior unanswerable-QA datasets address robustness in single-hop and multihop settings using missing knowledge or removed support paragraphs.MuSiQue’s approach is most similar to removing support paragraphs, but uses bridge entities rather than potentially incomplete support annotations.
  • Question Decomposition and Composition: Existing work decomposes multihop questions into simpler questions or special meaning representations, while MuSiQue’s pipeline naturally provides decompositions for interpretable models.
  • Question Decomposition and Composition: Recent bottom-up rule-based approaches primarily target data augmentation, and their generated questions have not been shown to be challenging or less cheatable.

3 Multihop Reasoning Desiderata

The paper frames multihop QA as interdependent reasoning over a graph and argues that datasets should require every step to use its predecessors rather than permit shortcut solutions.

  • Graph-based reasoning: A multihop question can be represented as a directed acyclic graph whose nodes are reasoning steps and whose edges encode critical dependencies.
  • Connected reasoning: Connected reasoning requires each step to depend on the outputs of all its predecessor steps.
  • Why connectedness matters: A model solving all k steps succeeds with probability at most p^k, whereas solving only the final step can succeed with probability as high as r, independent of k.
  • Why connectedness matters: Existing datasets often violate connected reasoning, allowing models to exploit artifacts and bypass intended reasoning steps while retaining high scores.
  • Illustrative example: The motivating example shows that a final sub-question may uniquely identify its answer without using its predecessor, so the construction must prevent this pattern.

4 Connected Reasoning via Composition

MuSiQue constructs connected multihop questions by composing dependent single-hop questions into reasoning graphs and testing whether predecessor information is necessary in a reading-comprehension context.

  • Construction by composition: The bottom-up approach composes multihop questions from a large pool of single-hop questions, enabling broad exploration and controlled selection of connected candidates.The resulting questions retain constituent questions, answers, and supporting paragraphs.
  • Construction by composition: Two single-hop questions are composable when the first answer is a named entity mentioned in the second question, the final answer is absent from the first, and their passages differ.Entity identity is checked using entity typing, Wikipedia search results, and wikification.
  • Construction by composition: Compositions can be chained into reasoning DAGs in which each node is a sub-question and the final node’s answer answers the multihop question.
  • Ensuring connected reasoning: Connectedness is tested by masking a predecessor answer and checking whether a strong QA model can still recover the successor answer.An edge is disconnected when the successor remains answerable without the predecessor information.
  • Reading-comprehension setting: The framework focuses empirically on reading comprehension with a fixed paragraph context, where artifacts and train-test leakage can otherwise enable context-free prediction or memorization.
  • Ensuring connected reasoning: The MuSiQue condition requires that the final answer and every constituent answer cannot be produced without the appropriate context and predecessor information.

5 Dataset Construction Pipeline

MuSiQue constructs connected 2–4-hop questions bottom-up from filtered single-hop pairs, then adds difficult contexts, human-composed wording, and unanswerable counterparts. The resulting MuSiQue datasets combine multiple reasoning structures with controls targeting disconnected-reasoning shortcuts.

  • Source selection: MuSiQue starts with single-hop questions from five English Wikipedia-based datasets and filters them before composition.The source instances contain a question, associated paragraph, and answer; erroneous or unsuitable contexts are removed.
  • Pair composition: Composable pairs share a bridge entity while using distinct paragraphs, and entity matching is checked with tagging, search, and wikification.Only 8% of pairs are later pruned during human validation, indicating that the automatic pairing step is effective.
  • Disconnected-reasoning control: Disconnection filtering retains only pairs whose constituent questions cannot be answered through the intended shortcuts.The pipeline uses model-based head and tail filtering, with thresholds chosen to trade off cheatability against dataset size.
  • Multihop construction: The pipeline builds connected 2–4-hop questions from six reasoning-graph types, while limiting question lengths and reuse of bridges and single-hop questions.These constraints provide mixed-hop and structural diversity without producing excessively long questions.
  • Context construction: Each context contains 20 paragraphs, combining supporting paragraphs with distractors retrieved from filtered single-hop gold paragraphs.The retrieval design makes distractors harder to identify from superficial overlap.
  • Human composition and unanswerability: Workers verify bridge-entity identity and compose natural-language questions that use every DAG step; adding unanswerable instances then yields MuSiQue-Full.MuSiQue-Ans is the answerable dataset, while MuSiQue-Full pairs answerable questions with contexts where one subquestion lacks its answer.

6 Dataset Quality Assessment

The authors assess MuSiQue-Ans through human annotation, agreement, and connected-reasoning analyses, and manually validate MuSiQue-Full’s answerability labels. These evaluations support dataset quality and indicate that many final questions satisfy the intended connected-reasoning condition.

  • Human evaluation: MuSiQue-Ans quality is evaluated with human answers and supporting paragraphs compared against gold labels.The evaluation measures both ordinary human performance and agreement across annotators.
  • Evaluation measures: Human Score, Human Upper Bound, and Human Agreement provide complementary measures of performance, maximal scoring, and inter-annotator consistency.Three annotations are collected per question, with the first two measures evaluated against gold labels and agreement evaluated across annotations.
  • MuSiQue-Ans results: Table 3 reports human performance and agreement for MuSiQue-Ans, which the authors characterize as a high-quality dataset.The authors also compare human performance with HotpotQA and 2WikiMultihopQA under these metrics.
  • MuSiQue-Full validation: 90% of 50 manually validated MuSiQue-Full instances received answerability judgments matching the gold labels.One author assessed whether each sampled question was answerable from its context.
  • Connected reasoning: 79.0% of tail questions would pass re-filtering on final validation data, versus 26.5% retained during original filtering.The comparison suggests that substantially more final MuSiQue-Ans questions satisfy the MuSiQue connectedness condition than the initial retained set.

7 Experimental Setup

The experiments compare MuSiQue with HotpotQA and 2WikiMultihopQA using matched training sizes, paragraph-level metrics, and several multihop and artifact-based models. The setup includes end-to-end, selection, explicit execution, and restricted-input baselines.

  • Datasets and splits: Experiments compare MuSiQue-Ans and MuSiQue-Full with HotpotQA and 2WikiMultihopQA using matched 20K-instance training sets.The prior datasets are randomly subsampled to 20K instances for fair comparison.
  • Tasks and metrics: MuSiQue-Ans, HotpotQA, and 2WikiMultihopQA use answer and paragraph-level support F1, while MuSiQue-Full adds sufficiency-aware metrics.MuSiQue-Full evaluates answerability jointly with answer and support prediction on paired sufficient and insufficient contexts.
  • Model families: The study evaluates Transformer-based multihop models and artifact-based models restricted from performing the desired reasoning.The two model categories probe current multihop capability and dataset susceptibility to shortcuts, respectively.
  • End-to-end baseline: The Longformer end-to-end model reaches 78.4 answer F1 on HotpotQA and 87.7 on 2WikiMultihopQA when trained on full datasets.These results establish the model as a strong multihop baseline before evaluating MuSiQue.
  • Selection model: The Select+Answer model first selects K relevant paragraphs and then predicts answers, support, and answerability from the selected context.K is chosen from {3,5,7}; the answerer’s outputs differ between MuSiQue-Ans and MuSiQue-Full according to task requirements.
  • Explicit execution: The Step Execution model decomposes questions into DAGs and repeatedly applies a single-hop model while traversing the graph and substituting intermediate answers.Its decomposer uses gold decompositions, while the executor can use End2End or Select+Answer single-hop models.
  • Artifact controls and evaluation: The study also includes Q-only, C-only, and one-paragraph models, plus DiRe scoring and matched human evaluations across datasets.The end-to-end model is not evaluated on HotpotQA because that dataset lacks ground-truth decompositions and intermediate answers.

8 Empirical Findings

MuSiQue-Ans is harder and less vulnerable to shortcut-based reasoning than prior multihop datasets. Ablations show that disconnection filtering, reduced train-test leakage, and positive distractors materially increase difficulty and reduce cheatability.

  • 8.1 MuSiQue is a Challenging Dataset: MuSiQue-Ans has a larger human-model gap and lower shortcut-based performance than HotpotQA and 2WikiMultihopQA.The comparison covers both answer performance and supporting-paragraph identification.
  • 8.1 MuSiQue is a Challenging Dataset: 14 pts separate human and model supporting-paragraph identification scores on MuSiQue-Ans, while its answer F1 gap is approximately 27 points.HotpotQA and 2WikiMultihopQA have answer F1 gaps of 10 and 5 points, respectively.
  • 8.1 MuSiQue is a Challenging Dataset: 32.0 answer score is achieved by the 1-Para artifact-based model on MuSiQue-Ans, versus 64.8 on HotpotQA and 60.1 on 2WikiMultihopQA.C-only support identification also fails on MuSiQue-Ans despite high scores on the prior datasets.
  • 8.1 MuSiQue is a Challenging Dataset: 37.8 is MuSiQue-Ans’s DiRe answer score, compared with 68.8 for HotpotQA and 63.4 for 2WikiMultihopQA.The lower score indicates less success from disconnected reasoning.
  • 8.1 MuSiQue is a Challenging Dataset: MuSiQue-Full is significantly more difficult and less cheatable than MuSiQue-Ans under a stricter instance-pair metric.MuSiQue-Full adds answerable and unanswerable instances with nearly identical questions and marginally different contexts.
  • 8.2 Dataset Construction Steps are Valuable: Without the Disconnection Filter, multihop and artifact-based models perform better, while a random split reaches An=87.3 and artifact-based models reach 1-para An=85.1 and C-only An=69.5.These ablations show that filtering and leakage reduction are crucial to difficulty and cheatability.
  • 8.2 Dataset Construction Steps are Valuable: Positive distractors are harder than Full Wikipedia distractors, and 20 paragraphs make the dataset more difficult and less cheatable than 10.The effects strengthen when positive distractors and 20 paragraphs are combined.
  • 8.3 Potential Avenues for Improvement: Ground-truth decompositions improve MuSiQue-Ans (An, Sp) by (9.4, 7.3) points and MuSiQue-Full (An+Sf, Sp+Sf) by (7.3, 6.9) points, but do not reach human parity.T5-based SA also modestly improves performance but was not used broadly because of high computational cost.

9 Conclusion

The paper argues that controlled bottom-up construction can produce multihop questions requiring connected reasoning. MuSiQue-Ans shows a larger human-model gap and less disconnected-reasoning cheatability than previous datasets, while its decompositions and unanswerable questions support further research.

  • 9 Conclusion: Bottom-up construction from single-hop questions enables systematic exploration and control over multihop candidates.The pipeline reduces shortcuts, minimizes train-test leakage, and adds harder distractor contexts.
  • 9 Conclusion: MuSiQue-Ans has a substantially higher human-model gap and is significantly less cheatable via disconnected reasoning than previous datasets.The dataset also includes unanswerable questions and question decompositions for future modeling work.
  • 9 Conclusion: MuSiQue’s unanswerable questions and question decompositions are intended to support models that get right answers for the right reasons.

A Appendix

The appendix shows the interfaces used for question composition and for collecting human scores across the evaluated multihop QA datasets.

  • A Appendix: Figure 3 presents the annotation interface for question composition, including the decomposition graph and passage associated with each subquestion.
  • A Appendix: Figure 4 presents the annotation interface used to establish human scores on MuSiQue-Ans, 2WikiMultihopQA, and HotpotQA.
Loading 2108.00573v3…