PaperlayerView exact revision on arXiv

Source-linked AI summary

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa

arXiv:2011.01060v2cs.CL

TL;DR

Existing multi-hop QA datasets may lack complete reasoning explanations, and many examples may not genuinely require multiple reasoning hops. The paper introduces 2WikiMultiHopQA, combining Wikipedia, Wikidata, evidence triples, logical rules, and controlled generation, and reports that the dataset is challenging while requiring multi-hop reasoning.

  • Problem

    Existing multi-hop datasets lack complete explanations of the reasoning path, and analyses found many examples that can be answered without multi-hop reasoning.

  • Method

    The authors combine Wikipedia and Wikidata, represent reasoning paths with evidence triples, and use templates, filtering, and verified logical rules to generate multi-hop questions.

  • Results

    An 8.7 F1 lower score than HotpotQA on a single-hop BERT evaluation indicates that many 2WikiMultiHopQA examples require multi-hop reasoning, while multi-hop models find the dataset challenging.

  • Takeaways & Limitations

    2WikiMultiHopQA provides a dataset for evaluating both answer predictions and reasoning paths through evidence information, while testing whether multi-hop reasoning is required.

  • Takeaways & Limitations

    The dataset contains unanswerable questions caused by mismatched information between Wikipedia articles and Wikidata knowledge.

Abstract

from arXiv · show

A multi-hop question answering (QA) dataset aims to test reasoning and inference skills by requiring a model to read multiple paragraphs to answer a given question. However, current datasets do not provide a complete explanation for the reasoning process from the question to the answer. Further, previous studies revealed that many examples in existing multi-hop datasets do not require multi-hop reasoning to answer a question. In this study, we present a new multi-hop QA dataset, called 2WikiMultiHopQA, which uses structured and unstructured data. In our dataset, we introduce the evidence information containing a reasoning path for multi-hop questions. The evidence information has two benefits: (i) providing a comprehensive explanation for predictions and (ii) evaluating the reasoning skills of a model. We carefully design a pipeline and a set of templates when generating a question-answer pair that guarantees the multi-hop steps and the quality of the questions. We also exploit the structured format in Wikidata and use logical rules to create questions that are natural but still require multi-hop reasoning. Through experiments, we demonstrate that our dataset is challenging for multi-hop models and it ensures that multi-hop reasoning is required.

1 Introduction

2WikiMultiHopQA addresses gaps in existing multi-hop datasets by combining structured and unstructured data with evidence that explains reasoning paths and by designing generation procedures to require multi-hop reasoning.

  • Existing multi-hop datasets often lack comprehensive explanations, while sentence-level supporting-fact classification cannot fully evaluate reasoning and inference skills.
  • 2WikiMultiHopQA combines Wikipedia and Wikidata to create a large, high-quality dataset with evidence represented as structured triples.Each evidence triple contains a subject entity, property, and object entity obtained from Wikidata.
  • The dataset includes evidence information that provides comprehensive explanations for predictions and supports evaluation of model reasoning and inference skills.
  • The dataset uses predefined templates and removes templates that permit single-hop or context-dependent multi-hop comparison questions.This design aims to ensure that comparison and bridge-comparison questions require multi-hop reasoning.
  • The dataset is more difficult for multi-hop models than HotpotQA, while human scores are comparable across the two datasets.
  • 8.7 F1 is the gap by which 2WikiMultiHopQA scores below HotpotQA on a single-hop BERT evaluation, indicating that many examples require multi-hop reasoning.

2 Task Overview

The task evaluates answer prediction, supporting-fact selection, and evidence generation from a question and documents, using joint metrics and several question types to test multi-hop reasoning.

  • 2.1 Task Formalization and Metrics: The task takes a question Q and document set D as input and predicts an answer, supporting-fact sentences, and evidence triples describing the reasoning path.
  • 2.1 Task Formalization and Metrics: Models are evaluated on answer spans, sentence-level supporting facts, and evidence using exact match and F1 score.
  • 2.1 Task Formalization and Metrics: Joint metrics combine answer, supporting-fact, and evidence performance, with joint EM equal to one only when all three tasks achieve exact match.
  • 2.2 Question Types: The dataset contains comparison, inference, compositional, and bridge-comparison questions.
  • 2.2 Question Types: Comparison questions compare entities from the same group on an attribute such as birth or death date.
  • 2.2 Question Types: Inference questions derive a new relation from two linked knowledge-base triples and use the resulting triple to form the question and answer.For example, mother followed by father yields a maternal-grandfather relation.
  • 2.2 Question Types: Compositional questions connect two triples when no single inference relation exists, asking about the entity reached through both relations.
  • 2.2 Question Types: Bridge-comparison questions require finding bridge entities across paragraphs and then comparing information about those entities.

3 Data Collection

The dataset combines Wikipedia text with Wikidata statements and uses templates, logical rules, filtering, and post-processing to construct verified multi-hop question-answer samples.

  • Wikipedia summaries and Wikidata statements are combined to represent entities with linked unstructured and structured information.
  • The generation framework creates templates, generates data from entity information, post-processes samples, and splits the data into train, development, and test sets.
  • Logical rules produce inference questions that remain simple in wording but require multi-hop reasoning; 28 rules were manually verified for Wikidata relations.
  • Comparison questions are generated from two entities sharing a relation, with Wikipedia evidence spans and Wikidata triples recorded alongside each question-answer pair.
  • Bridge questions require a bridge entity, answer spans in the second paragraph, and exclusion of direct two-hop shortcuts from both paragraphs.
  • Post-processing balances yes/no questions, removes ambiguous answers, and retrieves distractor paragraphs using bigram tf-idf and entity types.

4 Data Analysis

The analysis characterizes question and answer distributions, reasoning types, and answer categories across the dataset, including substantial variation by question type.

  • Compositional questions have the most examples, while inference questions have the fewest after ambiguous cases were removed.
  • Comparison and bridge-comparison questions have shorter average answers than inference and compositional questions because comparison includes many yes/no questions.
  • Comparison questions require quantitative or logical comparison, whereas compositional questions combine answers to sequential primitive questions.
  • The dataset contains 708 unique answer types; the five most frequent are yes/no (31.2%), date (16.9%), film (13.5%), human (11.7%), and big city (4.7%).

5 Experiments

Experiments show that 2WikiMultiHopQA is harder than HotpotQA for multi-hop models and that its evidence-generation task remains especially challenging.

  • Dataset quality: Overall joint-metric scores are lower on 2WikiMultiHopQA than on HotpotQA, although sentence-level supporting-fact scores are higher.The answer-prediction scores are lower on 2WikiMultiHopQA, while supporting-fact prediction performs better.
  • Dataset quality: 8.7 F1 separates the single-hop BERT scores on HotpotQA and 2WikiMultiHopQA, at 64.6 and 55.9 respectively.The lower score on 2WikiMultiHopQA indicates that many examples require multi-hop reasoning.
  • Baseline results: The baseline model adds an evidence-generation component that predicts relations from the question and entities from relations plus context.Evidence consists of triples containing subject entity, relation, and object entity.
  • Baseline results: Evidence generation is difficult: the model may predict one correct triple, but accurately recovering the complete triple set produces very low exact-match performance.Sentence-level supporting-fact prediction scores are high, but this binary task does not evaluate reasoning and inference skills adequately.
  • Baseline results: Inference questions have the lowest joint metric, while evidence generation has the lowest score across all four question types.The four types are comparison, inference, compositional, and bridge-comparison questions.
  • Human performance: Human answer performance reaches 91.0 EM and 91.8 F1, exceeding the model and leaving room for improvement.The human answer F1 upper bound is relatively close to HotpotQA’s 98.8 F1 upper bound.
  • Human performance: The human–model gap is especially large for evidence generation, where performance is 78.8 versus 16.7 F1.The authors identify ambiguity in Wikidata names as a possible reason for the low evidence-generation score.
  • Mismatch analysis: Some questions are unanswerable because Wikipedia article content and Wikidata triples can mismatch semantically.The generation process checks whether the object entity appears in a sentence, which does not guarantee that the sentence expresses the triple’s relation.

6 Related Work

Prior multi-hop datasets span textual, knowledge-base, and hybrid settings, but explanation-supporting resources remain limited. 2WikiMultiHopQA differs by using structured Wikidata triples for both justification and introspective explanations.

  • HybridQA combines Wikipedia tables and articles for multi-hop question answering over tabular and textual data.
  • Knowledge-base question-answering datasets have often focused on single-hop questions, with METAQA and ComplexQuestions extending to multi-hop or compositional settings.
  • Logical rules mined from knowledge bases can be used to test models’ reasoning skills.
  • HotpotQA and R4C provide explanations, while 2WikiMultiHopQA represents explanations as structured Wikidata triples rather than sentence-level or semi-structured data.R4C provides both justification and introspective explanations but contains 4,588 questions, limiting its use for end-to-end training with comprehensive explanations.

7 Conclusion

The conclusion presents 2WikiMultiHopQA as a large, high-quality dataset with comprehensive prediction explanations and questions designed to require multi-hop reasoning. Experiments show that it challenges multi-hop models while supporting dataset construction from Wikipedia and Wikidata.

  • 2WikiMultiHopQA provides comprehensive explanations for predictions in a large, high-quality multi-hop dataset.
  • Logical rules from the knowledge base create natural questions that still require multi-hop reasoning.
  • Experiments demonstrate that the dataset ensures multi-hop reasoning while challenging multi-hop models.
  • Bootstrapping from large-scale Wikipedia and Wikidata data is beneficial for building multi-hop MRC datasets.

A.1 Data Preprocessing

Data preprocessing combines English Wikipedia and Wikidata resources, categorizes entities by Wikidata types, and selects popular entities. Dataset quality depends on how consistently information intersects across the two sources.

  • English Wikipedia and Wikidata dumps were collected from January 1, 2020 and December 31, 2019, respectively.
  • The resources contained 5,950,475 entities categorized into 23,763 groups using Wikidata’s instance-of property.
  • The dataset focused on popular entities, including the top 50 for comparison questions.
  • Dataset quality depends on intersection consistency between Wikipedia and Wikidata, which is stronger for dates than occupations.
  • Stanford CoreNLP and Spacy were used for context sentence segmentation.

A.2 Comparison Questions

Comparison questions use selected popular entities and Wikidata properties, with multiple templates for each entity-property combination to ensure question diversity.

  • Comparison questions focus on the top 50 popular entities in Wikipedia and Wikidata.
  • A specified set of Wikidata properties is used to ensure dataset quality.
  • Multiple templates are provided for each entity-property combination to diversify the questions.

A.3 Inference Questions

Inference questions use logical rules to create multi-hop questions, but rule application requires checks to preserve valid reasoning and unique answers.

  • Rule constraints: Logical rules are difficult to apply because natural-language paragraphs may already state the inferred relation, undermining the intended multi-hop step.For example, deriving nationality from place of birth and country can fail when the person's paragraph already mentions nationality.
  • Answer uniqueness: The dataset restricts relations to those with a single value so each question has only one correct answer.This addresses ambiguity in rules such as child(a, b) ∧ child(b, c) ⇒ grandchild(a, c).
  • Rule validation: Because logical rules can be false in particular cases, inferred triples are checked against Wikidata before inclusion.The paper gives doctoral advisor and employer as an example where an apparent inference may not hold.
  • Question generation: Generated inference questions use predefined templates to convert selected relations into question–answer examples.The template set is identified as Table 11 in the source passages.

A.4 Compositional Questions

The question-generation procedures combine Wikidata relations with Wikipedia information, templates, and filtering rules to construct compositional and bridge-style multi-hop examples while reducing ambiguity.

  • Relation and template design: The generation process selects entity properties from Wikidata and uses predefined relation-based templates to form questions.The paper lists separate sets of first and second relations and reports 799 templates across 15 entity types.
  • Bridge construction: Bridge questions are built from two linked triples, while cases with multiple possible intermediate objects are discarded as ambiguous.The linked structure is (e, r1, e1) followed by (e1, r2, e2).
  • Comparison filtering: Comparison questions with equal numerical values are removed so the comparison does not yield a tie.The comparison procedure randomly selects two entities and retrieves their Wikidata triples before applying requirements.
  • Generated outputs: The procedures package each generated example with a question, answer, context, sentence-level support features, and evidence.These outputs are specified for both comparison and bridge-question generation.
  • Generation pipeline: The bridge-question procedure obtains Wikipedia articles for the linked entities, checks requirements, selects a relation-based template, and produces question, context, evidence, and answer outputs.The algorithm also computes sentence-level support features.
Loading 2011.01060v2…