Source-linked AI summary
Constructing Datasets for Multi-hop Reading Comprehension Across Documents
Johannes Welbl, Pontus Stenetorp, Sebastian Riedel
TL;DR
Reading-comprehension datasets rarely require combining evidence across documents, motivating a new multi-hop task and dataset-induction method. The paper induces two domain-specific datasets and evaluates existing models, finding cross-document integration but persistent difficulty selecting relevant information.
Problem
Existing reading-comprehension resources generally focus on answers recoverable from a single sentence, paragraph, or document rather than combining disjoint textual evidence.
Method
The paper defines cross-document multi-hop reading comprehension, induces WIKIHOP and MEDHOP from knowledge bases and linked documents, and uses candidate masking to reduce lexical cues.
Results
One evaluated model integrates information across documents, but both models struggle to select relevant information; masked settings also defeat baselines relying on lexical cues.
Takeaways & Limitations
The datasets and analyses provide resources for studying composite reasoning and expose the need for better relevance selection in multi-document reading comprehension.
Takeaways & Limitations
Spurious document-answer correlations can make WIKIHOP artificially easy, requiring filtering that substantially reduces dataset size.
Abstract
from arXiv · showhide
Most Reading Comprehension methods limit themselves to queries which can be answered using a single sentence, paragraph, or document. Enabling models to combine disjoint pieces of textual evidence would extend the scope of machine comprehension methods, but currently there exist no resources to train and test this capability. We propose a novel task to encourage the development of models for text understanding across multiple documents and to investigate the limits of existing methods. In our task, a model learns to seek and combine evidence - effectively performing multi-hop (alias multi-step) inference. We devise a methodology to produce datasets for this task, given a collection of query-answer pairs and thematically linked documents. Two datasets from different domains are induced, and we identify potential pitfalls and devise circumvention strategies. We evaluate two previously proposed competitive models and find that one can integrate information across documents. However, both models struggle to select relevant information, as providing documents guaranteed to be relevant greatly improves their performance. While the models outperform several strong baselines, their best accuracy reaches 42.9% compared to human performance at 74.0% - leaving ample room for improvement.
1 Introduction
The paper introduces cross-document multi-hop reading comprehension, motivated by the limits of single-document methods, and develops datasets and evaluations for combining distributed evidence.
- Motivation: Existing reading-comprehension datasets usually concentrate relevant information within a single sentence or document.This emphasizes locating and matching query-support text rather than combining disjoint evidence.
- Motivation: Cross-document reasoning could support information extraction, search, and question answering when required information is distributed across locations.The paper specifically connects this capability to protein interactions reported across publications and information unavailable in one place.
- Motivation: WIKIHOP requires identifying Mumbai from the Hanging Gardens article and then India from a separate Mumbai article.The example illustrates multi-hop inference when the answer is not stated explicitly in the entity’s article.
- Contributions: The proposed task combines evidence across documents, producing WIKIHOP and MEDHOP from WIKIDATA and DRUGBANK facts using distant supervision.WIKIHOP uses Wikipedia articles, while MEDHOP infers drug-drug interactions from MEDLINE abstracts about drugs, proteins, and their interactions.
- Evaluation: 74.1% and 68.0% of samples were judged inferable from the documents by human annotators.The paper also identifies dataset-construction pitfalls and proposes remedies.
- Evaluation: 54.5% was the best model accuracy on an annotated test set, versus 85.0% for humans.One model integrated information across documents, but both models struggled to select relevant documents from larger sets.
- Contributions: The contributions include the task and induction strategy, two domain datasets, bias remedies, baselines, and ablation-based model analysis.These components are presented as the paper’s central contributions.
2 Task and Dataset Construction Method
The paper formalizes multi-hop reading comprehension as selecting one correct candidate from documents and constructs datasets by traversing an entity-document graph.
- Task Formalization: The task gives a query, supporting documents, and candidate answers, with exactly one correct answer among the candidates and documents.All candidates are mentioned in the support documents.
- Dataset Assembly: Dataset assembly starts from knowledge-base triples and converts each fact into a query by leaving the object slot empty.For example, a subject-relation pair becomes a query whose answer is the missing object.
- Graph Construction: A directed bipartite graph links documents to mentioned entities, and breadth-first search identifies candidates and support documents.The traversal begins at the query subject and ends at type-consistent answer entities.
- Graph Construction: Visited endpoints define the candidate set, while visited documents define chains leading to the correct answer and type-consistent false candidates.Samples are discarded when the correct answer is not reached.
- Task Design: Including false candidates forces models to use distributed evidence instead of relying only on type-consistency heuristics.The support documents therefore contain chains to both correct and incorrect candidate endpoints.
- Assumptions: The graph traversal relies on a closed-world assumption that the knowledge base states all true facts.Type-consistent entities are determined from observed knowledge-base objects for the query relation.
3 WIKIHOP
WIKIHOP applies the graph-based induction method to Wikipedia and Wikidata, while addressing answer-distribution and document-answer biases that could make the task spuriously easy.
- Dataset Construction: WIKIHOP uses Wikipedia as its corpus and Wikidata triples to construct queries about properties whose answers are absent from the item article.The dataset is designed for cross-document multi-step reasoning.
- Dataset Construction: Wikipedia graph traversal follows hyperlinks from entity mentions to articles about those entities.Article-to-entity edges connect all mentioning articles, while entity-to-article edges connect each entity to its own article.
- Dataset Construction: WIKIHOP includes the correct answer and other candidates sharing the query property, traversing chains of up to 3 documents.Samples with more than 64 support documents or 100 candidates are removed, affecting approximately 1% of samples.
- Bias Mitigation: WIKIREADING has significant answer-distribution bias, including frequent United States of America answers for the country property.The authors subsample candidates to at most 0.1% of the dataset and omit United States articles.
- Bias Mitigation: Document-answer correlations could yield 74.6% accuracy without addressing them, because certain documents co-occur with answers independently of the query.For example, an article about London can signal United Kingdom regardless of the query type or entity.
- Bias Mitigation: The authors measure document-candidate cooccurrence and use it to filter samples with spurious correlations.This targets biases induced by the graph traversal process.
4 MEDHOP
MEDHOP applies cross-document multi-hop reading comprehension to drug–drug interactions, using linked biomedical evidence and constrained graph traversal. Its construction addresses dense document connectivity and candidate-frequency imbalance through document subsampling and entity masking.
- 4 MEDHOP: MEDHOP targets drug–drug interactions inferred from protein–protein interaction chains across MEDLINE abstracts.The dataset uses DRUGBANK’s interacts with relation and connects drugs, proteins, and abstracts in a traversal graph.
- 4.1 Assembly: MEDHOP links documents to proteins, drugs, and interacting proteins with precision-oriented edge constraints.Protein-to-document traversal requires another protein in the document to interact with the first according to REACTOME.
- 4.1 Assembly: Support sets are capped at 64 documents while preserving paths to the answer and alternative candidates.Documents connecting the query drug to its answer are added first, followed by documents connecting alternative candidates.
- 4.1 Assembly: MEDHOP’s dense graph creates large support sets that can make existing reading-comprehension models computationally infeasible.Document subsampling is therefore required after graph traversal.
- Mitigating Candidate Frequency Imbalance: Candidate-frequency imbalance remains because drugs differ greatly in the number of known interactions.Aspirin interacts with 743 drugs, whereas Isotretinoin interacts with 34; entity masking is used to mitigate this issue.
5 Dataset Analysis
The dataset analysis examines dataset scale, sample composition, and whether WIKIHOP answers are genuinely supported by document chains. It finds substantial multi-document evidence, but also ambiguity, single-document shortcuts, and limited distant-supervision violations.
- 5 Dataset Analysis: WIKIHOP shrinks from approximately 528K to approximately 44K training samples after document–answer correlation filtering.The full creation, filtering, and subsampling pipeline is applied separately to each split.
- 5 Dataset Analysis: WIKIHOP has 277 query types, whereas MEDHOP contains one query type: interacts with.MEDHOP samples commonly have 9 candidates because documents are selected up to the 64-document limit.
- 5.1 Qualitative Analysis: 45% of WIKIHOP samples have answers that uniquely follow from multiple texts or are suggested as likely.Another 26% plausibly support multiple candidates, often because hypernymy makes answer granularity difficult.
- 5.2 Crowdsourced Human Annotation: 74% of annotator judgments produced the correct answer after reading document sets, reaching 85% on a validated development subset.Annotators knew the answer before reading in 9% of questions.
- 5.2 Crowdsourced Human Annotation: Among majority judgments, 59.8% of facts followed from the texts, 14.2% were likely, and 25.9% did not follow.For facts judged to follow or likely, 55.9% required multiple documents and 44.1% required one.
6 Experiments
Experiments compare simple and neural baselines on WIKIHOP and MEDHOP, testing dataset biases, masking, cross-document reasoning, and relevant-document selection. Neural models generally outperform other baselines, but relevant-document selection remains a major weakness.
- Baseline models: Lexical matching with a single support document is insufficient for strong prediction across both datasets.TF-IDF beats random on WIKIHOP but performs worse than random on MEDHOP, where interacting drug pairs never co-occur within one document.
- Dataset biases: 74.6% accuracy could be reached on WIKIHOP before filtering by exploiting document-answer cooccurrence alone.Filtering reduced this bias, but aggressive filtering substantially reduced WIKIHOP’s size and was infeasible for smaller MEDHOP.
- Neural models: Both neural RC models outperform the other baselines, while BiDAF is strongest overall across WIKIHOP and MEDHOP.Unlike the other baselines, FastQA and BiDAF extract answer spans without using candidate options.
- Lexical abstraction: Masking removes lexical cues, yet both neural models largely retain or improve performance by leveraging candidate-expression context.Lexical baselines fail under randomized answer placeholders, making masking especially useful for MEDHOP.
- Multi-step inference: BiDAF performance drops by 3.3%/6.2% on MEDHOP and 10.0%/2.1% on WIKIHOP when documents without candidate mentions are removed.The result demonstrates that BiDAF can leverage cross-document information; FastQA’s changes are mixed across datasets.
7 Related Work
Prior work addresses multi-step reasoning through structured knowledge bases, related-document retrieval, and neural architectures, but this paper focuses on combining textual evidence across documents in an end-to-end RC setting.
- This work’s datasets target compositional inference from simple queries in a cross-document setting that goes beyond resolving coreference.
- Compositional Knowledge Base Inference: Knowledge-base inference methods combine multiple structured facts but face limited coverage and inefficient inference.
- Compositional Knowledge Base Inference: Earlier approaches typically rely on predefined schemas and pipeline components such as information extraction or human annotation.
- Learning Search Expansion: Related-document retrieval and query-expansion methods acquire or rerank evidence, rather than answer queries by combining facts across documents.
8 Conclusions and Future Work
The paper introduces a cross-document multi-hop RC task, a general dataset-construction strategy, and datasets spanning two domains. Experiments show that models can use cross-document information, but relevant-document selection and the human-performance gap remain central challenges.
- The paper introduces a cross-document multi-hop RC task, a generic dataset derivation strategy, and datasets from two domains.
- Contemporary RC models can leverage cross-document information, but their performance remains substantially below human performance.
- Selecting relevant document sets is identified as the most promising direction for future research.
- The datasets focus on factoid entity questions and assume extractive answers mentioned verbatim, limiting the question types they support.
A Appendix: Versions
The paper’s version corresponds directly to the TACL version, with only minor wording changes, additional footnotes, and appendices.
- The paper directly corresponds to the TACL version.
- Differences from the TACL version include minor wording changes and additional footnotes.
- The paper also adds appendices relative to the TACL version.
B Appendix: Candidate and Document statistics
The appendix describes support-document, document-length, and candidate-count distributions for WIKIHOP and MEDHOP, alongside histograms illustrating document lengths and WIKIHOP candidate counts.
- Candidate and Document statistics: WIKIHOP has a Poisson-like support-document distribution, whereas MEDHOP has a bimodal distribution linked to uneven interaction and study counts.
- Candidate and Document statistics: WIKIHOP document lengths measure first-paragraph lengths, while MEDHOP lengths reflect generally longer research-paper abstracts.
- Candidate and Document statistics: WIKIHOP’s candidate-count distribution decreases steadily; MEDHOP samples usually have 9 candidates because document selection stops at 64 documents.
- Versions: The appendix includes a link identifying the TACL article version.
- Candidate and Document statistics: Figure 5 presents document-length histograms for WIKIHOP and MEDHOP.
- Candidate and Document statistics: Figure 6 presents a histogram of candidates per WIKIHOP sample.
C Appendix: Document-Cue examples
The appendix illustrates document–answer cooccurrence patterns and gold evidence chains in WIKIHOP. Gold-chain lengths vary, reaching a maximum of three documents.
- Table 9 provides examples of document gold chains used in WIKIHOP.Article titles are boldfaced and the correct answer is underlined.
- Gold-chain lengths differ, with a maximum of 3 documents.
E Appendix: Query Types
The appendix catalogs WIKIHOP query types and their dataset proportions, alongside table documentation for answer–article cooccurrence examples and document gold chains. Across 277 query types, the distribution follows a power law.
- Query-type distribution: The 25 most frequent WIKIHOP query types are listed with their proportions in the training set.Their combined overview is framed as a subset of the dataset’s 277 query types.
- Query-type distribution: The distribution across 277 WIKIHOP query types follows a power law.
- Document-cue examples: Table 8 documents answer–article cooccurrence examples before filtering, using Count and relative training-sample proportion columns.
- Document gold chains: Table 9 documents example WIKIHOP gold chains, marking article titles in bold and correct answers with underlining.