Source-linked AI summary
Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs
Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, Bowen Zhou
TL;DR
Multi-hop reading comprehension must combine evidence across documents, unlike conventional single-document RC. This paper introduces a heterogeneous graph with query-aware nodes and edges, using attention-based representations and GNN message passing. On WikiHop, the single model is competitive, while the ensemble achieves state-of-the-art performance and exceeds reported human performance by 0.2%.
Problem
Multi-hop RC requires reasoning across multiple supporting documents to find an answer when a single document is insufficient.
Method
The HDE graph represents candidates, documents, and entities with query-aware nodes and heterogeneous edges, initialized using co-attention and self-attention and updated by GNN message passing.
Results
The ensemble model achieves state-of-the-art WikiHop performance and surpasses reported human performance by 0.2%.
Takeaways & Limitations
Heterogeneous graph reasoning provides a model for accumulating evidence across documents at candidate, document, and entity granularities.
Abstract
from arXiv · showhide
Multi-hop reading comprehension (RC) across documents poses new challenge over single-document RC because it requires reasoning over multiple documents to reach the final answer. In this paper, we propose a new model to tackle the multi-hop RC problem. We introduce a heterogeneous graph with different types of nodes and edges, which is named as Heterogeneous Document-Entity (HDE) graph. The advantage of HDE graph is that it contains different granularity levels of information including candidates, documents and entities in specific document contexts. Our proposed model can do reasoning over the HDE graph with nodes representation initialized with co-attention and self-attention based context encoders. We employ Graph Neural Networks (GNN) based message passing algorithms to accumulate evidences on the proposed HDE graph. Evaluated on the blind test set of the Qangaroo WikiHop data set, our HDE graph based single model delivers competitive result, and the ensemble model achieves the state-of-the-art performance.
1 Introduction
Multi-hop reading comprehension requires reasoning across multiple supporting documents, a gap left by research focused mainly on single-document or paragraph settings. The HDE graph addresses this with query-aware heterogeneous nodes and edges, achieving strong WikiHop results.
- Motivation: Multi-hop RC requires collecting evidence across multiple documents when one document cannot provide the answer.
- Proposed approach: The HDE graph represents candidates, documents, and entities as query-aware nodes at different information granularities.Co-attention and self-attention learn the node representations.
- Proposed approach: Different edge types connect query-, document-, and candidate-related nodes to encode structural information for reasoning.
- Results: The ensemble ranks first on the WikiHop leaderboard and exceeds reported human performance by 0.2%.The result is achieved without pretrained contextual ELMo embeddings.
2 Related Work
Multi-hop reading comprehension has been studied across several datasets, but this paper focuses specifically on gathering evidence across multiple documents. Related approaches use neural RC models or GNNs to connect entities and candidates for multi-hop reasoning.
- Related approaches: Neural RC approaches concatenate supporting documents or combine document-level and entity-level information to exploit cross-document connections.
- Related approaches: GNN approaches use extracted entities or exact-matched candidate mentions as graph nodes for multi-hop reasoning across documents.
- Multi-hop RC: Earlier multi-hop datasets require reasoning across multiple sentences or common knowledge, whereas this paper targets evidence collection across multiple documents.
- GNN for NLP: GNNs have also been applied in NLP to integrate syntactic and semantic information, perform relation extraction, and classify text over heterogeneous graphs.
3 Methodology
The model builds a heterogeneous graph whose query-aware document, candidate, and entity nodes are initialized by context encoding, then reasons through typed edges with GNN message passing.
- Context encoding: The model encodes queries, documents, candidates, and extracted entities with bidirectional GRUs before constructing query-aware representations.Co-attention combines query and document-side context, while self-attentive pooling summarizes sequential representations into fixed-dimensional vectors.
- Context encoding: Entities are extracted by exact matching mentions of the query subject and candidates in support documents, then represented from document encodings.The extracted mentions serve as entity nodes and connect information across documents.
- Graph construction: The HDE graph represents each document, candidate, and extracted entity as a node carrying a different granularity of query-aware information.Document nodes encode global document information, candidate nodes encode candidate information, and entity nodes encode information in specific document contexts.
- Graph construction: Typed edges connect documents, candidates, and entities according to document occurrence, extraction, mention identity, shared documents, cross-document identity, and candidate relations.The graph includes document–candidate, document–entity, candidate–entity, entity–entity, and candidate–candidate connections.
- Reasoning over the HDE graph: GNN message passing aggregates and combines information from neighbors over the HDE graph, using edge types to control how information propagates.The model follows a GCN-style aggregation and combination process for multi-hop reasoning.
4 Experiments
Experiments evaluate the HDE model on WIKIHOP using development and blind test comparisons, ablations, sample-type analyses, and result analyses by support-document and candidate counts. The results show gains over published baselines and that graph structure, edge types, node scores, and document connectivity affect performance.
- Results: The HDE model improves published WIKIHOP accuracy from 67.1% to 68.1% on development data and from 70.6% to 70.9% on the blind test set.Compared with DynSAN, the single model remains 0.5% worse.
- Results: The ensemble achieves state-of-the-art performance and exceeds reported human performance by 0.2%.The ensemble uses 15 models with different hyperparameter settings and random seeds combined by majority voting.
- Ablation studies: Removing the HDE graph decreases WIKIHOP development accuracy by 2.6 percentage points.The ablation directly replaces graph-based score accumulation with candidate and entity representations.
- Ablation studies: Treating all edge types equally decreases accuracy by 1.4%, while using only entity or candidate scores decreases it by 1.0% or 1.5%, respectively.Different edge types and both candidate and entity scores contribute to classification.
- Ablation studies: Removing entity nodes causes the largest degradation, whereas removing document nodes causes the least degradation.The authors attribute the limited contribution of document nodes partly to information loss during self-attentive pooling over long sequences.
- Result analysis: The HDE graph improves accuracy by 1.1% for single-document-sufficient samples and by 4.0% when multiple documents are required.The development set contains 2,069 Single-follow and 2,601 Multi-follow samples.
- Result analysis: Accuracy varies with the number of support documents and candidates, while more candidates increase selection confusion and enlarge the HDE graph.Figure 4 plots support-document counts and accuracy; Figure 5 plots candidate counts and accuracy.
5 Conclusion
The paper proposes a GNN-based HDE graph for multi-hop reading comprehension across multiple documents, encoding candidates, documents, and entities at different granularities. On WIKIHOP, the single model delivers competitive results, while the ensemble achieves state-of-the-art performance.
- The proposed method uses an HDE graph for multi-hop reading comprehension across multiple documents.
- HDE graph nodes represent candidates, documents, entities of candidate mentions, and query subjects at different information granularities.
- Co-attention and self-attention encode these elements into query-aware representations that initialize the graph nodes.
- Graph Neural Networks perform multi-hop reasoning over the HDE graph using message passing and updated node representations.
- On WIKIHOP, the end-to-end single model delivers competitive results, while the ensemble achieves state-of-the-art performance.