Source-linked AI summary
Question Answering by Reasoning Across Documents with Graph Convolutional Networks
Nicola De Cao, Wilker Aziz, Ivan Titov
TL;DR
Reading comprehension has largely focused on individual documents, despite applications that require combining evidence across multiple documents. Entity-GCN models this as graph inference with message passing over entity mentions and uses pre-trained embeddings for efficient computation. It outperforms published results on WikiHop, while the reported evaluation has limitations related to hidden test data and comparisons involving ELMo.
Problem
Existing reading-comprehension systems often rely on local information, whereas multi-document question answering requires aggregating information across documents.
Method
Entity-GCN uses graph neural networks to propagate information across entity mentions in multiple documents, with pre-trained embeddings and lightweight learned components.
Results
Entity-GCN outperforms published results on WikiHop, with the full model improving over the best previously published results and an ensemble adding further improvement.
Takeaways & Limitations
Graph-based message passing provides a compact approach for gathering evidence from different documents, while pre-trained embeddings support fast computation.
Takeaways & Limitations
The authors could not analyze a test-set difference further because the test set was hidden, and could not combine comparison methods with ELMo without additional dimensionality or distributed implementation.
Abstract
from arXiv · showhide
Most research in reading comprehension has focused on answering questions based on individual documents or even single paragraphs. We introduce a neural model which integrates and reasons relying on information spread within documents and across multiple documents. We frame it as an inference problem on a graph. Mentions of entities are nodes of this graph while edges encode relations between different mentions (e.g., within- and cross-document co-reference). Graph convolutional networks (GCNs) are applied to these graphs and trained to perform multi-step reasoning. Our Entity-GCN method is scalable and compact, and it achieves state-of-the-art results on a multi-document question answering dataset, WikiHop (Welbl et al., 2018).
1 Introduction
Reading-comprehension systems often rely on local information, while WikiHop requires reasoning across multiple documents. Entity-GCN frames this challenge as graph inference, using graph neural networks and efficient preprocessing to improve performance without expensive recurrent document encoders.
- Motivation: WikiHop targets question answering that requires aggregating information across multiple documents, although some questions may be answerable from a single sentence.Each example contains documents, a query, and candidate answers.
- Prior approaches: Prior WikiHop methods mostly concatenate documents and apply standard RNN-based reading-comprehension models, while related work adds co-reference jump-links to RNNs.These RNN states still pass information across documents and perform multi-hop reasoning.
- Approach: Entity-GCN represents document collections as graphs whose entity nodes are connected by within- and cross-document relations, then propagates local context with GCNs.The graph includes co-occurrence and co-reference-style relations between entity mentions.
- Approach: The model avoids training expensive document encoders by using pre-trained contextual embeddings, learning only lightweight query, GCN, and answer-selection components.Most computation is preprocessed, leaving lightweight online computation for training and testing.
- Efficiency: At least 5 times faster training than BiDAF is reported for Entity-GCN in WikiHop.The comparison uses the somewhat contrived WikiHop setting with relatively small candidate sets.
- Results: Over 2% improvement over the best previously published results is achieved by the full Entity-GCN model without recurrent document encoders.An ensemble adds a further 3.6% improvement and reaches within 3% of reported human performance.
2 Method
The method represents multi-document question answering as inference over an entity graph, using typed relations and relational GCN message passing to combine local evidence. Fixed contextualized document representations and lightweight online components support efficient candidate selection.
- Dataset and task: WIKIHOP provides a query, supporting documents, candidate entities, and a correct answer that must be inferred from the documents.Queries use the form ⟨s, r, ?⟩, and the system selects the correct candidate entity from Cq.
- Entity graph: Entity-GCN creates one node for each mention of the candidate entities or query subject in the supporting documents.Mentions are gathered using exact matching and predictions from an external coreference resolver, while ambiguous chains are discarded.
- Entity graph: The graph uses DOC-BASED, MATCH, COREF, and COMPLEMENT relations to connect co-occurring, identical, coreferent, or otherwise disconnected mentions.These relation types distinguish more reliable exact matches from sparser or less reliable coreference links and ensure disconnected graphs are connected.
- Multi-step reasoning: Relational GCN layers propagate relation-specific messages from neighboring nodes, updating all node representations in parallel for multi-step reasoning.Each layer corresponds to a hop, allowing information to travel through the entity graph across connected mentions.
- Answer selection: The model scores candidates by combining the query representation with mention representations and taking the highest predicted probability among each candidate’s mentions.The max operator handles candidates realized at multiple locations in the graph.
- Efficient computation: ELMo supplies fixed contextualized mention representations that can be precomputed, leaving only the query encoder, GCN layers, and answer selector to run online.This design makes the model at least 5 times faster to train than BiDAF in the WIKIHOP setting.
3 Experiments
Experiments evaluate Entity-GCN against prior systems and through ablations on WIKIHOP, testing scalability, contextual embeddings, graph reasoning, and edge types. The model outperforms prior work while showing that both ELMo and structured graph propagation contribute to performance.
- 3.1 Comparison: Entity-GCN outperforms all previous work by over 2% points on WIKIHOP and processes 57.8 document sets per second versus 12.5 for BiDAF.The comparison uses the best single model without coreference edges and a single Titan X GPU for training-speed measurements.
- 3.1 Comparison: The evaluation compares Entity-GCN with prior models on both masked and unmasked WIKIHOP, using closed-test and public-validation accuracy.The masked setting removes lexical cues by replacing candidates and mentions with consistent placeholder tokens, and omits coreference resolution.
- 3.2 Ablation Study: Replacing ELMo with GloVe remains competitive, but removing R-GCN loses 8.0 points on unmasked validation accuracy.Without contextual document processing, R-GCN still improves mention representations through their relations to other mentions.
- 3.2 Ablation Study: The full model is 6.1% and 8.4% more accurate than the local No R-GCN model in unmasked and masked settings, respectively.This supports the need for multihop inference while indicating that ELMo representations capture predictive context features.
- 3.2 Ablation Study: A fully connected graph without relation types yields only marginal improvements over ELMo alone, indicating that informative graph construction and relation parameterization are needed.The result compares the naive graph against the No R-GCN condition in both masked and unmasked settings.
- 3.2 Ablation Study: DOC-BASED connections help more than MATCH or COREF connections, while adding all relation types together gives Entity-GCN a clear advantage.Coreference links appear more marginal, and adding coreference caused test-set performance degradation, likely because the test documents were harder for the external system.
4 Error Analysis
Entity-GCN performs less well on questions about places and degrades as candidate-answer counts or graph size increase, though graph-size degradation is comparatively gentle.
- Questions about places, including birth and death, are harder for Entity-GCN.The analysis filters query types with at least 50 supporting documents and 5 candidates.
- -0.687 Pearson correlation links accuracy negatively with the number of candidate answers.Candidate counts peak at 5 and average approximately 20 in the dataset.
- Accuracy drops as graph node count increases, but the negative Pearson correlation is closer to zero than for candidate count.The authors describe this gentler degradation as important because practical document sets can be large.
5 Related Work
Prior multi-document QA systems largely concatenate documents or augment recurrent models, while graph neural networks provide a related basis for modeling reasoning over structured relations.
- BiDAF, FastQA, Coref-GRU, MHPGM, and Weaver/Jenga have been applied to multi-document question answering.BiDAF and FastQA concatenate documents, Coref-GRU adds coreference jump links, MHPGM uses external commonsense relations, and Weaver/Jenga uses alternating bi-LSTMs.
- Earlier WIKIHOP systems process documents by concatenating them in random order and use the first answer mention for training and exact-match evaluation.This applies to the BiDAF and FastQA adaptations described in the passage.
- Graph neural networks have succeeded across NLP, document-level modeling, and knowledge-base question answering, including GCN-based reasoning chains.The paper positions its work among the first applications of graph neural networks to multi-document QA.
6 Conclusion
The paper presents Entity-GCN as a compact graph-based model that gathers evidence across documents through differentiable message passing and achieves strong multi-document QA performance.
- Entity-GCN represents document collections as compact graphs whose nodes are entity mentions and whose edges encode within- and cross-document relations.The model uses this representation to gather evidence from different documents.
- Differentiable message passing updates node representations from their neighborhoods to answer questions using evidence distributed across documents.The graph neural network is designed to support multi-step reasoning.
- Entity-GCN outperforms published results, while ablations provide substantial evidence supporting multi-step reasoning.Pre-trained contextual embeddings also make the model fast.
A.1 Architecture
The architecture builds query-aware entity representations, propagates them through recurrent graph-convolutional layers, and predicts candidate-answer probabilities from the final node states.
- A.1 Architecture: ELMo embeddings provide 3072-dimensional input vectors for the entity mentions.Each embedding concatenates three 1024-dimensional vectors.
- A.1 Architecture: A two-layer bi-LSTM encodes the query into a 256-dimensional question representation.The layers use 256 and 128 hidden units, whose forward and backward states are concatenated.
- A.1 Architecture: Candidate embeddings are projected, combined with the query, and transformed by an MLP into 512-dimensional query-aware entity representations.The MLP has hidden layers of 1024 and 512 units.
- A.1 Architecture: R-GCN layers use affine transformations that preserve 512-dimensional node representations.The input and output dimensionality remains the same across these layers.
- A.1 Architecture: A two-layer MLP combines final node representations with the query to predict each candidate node's answer probability.The prediction MLP uses hidden units of 256 and 128.
- A.1 Architecture: With WIKIHOP, models using at least three R-GCN layers reached essentially the same performance, while additional layers increased training time.The gating mechanism retained increasing amounts of past information, making extra layers unnecessary beyond the required depth.
A.2 Training Details
Training uses Adam with early stopping and dropout, selecting each experiment’s best model by validation accuracy.
- Training runs for at most 20 epochs with batch size 32 and Adam at learning rate 10^-4.Adam uses β1 = 0.9 and β2 = 0.999.
- Dropout rates from 0 to 0.25 and early stopping on validation accuracy are used to reduce overfitting.
- Reported results use the best experiment checkpoint according to validation-set accuracy.
B Error Analysis
The error analysis examines three WIKIHOP development examples where Entity-GCN fails, including both confident and non-confident predictions.
- Three WIKIHOP development-set failure cases are analyzed for Entity-GCN.The examples include two high-confidence incorrect answers and one lower-confidence failure.
- The authors comment on possible reasons for each observed failure case.
C Ablation Study
Figure 3 evaluates Entity-GCN as candidate-set and graph-node sizes increase, relating accuracy to the distribution of input sizes.
- Figure 3 plots validation accuracy against candidate-set size and node-set size for the best single model.
- The analysis considers scalability by testing performance as the number of candidate answers or graph nodes increases.
- The figure overlays accuracy in blue with rescaled input-size distributions in orange, while dashed lines mark average accuracy.