Source-linked AI summary
Hierarchical Graph Network for Multi-hop Question Answering
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, Jingjing Liu
TL;DR
Multi-hop QA must aggregate scattered evidence across documents and granularities for joint answer and supporting-fact prediction. HGN builds a unified hierarchical graph, encodes and propagates representations across its heterogeneous nodes, and achieves state-of-the-art HotpotQA performance. The approach remains mainly tailored to HotpotQA, with two-hop reasoning and an off-the-shelf Fullwiki retriever.
Problem
Aggregating scattered evidence across paragraphs, sentences, and entities for joint answer and supporting-fact prediction remains challenging in multi-hop QA.
Method
HGN builds a hierarchical graph of question, paragraph, sentence, and entity nodes, initializes them with contextual encoding, and applies graph propagation for multiple prediction sub-tasks.
Results
HGN achieves state-of-the-art performance on HotpotQA in both Distractor and Fullwiki settings, including a Distractor Joint EM/F1 of 47.11/74.21.
Takeaways & Limitations
Weaving heterogeneous granularities into one graph supports joint evidence and answer prediction for multi-hop QA.
Takeaways & Limitations
The analysis reports that HGN cannot perform commonsense or discrete reasoning and that errors mainly arise from multi-hop reasoning or MRC span selection.
Abstract
from arXiv · showhide
In this paper, we present Hierarchical Graph Network (HGN) for multi-hop question answering. To aggregate clues from scattered texts across multiple paragraphs, a hierarchical graph is created by constructing nodes on different levels of granularity (questions, paragraphs, sentences, entities), the representations of which are initialized with pre-trained contextual encoders. Given this hierarchical graph, the initial node representations are updated through graph propagation, and multi-hop reasoning is performed via traversing through the graph edges for each subsequent sub-task (e.g., paragraph selection, supporting facts extraction, answer prediction). By weaving heterogeneous nodes into an integral unified graph, this hierarchical differentiation of node granularity enables HGN to support different question answering sub-tasks simultaneously. Experiments on the HotpotQA benchmark demonstrate that the proposed model achieves new state of the art, outperforming existing multi-hop QA approaches.
1 Introduction
Multi-hop QA requires aggregating scattered evidence across paragraphs and granularity levels to select relevant context, supporting facts, and an answer. HGN addresses this with a unified hierarchical graph for joint sub-task prediction.
- Multi-hop QA requires reasoning across multiple documents or paragraphs rather than deriving answers from a single paragraph.
- A HotpotQA example requires selecting a relevant paragraph, identifying a supporting sentence leading to the next hop, and extracting the answer span.
- Existing approaches may retrieve answer-containing paragraphs and apply MRC, but aggregating scattered paragraph-, sentence-, and entity-level evidence remains challenging.
- HGN constructs a hierarchical graph with question, paragraph, sentence, and entity nodes to capture clues at different granularities.
- Pre-trained contextual encoders initialize node representations, graph propagation updates them, and task-specific modules use them for paragraph selection, supporting-fact prediction, entity prediction, and answer prediction.
- HGN achieves new state-of-the-art results on HotpotQA in both Distractor and Fullwiki settings.
2 Related Work
Related work addresses multi-hop QA with retrieval, entity graphs, recurrent reasoning paths, and hierarchical coarse-to-fine models. HGN differs by organizing multiple granularities hierarchically and using graph neural networks for downstream tasks.
- Multi-Hop QA: WikiHop and HotpotQA are datasets designed for aggregating scattered evidence across multiple documents to answer multi-hop questions.
- Graph Neural Network: Prior graph-based approaches construct entity or sentence graphs and use graph neural networks, masking, attention, or recurrent reasoning to identify evidence and answer spans.
- Hierarchical Coarse-to-Fine Modeling: Earlier hierarchical QA methods commonly use coarse-to-fine selection, reinforcement learning, minimal-context identification, cascades, or hierarchical attention across documents.
- Hierarchical Coarse-to-Fine Modeling: HGN organizes different granularities hierarchically and uses graph neural networks to obtain representations for different downstream tasks.
3 Hierarchical Graph Network
HGN constructs and reasons over a hierarchical graph connecting questions, paragraphs, sentences, and entities, then jointly predicts paragraphs, supporting facts, and answers. Its graph uses contextual encoding, attention-based propagation, and multi-task objectives to support these downstream tasks.
- Graph Construction: HGN constructs a hierarchical graph connecting question, paragraph, sentence, and entity nodes, with edges linking clues across granularity levels.The graph includes question-to-paragraph, paragraph-to-sentence, sentence-to-entity, and hyperlink-based sentence-to-paragraph connections.
- Graph Construction: Paragraph selection retrieves title-matching candidates, supplements them with entity-based search when needed, and ranks candidates with a RoBERTa-based paragraph ranker.At most two paragraphs are selected in the first hop, followed by top-N ranking during the two-hop selection process.
- Graph Construction: Hyperlinks in first-hop paragraphs identify second-hop paragraphs, with bidirectional edges connecting source sentences to linked target paragraphs.This avoids relying on entity linking for discovering second-hop paragraphs and permits information flow in both directions.
- Context Encoding: Selected paragraphs are concatenated with the question and encoded by RoBERTa with bi-attention before node representations are extracted from the contextual sequence.Paragraph, sentence, and entity nodes are represented from span boundaries, while the question node uses max pooling.
- Graph Reasoning: A Graph Attention Network propagates messages among all hierarchical nodes, producing updated paragraph, sentence, entity, and question representations.The graph reasoning module uses edge-type-aware attention coefficients to update each node from its neighbors.
- Multi-task Prediction: Multi-task prediction uses paragraph nodes for paragraph selection, sentence nodes for supporting facts, and graph-enhanced context for answer span and answer-type prediction.The final training objective jointly combines the task losses, while entity prediction acts as regularization when answers are not entity nodes.
4 Experiments
Experiments on HotpotQA evaluate HGN’s answer and supporting-facts prediction in Distractor and Fullwiki settings. HGN achieves state-of-the-art results, while ablations and error analyses identify contributions and remaining weaknesses.
- Evaluation setup: HotpotQA evaluates answer prediction and supporting-facts prediction using exact match, F1, and joint metrics in Distractor and Fullwiki settings.The joint score measures answer and evidence prediction together.
- Test-set results: 47.11/74.21 Joint EM/F1 in Distractor improves over previous state of the art by 2.44/1.48 absolute points.HGN outperforms published and unpublished work on every metric in this setting.
- Test-set results: 2.57/1.08 Joint EM/F1 improvement is achieved in Fullwiki despite using an inferior retriever.Using the same retriever as SemanticRetrievalMRS yields a significant margin over that system.
- Ablation analysis: 2.81 joint F1 points separate the PS Graph from plain RoBERTa, while adding entity nodes contributes 0.30 points and the final graph adds 0.24 points.The authors associate the limited entity-node gain with possible noise and hypothesize that sentence connections improve representations.
- Encoder analysis: HGN variants outperform DFGN, EPS, and SAE when compared using the same pre-trained language models, indicating gains from model design.This comparison is intended to separate architecture effects from the choice of pre-trained encoder.
- Error analysis: 88.5 joint F1 makes comp-yn the easiest reasoning type, while bridge and comp-span reach approximately 74 joint F1.Error analysis also reports mistakes involving multiple equivalent answers, commonsense or discrete reasoning, multi-hop reasoning, and MRC span selection.
- Error analysis: The model remains limited on commonsense and discrete reasoning, often randomly selecting an entity, while most errors arise from multi-hop reasoning or MRC span selection.The authors state that these error patterns leave room for further improvement.
- Scope boundary: The graph is mainly tailored to HotpotQA, uses Wikipedia hyperlinks for connections, and restricts HotpotQA reasoning to two multi-hops.The authors suggest entity linking as an alternative for greater generalizability.
5 Conclusion
The paper introduces HGN, which unifies heterogeneous nodes in a hierarchical graph to capture clues across granularity levels for multi-hop question answering. Experiments show state-of-the-art performance on HotpotQA, while Fullwiki still relies on an off-the-shelf paragraph retriever.
- HGN weaves heterogeneous nodes into a single unified graph to capture clues from different granularity levels.
- The model achieves state-of-the-art performances on the HotpotQA benchmark.
- In the Fullwiki setting, HGN uses an off-the-shelf paragraph retriever to select relevant context from a large text corpus.
- Future work includes investigating interaction and joint training between HGN and the paragraph retriever.
A Datasets
HotpotQA provides Distractor and Fullwiki settings with comparable training and development sizes but different paragraph-search requirements. Fullwiki is more challenging because relevant documents must be found across all of Wikipedia.
- HotpotQA has Distractor and Fullwiki settings, each with 90k training samples and 7.4k development samples.
- The Distractor setting contains 2 gold paragraphs and 8 distractors.
- Fullwiki may not provide 2 gold paragraphs and requires searching the entire Wikipedia for relevant documents.
- Both settings include 90K hidden test samples.
B Implementation Details
The implementation builds bounded hierarchical graphs for HotpotQA and trains HGN with specified Transformer-based components and regularization settings. Training uses four Quadro RTX 8000 GPUs and takes around 12 hours for the best model.
- The graph limits each example to 60 entities, 40 sentences, and 4 paragraphs.
- Because HotpotQA requires two-hop reasoning, each question connects to up to two paragraphs.
- Entities are extracted with spaCy, while paragraph ranking uses a RoBERTa-large binary classifier.
- HGN is trained for 8 epochs with batch size 8, learning rate 1e-5, and LSTM and GNN dropout rates of 0.3.
- Training uses 4 Quadro RTX 8000 GPUs, and the best HGN model takes around 12 hours to train.
D Effectiveness of Paragraph Selection
The paragraph-selection study compares threshold-based, Top-N, and two-step ranking strategies, then examines graph-supported evidence discovery. The two-step selector achieves balanced precision and recall, while HGN recovers supporting facts missed by plain BERT.
- Effectiveness of Paragraph Selection: DFGN’s threshold-based selection reaches 98.27% recall but only 60.28% precision.
- Effectiveness of Paragraph Selection: The two-step paragraph selection method outperforms threshold-based selection in both DFGN and the plain BERT-base model.
- Effectiveness of Paragraph Selection: Adding graph connections enables HGN to discover both supporting-evidence pieces when plain BERT misses one.
- Effectiveness of Paragraph Selection: HGN predicts a missing supporting sentence in the second case study, whereas plain BERT predicts the other evidence sentences correctly.
F Additional Examples for Error Analysis
The section presents additional error-analysis examples comparing each question’s dataset answer with the proposed model’s prediction. The examples include errors across factual, temporal, entity-linking, and film-related questions.
- Error-analysis setup: The examples explicitly distinguish the question, dataset-provided answer, and proposed-model prediction.The section notes that a full set of 100 examples appears in Tables 10 and 11.
- Sample errors: The model predicts Ronald Reagan instead of George H. W. Bush for a question linking Richard Darman to an actor-president.
- Sample errors: It predicts Hamburger SV instead of Mehdi Mahdavikia when asked for the sports club associated with the football player.
- Sample errors: Other examples show incorrect predictions for an investment technique, a television drama, and the founding year of a brewery.The listed predictions are Magic formula investing, Cinderella and Four Knights, and 1698, respectively, against different dataset answers.
Category: Multiple Answers
This category collects examples involving multiple valid answers, aliases, morphology, comparison, and answer-format variation in supporting-fact prediction and question answering.
- Supporting-facts prediction: Figure 3 illustrates supporting-facts prediction examples in the HotpotQA Distractor setting.
- Multiple answers: The model gives semantically related alternatives such as “third season” for “three” and “singer-songwriter” for “singer, songwriter.”
- Aliases: One example identifies Ti Lung5 instead of the dataset answer Tommy Tam Fu-Wing, whose alias is separately noted.
- Reasoning errors: The category also includes incorrect ordering of people by birth year and a wrong shared description for two historical figures.
- Answer variation and comparison: Other errors reverse yes/no judgments, select the wrong genus, or choose an incomplete cathedral name.The examples concern Dashboard Confessional and World Party, Quesnelia and Honeysuckle, and St Chad’s Cathedral versus its longer answer name.
Category: Commonsense & External Knowledge
This category presents examples where answering requires external factual knowledge, commonsense connections, geographic relations, dates, sports information, or recognizing answer forms.
- External knowledge: The examples include correct-country answers rendered as demonyms, such as Ghanaian for Ghana and English for England.
- External knowledge: Several predictions miss factual relations involving a portrait subject, shared elevation, or a song’s release year.The listed predictions are Queen Victoria, kilometres, and 1985, differing from the dataset answers.
- Dates and geography: Temporal and geographic questions produce errors such as 1946 instead of 1945 and Sudan instead of the Republic of Chad.
- Numerical and sports knowledge: The category contains numerical and biographical failures, including an incorrect Roud Folk Song Index, race-related year, college team, and sports team.
- Entity and commonsense knowledge: Additional examples concern identifying a business unit, the mechanism used with gunpowder, a comic-book character, and an actress.The predictions differ from the dataset answers in each case.
- Error-analysis materials: Table 11 provides the full index list for the 100 samples selected for error analysis.