Source-linked AI summary
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, Xiang Ren
TL;DR
Knowledge-augmented QA models either struggle with efficient multi-hop reasoning or provide limited transparency into their rationales. MHGRN equips PTLMs with multi-hop, multi-relational reasoning over extracted knowledge-graph subgraphs, combining path-based reasoning with GNNs; it outperforms existing methods with improved scalability and interpretability.
Problem
Existing knowledge-augmented QA models struggle to model multi-hop relations efficiently or lack transparent prediction rationales.
Method
MHGRN performs multi-hop, multi-relational message passing over contextualized subgraphs extracted from external knowledge graphs, combining GNN and path-based reasoning.
Results
MHGRN outperforms existing graph-encoding methods with superior scalability and interpretability across knowledge-aware QA evaluations.
Takeaways & Limitations
Structured knowledge and textual knowledge can potentially be complementary, as MHGRN continues to benefit textual-knowledge-empowered systems.
Takeaways & Limitations
GNN-based reasoning is performed at the node level, limiting direct path modeling and path-level interpretability.
Abstract
from arXiv · showhide
Existing work on augmenting question answering (QA) models with external knowledge (e.g., knowledge graphs) either struggle to model multi-hop relations efficiently, or lack transparency into the model's prediction rationale. In this paper, we propose a novel knowledge-aware approach that equips pre-trained language models (PTLMs) with a multi-hop relational reasoning module, named multi-hop graph relation network (MHGRN). It performs multi-hop, multi-relational reasoning over subgraphs extracted from external knowledge graphs. The proposed reasoning module unifies path-based reasoning methods and graph neural networks to achieve better interpretability and scalability. We also empirically show its effectiveness and scalability on CommonsenseQA and OpenbookQA datasets, and interpret its behaviors with case studies.
1 Introduction
Knowledge-aware QA uses external knowledge graphs to support relational reasoning and more transparent evidence, but existing approaches trade off scalable multi-hop modeling against interpretability. MHGRN is proposed to combine these strengths and improve performance on two QA datasets.
- Motivation: Knowledge-aware QA requires relational reasoning over entities and relationships beyond textual comprehension, using external knowledge such as ConceptNet.CommonsenseQA examples require inferring latent relations among concepts that may not be explicitly stated in the question.
- Motivation: Knowledge graphs can make reasoning evidence explicit, such as a relational path connecting CHILD to SCHOOLROOM through ConceptNet relations.The cited path is presented as evidence supporting the answer SCHOOLROOM.
- Existing limitations: Path-based models offer interpretable multi-hop reasoning but are difficult to scale because possible paths grow polynomially with nodes and exponentially with path length.KagNet and MH-PGM encode extracted relational paths with sequence models and attention.
- Existing limitations: GNNs scale through message passing, but common variants may ignore relation types or fail to distinguish the importance of neighbors and relations.GCNs aggregate neighborhood information, while RGCNs add relation-specific aggregation without fully modeling differing importance.
- Proposed approach: MHGRN combines GNN scalability with path-based interpretability through multi-hop message passing and structured relational attention.The model is designed to let each node attend directly to multi-hop neighbors within a single layer.
- Results: Experiments on CommonsenseQA and OpenbookQA report significant gains over knowledge-agnostic PTLMs and large-margin improvements over other graph encoders.The paper summarizes these experiments as a principal contribution.
2 Problem Formulation and Overview
The framework scores each answer option by combining textual understanding of the question–answer statement with relational reasoning over a question-guided knowledge-graph subgraph. It selects the option with the highest plausibility score.
- Task formulation: The task identifies the correct answer from multiple-choice options by assigning each option a plausibility score relative to the question.The paper limits its scope to multiple-choice question answering.
- Graph construction: For each question–answer pair, the framework concatenates their representations into a statement vector and extracts a relevant contextualized subgraph from an external knowledge graph.The subgraph is a multi-relational graph containing entities relevant to the statement.
- Architecture: The graph encoder performs relational reasoning over the contextualized subgraph, while the text encoder captures textual information.Figure 3 presents these as complementary components of the framework.
- Scoring: The encoded statement and graph representation are concatenated to calculate the plausibility score for each answer option.The highest-scoring option is selected as the answer.
3 Background: Multi-Relational Graph Encoding Methods
Multi-relational graph encoders either operate directly on graph structure with GNN message passing or decompose graphs into relational paths. These families differ in scalability, path-level reasoning, and interpretability.
- Path-based models: Path-based models represent a graph as relational paths connecting entity pairs, then encode and pool those paths.Relation Networks encode one-hop triples, while KagNet extends this idea to paths of length up to K with LSTMs and attention.
- GNNs: GNNs transform node features into node embeddings through message passing and obtain a graph representation by pooling the node embeddings.GCNs aggregate direct-neighbor information, while RGCNs use relation-specific weight matrices for multi-relational graphs.
- Comparison: GNNs provide scalability but reason at the node level, which makes them incompatible with explicit path modeling and limits path-level interpretability.This contrasts with path-based methods that directly represent relational paths.
- Path-based models: Relation Networks encode triples whose heads are question entities and tails are answer entities, pooling their embeddings into a graph representation.Triple embeddings combine head-node features, relation embeddings, and tail-node features.
- Path-based models: KagNet models nondegenerate paths by encoding question-to-answer paths of length no more than K and aggregating them with attention.This extends one-hop relation modeling to multi-hop paths.
4 Proposed Method: Multi-Hop Graph Relation Network (MHGRN)
MHGRN is a GNN architecture that combines path-level reasoning and interpretability with GNN scalability. It performs multi-hop relational message passing, structured attention, and plausibility scoring for question-answer statements.
- MHGRN unifies GNNs and path-based models, preserving scalable message passing while enabling interpretable path-level reasoning.
- The architecture takes a multi-relational graph and question-answer statement vector as input, then outputs a statement plausibility score.
- Type-Specific Transformation: Type-specific linear transformations make node representations aware of each node’s type.
- Multi-Hop Message Passing: MHGRN passes messages directly over valid relational paths with lengths up to K within a multi-hop message-passing layer.
- Structured Relational Attention: Structured relational attention assigns importance to relation paths and message transfers between node types while avoiding O(m^k) parameterization.
- Dynamic programming computes message passing and attention in linear time, with sparse-graph time and space complexity linear in K or the number of nodes n.
- Expressive Power of MHGRN: MHGRN can represent K-hop relational networks, generalizing single-hop RGCN-style message passing while directly modeling multi-hop paths.
- Learning, Inference and Path Decoding: For QA, answer-entity embeddings are pooled, combined with the text representation, and scored by an MLP; inference can decode a highest-attention reasoning path.
5 Experimental Setup
The experiments construct contextualized ConceptNet subgraphs and evaluate knowledge-aware QA models on CommonsenseQA and OpenbookQA using pretrained language encoders and multiple baselines.
- Extracting G from External KG: The study uses ConceptNet as an external general-domain knowledge graph and constructs a 34-relation multi-relational graph with reverse relations.
- Evaluation: The CommonsenseQA evaluation reports in-house Dev and Test accuracy over four runs, alongside an official-split comparison with leaderboard methods.
- Extracting G from External KG: Contextualized graphs include linked question entities, entities on two-hop paths between them, and all edges among the selected nodes without pruning.
- Datasets: Evaluation covers the multiple-choice CommonsenseQA and OpenbookQA datasets, both of which require world knowledge beyond textual understanding.
- Datasets: CommonsenseQA probes latent compositional relations among ConceptNet entities, while OpenbookQA combines elementary science questions with an open book of facts and broader common sense.
- Baseline Methods: Baselines include pretrained-language-model fine-tuning, KG-augmented models, text-based external-knowledge systems, and RGCN, RN5, KagNet, and GconAttn.
6 Results and Discussions
MHGRN improves knowledge-aware QA across CommonsenseQA and OpenbookQA while combining interpretable reasoning paths with scalable graph computation. Analyses show complementary textual and structured knowledge, useful hop-depth behavior, near-linear cost, and plausible decoded evidence.
- Main results: MHGRN achieves the best performance among compared models on CommonsenseQA’s official split and improves over knowledge-agnostic pre-trained language models.The comparison includes single-model and ensemble settings, excluding the substantially larger UnifiedQA from direct comparison because of training cost.
- Main results: MHGRN surpasses all implemented OpenbookQA baselines, with an absolute Test increase of ∼2%.Replacing ROBERTA-LARGE with AristoRoBERTaV76 still yields benefits, suggesting textual and structured knowledge can be complementary.
- Analysis: Across training-data fractions, MHGRN consistently improves more over knowledge-agnostic fine-tuning than other graph encoding methods.The analysis jointly trains the text and graph encoders or fine-tunes text encoders alone on CommonsenseQA.
- Analysis: Increasing MHGRN’s hop parameter helps until K = 4, while performance begins declining when K > 3, possibly because longer paths introduce exponential noise.The study evaluates IHDev accuracy on CommonsenseQA with respect to the number of hops.
- Scalability: Training time for MHGRN and RGCN grows linearly with K, while MHGRN’s empirical cost ratio approaches 2 despite a theoretical MultiRGN cost m times that of RGCN.The authors attribute this gap to better parallelization.
- Interpretability: Decoded CommonsenseQA examples show MHGRN producing reasonable path evidence, including chains through mentioned entities and bridges through unmentioned entities.Both examples answer the questions correctly.
7 Related Work
Related work augments language understanding with structured or textual external knowledge, but existing graph and path encoders leave trade-offs in scalability, relational modeling, or path-level transparency.
- Knowledge-Aware Methods for NLP: Knowledge-aware NLP methods encode external knowledge as nodes, triples, paths, or subgraphs to augment textual understanding.Other approaches provide textual knowledge directly to pre-trained language models through fine-tuning or retrieved evidence.
- Neural Graph Encoding: GAT and RGCN support graph-based aggregation, but the cited neural graph encoders perform only single-hop message passing or lack path-level interpretation.RGCN handles multi-relational graphs through relational message passing.
- Neural Graph Encoding: The related-work landscape therefore contrasts explicit path reasoning with graph message passing for knowledge-aware QA.The supplied passages identify interpretability and scalability as distinguishing concerns across these approaches.
8 Conclusion
The paper concludes that MHGRN provides scalable multi-hop relational reasoning over interpretable knowledge structures. It combines graph neural networks’ scalability with path-based models’ interpretability and empirically outperforms existing methods.
- Conclusion: MHGRN leverages general knowledge through multi-hop reasoning over interpretable structures such as ConceptNet.The method is presented as a principled and scalable approach.
- Conclusion: MHGRN generalizes and combines the advantages of graph neural networks and path-based reasoning models.Its design explicitly performs multi-hop relational reasoning.
- Conclusion: MHGRN is empirically shown to outperform existing methods with superior scalability and interpretability.This conclusion summarizes the paper’s reported empirical findings.
A Merging Types of Relations in ConceptNet
The appendix describes preprocessing that merges ConceptNet relations with similar semantics and general usage patterns.
- Merging Types of Relations in ConceptNet: ConceptNet relations are merged when they are close in semantics and in the general usage of their triple instances.Table 7 lists the relations merged during preprocessing and marks reverse relations with an asterisk.
B Dataset Split Specifications
The experiments use publicly available dataset splits and an in-house CommonsenseQA split for model analysis. CommonsenseQA’s in-house split contains 1,241 test examples and 8,500 training examples.
- CommonsenseQA and OpenbookQA provide publicly available training and development splits with public leaderboards.
- The CommonsenseQA in-house split uses 1,241 official training examples as test data and the remaining 8,500 as training data.
- The in-house CommonsenseQA split supports model analysis because official ground-truth labels are not readily available.
C Implementation Details
The implementation uses pretrained text encoders and graph encoders with separately tuned learning rates, standardized node features, and small graph architectures. MHGRN’s multi-hop computation is implemented with dynamic programming and has linear complexity on sparse graphs.
- Optimization and training: Text encoders use dataset-specific learning rates selected from a predefined candidate set, while graph encoders use separately tuned learning rates.The text-encoder candidates range from 1×10^-5 to 1×10^-4, and graph-encoder candidates range from 1×10^-4 to 3×10^-3.
- Optimization and training: Training uses PyTorch, cross-entropy loss, RAdam, early stopping, and dataset-dependent batch sizes and maximum sequence lengths.Most models use batch size 32 and sequence length 64; the AristoRoBERTaV7+MHGRN comparison uses batch size 16 and sequence length 256.
- Input representations: ConceptNet node features are 1024-dimensional mean-pooled BERT-LARGE embeddings derived from templated knowledge-triple sentences.The same node-feature set is used for all implemented models.
- Graph encoders: The experiments use two-layer RGCN and single-layer MHGRN graph encoders, with parameter counts reported separately.The parameter-count comparison is listed in Table 11.
- Multi-hop computation: Dynamic programming computes multi-hop message passing in linear time, with sparse-graph time and space complexity linear in maximum path length K or node count n.The method avoids explicitly enumerating the potentially exponential number of paths.
- Formal graph encoding: A K-hop Relation Network maps a multi-relational graph to a fixed-size vector by pooling weighted multi-hop relational-path representations.The formal construction combines endpoint features with relation embeddings along paths and applies pooling weights.
- Formal graph encoding: MHGRN can express the K-hop Relation Network for arbitrary graphs through an appropriate parameter setting.The construction disables relation-type attention, restricts message passing from question entities to answer entities, and uses identity activation with pooling over answer entities.