Source-linked AI summary
KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, Xiang Ren
TL;DR
Commonsense reasoning systems must answer questions involving ordinary situations, yet language-model baselines remain difficult to interpret and knowledge structures are not well defined for explainable inference. KagNet grounds each question–answer pair in an external commonsense graph and applies a graph-based attentive encoder, achieving state-of-the-art CommonsenseQA performance while exposing intermediate attention scores. The framework’s scope remains limited for negation, comparative questions, and subjective judgments.
Problem
Language-model pre-training alone does not provide well-defined reusable structures for explainable commonsense reasoning, while neural inferences remain difficult to interpret.
Method
KagNet grounds question–answer pairs as schema graphs from external knowledge graphs and models them with a GCN-LSTM-HPA architecture using hierarchical path-based attention.
Results
KagNet achieves state-of-the-art performance on CommonsenseQA and improves test accuracy by 2.2% over the compared baseline.
Takeaways & Limitations
Intermediate attention scores provide a readable view of question–answer concept pairs and paths involved in the inference process.
Takeaways & Limitations
The framework is not capable of comparative reasoning, and its grounding stage is not sensitive to negation words.
Abstract
from arXiv · showhide
Commonsense reasoning aims to empower machines with the human ability to make presumptions about ordinary situations in our daily life. In this paper, we propose a textual inference framework for answering commonsense questions, which effectively utilizes external, structured commonsense knowledge graphs to perform explainable inferences. The framework first grounds a question-answer pair from the semantic space to the knowledge-based symbolic space as a schema graph, a related sub-graph of external knowledge graphs. It represents schema graphs with a novel knowledge-aware graph network module named KagNet, and finally scores answers with graph representations. Our model is based on graph convolutional networks and LSTMs, with a hierarchical path-based attention mechanism. The intermediate attention scores make it transparent and interpretable, which thus produce trustworthy inferences. Using ConceptNet as the only external resource for Bert-based models, we achieved state-of-the-art performance on the CommonsenseQA, a large-scale dataset for commonsense reasoning.
1 Introduction
CommonsenseQA requires models to distinguish plausible answers from closely related distractors, while existing language-model baselines remain less capable than humans and difficult to interpret. KagNet addresses this gap by grounding question–answer pairs in external commonsense knowledge and modeling the resulting schema graphs with interpretable attention.
- CommonsenseQA challenges reasoners to distinguish the correct answer from highly related but less plausible distractors.The example asks where adults use glue sticks and contrasts plausible and distractive choices.
- Pre-trained language models provide strong baselines, but their performance remains below humans and their reasoning processes lack transparency.The paper describes these inferences as difficult to understand and potentially dubious.
- Knowledge bases offer reusable relational structures that can enhance reasoning capacity and make model behavior more transparent.The paper motivates knowledge-centric approaches as extensible through commonsense knowledge acquisition.
- KagNet retrieves a schema graph for each question–answer pair from external knowledge graphs such as ConceptNet, then models it for inference.The framework has schema graph grounding followed by graph modeling for inference.
- KagNet combines graph convolutional networks, LSTMs, and hierarchical path-based attention to represent relational schema graphs.The resulting architecture is described as GCN-LSTM-HPA.
- Intermediate attention scores make the framework’s inferences more readable and interpretable.The paper presents these scores as a basis for understanding model behavior.
2 Overview
The framework treats commonsense question answering as selecting one candidate answer and adds external knowledge by grounding each question–answer pair as a schema graph. It encodes that graph alongside a statement representation and scores each candidate’s inference plausibility.
- Given a question and candidate answers, the task is to select one answer, while the knowledge-aware formulation grounds them as a schema graph from an external knowledge graph.The knowledge graph contains concepts connected by typed semantic relations.
- The framework recognizes concepts in the question and answer, then constructs a schema graph by finding paths between mentioned-concept pairs.This workflow is illustrated as the first stage before graph encoding.
- A language encoder represents the question–answer pair as a statement vector that supplies an additional input to graph modeling.The encoder may be trainable or a fixed feature extractor.
- The GCN-LSTM-HPA architecture produces a graph vector, which a multilayer perceptron converts into a scalar plausibility score for the candidate.The final answer is the candidate with the maximum plausibility score for the question.
3 Schema Graph Grounding
Schema graph grounding maps question and answer text to ConceptNet concepts, retrieves connecting paths, and reduces noise before graph modeling. The procedure uses matching rules, path finding, pruning, and knowledge-graph-embedding scores.
- The grounding stage recognizes concepts, constructs schema graphs by retrieving knowledge-graph paths, and prunes noisy paths.These are the three stated grounding components.
- 3.1 Concept Recognition: Concept recognition matches question and answer tokens to mentioned concepts in ConceptNet, using lemmatization and stop-word filtering to improve naive matching.The paper notes that exact n-gram matching can retrieve irrelevant or polysemous concepts.
- 3.2 Schema Graph Construction: ConceptNet represents commonsense knowledge as typed triples, and the framework merges 42 original relation types into 17 to increase graph density.A triple contains a head concept, relation type, and tail concept.
- 3.2 Schema Graph Construction: A schema graph is a related subgraph for a question–answer pair, constructed by finding short paths among mentioned question and answer concepts.The approach avoids comprehensive subgraph retrieval because ConceptNet is incomplete and very large.
- 3.2 Schema Graph Construction: Knowledge-graph embeddings such as TransE score decomposed path triples so irrelevant paths can be pruned according to their confidence.The pretrained concept and relation embeddings also initialize KagNet.
4 Knowledge-Aware Graph Network
KAGNET models schema graphs with a GCN-LSTM-HPA architecture that combines contextualized graph encoding, relational path modeling, and hierarchical attention. It represents question-answer plausibility using information from both knowledge-symbolic and language-semantic spaces.
- GCN Encoding: KAGNET first applies GCNs to schema graphs to contextualize pre-trained concept embeddings and capture structural patterns.The GCN updates each concept using its neighboring nodes and its own representation.
- Relational Path Encoding: LSTMs encode sequences of triple vectors along paths between question concepts and answer concepts, capturing multi-hop relational information.Each triple combines corresponding concept and relation vectors, and path representations use the first and last hidden states.
- Semantic-Symbolic Integration: The model combines schema-graph relational representations with language-side representations derived from the question-answer statement.The statement vector is combined with path-based representations before graph aggregation.
- Answer Scoring: The final graph representation is aggregated and passed through a multilayer perceptron to produce an answer plausibility score.The candidate with the highest plausibility score is selected for the question.
- Hierarchical Attention: Hierarchical path-based attention selectively aggregates important paths and then important question-answer concept pairs instead of treating all equally.This addresses the limitations of mean pooling when paths or concept pairs contribute unequally to reasoning.
5 Experiments
Experiments evaluate KagNet on CommonsenseQA against knowledge-agnostic and knowledge-aware baselines, across official and in-house data settings. Results show stronger benchmark performance, benefits across training-data conditions, and interpretable attention-based analyses, while exposing difficulties with several reasoning types.
- Dataset and experiment setup: 12,102 questions with five candidate answers define the hard CommonsenseQA evaluation, covering spatial, social, causal, physical, and temporal commonsense.The experiments use both official and in-house splits for benchmark comparisons and ablations.
- Comparison with standard baselines: KagNet improves official CommonsenseQA test accuracy by an absolute 2.2% over reported baseline methods, achieving state-of-the-art performance.The official split is used for comparison with leaderboard and dataset-paper results.
- Performance comparisons and analysis: KagNet-based methods with fixed GPT or BERT encoders outperform fine-tuning those encoders across 10%, 50%, and 100% training-data settings.The improvement is relatively limited when only 10% of the training data is available.
- Performance comparisons and analysis: KagNet outperforms all evaluated ConceptNet-based knowledge-aware baselines by a large margin in accuracy under both easy and hard modes.The comparison uses an in-house split with BLSTM sentence encoders and does not add human-generated explanations as supervision.
- Ablation study: Ablations show that the GCN-LSTM-HPA architecture, both attention levels, and path pruning contribute to performance, while relational GCN replacement degrades results through overparameterization.The study attributes additional value to filtering noisy paths.
- Error analysis: KagNet remains challenged by negation, comparisons among multiple plausible answers, and subjective judgments reflected in annotator preferences.The grounding stage can miss negation, the training method does not explicitly compare candidates, and some labels depend on reasoner personality.
- Case study on interpretability: Hierarchical attention supports interpretation by selecting high-attention question-answer concept pairs and top-ranked paths, while diminishing noisy concepts such as “fountain.”The analysis uses pair-level and path-level attention scores to connect selected paths with the inference process.
6 Related Work
The paper situates KagNet among commonsense datasets, external-knowledge methods, and relational reasoning models. Its distinguishing feature is explicit graph-structured knowledge with scored triples and paths for interpretable inference.
- Commonsense knowledge and reasoning: CommonsenseQA and related datasets have expanded supervised evaluation across general commonsense, social behavior, situation prediction, and visual scene understanding.
- Commonsense knowledge and reasoning: Earlier language-model approaches show promise but do not generalize readily to broad question answering or expose the knowledge used in inference.
- Injecting external knowledge for NLU: Unlike methods that use knowledge-base retrieval mainly to augment embeddings, KagNet imposes graph-structured knowledge directly into the model.
- Injecting external knowledge for NLU: KagNet injects external knowledge as an explicit graph structure, supporting relational reasoning over concepts rather than only transforming word embeddings into concept embeddings.
- Relational reasoning: KagNet treats question and answer concepts as objects and models their relations from semantic and symbolic spaces, connecting it to knowledge-augmented relational reasoning.
7 Conclusion
The paper presents a knowledge-aware framework that grounds relevant commonsense into schema graphs and models them with KagNet for transparent relational reasoning. It reports new state-of-the-art results while identifying question parsing and visual reasoning as future directions.
- Conclusion: The framework constructs schema graphs for relevant commonsense knowledge and models them with the KAGNET module.
- Conclusion: KAGNET uses a GCN-LSTM-HPA architecture to represent graphs for relational reasoning transparently and interpretably.
- Conclusion: The framework yields new state-of-the-art results on a large-scale general dataset for testing machine commonsense.
- Conclusion: Future directions include improved parsing for negation and comparative questions and incorporating knowledge into visual reasoning.