Source-linked AI summary
Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering
Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Songlin Hu
TL;DR
Commonsense question answering needs external evidence that prior methods obtain from annotations or a single knowledge source. This paper automatically combines ConceptNet and Wikipedia evidence through graph-based representation learning and inference, reaching 75.3% accuracy on CommonsenseQA.
Problem
Commonsense question answering requires background knowledge, while prior approaches rely on human-annotated evidence or a single structured or unstructured knowledge source.
Method
The approach automatically extracts evidence from ConceptNet and Wikipedia, constructs graphs for both sources, and applies graph-based contextual representation learning and graph-based inference.
Results
75.3% accuracy on CommonsenseQA is reported as state-of-the-art performance on the dataset.
Takeaways & Limitations
Combining both knowledge sources and using both graph-based modules brings further performance improvements on CommonsenseQA.
Takeaways & Limitations
Error analysis identifies insufficient evidence and similar evidence among choices as remaining error categories.
Abstract
from arXiv · showhide
Commonsense question answering aims to answer questions which require background knowledge that is not explicitly expressed in the question. The key challenge is how to obtain evidence from external knowledge and make predictions based on the evidence. Recent works either learn to generate evidence from human-annotated evidence which is expensive to collect, or extract evidence from either structured or unstructured knowledge bases which fails to take advantages of both sources. In this work, we propose to automatically extract evidence from heterogeneous knowledge sources, and answer questions based on the extracted evidence. Specifically, we extract evidence from both structured knowledge base (i.e. ConceptNet) and Wikipedia plain texts. We construct graphs for both sources to obtain the relational structures of evidence. Based on these graphs, we propose a graph-based approach consisting of a graph-based contextual word representation learning module and a graph-based inference module. The first module utilizes graph structural information to re-define the distance between words for learning better contextual word representations. The second module adopts graph convolutional network to encode neighbor information into the representations of nodes, and aggregates evidence with graph attention mechanism for predicting the final answer. Experimental results on CommonsenseQA dataset illustrate that our graph-based approach over both knowledge sources brings improvement over strong baselines. Our approach achieves the state-of-the-art accuracy (75.3%) on the CommonsenseQA leaderboard.
Introduction
Commonsense question answering requires collecting and reasoning over background knowledge, yet prior approaches typically use human-annotated evidence or only one knowledge source. This work combines heterogeneous evidence with graph-based representation learning and inference, achieving state-of-the-art CommonsenseQA performance.
- Commonsense question answering requires machines to collect background knowledge and reason over it to answer questions.
- CommonsenseQA questions are designed so answer choices share a relation with the question concept, while only one choice is correct through background knowledge.
- ConceptNet evidence identifies choices A and C, whereas Wikipedia evidence identifies C and E; combining both yields the correct answer C.
- Prior approaches either generate evidence from human annotations or extract it from homogeneous structured or unstructured sources, without simultaneously using both.
- The proposed approach automatically extracts heterogeneous evidence and uses graph-based contextual representation learning plus graph-based inference.
- 75.3% accuracy on CommonsenseQA establishes the approach as state-of-the-art on the dataset.
Task Definition and Dataset
CommonsenseQA evaluates whether a model can distinguish one correct answer from four incorrect choices using background knowledge. Accuracy is the evaluation metric.
- CommonsenseQA contains natural-language questions with five answer choices, requiring the model to distinguish the correct choice from the others.
- Annotators use background knowledge to write questions in which only one answer choice is correct, making the task challenging.
- Accuracy is used as the evaluation metric for CommonsenseQA.
Approach Overview
The approach has two stages: extracting knowledge from ConceptNet and Wikipedia, then performing graph-based reasoning over the resulting structures.
- The approach consists of knowledge extraction followed by graph-based reasoning.
- Knowledge is extracted from the structured ConceptNet knowledge base and Wikipedia plain texts according to each question and its choices.
- Graphs are constructed to exploit the relational structures present in both knowledge sources.
- Figure 2 provides an overview of the proposed approach.
Knowledge Extraction
The extraction stage retrieves structured paths from ConceptNet and ranked sentences from Wikipedia, then converts both evidence sources into graphs that expose their relational structure.
- Knowledge Extraction from ConceptNet: ConceptNet evidence is extracted by identifying question and choice entities and searching for paths shorter than three hops between them.
- Knowledge Extraction from ConceptNet: Covered ConceptNet triples are merged into a Concept-Graph whose nodes are triples and whose edges connect successive triples sharing an entity.
- Knowledge Extraction from ConceptNet: ConceptNet triples are converted into natural-language sequences using relation templates to obtain contextual representations for their nodes.
- Knowledge Extraction from Wikipedia: Wikipedia evidence is retrieved by searching indexed sentences after removing stopwords from the question and choices, selecting the top 10 sentences.
- Knowledge Extraction from Wikipedia: Semantic Role Labeling converts Wikipedia sentences into subject-predicate-object triples, with arguments and predicates represented as graph nodes.
- Knowledge Extraction from Wikipedia: The Wiki-Graph adds subject-predicate and predicate-object edges, plus enhanced connectivity edges based on phrase containment or one-word differences.
Graph-Based Reasoning
The model combines graph-based contextual representation learning with graph-based inference over ConceptNet and Wikipedia evidence. It reorders evidence using graph structure, encodes graph neighborhoods with GCNs, and aggregates node information for answer prediction.
- Graph-Based Reasoning: The architecture contains graph-based contextual representation learning and graph-based inference modules.The first module uses graph information to redefine word distances; the second uses GCNs and graph attention for prediction.
- Graph-Based Contextual Representation Learning Module: Graph structure redefines relative word distances so semantically related evidence words receive shorter positions in XLNet representations.Topology sorting reorders ConceptNet triples and Wikipedia evidence sentences before they are concatenated with the question and answer choice as XLNet input.
- Graph-Based Reasoning: ConceptNet and Wikipedia evidence are converted into graph structures and combined for reasoning.ConceptNet paths form a Concept-Graph, while Wikipedia evidence is represented through SRL-derived triples and a Wiki-Graph; the two graphs are treated as one graph.
- Graph-Based Inference Module: GCNs propagate information from adjacent evidence nodes, while graph attention aggregates final-layer node representations into a graph representation.The graph representation is combined with XLNet’s input representation and passed to an MLP to compute answer confidence scores.
- Graph-Based Contextual Representation Learning Module: The attempted graph-based position matrix was discarded because it consumed too much memory and could not execute efficiently.The paper instead uses topology sorting to reorder evidence according to the constructed graphs.
Experiments
Experiments evaluate the model on CommonsenseQA, compare it with baselines, analyze reasoning and knowledge-source components, and examine illustrative successes and errors.
- Benchmark Results: The model achieves the best performance on CommonsenseQA development and blind test datasets, outperforming RoBERTa (ensemble) by 2.8% absolute accuracy.It also exceeds group 1 methods by more than 10% absolute accuracy.
- Benchmark Results: The model gains 13.1% over BERT + AMS, 5.7% over RoBERTa + CSPT, and 3.2% over RoBERTa + IR by combining graph reasoning with ConceptNet and Wikipedia evidence.It uses ConceptNet graph paths for reasoning and Wikipedia for unstructured evidence.
- Reasoning Ablations: Adding topology sort yields a 1.9% gain, graph inference yields a 1.4% benefit, and combining both produces a 3.5% improvement over the baseline.Topology sort changes relative word positions, while GCN and graph attention encode and aggregate node and word representations.
- Knowledge-Source Ablations: ConceptNet alone improves the baseline by 6.4%, Wikipedia alone by 4.6%, and their combination by 9.4% absolute gain.The results indicate complementary contributions from the two heterogeneous knowledge sources.
- Case Study and Error Analysis: In a mammals example, ConceptNet supplies animal and hair relations while Wikipedia supplies evidence that very few mammals lay eggs; 38 of 50 sampled errors involve similar evidence.The remaining sampled errors include 10 cases lacking evidence and 2 cases with duplicate choices.
Related Work
Related work frames CommonsenseQA as a commonsense reasoning benchmark requiring external knowledge and situates this paper among transfer-learning and graph-neural-network approaches.
- Commonsense Reasoning: Commonsense reasoning tasks require reasoning over external knowledge beyond the input to predict the correct answer.
- Commonsense Reasoning: CommonsenseQA derives from ConceptNet, gives answer choices the same relation to the question concept, and has prompted work using explanations and extracted ConceptNet evidence.
- Knowledge Transfer in NLP: Pre-trained language models such as ELMo, GPT, BERT, XLNet, and RoBERTa have improved many NLP tasks; this paper uses XLNet as its backend.
- Graph Neural Networks for NLP: Graph neural networks, including GCNs, have been applied to entity-relation extraction, dependency-tree relation extraction, and multi-hop reading comprehension.
Conclusion
The paper studies CommonsenseQA using heterogeneous external evidence and graph-based reasoning, reporting state-of-the-art performance on the dataset.
- The approach extracts evidence from ConceptNet and Wikipedia, constructs graphs for both sources, and uses graph-based contextual representation learning and graph-based inference.
- Results show that the model achieves state-of-the-art performance on CommonsenseQA.