Source-linked AI summary
Improving Question Answering by Commonsense-Based Pre-Training
Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, Jian Yin
TL;DR
Many neural QA systems struggle with commonsense questions because they lack connections between concepts. The paper pre-trains direct and indirect relational functions from ConceptNet and integrates them with existing models, improving three commonsense QA tasks while helping retrieve useful evidence.
Problem
Existing neural systems lack true language understanding and reasoning capabilities, and machines struggle with commonsense questions when training data are limited.
Method
The paper pre-trains direct and indirect relational functions between concepts using external commonsense knowledge and integrates them with neural QA systems.
Results
The pre-trained models improve strong baselines on three commonsense question-answering datasets and help discover useful evidence from ConceptNet.
Takeaways & Limitations
Commonsense-based pre-training is complementary to document-based neural QA approaches and can further improve performance through model combination.
Takeaways & Limitations
The system fails on difficult questions requiring logical reasoning and can fail to identify the most useful retrieved concept for distinguishing answers.
Abstract
from arXiv · showhide
Although neural network approaches achieve remarkable success on a variety of NLP tasks, many of them struggle to answer questions that require commonsense knowledge. We believe the main reason is the lack of commonsense \mbox{connections} between concepts. To remedy this, we provide a simple and effective method that leverages external commonsense knowledge base such as ConceptNet. We pre-train direct and indirect relational functions between concepts, and show that these pre-trained functions could be easily added to existing neural network models. Results show that incorporating commonsense-based function improves the baseline on three question answering tasks that require commonsense reasoning. Further analysis shows that our system \mbox{discovers} and leverages useful evidence from an external commonsense knowledge base, which is missing in existing neural network models and help derive the correct answer.
1 Introduction
Existing neural systems can perform strongly with supervision yet struggle with commonsense reasoning, especially when training data are limited. The paper addresses this gap by pre-training direct and indirect concept-relational functions from external commonsense knowledge and combining them with QA models.
- Existing systems can achieve strong supervised performance but lack language understanding and reasoning capabilities crucial for commonsense reasoning.
- The proposed method pre-trains generic functions over external commonsense knowledge to broaden concept coverage and access relational context.
- These pre-trained functions can be integrated with existing neural QA systems while preserving their semantic matching ability.
- The paper attributes this limitation to missing connections between concepts, including both direct and indirect relations.
- ConceptNet illustrates direct relations such as driving–a license and indirect relations linking a car with getting to a destination through driving.
- Experiments on three commonsense question-answering tasks report improvements over strong baselines and show that the system can discover useful external evidence.
2 Tasks and Datasets
The paper evaluates commonsense reasoning in three multiple-choice QA settings with different supporting-context structures. Systems rank candidate answers using questions and, when available, supporting passages.
- The experiments use three multiple-choice QA tasks requiring commonsense reasoning as testbeds.
- The tasks predict the correct answer from candidate answers given a question and optionally a supporting passage.
- Following prior work, the problem is treated as ranking, with the highest-scoring candidate returned at test time.
- SemEval 2018 Task 11: SemEval 2018 Task 11 uses daily-activity events, two candidate answers, and documents requiring commonsense inference beyond explicitly stated facts.
- ARC: ARC contains scientific questions with mostly four candidate answers and separates examples into easy and challenging sets.
- OpenBook QA: OpenBook QA combines an elementary-science fact set with commonsense knowledge and contains 5,957 multiple-choice questions.
3 Commonsense Knowledge
The commonsense source is ConceptNet, a semantic network that represents words and phrases together with commonsense relationships between them.
- ConceptNet contains 657,637 instances spanning 39 relationship types represented as subject–relation–object triples.
- Its relations include IsA, Causes, and CapableOf, which encode relationships such as car–vehicle, car–pollution, and car–go fast.
- The approach is presented as generic enough to apply to other commonsense knowledge bases, although those extensions are left for future work.
4 Approach Overview
The framework scores candidate answers by combining evidence from the question and supporting passage with evidence retrieved from commonsense knowledge. It then selects the candidate with the highest resulting score.
- The system formulates commonsense QA as designing a scoring function that captures document evidence and retrieved commonsense evidence.
- The candidate score combines a document-based scoring function fdoc(ai) with a commonsense-based scoring function fcs(ai).
- The final score considers the question, any given passage, and commonsense knowledge related to the instance.
5 Commonsense-based Model
The model pre-trains commonsense-based functions over ConceptNet to capture direct and indirect relations between concepts, then uses them to score relevance between question and answer text.
- Pre-training commonsense functions: The approach uses graph neural networks to integrate commonsense-graph context before calculating relevance between question and candidate-answer text.Concept representations combine word-level and neighbor-based information from the knowledge graph.
- Concept representation: The model represents each concept using its contained words and connected neighbors, with relation-specific transformations over neighboring concept representations.The final concept representation concatenates word-based and neighbor-based representations.
- Direct and indirect relations: Direct and indirect relational functions are learned by defining different positive instances and sampling strategies for each relation type.The training objective uses a margin of 0.1, with directly adjacent concepts serving as positive examples for direct relations.
- Commonsense fact retrieval: ConceptNet facts are retrieved for each sentence by matching extracted 1-, 2-, and 3-grams to fact triples.Each fact is represented as a subject, relation, and object, and the retrieved facts form the sentence-level evidence sets.
- Text relevance scoring: For question-answer scoring, the model selects the most relevant concept in the candidate evidence for each question concept and averages those scores.This relevance score is designed to capture both direct relations, such as electrons and negative charge, and indirect relations connected through a pivot concept.
- Relation to TransE: The method outputs a parameterized scoring function and incorporates concept neighbors through a graph neural network, unlike TransE's entity and relation embeddings.The comparison distinguishes the models by goals, outputs, and the evidence used to represent entities.
6 Experiment
Experiments on ARC, OpenBook QA, and SemEval evaluate separately trained document-based and commonsense-based models. Commonsense scores improve document-based baselines, while error analysis identifies retrieval relevance, concept disambiguation, and logical reasoning as remaining challenges.
- Model Comparisons and Analysis: The experiments evaluate commonsense-based models on ARC Challenge, OpenBook QA, and SemEval 2018 Task 11 using accuracy.The document-based and commonsense-based models are trained separately to improve generality and facilitate incorporation into other tasks.
- Model Comparisons and Analysis: Commonsense-based scores improve the document-based TriAN model, and combining both scores yields further improvements on ARC and SemEval.The results indicate that commonsense-based models are complementary to standard document-based models.
- Model Comparisons and Analysis: ConceptNet evidence helps solve examples that the document-based model mispredicts by connecting questions, candidate answers, and passages through direct and indirect relations.In one SemEval example, driving–license, license–permit, and driver–care relations provide complementary evidence.
- Error Analysis and Discussion: The dominant error arises when the system fails to identify the most useful retrieved concept based on question relevance, document relevance, and candidate discrimination.A context-aware module is proposed to model retrieved-concept importance for each instance.
- Error Analysis and Discussion: Entity or concept ambiguity limits linking to the knowledge base, motivating a disambiguation model that considers both the question and passage.The example contrasts linking “Micheal Jordan” to a machine-learning expert versus a basketball player.
- Error Analysis and Discussion: The current system fails on difficult questions requiring logical reasoning, such as questions about durations or counting people.The authors identify semantic parsing with predefined grammar and operators as a future direction.
7 Related Work
The work positions commonsense-based pre-training within research that integrates external structured knowledge and within broader NLP model-pretraining approaches. Unlike sequence-oriented text pre-training, it models concept relatedness using graph structure and applies the result to question answering.
- Related Work: Related approaches use external knowledge such as WordNet and NELL either to enhance computational units or to provide external signals before final prediction.The paper places most existing studies in the first group.
- Related Work: Text-based pre-training learns representations for continuous word sequences, whereas this work models concept relatedness using knowledge-base graph structure.The paper distinguishes unstructured text-based and structured knowledge-based pre-training.
- Related Work: Previous knowledge-based pre-training is typically evaluated on knowledge-base completion or link prediction, while this work applies pre-trained commonsense models to question answering.The paper characterizes its approach as supporting external knowledge at the model’s top layer.
8 Conclusion
The paper presents ConceptNet-based pre-training for commonsense QA, combining concept-internal words with external graph context to measure relations. Across ARC, SemEval, and OpenBook QA, the approach complements document-based neural systems, improves them through combination, and retrieves useful commonsense evidence.
- The method pre-trains models to measure relations between concepts using ConceptNet as an external commonsense knowledge base.Each concept incorporates both its internal words and the external context supplied by neighboring concepts in the knowledge graph.
- The pre-trained models are applied to three commonsense question answering tasks: ARC, SemEval, and OpenBook QA.
- The commonsense-based models complement standard document-based neural approaches and yield further improvement when combined with them.
- Model analysis shows that the system can discover useful evidence from an external commonsense knowledge base.The paper identifies context-aware concept ranking and logical reasoning operations as issues for future work.