Source-linked AI summary
Improving Natural Language Inference Using External Knowledge in the Science Questions Domain
Xiaoyan Wang, Pavan Kapanipathi, Ryan Musa, Mo Yu, Kartik Talamadupula, Ibrahim Abdelaziz, Maria Chang, Achille Fokoue, Bassem Makni, Nicholas Mattei, Michael Witbrock
TL;DR
NLI systems have largely relied on text despite the availability of structured knowledge bases. This paper introduces knowledge-graph augmentation for science-domain entailment and evaluates text-only, graph-only, and combined models, reaching 85.2% accuracy on SciTail.
Problem
NLI research has mostly used textual information, while the use of structured external knowledge for distinguishing entailment, contradiction, and neutrality remains underexplored.
Method
ConSeqNet retrieves knowledge relevant to premises and hypotheses from external sources and compares graph-only, text-only, and text-and-graph entailment models.
Results
85.2% accuracy is achieved on the SciTail entailment dataset, with graph information improving accuracy when text features are also present.
Takeaways & Limitations
External knowledge graphs such as ConceptNet show promise for textual entailment in the science questions domain.
Takeaways & Limitations
The approach remains constrained by noisy external knowledge sources, motivating methods that extract task-relevant knowledge.
Abstract
from arXiv · showhide
Natural Language Inference (NLI) is fundamental to many Natural Language Processing (NLP) applications including semantic search and question answering. The NLI problem has gained significant attention thanks to the release of large scale, challenging datasets. Present approaches to the problem largely focus on learning-based methods that use only textual information in order to classify whether a given premise entails, contradicts, or is neutral with respect to a given hypothesis. Surprisingly, the use of methods based on structured knowledge -- a central topic in artificial intelligence -- has not received much attention vis-a-vis the NLI problem. While there are many open knowledge bases that contain various types of reasoning information, their use for NLI has not been well explored. To address this, we present a combination of techniques that harness knowledge graphs to improve performance on the NLI problem in the science questions domain. We present the results of applying our techniques on text, graph, and text-to-graph based models, and discuss implications for the use of external knowledge in solving the NLI problem. Our model achieves the new state-of-the-art performance on the NLI problem over the SciTail science questions dataset.
1 Introduction
NLI classifies the relationship between premises and hypotheses, and this paper augments text-based inference with external knowledge graphs for science-domain questions. ConSeqNet combines text and graph information, achieving 85.2% accuracy on SciTail.
- NLI determines whether a hypothesis follows from a premise by classifying their relationship as entailment, contradiction, or neutral.
- The framework uses external knowledge from WordNet, ConceptNet, and DBpedia to assess whether a premise entails its hypothesis.
- ConceptNet subgraphs connect concepts from the premise and hypothesis through additional intermediate concepts, enriching the text for learning-based NLI models.
- ConSeqNet compares graph-only, text-only, and text-and-graph approaches across ConceptNet, WordNet, and DBpedia.
- 85.2% accuracy is achieved on the SciTail entailment dataset, which is derived from science-domain question-answering data.
2 Background and Related Work
Prior NLI research primarily uses neural models over text, while external knowledge graphs represent structured concepts and relations. This paper positions its approach as enriching premise and hypothesis text with such graphs for textual entailment.
- 2.1 Natural Language Inference: Large-scale NLI datasets, including SNLI, MultiNLI, and SciTail, support neural classifiers and address settings such as domain adaptation and standardized-test question answering.
- 2.1 Natural Language Inference: Common NLI systems use encoder-attention-classifier architectures, often encoding premises and hypotheses with recurrent neural networks.
- 2.2 External Knowledge (Knowledge Graphs): Knowledge graphs represent concepts as nodes connected by labeled relationships, encoding facts as structured triples.
- 2.2 External Knowledge (Knowledge Graphs): Relevant knowledge availability depends on the task and domain, with DBpedia emphasizing entity facts and ConceptNet emphasizing crowdsourced common-sense knowledge.
- 2.3 Knowledge Graphs and NLI: Earlier graph-based entailment methods generally used syntactic structures rather than external knowledge sources to enrich premise and hypothesis text.
3 Approach
ConSeqNet combines text-based and graph-based models to determine entailment from premise–hypothesis pairs. It maps text to knowledge-graph subgraphs and uses specialized matching architectures for sequential and general graph structures.
- Overall architecture: ConSeqNet has separate text-based and graph-based components for determining textual entailment.The text model receives premise and hypothesis text, while the graph model receives knowledge derived from both.
- Text-based model: The text model encodes premise and hypothesis representations, computes word-by-word attention, matches aligned features, and pools the resulting states for classification.Feature vectors combine contextual and aligned representations before sequential matching and fixed-size pooling.
- Graph construction: Premise and hypothesis text are transformed into knowledge-graph subgraphs whose concepts and labeled edges become inputs to the graph entailment model.The subgraphs are generated separately for the premise and hypothesis from concepts mapped to an external knowledge graph.
- Graph construction: Three graph constructions are used: Concepts Only, One-Hop, and Two-Hop, with the latter adding connecting neighbors while limiting new concepts.One-Hop expands all mapped concepts by their neighbors, whereas Two-Hop adds paths between concepts already present.
- Graph-based models: Gmatch-LSTM orders Concepts Only nodes as sequences, while GconAttn handles general graphs whose concepts lack an obvious sequential order.GconAttn uses two-way attention and pooling rather than relying on a sequence arrangement.
- Graph-based models: Graph concepts are initialized with embeddings trained using knowledge-graph embedding techniques such as TransH and CompleX.This supplies concept representations derived from the corresponding external knowledge graph.
4 Experiments and Results
Experiments evaluate ConSeqNet on SciTail using different knowledge sources, graph-construction strategies, and text–graph model combinations. The final jointly trained model reaches 89.6% dev accuracy and 85.2% test accuracy, with experiments indicating that external knowledge contributes to performance.
- Datasets: SciTail contains 27,026 premise–hypothesis pairs with binary entailment or neutral labels derived from science-domain multiple-choice question answering.The hypothesis combines each question with its correct answer, while premises come from the ARC corpus.
- Experimental setup: ConSeqNet experiments vary text models, graph models, knowledge sources, graph-construction techniques, and graph embeddings.Evaluated knowledge sources include DBpedia, WordNet, and ConceptNet; graph construction includes Concepts Only, One-Hop, and Two-Hop methods.
- Selecting external knowledge: ConceptNet maps 9 concepts per sentence on average, compared with 6 for DBpedia, and was selected for subsequent graph-based experiments.The authors also judged ConceptNet’s common-sense relationships more suitable for SciTail than DBpedia’s factual relationships.
- Graph generation experiments: An Oracle reached at least 91.6% dev accuracy versus 88.5% for text-only, and One-Hop corrected almost 40% of text-only errors, indicating value from external knowledge.Graph-only models were weaker, while graph-plus-text and text-only models had comparable accuracy.
5 Conclusion & Future Work
ConSeqNet uses ConceptNet as an external knowledge source for NLI and achieves 85.2% accuracy on SciTail. The authors identify noisy knowledge sources and knowledge representation as areas for future work.
- 85.2% accuracy is reported for ConSeqNet on the SciTail dataset.
- The system analyzes how different external knowledge sources affect NLI and finds promise in knowledge graphs such as ConceptNet for textual entailment.
- Existing external knowledge sources are extremely noisy, motivating techniques that extract knowledge relevant to specific tasks such as NLI.
- Future work includes selecting multiple relevant knowledge sources based on the dataset and context.
- Future work also targets improved representations of premise and hypothesis subgraphs and systematic use of their concept relations.