Source-linked AI summary
Neural Natural Language Inference Models Enhanced with External Knowledge
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Diana Inkpen, Si Wei
TL;DR
The paper addresses whether neural NLI models can learn all required inference knowledge from annotated data alone. It enriches neural models with external knowledge in key inference components and reports state-of-the-art performance on SNLI and MultiNLI, with stronger benefits when training data are restricted.
Problem
Although large annotated NLI datasets enable strong neural models, the paper asks whether they provide all inference knowledge needed for NLI.
Method
The paper enriches neural NLI models with external knowledge in co-attention, local inference collection, and inference composition.
Results
The proposed KIM model achieves 88.6% test-set accuracy on SNLI, improving over ESIM’s 88.0% and reaching the best reported single-model performance.
Takeaways & Limitations
External knowledge improves neural NLI performance on SNLI and MultiNLI, with larger advantages when training data are restricted.
Takeaways & Limitations
Adding 15 WordNet relation features beyond the five semantic relations did not further improve development-set performance.
Abstract
from arXiv · showhide
Modeling natural language inference is a very challenging task. With the availability of large annotated data, it has recently become feasible to train complex models such as neural-network-based inference models, which have shown to achieve the state-of-the-art performance. Although there exist relatively large annotated data, can machines learn all knowledge needed to perform natural language inference (NLI) from these data? If not, how can neural-network-based NLI models benefit from external knowledge and how to build NLI models to leverage it? In this paper, we enrich the state-of-the-art neural natural language inference models with external knowledge. We demonstrate that the proposed models improve neural NLI models to achieve the state-of-the-art performance on the SNLI and MultiNLI datasets.
1 Introduction
NLI asks models to determine whether a hypothesis is entailed, contradicted, or neutral relative to a premise, but learning the needed inference knowledge remains difficult. This paper enriches neural NLI models with external knowledge and reports improved performance on SNLI and MultiNLI.
- NLI determines the inferential relationship between a premise and hypothesis, such as entailment, contradiction, or neutral.The paper frames informal language inference as a challenging problem toward natural language understanding.
- Large annotated datasets made it feasible to train complex neural models that achieve state-of-the-art NLI performance.SNLI and MultiNLI are cited as examples of datasets enabling parameter-rich neural modeling.
- The paper questions whether NLI training data contain all required inference knowledge and explores using accumulated human knowledge instead.The motivation is that machines may be unable to learn all complicated inference knowledge by themselves.
- Missing relationships such as wheat–corn knowledge can prevent a model from recognizing that one sentence contradicts another.The example concerns a lady in a wheat field versus a person in a corn field when training data lack the relevant relationship.
- External knowledge is incorporated into co-attention, local inference collection, and inference composition components of neural NLI models.The proposed models improve state-of-the-art NLI models on SNLI and MultiNLI, with larger advantages under restricted training data.
2 Related Work
NLI research progressed from early work on relatively small datasets to complex neural models enabled by larger annotations. This paper combines neural modeling with external inference knowledge and evaluates that combination on SNLI and MultiNLI.
- Early NLI and textual-entailment research largely used relatively small datasets.
- Larger annotated datasets enabled more complex neural NLI models, including sentence-encoding and inter-sentence-attention approaches.
- Inter-sentence attention models compare premise and hypothesis representations through attention, with ESIM serving as a strong baseline here.ESIM is identified as one of the best models at the time and as a baseline for this paper.
- Unlike earlier conventional NLI work using external knowledge on smaller datasets, this paper merges neural modeling with extra inference knowledge.
- External knowledge has also been applied to neural methods for word embeddings, machine translation, language modeling, and dialogue systems.
3 Neural-Network-Based NLI Models with External Knowledge
The paper incorporates lexical external inference knowledge into major neural NLI components, spanning co-attention, local inference collection, and sentence-level inference composition. The resulting framework uses this knowledge to enrich alignment and local inference representations before composing the overall relationship.
- Framework overview: External lexical inference knowledge is represented as relation features between premise and hypothesis words and incorporated into three NLI components.The components are co-attention, local inference collection, and inference composition.
- Encoding Premise and Hypothesis: BiLSTM encoders transform embedded premise and hypothesis words into context-dependent hidden states.The premise and hypothesis are embedded with E and encoded using BiLSTMs.
- Knowledge-Enriched Co-Attention: Knowledge-enriched co-attention uses relation features to compute soft alignment and identify relevant hypothesis or premise context for each word.The relation features are added through a function F, with the paper using F(r_ij) = λ1(r_ij), where the indicator is nonzero when r_ij is not a zero vector.
- Local Inference Collection with External Knowledge: Local inference collection compares aligned contextual representations and combines soft-aligned external relation features to obtain word-level inference information.The model uses difference and element-wise product matching, while G reduces dimensionality through a nonlinear mapping.
- Knowledge-Enhanced Inference Composition: A composition layer reads the collected local inference vectors and learns which local relationships matter for the overall sentence-level inference decision.BiLSTMs serve a different role here than in input encoding: they process local inference vectors and distinguish crucial ones for the final relationship.
- Knowledge-Enhanced Inference Composition: The complete model pools composed representations, classifies them with an MLP and softmax layer, and trains end-to-end with cross-entropy loss.Mean, max, and weighted pooling vectors are concatenated before classification.
4 Experiment Set-Up
The experiment represents external lexical knowledge through WordNet relations and relation embeddings, then evaluates knowledge-enhanced NLI models on SNLI and MultiNLI.
- Lexical Semantic Relations: WordNet supplies lexical relation features for synonymy, antonymy, hypernymy, hyponymy, and co-hyponymy.These relations are represented as a five-dimensional vector and incorporated as external knowledge.
- Lexical Semantic Relations: Hypernymy and hyponymy are intended to support entailment modeling, while antonymy and co-hyponyms support contradiction modeling.The feature contributions are learned end-to-end during model training.
- Lexical Semantic Relations: Additional WordNet relations increase the feature set to 15, but they do not further improve development-set performance.The added relations include instance, entailment, meronymy, and related features.
- Relation Embeddings: TransE relation embeddings capture relationships between WordNet word pairs using either pretrained WN18 embeddings or embeddings trained for WordNet.The pretrained approach uses 18 relation embeddings from WN18.
- Experiments: The study evaluates entailment, contradiction, and neutral classification on SNLI and MultiNLI using classification accuracy and established data splits.Models are selected on development sets and tested on the corresponding test sets.
5 Experimental Results
KIM improves ESIM on SNLI and MultiNLI, with gains especially pronounced when training data are restricted. Ablations show local inference collection is the most consistently useful component for incorporating external knowledge.
- Overall Performance: 88.6% accuracy makes KIM the best single-model result reported on SNLI, compared with ESIM's 88.0%.The difference is statistically significant under a one-tailed paired t-test at the 99% significance level.
- Overall Performance: 77.2% in-domain and 76.4% cross-domain accuracy improve on ESIM's 76.8% and 75.8% on MultiNLI.The gains are significant and consistent with the SNLI results.
- Ablation Results: With 0.8% of training data, local inference collection raises accuracy from ESIM's 62.4% to 70.3%, while co-attention reaches 66.6% and composition 63.4%.The local inference component provides the largest individual gain among the three components.
- Ablation Results: 72.6% accuracy is achieved when external knowledge is added to co-attention, local inference collection, and inference composition together, a +10.2% absolute gain.With more training data, only local inference collection continues to provide a significant gain, whereas co-attention and composition help mainly under restricted data.
- External Knowledge Analysis: More external knowledge produces higher accuracies, with a particularly large gain when more than half is used under 0.8% training data.The analysis uses external knowledge in local inference collection, which works across training-data scales.
- Generalization Analysis: On the Glockner test set, KIM reaches 83.5% accuracy with a 5.1% drop, while baseline models drop 22.3%–32.8% from SNLI performance.KIM outperforms ESIM in 13 of 14 replacement-word categories and is worse only on synonyms.
6 Conclusions
KIM enriches neural NLI models with external knowledge in co-attention, local inference collection, and inference composition, achieving state-of-the-art accuracies.
- KIM integrates external knowledge into co-attention, local inference collection, and inference composition.
- KIM achieves state-of-the-art accuracies on natural language inference.
- The authors provide detailed analyses of KIM and its results.