Source-linked AI summary
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
Max Glockner, Vered Shwartz, Yoav Goldberg
TL;DR
NLI systems achieve high accuracy on SNLI, but it remains unclear whether they learn lexical knowledge or exploit simpler benchmark properties. The paper builds a minimally changed, vocabulary-controlled test set of lexical inferences and finds substantially worse performance across systems, indicating limited generalization. The test set is intended to evaluate lexical inference ability, with use constrained by its contradiction bias.
Problem
It is unclear whether neural NLI methods learn lexical knowledge, whether large training sets teach it implicitly, or whether existing datasets rarely require it.
Method
The paper automatically generates and manually verifies NLI pairs by replacing words in SNLI premises with synonyms, hypernyms, co-hyponyms, and antonyms already known to training data and embeddings.
Results
Performance is substantially worse across systems on the simpler new test set, despite examples differing by at most one word from training sentences.
Takeaways & Limitations
The test set can assess lexical inference abilities and distinguish otherwise similarly performing NLI systems.
Takeaways & Limitations
Because the new test set is biased toward contradiction, it should supplement rather than replace the original NLI test set and should not be used for training.
Abstract
from arXiv · showhide
We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set. Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.
1 Introduction
The paper asks whether high-performing neural NLI systems truly learn lexical knowledge and shows that they fail on simple lexical and world-knowledge inferences despite familiar vocabulary and minimal sentence changes.
- NLI systems classify whether a hypothesis is entailed, contradicted, or neutral with respect to a premise.
- The paper constructs a test set covering lexical relations such as hypernymy and co-hyponymy, using examples differing by one word from training sentences.All words occur in the training set and pre-trained embeddings, isolating lexical knowledge from vocabulary novelty.
- Performance is substantially worse across systems on the new test set, showing that the SNLI test set alone is insufficient to measure language understanding.
- Similar training examples mostly determine whether systems correctly predict new test examples, making this an inefficient way to learn lexical inferences.The authors propose the test set for evaluating methods that learn lexical inferences more effectively.
2 Background
SNLI and related NLI datasets supported strong neural-model results, while the limited benefit of WordNet integration left unclear whether models or benchmarks explain the apparent lack of lexical knowledge.
- NLI Datasets: SNLI contains 570k manually labeled premise–hypothesis pairs covering entailment, contradiction, and neutrality.Premises are image captions, and crowd workers generated hypotheses describing the same event.
- NLI Datasets: MultiNLI broadened evaluation across genres, while SciTail is smaller, science-based, and labeled only for entailment or neutrality.
- Neural Approaches for NLI: Neural NLI models achieved high accuracy on SNLI while relying primarily on pre-trained word embeddings rather than explicit lexical resources.
- External Lexical Knowledge: KIM’s WordNet integration added only 0.6 points on SNLI, raising whether the model missed lexical knowledge or SNLI rarely required it.
3 Data Collection
The test set is generated by replacing words in SNLI premises with semantically related alternatives, constraining vocabulary and filtering examples before manual verification.
- The authors automatically generate sentence pairs and then manually verify them using crowd-sourced workers.
- Each hypothesis replaces one premise word with a synonym, hypernym, mutually exclusive co-hyponym, or antonym.Some replacements use multi-word noun phrases, with determiners and prepositions adapted when needed.
- Replacement words are already present in SNLI training data and pre-trained embeddings to test generalization over known vocabulary.Their training-set frequency ranges from one occurrence to 248,051, with mean 3,663.1 and median 149.5.
- The authors sample SNLI training premises containing listed words and filter replacements that produce implausible or ungrammatical sentences.Wikipedia bigrams are used during filtering.
- Workers were required to meet a 99% approval rate across at least 1,000 prior tasks, and each annotation was assigned to three workers.
1. Do the sentences describe the same event?
Workers label generated pairs through simple event-consistency and information-addition judgments, while agreement is substantial and estimated human performance is high.
- Annotators first judge whether the premise and hypothesis describe the same event.
- If the event is not the same, the pair receives a contradiction label; otherwise, added information distinguishes neutral from entailment.
- Pairs judged ungrammatical by any worker are discarded, and majority vote determines the gold label.
- Fleiss’ Kappa κ = 0.61 indicates substantial annotation agreement, while estimated human performance is 94.1%.The new test set is reported as substantially easier for humans than SNLI.
4 Evaluation
The evaluation compares neural NLI models with and without WordNet-based knowledge across SNLI and augmented training data. On the new test set, KIM substantially outperforms other neural models, while MultiNLI training partly reduces the performance drop.
- Models: The evaluation includes three neural models without external knowledge, using sentence encoding and/or attention, plus WordNet-based and KIM systems.RESIDUAL-STACKED-ENCODER, ESIM, and DECOMPOSABLE ATTENTION use pretrained GloVe embeddings; KIM extends ESIM with WordNet relations.
- Evaluation scope: The new test set cannot serve as training data or the main NLI evaluation set because it is biased toward contradiction.The authors recommend using it alongside the original test set to assess lexical-inference handling.
- Training data: Models are trained on SNLI alone or with MultiNLI or SciTail added, testing whether broader training data supplies lexical knowledge.The datasets are intended to assess whether SNLI lacks examples needed to learn the required lexical knowledge.
- Results: 11 to 33 points: accuracy drops by this range from the original SNLI test set to the new test set despite the latter being considerably simpler.Adding MultiNLI somewhat mitigates the drop, whereas adding SciTail does not similarly improve performance.
- Results: KIM substantially outperforms the other neural models on the new test set, indicating the importance of lexical knowledge for this evaluation.The authors also identify limited WordNet coverage and applying lexical inferences within context as remaining improvement areas.
5 Analysis
Analysis by category, word similarity, and training frequency shows that neural models often rely on training exposure and lexical overlap rather than robust lexical inference. Fine-tuned embeddings can benefit from replacement-pair examples, but explicit coverage remains inefficient.
- 5.1 Accuracy by Category: Neural models perform well on frequent categories such as colors but poorly on rare categories such as planets.The analysis groups accuracy by replacement-word category and finds performance aligned with category frequency in SNLI.
- 5.1 Accuracy by Category: On entailment examples, neural models outperform the WordNet baseline partly through high premise–hypothesis lexical overlap rather than recognizing synonymy.The discussion therefore focuses on contradiction examples.
- 5.2 Accuracy by Word Similarity: Accuracies for ordinals, nationalities, and countries are especially low, possibly because contradicting words are close in embedding space.The fixed-embedding Decomposable Attention model seems to suffer the most.
- 5.2 Accuracy by Word Similarity: Models err more on contradicting pairs whose pretrained vectors have higher cosine similarity.This trend emerges when prediction accuracy is grouped by cosine similarity between the contradicting words.
- 5.3 Accuracy by Frequency in Training: ESIM accuracy on test pairs increases with the frequency of their replacement words in contradiction examples in training data.This suggests that fine-tuning can learn lexical knowledge when sufficient pair-specific examples are available.
- 5.3 Accuracy by Frequency in Training: Explicit training examples can teach lexical knowledge, but relying on them is a very inefficient way to obtain simple lexical knowledge.The analysis links this inefficiency to the limited amount of training data available.
6 Conclusion
The paper introduces a simpler-than-SNLI NLI test set for simple lexical inferences, finding that state-of-the-art systems perform poorly despite no new vocabulary.
- The test set evaluates whether NLI systems can make inferences requiring simple lexical knowledge.
- Despite being simpler than SNLI and introducing no new vocabulary, the test set exposes limited generalization ability in state-of-the-art systems.
- The benchmark can assess lexical inference abilities and distinguish systems with otherwise similar performance.