Source-linked AI summary
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts, Christopher D. Manning
TL;DR
Existing NLI corpora were too limited to support broad evaluation of data-intensive semantic models. This paper introduces the 570,152-pair SNLI corpus and finds that lexicalized and neural models perform well, with transfer yielding the best reported neural performance and approaching state of the art.
Problem
Existing NLI corpora were generally too small, partly algorithmically generated, and affected by coreference indeterminacies that limited evaluation of semantic representations.
Method
The paper constructs the 570,152-pair SNLI corpus from human-written, grounded sentence pairs and evaluates scalable rule-based, feature-based, and neural NLI models.
Results
Lexicalized and neural models perform well, with comparable feature-rich and LSTM performance; transfer makes the LSTM the best reported neural model and near overall state of the art.
Takeaways & Limitations
SNLI provides a large-scale, naturalistic resource for evaluating learning-centered approaches to natural language inference and semantic representation.
Takeaways & Limitations
Models still misanalyze lexical and phrasal relationships and struggle with inferences requiring world knowledge and context.
Abstract
from arXiv · showhide
Understanding entailment and contradiction is fundamental to understanding natural language, and inference about entailment and contradiction is a valuable testing ground for the development of semantic representations. However, machine learning research in this area has been dramatically limited by the lack of large-scale resources. To address this, we introduce the Stanford Natural Language Inference corpus, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning. At 570K pairs, it is two orders of magnitude larger than all other resources of its type. This increase in scale allows lexicalized classifiers to outperform some sophisticated existing entailment models, and it allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.
1 Introduction
Natural language inference is central to computational semantics, but existing corpora are too limited for evaluating modern, learning-centered approaches. The paper introduces SNLI, a large human-authored corpus, and uses it to compare rule-based, linear, and neural models.
- Natural language inference characterizes entailment and contradiction, relations central to natural language meaning and relevant to tasks from information retrieval to semantic parsing.
- Existing NLI corpora are generally too small for modern data-intensive, wide-coverage models, often algorithmically generated, and frequently ambiguous about event and entity interpretations.
- 570,152 sentence pairs make SNLI two orders of magnitude larger than other resources of its type, with human-written sentences and labels created in a grounded, naturalistic context.The corpus labels pairs for entailment, contradiction, and semantic independence.
- Two models achieve comparable performance: a feature-rich classifier and a neural model centered on a Long Short-Term Memory network.The LSTM model is also evaluated for transfer learning by adapting it to an existing NLI challenge.
2 A new corpus for NLI
SNLI addresses the limited size and label indeterminacy of prior NLI resources by grounding crowdsourced sentence pairs in shared scenarios and collecting balanced entailment, neutral, and contradiction hypotheses. Validation found broad annotation consensus, while the resulting sentences were generally fluent and structurally varied.
- Motivation: Prior RTE-based NLI datasets typically contained fewer than a thousand examples each, limiting their usefulness for learning distributed representations.
- Corpus design: SNLI grounded premise–hypothesis pairs in specific scenarios and constrained both sentences to describe each scenario from the same perspective, reducing coreference indeterminacy.The approach mitigated, but did not completely solve, coreference issues.
- Data collection: Mechanical Turk workers generated hypotheses for entailment, neutral, and contradiction, forcing the corpus to remain balanced across the three labels.About 2,500 workers contributed to data collection.
- Validation: Validation produced broad consensus: nearly all examples received majority labels, and gold-labeled examples were nearly evenly distributed across the three categories.Pairs were validated by presenting them to four additional annotators alongside the author’s label.
- Corpus characteristics: The collected sentences were largely fluent and correctly spelled, while their substantial structural differences and commonsense content leave room beyond superficial word alignment.The corpus included both full sentences and caption-style noun-phrase fragments.
3 Our data as a platform for evaluation
SNLI serves as a large-scale evaluation platform for comparing scalable NLI systems, lexicalized classifiers, and neural sentence-embedding models. Results show that lexicalized features and cross-bigrams are effective with sufficient data, while neural models benefit strongly from corpus scale but still share substantial semantic errors.
- Baseline evaluation: EOP baselines include edit-distance and classifier-based algorithms, with the latter tested both bare and with EOP’s full lexical-resource suite.EOP is an open-source platform for developing RTE systems and sharing lexical resources and evaluation sets.
- Baseline evaluation: The evaluation compares scalable NLI approaches across SNLI, SICK, and RTE-3, using two-class entailment accuracy and corpus-specific training.Contradiction and unknown are mapped to nonentailment for the two-class setting.
- Lexicalized models: The lexicalized classifier uses 6 feature types, and accuracy rises substantially with lexicalized features and again with sparse cross-bigrams.Cross-bigrams can help the classifier learn structures such as explicit negations and adjective modification.
- Neural models: SNLI’s scale enables neural sentence-embedding models, with LSTMs benefiting from stronger long-term-dependency modeling than plain RNNs.The evaluation feeds each sentence through an embedding model and uses the resulting vectors for NLI classification.
- Lexicalized models: With only 100 training examples, the cross-bigram classifier already outperforms its unlexicalized counterpart.High-weight features at this size tend to be high-precision entailments that improve entailment identification when activated.
- Error analysis: Despite large-scale training, models still make lexical, phrasal, and shortcut-based semantic errors, including failures involving coreference and Winograd-like inferences.These errors occur even for relationships common in training data and can arise from misleading distributional associations.
4 Transfer learning with SICK
The SNLI-trained LSTM transferred to SICK through simple parameter initialization and achieved competitive performance, including the best reported result for an unaugmented neural model. The same approach did not yield competitive performance on RTE-3 because its small training set could not support adaptation to the unfamiliar genre.
- SICK transfer: SNLI-trained LSTM representations transferred to SICK using a simple method that initialized a new model from the SNLI parameters and trained it only on SICK.Only the softmax parameters and embeddings for SICK-only words were newly initialized.
- SICK transfer: SICK-only training performed poorly, while the SNLI-trained model failed on SICK by labeling more neutral examples as contradictions than correctly.The authors suggest subtle differences in how the labeling task was presented as a possible cause.
- SICK transfer: Transferring SNLI representations to SICK achieved the best performance yet reported for an unaugmented neural network model, surpassed available EOP models, and approached the overall state of the art.The comparison is based on the results reported in Table 7.
- RTE-3 transfer: RTE-3’s 800-example training set did not allow adaptation to its unfamiliar text genre, so no training configuration achieved competitive performance.The authors suggest that better transfer learning for small data sets could enable improvements.
5 Conclusion
Natural language reasoning can be framed around entailment and contradiction in context, making NLI a promising testbed for semantic representation theories. Its broader potential has been constrained by the limited nature of existing NLI resources.
- 5 Conclusion: Entailment and contradiction in context underlie questions about meaningfulness in natural language.The passage presents natural languages as powerful vehicles for reasoning.
- 5 Conclusion: NLI is an ideal testing ground for theories of semantic representation.The passage links NLI’s focus on entailment and contradiction to the evaluation of semantic representations.
- 5 Conclusion: Training for NLI tasks can provide rich domain-general semantic representations.
- 5 Conclusion: Existing NLI resources have been too limited to fully realize this potential.