Source-linked AI summary
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Adina Williams, Nikita Nangia, Samuel R. Bowman
TL;DR
Existing NLU evaluations can conflate sentence understanding with other machine-learning challenges and lack broad coverage of English. MultiNLI addresses this with a ten-genre NLI corpus and explicit matched/mismatched evaluation, yielding greater difficulty and diversity than SNLI.
Problem
Applied NLU evaluations can mix sentence understanding with other hard machine-learning problems, while existing NLI resources have limited coverage of English.
Method
MultiNLI constructs 433k NLI pairs from ten written and spoken genres, with five training genres enabling matched and mismatched cross-genre evaluation.
Results
MultiNLI has greater linguistic difficulty and diversity than SNLI, reflected in dramatically lower baseline performance and comparable inter-annotator agreement.
Takeaways & Limitations
MultiNLI provides a benchmark for evaluating sentence understanding across diverse language and for cross-genre domain adaptation.
Takeaways & Limitations
Coreference ambiguity can make annotation agreement difficult because explicit rules may be unintelligible to untrained annotators.
Abstract
from arXiv · showhide
This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. In addition to being one of the largest corpora available for the task of NLI, at 433k examples, this corpus improves upon available resources in its coverage: it offers data from ten distinct genres of written and spoken English--making it possible to evaluate systems on nearly the full complexity of the language--and it offers an explicit setting for the evaluation of cross-genre domain adaptation.
1 Introduction
The paper motivates NLI as a focused benchmark for evaluating sentence-level meaning representations and introduces MultiNLI to address SNLI’s narrow coverage. MultiNLI provides 433k sentence pairs across ten written and spoken English genres, enabling broader NLU evaluation and explicit cross-domain transfer assessment.
- Motivation: NLI evaluates sentence-meaning representations by classifying sentence pairs as entailment, neutral, or contradiction while requiring broad semantic understanding.The task involves phenomena including lexical entailment, quantification, coreference, tense, belief, modality, and lexical or syntactic ambiguity.
- Limitations of SNLI: SNLI’s image-caption genre restricts examples to concrete visual scenes, making temporal reasoning, belief, and modality too rare to meaningfully challenge models.Its hypotheses are consequently short and simple, limiting the corpus’s demands on sentence understanding.
- MultiNLI: MultiNLI contains 433k sentence pairs collected from ten genres of written and spoken English spanning varied styles, formality levels, and topics.The genres are intended to approximate the diversity of modern standard American English, with all genres represented in the test set and presumably the development set according to the supplied passage’s continuation.
- Research goals: MultiNLI is designed to evaluate sentence representations both within the training domain and in unfamiliar domains, supporting explicit research on domain adaptation and cross-domain transfer learning.This addresses the limited success of general-purpose representation learning approaches in NLU.
2 The Corpus
MultiNLI builds sentence pairs from ten diverse written and spoken English genres, using crowdworkers to create entailment, contradiction, and neutral hypotheses. Validation assigns majority-vote gold labels, while corpus statistics indicate longer, more complete premises and limited reliance on trivial hypothesis edits.
- Premise Text Sources: Ten freely available text sources were selected to maximize diversity and roughly represent the full range of American English.Nine sources came from OANC, while FICTION combined contemporary works spanning mystery, humor, western, science fiction, and fantasy.
- Hypothesis Collection: Crowdworkers composed three hypotheses per premise: ENTAILMENT, CONTRADICTION, and NEUTRAL.Prompts were tailored to premise genres, with five unique prompts covering written non-fiction, spoken genres, and other source types.
- Validation: Each validated pair received four additional labels, producing five labels per example and a majority-vote gold label.Examples without a three-vote consensus were retained but marked ‘-’ and excluded from standard evaluations.
- Statistics: 91% of MultiNLI premises were parsed as complete sentences, compared with 74% in SNLI, while MultiNLI premises averaged 22.3 words versus 14.1 in SNLI.MultiNLI premises had a maximum length of 401 words, and hypotheses a maximum of 70 words.
- Statistics: 0.9% of MultiNLI hypotheses differed from premises by deletion, versus 2.5% in SNLI, while single-word edits affected 1.2% versus 1.6%.High token overlap (>37%) occurred in 30% of MultiNLI pairs and 29% of SNLI pairs.
3 Baselines
The study benchmarks three neural NLI models—CBOW, BiLSTM, and ESIM—under SNLI, MultiNLI, and mixed training settings. Genre-specific CBOW experiments assess cross-genre transfer, revealing strong performance on similar genres but weaker results for SLATE.
- Model baselines: Three baselines are evaluated: continuous bag-of-words, bidirectional LSTM, and Enhanced Sequential Inference Model.CBOW sums word embeddings, while BiLSTM averages bidirectional recurrent states over words.
- Model baselines: CBOW and BiLSTM classify sentence pairs using concatenated premise and hypothesis representations, their difference, and their element-wise product.The combined representation is passed through a tanh layer and three-way softmax classifier.
- Training settings: Models are trained on SNLI, MultiNLI, or a mixture containing all MultiNLI data plus 15% of SNLI sampled each epoch.This mixture exposes each genre with roughly equal frequency during training.
- Cross-genre transfer: Separate genre-specific CBOW models test transfer across genres, with the best model for each genre trained on that genre; SNLI-only training performs worse on every genre.The genre-specific experiments use dropout 0.2, while SNLI training uses a single random 15% sample.
- Cross-genre transfer: 63% accuracy on FACE-TO-FACE is achieved by the TELEPHONE-trained model, nearly one point above its TELEPHONE accuracy.By contrast, average matched-section performance on SLATE is only 57.5% when averaging runs trained on SNLI and all MultiNLI genres.
4 Discussion and Analysis
The analysis finds that MultiNLI remains difficult despite relatively high inter-annotator agreement, while annotation ambiguity and linguistic phenomena shape both labels and model performance. Baselines show limited specialization across tagged subsets, with particular weaknesses on comparison, discourse, and longer sentences.
- Annotation ambiguity: Coreference assumptions can change an NLI pair’s label between CONTRADICTION and NEUTRAL, creating serious challenges for inter-annotator agreement.The paper notes that explicit coreference rules may be difficult for untrained or non-expert annotators to understand.
- Difficulty and transfer: about 15%: all three baseline models perform better on SNLI than MultiNLI when trained on their respective datasets, reflecting MultiNLI’s greater difficulty.The analysis attributes this difficulty to MultiNLI’s greater diversity of linguistic phenomena and longer average sentence length.
- Difficulty and transfer: Including downsampled SNLI in MultiNLI training significantly improves SNLI performance but produces no significant change on MultiNLI test sets.This suggests that adding SNLI does not substantially improve performance on MultiNLI and leaves considerable headroom for future work.
- Corpus phenomena: Negated sentences are slightly more likely to receive CONTRADICTION labels, whereas sentences exceeding 20 words are slightly more likely to receive ENTAILMENT labels.The distributions of labels in tagged subsets otherwise roughly mirror the corpus-wide balanced distribution.
- Model behavior: 3-4 points: baseline accuracy drops most on sentences containing comparatives or superlatives, while discourse markers reduce performance by roughly 2 to 3 points.The attention-based ESIM model performs better than the other baselines on sentences longer than 20 words, and baselines show slight gains on negation.
5 Conclusion · http://arxiv.org/ps/1704.05426v4
MultiNLI is presented as a more linguistically diverse and difficult NLI resource than SNLI, covering ten genres and supporting evaluation of cross-genre adaptation. Its lower baseline performance reflects this challenge, while later work shows models trained on SNLI and MultiNLI can substantially improve transfer-learning benchmarks.
- 5 Conclusion: MultiNLI is introduced as a dataset for evaluating whether sentence-understanding models capture the full meanings of natural-language sentences.The paper frames NLI as a way to assess sentence understanding and presents MultiNLI as a new resource.
- 5 Conclusion: MultiNLI offers greater linguistic difficulty and diversity than existing NLI datasets such as SNLI.The conclusion characterizes existing datasets as having limited headroom and coverage of English meaning diversity.
- 5 Conclusion: Ten genres provide MultiNLI with representative text and speech coverage, compared with SNLI’s simple image captions.This empirical coverage is identified as a principal improvement over SNLI.
- 5 Conclusion: MultiNLI contains a higher percentage of sentences tagged with one or more of thirteen difficult linguistic phenomena.The tag set is used to characterize the corpus’s increased difficulty.
- 5 Conclusion: Baseline model performance is dramatically lower on MultiNLI than on SNLI, reflecting the corpus’s greater diversity.The conclusion directs readers to Table 5 for this comparison.
- 5 Conclusion: Subsequent work showed that NLI can serve as an effective source task for pre-training and transfer learning in sentence-to-vector models.This work appeared after MultiNLI’s draft release in the first half of 2017.
- 5 Conclusion: Models trained on SNLI and MultiNLI substantially outperformed all prior models on established transfer-learning benchmarks.The result is reported across a suite of established benchmarks.
- 5 Conclusion: The authors hope MultiNLI will remain a resource for developing and evaluating sentence-understanding methods.The stated intended use extends many years into the future.