Source-linked AI summary
Hypothesis Only Baselines in Natural Language Inference
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, Benjamin Van Durme
TL;DR
NLI datasets are intended to test whether a hypothesis follows from a context, yet models may exploit hypothesis-side regularities instead. This paper evaluates a hypothesis-only baseline across ten datasets and finds that it significantly beats the majority baseline in many of them, suggesting that statistical irregularities can make NLI appear easier than intended. The authors argue for testing NLI systems against hypothesis-only counterparts while noting that some irregularities remain unexplored.
Problem
NLI evaluations assume inference depends on both context and hypothesis, but dataset biases may permit strong performance without using the context.
Method
The paper trains a hypothesis-only neural baseline and evaluates it across ten NLI-style datasets, analyzing statistical giveaways and construction effects.
Results
In six of ten datasets, the hypothesis-only model significantly outperforms the majority-class baseline, with exploitable signals including word choice and grammaticality.
Takeaways & Limitations
NLI models should be compared with hypothesis-only baselines, and future datasets should reduce exploitable irregularities.
Takeaways & Limitations
The study leaves statistical irregularities such as sentence-length correlations and construction methods that minimize giveaway words for future work.
Abstract
from arXiv · showhide
We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on ten distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context.
1 Introduction
NLI predicts an entailment relation between a context and hypothesis, but hypothesis-only models test whether datasets contain exploitable signals that bypass the context. The paper evaluates this baseline across ten datasets and finds it often exceeds majority-class performance.
- NLI predicts whether a hypothesis follows from a premise using labels such as entailment, neutral, and contradiction.
- A hypothesis-only model ignores the premise and classifies the hypothesis sentence using learned or prior information.
- The paper asks whether NLI datasets contain statistical irregularities that let hypothesis-only models outperform their dataset-specific prior.
- Experiments evaluate a hypothesis-only baseline across ten NLI-style datasets and examine hypothesis giveaways, construction methods, and majority baselines.
- The study builds on earlier concerns about dataset hypotheses and prior evaluations of NLI difficulty using simple models.
2 Motivation
The paper is motivated by evidence that dataset biases can support strong performance without understanding text meaning. It extends this concern to NLI, where context-free success would undermine the intended premise-hypothesis reasoning challenge.
- Prior work found that models could solve Story Cloze without the narrative using writing-style features rather than ending content.
- Sentence length, sentiment, and negation features were reported as sufficient for high accuracy on Story Cloze.
- NLI is treated as central to natural language understanding because it requires many distinct types of reasoning.
- The motivation is reinforced by earlier independent observations concerning SNLI and by prior RTE evaluations with simple bag-of-words models.
3 Methodology
The baseline adapts InferSent to encode only hypotheses and predict NLI labels, using standard neural sentence representations and supervised training procedures.
- The model modifies InferSent so a BiLSTM encodes only the hypothesis before an MLP predicts the NLI tag.
- Hypothesis representations are formed by max-pooling over BiLSTM hidden states and passed to the classifier.
- Inputs are tokenized and mapped to 300-dimensional Common Crawl GloVe vectors, with an out-of-vocabulary vector for unknown words.
- Training uses SGD with an initial learning rate of 0.1, decay rate 0.99, and at most 20 epochs with optional early stopping.
4 Datasets
The study covers ten NLI datasets grouped by construction protocol, including elicited, judged, and automatically recast data. These datasets vary in domains, labels, premise structure, and how hypotheses are generated.
- Dataset overview: The investigation collects ten datasets and categorizes them by construction method, with Table 1 reporting size, protocol, classes, and example hypotheses.
- Elicited datasets: Elicited datasets ask humans to generate hypotheses and labels from contexts; SNLI uses Flickr image-caption premises and three entailment labels.
- Elicited datasets: SNLI contains reported gender, race, and ethnic stereotypes, while Multi-NLI expands SNLI with premises from multiple genres.
- Judged datasets: Judged datasets automatically pair hypotheses and premises before humans assign labels, including SICK, Add-one RTE, and SciTail.
- Judged datasets: SciTail pairs science-derived hypotheses with domain-specific premises and allows one hypothesis to appear with multiple premises.
- Other datasets: MPE uses four image captions as premises, while JOCI uses an ordinal likelihood scale mapped into three NLI classes.
- Automatically recast: Recast datasets are automatically generated from other NLP tasks with minimal human intervention, including SPR, DPR, and FN+.
5 Results
Across ten NLI-style datasets, the hypothesis-only model is evaluated against majority baselines to test whether performance can arise without premise access. It significantly outperforms the majority baseline on six datasets, with larger gains in some elicited datasets and reduced margins in Multi-NLI compared with SNLI.
- Evaluation setup: The study evaluates whether a hypothesis-only model outperforms the majority baseline across ten NLI-style datasets.Except for JOCI, experiments use each dataset’s standard train, development, and test splits.
- Results: Six of ten datasets show significant hypothesis-only improvements over the majority baseline, including performance above the best reported result on recast SPR.
- Results: The largest relative gains occur on human-elicited datasets, where the hypothesis-only model more than doubles the majority baseline.
- Results: Judged and recast datasets have smaller margins between hypothesis-only and majority models than elicited datasets, despite noticeably higher baseline performances than SNLI and Multi-NLI.
- Results: The drop between SNLI and Multi-NLI suggests that multiple genres may reduce biases, but the hypothesis-only model still drastically outperforms the majority baseline.
- Results: No general trend links dataset construction methodology to hypothesis-only gains, while the results indicate exploitable signals unrelated to NLI contexts.
6 Statistical Irregularities
The analysis examines dataset-level statistical irregularities that let hypothesis-only models exploit label-correlated words, grammaticality, or lexical semantics instead of premise–hypothesis relations. These irregularities vary across datasets and can produce substantial gains over majority baselines.
- Analyses of irregularities: The study investigates specific words, grammaticality, and lexical semantics as possible sources of hypothesis-only performance.These analyses target characteristics that may explain why the hypothesis-only model often outperforms the majority baseline.
- Give-away words: At X = 1.0, Figure 2 counts sentences containing a word that deterministically predicts a label; lower thresholds show broader potential coverage.The figure plots the number of sentences per label containing a word w with p(l|w) >= x.
- Give-away words: DPR shows fewer sentences with highly label-specific words than SNLI, while SNLI declines gradually from the three-class uncertainty point, 0.33.For DPR, the sharp drop away from p(l|w) = 0.5 indicates a weaker effect; SNLI uses three-way classification.
- Give-away words: SNLI contains strongly label-deviating words, whereas MPE’s word scores are closer to uniform; higher frequency increases their potential effect on accuracy.Figure 3 reports highly correlated words, thresholded to the top 10 and ranked by frequency.
- Give-away words: In SNLI, contradictory hypotheses include frequent sleeping-related and universal-negation terms, which may reflect crowd-sourcing strategies and insufficient control of elicitation.The authors report that Flickr-derived contexts naturally involve activities and that workers were not guided or incentivized to produce less obvious contradictions.
- Grammaticality: FN+ yields hypothesis-only gains despite few high-frequency words with high p(l|w), with performance improving by over 34% on non-entailed examples but not entailed ones.This motivates examining another irregularity: the relationship between grammaticality and labels.
7 Related Work
Prior NLI work showed that syntactic and other non-semantic features can provide strong signals, while related work emphasized inference involving both premise and hypothesis. This paper surveys existing NLI datasets using a hypothesis-only model.
- Non-semantic information to help NLI: Sentence structure and syntactic features have previously provided strong signals for NLI, including improvements on recognizing textual entailment tasks.Earlier approaches used sentence structure or combined syntactic features rather than relying only on external knowledge.
- World Knowledge and NLI: Prior work argues that textual entailment is relevant only when hypotheses are evaluated in relation to their premises.This motivates treating premise access as central to interesting NLI.
- NLI’s resurgence: NLI developed through shared RTE challenges, with early datasets containing thousands to tens of thousands of labeled sentence pairs.Early models often used alignment-based techniques.
8 Conclusion
The paper introduces a stronger hypothesis-only baseline for ten NLI datasets and finds that statistical irregularities can make the task easier than intended. It recommends using this baseline when evaluating NLI models and developing future datasets.
- 8 Conclusion: In six of ten datasets, the hypothesis-only model significantly outperformed the majority-class label baseline.The result indicates that majority-class prediction is not a strong baseline for those datasets.
- 8 Conclusion: Word choice and grammaticality may reduce NLI difficulty by not fully testing whether a hypothesis follows from its corresponding premise.These statistical irregularities can provide information without requiring the intended premise–hypothesis inference.
- 8 Conclusion: Future NLI datasets should exhibit fewer exploitable irregularities, and new models should be compared with corresponding hypothesis-only baselines.The authors also plan to extend this diagnostic to multimodal tasks involving relationships between two inputs.