Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches

1. Introduction

• Human readers use world knowledge and reasoning to infer unstated conclusions, while machines remain far from this natural language inference ability.The survey uses the silent piggy bank example to illustrate how readers infer both that no money was found and that Jack is disappointed.Source: §1 · pp. 1–2

1.1 Scope of Benchmark Datasets

• Recent benchmarks move beyond linguistic context, requiring reasoning and knowledge not explicitly stated in text.Benchmark scope also varies by task breadth, knowledge type, and format, from focused reference resolution to broader tasks and multiple-choice or open-ended questions.Source: §1.1 · pp. 3–4

1.2 Scope of Knowledge Resources

• The survey distinguishes linguistic, common, and commonsense knowledge, with commonsense covering everyday assumptions and intuitive physical and social reasoning.Linguistic knowledge includes word meanings, grammar, syntax, semantics, and discourse; common knowledge is often explicit, while commonsense is typically unstated.Source: §1.2 · pp. 4–5

1.3 Scope of Approaches

• Approaches span symbolic and statistical methods to deep neural models, whose strong benchmark performance leaves genuine inference progress debated amid opacity and dataset bias.Some recent neural approaches reach or exceed human performance, but their explainability and benchmark biases remain concerns.Source: §1.3 · p. 5

2. Benchmarks and Tasks

• Benchmark datasets offer training and testing data, evaluation frameworks, and continuous numerical feedback as alternatives to the Turing Test.The section introduces recent benchmarks designed to evaluate natural language inference and then discusses benchmark-creation considerations.Source: §2 · p. 5
• The section covers recent inference benchmarks alongside considerations and lessons for creating them.The considerations include task formulation, data bias, collection methods, and benchmark complexity.Source: §2 · p. 5
• In 2018, more NLI benchmarks were created, and they were larger than ever before.The figure describes a surge in benchmark tasks since the early 2000s.Source: Figure 2 · p. 6

2.1 An Overview of Existing Benchmarks

• Most benchmarks developed after 2015 are much larger than earlier ones, with some exceeding 100,000 instances and enabling deep-learning approaches.The survey describes an overall surge in benchmark variety since the RTE Challenges introduced textual entailment in the early 2000s.Source: §2.1 · p. 6

2.1.1 REFERENCE RESOLUTION

• Reference resolution remains unsolved and data-scarce; WSC requires commonsense to resolve pronouns, WinoGrande scales adversarial schemas to nearly 44,000 examples (Sakaguchi et al., 2019), and event-reference benchmarks remain limited.The surveyed benchmarks primarily address entity references, while event-reference resolution requiring outside knowledge and reasoning is anticipated as future work.Source: §2.1.1 · pp. 6–8

2.1.2 QUESTION ANSWERING

• Question-answering benchmarks range from passage-based comprehension to explicit external-knowledge and reasoning tests, including OpenBookQA’s 1,300 science facts and Cosmos QA’s 94% commonsense-requiring questions.Some QA tasks can be solved from linguistic context alone, while others require reasoning beyond the text; CoQA also includes out-of-domain and unanswerable test questions.Source: §2.1.2 · pp. 8, 12, 13

2.1.3 TEXTUAL ENTAILMENT

• Textual entailment asks whether a typical person would infer a hypothesis from a text, making commonsense central; SNLI offers nearly 600,000 labeled sentence pairs, while SherLIiC targets lexical inference in context.RTE later added contradiction recognition, and SherLIiC focuses particularly on entailment relations between event and action verbs.Source: §2.1.3 · pp. 13, 15, 16

2.1.4 PLAUSIBLE INFERENCE

• Plausible-inference benchmarks test hypothetical or uncertain conclusions using context and external knowledge; COPA evaluates causal reasoning, while CBT uses cloze questions about children’s-book stories.COPA asks systems to choose a cause or effect, and CBT’s four tasks predict a named entity, common noun, verb, or preposition.Source: §2.1.4 · p. 16
• Other plausible-inference benchmarks range from cloze prediction and ordinal plausibility ratings to multiple-choice endings and passage-based questions.LAMBADA requires a context-dependent final word; JOCI rates hypotheses on a five-point scale; CLOTH labels reasoning types; SWAG uses adversarial filtering; ReCoRD uses named entities to fill cloze blanks; ROCStories offers alternate endings.Source: §2.1.4 · pp. 16, 18
• HellaSWAG refreshes SWAG with new sources and adversarially filtered endings, while AlphaNLI tests abductive explanations between two observations.HellaSWAG has 70,000 examples and uses a better language model to generate negative endings. AlphaNLI pairs two partial observations with competing hypotheses and requires commonsense implications.Source: §2.1.4 · p. 19

2.1.5 INTUITIVE PSYCHOLOGY

• Story Commonsense and Event2Mind provide annotated resources for predicting characters’ motivations, emotions, intentions, and reactions.Story Commonsense adds about 160,000 annotations and classification tasks for needs, motives, and emotions; Event2Mind provides about 57,000 annotations across three intention and reaction prediction tasks.Source: §2.1.5 · p. 21
• SocialIQA tests social-interaction commonsense, while Triangle-COPA focuses on psychological questions and supplies logical forms.SocialIQA has 45,000 three-option questions, and fine-tuning on it improved later performance on smaller COPA and Winograd Schema Challenge benchmarks, exceeding state of the art on both.Source: §2.1.5 · pp. 19, 21

2.1.6 MULTIPLE TASKS

• These suites can support consistent skill testing and diagnostics, and many SuperGLUE tasks require multi-sentence reasoning or external knowledge.The section describes multi-task benchmarks as diagnostics for performance across areas; SuperGLUE includes tasks involving word-sense disambiguation and reference resolution.Source: §2.1.6 · pp. 22, 25
• bAbI isolates 20 reasoning and language-processing skills, but its synthetic, simple structure limits its value as a test of natural-language comprehension.Each task has 1,000 training and 1,000 test examples; systems can learn much of the test inference from training and do not generalize well to naturally generated data. The paper presents it as a gauge of learned reasoning skills.Source: §2.1.6 · pp. 22–23
• IIE, GLUE, DNC, and SuperGLUE combine tasks to evaluate varied language-processing and entailment capabilities.IIE recasts about 300,000 examples into entailment tasks; GLUE has nine tasks and over one million examples; DNC covers nine entailment tasks and seven reasoning types; SuperGLUE reframes a harder selection across about 23,000 examples.Source: §2.1.6 · pp. 23–24

2.2.1 TASK FORMAT

• Inference benchmarks use classification formats including entailment, cloze, and traditional multiple choice, as well as open-ended responses restricted to spans, label sets, or unrestricted answers.Entailment decisions range from binary or ternary to five-way formats, and the classic RTE format can be adapted to diverse reasoning tasks.Source: §2.2.1 · pp. 25–26

2.2.2 EVALUATION SCHEMES

• Evaluation should provide objective, informative measures suited to each task, while human comparisons and high scores require caution when data biases or opaque models may inflate performance.Exact match and F-measure suit many classification and short-answer tasks; longer generated responses often use BLEU, Meteor, or ROUGE, which may not correlate well with human judgments.Source: §2.2.2 · pp. 29–30
• Baselines and human performance provide reference points for progress, but simple baselines should expose trivial solutions and new benchmarks should report comparable model baselines.Human performance may be averaged across judges, determined by majority vote, or estimated through annotator agreement; expert performance can provide a ceiling.Source: §2.2.2 · pp. 29–30

2.2.3 DATA BIASES

• Up to 72.4% accuracy was possible on the Story Cloze Test by judging endings alone, while hypothesis-only models outperformed majority-class baselines on several entailment benchmarks.These shortcuts exploit writing-style and lexical correlations rather than realistic reasoning, including negative wording in wrong endings and words such as “nobody” in contradictory hypotheses.Source: §2.2.3 · p. 32
• Benchmark creators should balance answer labels and question types, inspect for superficial correlations, and use bias-reduction methods that evolve with stronger models.Adversarial filtering can remove easy stylistic cues, but the continuing need to strengthen it raises uncertainty about whether all statistical biases can be eliminated; predicting premises from hypotheses and labels is another proposed technique.Source: §2.2.3 · pp. 30–33

2.2.4 COLLECTION METHODS

• Data collection trades off scale, quality, and validation: expert curation is high quality but not scalable, automatic generation is efficient but variable, and mining works best from reliable sources.For example, bAbI’s simple generated language made most tasks solvable at 100% accuracy by baseline systems, while mined data from less reliable sources may need filtering or validation.Source: §2.2.4 · pp. 33–34
• Crowdsourcing can scale benchmark creation, but task instructions, worker qualification, and data validation must match task difficulty and support reliable submissions.Complex writing tasks may require clear instructions and pilot studies, careful validation, and sometimes paired workers who review one another’s work.Source: §2.2.4 · pp. 34–35
• Collection methods trade quality, speed, scalability, representativeness, verification needs, and cost, so benchmark creators must weigh their respective drawbacks.Manual authoring yields high-quality, expert-verified data but is slow and unscalable; automated authoring is faster but depends on the language model and may need manual checks, while text mining can better reflect human language. Crowdsourcing is quick and flexible but requires attention to task difficulty, worker qualification, validation, and cost; ROCStories cost about 26 cents per story plus 10 cents per test case, near 15,000 USD total.Source: §2.2.4 · p. 35

2.2.5 BENCHMARK COMPLEXITY

• Human performance measurements depend on benchmark evaluation frameworks and collection methods; therefore, low scores may reflect response-span scoring or crowdsourcing effects rather than extraordinary task difficulty.SQuAD 1.1 human accuracy was 77.0%, with errors typically involving omitted unnecessary answer-span words; RACE crowd workers scored 73.5% versus experts’ 94.5%, and SWAG scores ranged from 82.8% for one worker to 88.0% for an ensemble of five.Source: §2.2.5 · pp. 36–37
• Machine scores above human performance may reflect task-specific capabilities rather than general superiority, and statistical biases and limited interpretability further weaken that inference.On CBT, neural language models beat humans at predicting prepositions, a subtask difficult for people (67.6% accuracy without context), because models capture patterns that aid this prediction (Hill et al., 2015).Source: §2.2.5 · p. 37
• Benchmarks with more than 10,000 examples tend to show the highest machine performance and improvement, while those under 1,000 examples show limited improvement.The regression found no statistically significant correlation, and exceptions include small synthetic bAbI reaching 100% accuracy and large Story Cloze Test and Cosmos QA benchmarks with limited performance or improvement.Source: §2.2.5 · pp. 37–38, Figure 12 · p. 38
• Older benchmarks have qualitatively seen more progress, but this is not statistically significant and does not establish that newer benchmarks are harder.Adversarial construction may contribute to larger gaps on newer benchmarks, while Winograd Schema Challenge’s gap exceeds WinoGrande’s, possibly because it is smaller and expert-constructed; Story Cloze Test’s persistent gap may reflect writing-style bias and waning research interest.Source: §2.2.5 · pp. 38–39, Figure 13 · p. 39
• Popularity has no clear relationship with progress among benchmarks with few submissions, whereas the most popular benchmarks have seen machine performance surpass human performance.SQuAD 1.1 and 2.0 have received the most progress.Source: §2.2.5 · p. 40, Figure 14 · p. 40
• Human and machine performance can suggest benchmark complexity, but measurement limitations and benchmark characteristics make direct comparisons unreliable.For example, machine performance exceeds human performance on SQuAD 1.1 and SWAG, although both have relatively low human performance; high machine scores may also reflect statistical biases or opaque models rather than genuine superiority.Source: §2.2.5 · pp. 35, 36, 37, 40

3.1.1 LINGUISTIC KNOWLEDGE RESOURCES

• Linguistic resources include annotated corpora, lexical and frame resources, and pretrained semantic vectors, but static word vectors represent a word identically across contexts and senses.The Penn Treebank, WordNet, FrameNet, and word2vec illustrate these resource types; contextual representations instead depend on where a word appears.Source: §3.1.1 · pp. 41–43

3.1.2 COMMON KNOWLEDGE RESOURCES

• Common and commonsense knowledge resources include Wikipedia-derived, web-mined, probabilistic, and collaboratively maintained knowledge bases.Examples include YAGO and DBpedia, NELL and Probase, and Wikidata; these resources encode facts or relations among entities and concepts.Source: §3.1.2 · pp. 43–44

3.1.3 COMMONSENSE KNOWLEDGE RESOURCES

• Commonsense resources encode unstated everyday knowledge, including event inferences, object relations, and spatial co-occurrence.ATOMIC represents event-based if-then relations, while ASER automatically extracts events and relations from text; LocatedNear supplies two datasets for object co-occurrence.Source: §3.1.3 · pp. 45–46

3.2 Approaches to Creating Knowledge Resources

• Knowledge resources are built through expert encoding, text mining, and crowdsourcing, each balancing cost against the quality of acquired knowledge.Manual encoding is costly but expert-based; text mining scales efficiently but can be noisy and inconsistent, while crowdsourced knowledge may require expert validation because contributors may not follow engineers’ representations. It remains unclear how well these bottom-up resources support NLI in practice.Source: §3.2 · pp. 46–48

3.3 Leveraging Incomplete Resources

• Because even large knowledge bases omit facts needed for NLI benchmarks, completion methods infer or extract new relations from existing resources or text.Dimensionality reduction can smooth missing or noisy relations but cannot create new terms; other approaches learn to score proposed relations, mine relations from text, or infer relations from existing ones.Source: §3.3 · p. 48
• Up to 92.1% of relations generated by a ConceptNet-trained transformer were rated accurate by human workers (Bosselut, Rashkin, Sap, Malaviya, Celikyilmaz, and Choi, 2019).Source: §3.3 · p. 49
• Memory comparison networks generalize typical event orderings to acquire new temporal relations, while transformers trained on existing relations infer novel ones (Andrade, Bai, Rajendran, & Watanabe, 2018; Bosselut, Rashkin, Sap, Malaviya, Celikyilmaz, and Choi, 2019).Source: §3.3 · p. 49
• Future work must improve methods for handling the long-tail phenomenon in knowledge resources.Source: §3.3 · p. 49

4.1 Symbolic Approaches

• Symbolic NLI systems used logical forms, abduction, external semantic knowledge, or natural logic, but manually authored rules and language mappings do not scale well to larger datasets.Combining natural logic with a statistical RTE model exceeded state-of-the-art performance on the third RTE Challenge, while the Triangle-COPA baseline reached 91% accuracy using manually authored logic and commonsense rules.Source: §4.1 · pp. 49–50

4.2 Early Statistical Approaches

• Early statistical RTE systems moved from lexical matching, often only slightly better than random, toward richer linguistic features, external knowledge, and web co-occurrence evidence.These systems have not remained competitive on recent large benchmarks, though they can serve as baselines, as shown for JOCI.Source: §4.2 · pp. 50–51

4.3 Neural Approaches

• Neural approaches learn task-relevant features rather than specifying them all manually, using embeddings with task-specific RNN, CNN, or transformer architectures and output layers suited to classification or generation.Pre-trained representations can be used as features or fine-tuned for downstream tasks; attention and memory augmentation can further support alignment and reasoning over multiple facts.Source: §4.3 · pp. 51–52

4.3.1 ATTENTION MECHANISM

• Transformers use multi-head self-attention to let each input position attend across the sequence, potentially capturing long-range syntactic, semantic, and coreference relations while enabling parallel computation.The section also describes attention as useful for input-output alignment and distant input states; RNNs with attention improved tasks such as entailment, but struggled with MCScript yes/no questions requiring negation handling and deeper understanding.Source: §4.3.1 · pp. 52–53

4.3.2 MEMORY AUGMENTATION

• Memory-augmented approaches represent changing world states with dynamic memory, while variants such as QRN, DNC, neural process networks, and KG-MRC adapt this strategy to procedural text and entity tracking.The optimized bidirectional DNC achieved a response word error rate of about 0.4 on bAbI; neural process networks learn action operators to represent entity state changes and predict future actions.Source: §4.3.2 · pp. 53–55
• 99.5% accuracy over all bAbI questions made ENTNET one of the first models to pass all 20 subtasks, and it outperformed memory-network baselines on bAbI and CBT.ENTNET updates parallel entity-specific memory cells as text arrives, so it can answer multiple questions without rereading the text; unlike memory networks, it underperformed on ProPara, where entity-state tracking and coreference are harder.Source: §4.3.2 · pp. 54–55
• 62.50 F-measure makes NCET the current state of the art on ProPara, tracking entity states with per-entity bidirectional LSTMs and a neural CRF that enforces global physical constraints.KG-MRC uses sentence-by-sentence knowledge graphs and achieves a 57.60 F-measure, while learning commonsense constraints that PROSTRUCT manually enforces.Source: §4.3.2 · p. 56

4.3.3 CONTEXTUAL MODELS AND REPRESENTATIONS

• GPT’s release advanced the state of the art on 12 benchmarks, while contextual models such as ELMO, GPT, BERT, and XLNET provide context-sensitive representations unlike word2vec and GloVe.ELMO embeddings improved earlier systems and supported word-sense and part-of-speech identification; GPT is unsupervisedly pre-trained and fine-tuned for downstream tasks, but has high computational demands and generalization issues under high lexical variation.Source: §4.3.3 · pp. 56–57
• GPT 2.0 exceeded three of four baseline approaches to CoQA in an unsupervised setting, trained only on documents and questions, not answers, and learned from natural-language demonstrations without explicit question-answer locations.Source: §4.3.3 · p. 57
• BERT surpassed earlier state-of-the-art results on several benchmarks, and its variants later topped many surveyed leaderboards and transferred to COPA and the Winograd Schema Challenge.BERT was also used as a baseline on AlphaNLI and SocialIQA; its deep structure is difficult to explain, and early analysis found that it could exploit superficial correlations, particularly on ARCT.Source: §4.3.3 · pp. 58–59
• MT-DNN outperformed the original BERT on several leaderboards, with gains attributed mainly to task-specific layers and multi-task fine-tuning that reduces task overfitting and leverages cross-task data.The reported benchmarks include SciTail, SNLI, and GLUE.Source: §4.3.3 · p. 59
• XLNET exceeded BERT on several benchmarks, using permutation-based autoregressive pre-training with Transformer-XL to capture left- and right-context and dependencies between predicted tokens.The approach addresses BERT’s masked-token pre-training and fine-tuning discrepancy and its independence assumption for masked predictions.Source: §4.3.3 · pp. 59–60
• ROBERTA returned to the top of RACE, SWAG, and GLUE leaderboards, while ALBERT combined parameter reduction and a discourse-coherence objective with superior results on several leaderboards.ROBERTA exceeded human performance on SWAG and GLUE; ALBERT’s techniques enabled deeper scaling with fewer parameters than BERT-large.Source: §4.3.3 · p. 60
• Contextual embeddings can be used as features or fine-tuned directly, and the better choice varies by model and task: ELMO features helped on sentence-pair classification, while fine-tuning helped on sentence similarity.BERT often outperformed ELMO when used as features; ELMO could be fine-tuned successfully, though doing so was more difficult.Source: §4.3.3 · p. 61

4.4 Incorporating External Knowledge

• Despite benchmarks’ need for external knowledge, surveyed approaches—especially neural ones—usually relied on benchmark training data and linguistic resources, and common and commonsense knowledge resources rarely supported state-of-the-art results.Large pre-training texts may provide some knowledge, but are unlikely to contain unstated commonsense knowledge.Source: §4.4 · pp. 62–63
• External-resource methods include WordNet-based inference and ConceptNet-assisted reasoning, but adding ConceptNet facts can distract; selecting appropriate knowledge from noisy resources remains a challenge.WordNet relations supported entailment models, while ConceptNet paths facilitated multistep reasoning in NarrativeQA; Cyc was not used in surveyed approaches and ConceptNet was only occasionally applied.Source: §4.4 · pp. 62–63

• Related language benchmarks include corpora and multi-task sets involving commonsense, dialogue benchmarks with different commonsense demands, and multilingual RTE datasets.Source: §5 · p. 64
• Visual benchmarks cover knowledge-based VQA, commonsense action reasoning, and targeted visual reasoning; video action datasets and navigation tasks extend this work toward physically grounded inference.Source: §5 · pp. 64–65

• Jointly developing benchmarks and knowledge resources could formalize the knowledge scope needed and support evaluation of approaches that incorporate external knowledge.Source: §6 · p. 65
• Interactive or embodied agents could help researchers understand commonsense knowledge practically and address knowledge acquisition and language inference jointly.Source: §6 · pp. 65–66
• Most benchmarks do not systematically distinguish reasoning abilities, so the authors recommend curating data to evaluate different capabilities and enable targeted improvement.Existing exceptions include bAbI’s reasoning sub-tasks and Event2Mind’s psychological-state predictions.Source: §6 · p. 66
• RoBERTa surpassed earlier variants on several benchmarks using BERT’s architecture with small pre-training changes and careful hyperparameter selection, underscoring the importance of design choices.Source: §6 · pp. 66–67
• Task-competence leaderboards can reward marginal gains without showing broader question-answering progress, motivating broader, multidimensional evaluation.Source: §6 · p. 67
• 40% fewer parameters, 60% faster prediction, and 97% of GLUE performance for DISTILBERT illustrate efficiency gains that current leaderboards do not reward.Efficiency-aware evaluation could reward better use of available training data, broaden participation, and encourage greener AI practices.Source: §6 · p. 68
• Because deep models’ reasoning is opaque and may exploit benchmark biases, the authors suggest evaluating explanations, as VCR asks models to justify answers.The VCR setup may itself be vulnerable to statistical artifacts of human language.Source: §6 · pp. 68–69
• Generalization evaluation should test adaptation to new distributions and tasks with minimal training; the authors propose few-example assessment while noting large training-data needs remain limiting.Source: §6 · p. 69

• The comparison records five construction stages and data size, with M/A/T/C denoting manual methods, automatic generation, text mining, and crowdsourcing.Source: Appendix A · pp. 69–70
• RTE-1 has 1.37K examples, compared with 570K for SNLI and 1.44M for GLUE, illustrating the range of listed benchmark sizes.Source: Appendix A · p. 70
• Creation pipelines range from manual construction in COPA to mixed text-mining and crowdsourcing in SNLI and automatic generation in bAbI.Source: Appendix A · p. 70