PaperlayerView exact revision on arXiv

Source-linked AI summary

Atomic-SNLI: Fine-Grained Natural Language Inference through Atomic Fact Decomposition

Minghui Huang

arXiv:2601.06528v1cs.CLcs.AI

TL;DR

Sentence-level NLI can obscure which hypothesis details support its prediction, and models struggle with atomic-level reasoning. Atomic-SNLI decomposes SNLI hypotheses and adds generated atomic examples; fine-tuning improves atomic reasoning while maintaining sentence-level performance.

  • Problem

    Sentence-level NLI often does not reveal which parts of a hypothesis drive its decision, and current models perform comparably or worse on atomic-level inference.

    Source: §1 · p. 1, §6 · p. 8

  • Method

    Atomic-SNLI is constructed by decomposing SNLI and enriching it with generated atomic examples, while its test format supports aggregation of atomic predictions into sentence-level judgments.

    Source: §4 · p. 4, §6 · p. 8

  • Results

    Fine-tuning on Atomic-SNLI improves atomic reasoning while maintaining sentence-level performance; for 3-fact cases, DeBERTa-v3-xsmall gains +10.07% accuracy and +8.16% F1.

    Source: §5.2 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    The dataset establishes a benchmark for granular NLI and offers a path toward more transparent and robust reasoning systems.

    Source: §6 · p. 8

    The approach assumes independence between atomic facts, an assumption that may not hold in all cases.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Current Natural Language Inference (NLI) systems primarily operate at the sentence level, providing black-box decisions that lack explanatory power. While atomic-level NLI offers a promising alternative by decomposing hypotheses into individual facts, we demonstrate that the conventional assumption that a hypothesis is entailed only when all its atomic facts are entailed fails in practice due to models' poor performance on fine-grained reasoning. Our analysis reveals that existing models perform substantially worse on atomic level inference compared to sentence level tasks. To address this limitation, we introduce Atomic-SNLI, a novel dataset constructed by decomposing SNLI and enriching it with carefully curated atomic level examples through linguistically informed generation strategies. Experimental results demonstrate that models fine-tuned on Atomic-SNLI achieve significant improvements in atomic reasoning capabilities while maintaining strong sentence level performance, enabling both accurate judgements and transparent, explainable results at the fact level.

1 Introduction

Sentence-level NLI predicts entailment, contradiction, or neutrality for a premise–hypothesis pair, but often does not reveal which parts of the hypothesis drove its decision.

  • NLI determines whether a premise and hypothesis are in entailment, contradiction, or neutral relation.Source: §1 · p. 1
  • Sentence-level predictions often leave unclear which specific parts of the hypothesis led to the decision.Source: §1 · p. 1
  • The example pairs a premise about two women, a camera, and artwork with a hypothesis about camera-holding and shark pictures.Source: §1 · p. 1

Label: CONTRADICTION

Atomic-level NLI decomposes hypotheses into individual facts and aggregates their predictions, offering a more transparent view of reasoning. The paper finds that current models struggle with atomic inference and that the assumption that all facts must be entailed does not reliably produce correct sentence-level judgments.

  • Atomic-level analysis breaks a hypothesis into facts, checks each against the premise, and combines the checks into an overall label.Source: Label: CONTRADICTION · p. 1, Figure 1 · p. 1
  • Current models’ atomic-level accuracy is comparable to or lower than sentence-level accuracy, and the standard inference rule often fails even with perfect atomic judgments.Source: Label: CONTRADICTION · p. 2
  • The paper investigates compositional reasoning limits and introduces Atomic-SNLI, a dataset that enriches decomposed SNLI with generated neutral and contradictory atomic examples.Source: Label: CONTRADICTION · p. 2
  • Fine-tuning on Atomic-SNLI improves atomic reasoning while maintaining strong sentence-level performance.Source: Label: CONTRADICTION · p. 2
  • Atomic analysis can expose which fact drives a contradiction: in the example, one camera-holding claim conflicts while other components are entailed or neutral.Source: Figure 1 · p. 1, Label: CONTRADICTION · p. 2

2 Related Work

Prior work uses fact decomposition across NLP applications, while research on fine-grained NLI has begun examining atomic reasoning. This paper focuses on atomic fact-level entailment and provides a dedicated dataset and evaluation framework.

  • Fact decomposition supports factuality evaluation, automated fact-checking, retrieval, and downstream question answering in prior work.Source: §2 · p. 2
  • Huang (2025) argues that LLM-generated decompositions lack stability and reliability, and proposes DECMETRICS and a specialized DECMODEL.Source: §2 · p. 2
  • Unlike recent fine-grained NLI work, this approach targets atomic fact-level entailment with a dedicated dataset and evaluation framework.Source: §2 · p. 2

3 Atomic-Level NLI Analysis

The analysis decomposes SNLI hypotheses into atomic facts, applies compositional rules, and compares atomic- and sentence-level NLI. Atomic-level accuracy lags across models, with performance varying by fact count and label.

3.1 Hypothesis Decomposition

  • The 9,824 valid decomposed test hypotheses contain 1–5 facts; 8,767 (89.2%) have one fact, while only 27 (0.3%) have four or more.Source: §3.1 · p. 2

3.2 Inference Framework

  • The framework entails a hypothesis only when every fact is entailed, marks contradiction when any fact contradicts, and marks neutral when at least one is neutral and none contradicts.Source: §3.2 · p. 3

3.4 Results and Analysis

  • Atomic-level accuracy trails sentence-level accuracy by 0.61 to 0.79 percentage points across all evaluated models.Source: Table 2 · p. 3, §3.4 · p. 4

3.3 Experimental Setup

  • The experiment compares sentence-level and atomic-level SNLI performance using four pretrained NLI models.Source: §3.3 · pp. 3–4

3.4 Results and Analysis

  • For multi-fact hypotheses, contradiction recall and entailment precision increase; models also tend to over-predict contradiction, yielding artificially high contradiction recall under deterministic rules.Source: §3.4 · p. 4

4 Atomic-SNLI Dataset Construction

Atomic-SNLI decomposes SNLI hypotheses into labeled atomic facts, with separate construction processes for entailment, neutrality, and contradiction. The resulting dataset retains SNLI premises while expanding the number of inference instances and balancing relation types.

4 Atomic-SNLI Dataset Construction

  • Atomic-SNLI turns SNLI into premise–atomic-fact–label examples, enabling localized reasoning over individual claims.The training and validation splits provide supervised atomic data; test examples retain the decomposed fact set for aggregation into a sentence-level judgment.Source: §4 · p. 4

4.1 Data Construction Methodology

  • Each SNLI hypothesis is decomposed into one or more self-contained atomic facts, with specialized processing for multi-fact hypotheses.Single-fact hypotheses are directly converted into atomic triplets.Source: §4.1 · p. 5
  • Entailment examples come from decomposed facts filtered for entailment confidence, while neutral examples combine direct filtering with retrieval and neutrality validation.Retrieved neutral candidates are lexically selected, reranked by neutrality score, and filtered; entailment pairs must exceed the entailment-confidence threshold.Source: §4.1 · p. 5
  • Contradiction examples combine direct extraction with LLM-generated minimally altered facts validated by an ensemble of NLI models.Generated facts are retained when all validation models predict contradiction above the confidence threshold.Source: §4.1 · p. 5

4.2 Dataset Statistics

  • Atomic-SNLI keeps the original SNLI premises across splits but contains more inference instances because hypotheses yield multiple atomic facts.Balanced sampling of generated neutral and contradictory hypotheses is intended to prevent any relation from dominating the training data.Source: §4.2 · p. 5

5 Experiments

The experiments test atomic reasoning on multi-fact hypotheses and whether atomic training transfers to sentence-level NLI. Atomic-SNLI improves multi-fact results, while probability summation supports class-balanced predictions and fact-level explanations.

5.1 Experimental Setup

  • The evaluation focuses on multi-fact hypotheses and compares pretrained models fine-tuned on Atomic-SNLI with the same models fine-tuned on SNLI.Sentence-level labels are formed by summing predicted probabilities over the atomic facts and taking the argmax.Source: §5 · p. 5, §5.1 · p. 6

5.2 Main Results

  • +10.07% accuracy and +8.16% F1 for DeBERTa-v3-xsmall on 3-fact hypotheses; Atomic-SNLI training also consistently improves 2- and 3-fact results.The reported gains compare Atomic-SNLI-trained models with their SNLI-trained counterparts.Source: §5.2 · p. 6, Table 4 · p. 7
  • Atomic-SNLI benefits appear across DeBERTa, ELECTRA, and BERT, but performance degrades on 4-fact hypotheses with only 25 test instances.DeBERTa variants show the largest gains, followed by ELECTRA and BERT.Source: §5.2 · p. 6

5.3 Further Analysis

  • Probability summation maintains balanced precision and recall across NLI classes, unlike simple logical rules’ extreme class-wise patterns.The approach aggregates soft atomic prediction probabilities rather than applying strict hard-label rules.Source: §5.3 · pp. 6–7

5.4 Atomic-Level Explainability

  • Atomic decomposition makes model decisions traceable to individual claims, including the unsupported swim-meet detail in a neutral example.The examples also show how a contradiction can be isolated to one conflicting fact while other decomposed facts are supported.Source: §5.4 · pp. 7–8

6 Conclusion

The paper finds that current NLI models struggle with atomic reasoning and that universal entailment of a hypothesis’s atomic facts does not reliably hold in practice. Atomic-SNLI improves atomic reasoning while maintaining sentence-level performance, but its construction and inference rely on assumptions with stated limitations.

6 Conclusion

  • Current models perform worse on atomic-level inference than sentence-level tasks, and universal atomic entailment fails in practice.The conclusion identifies these findings as evidence of limitations in compositional reasoning.Source: §6 · p. 8
  • Atomic-SNLI decomposes SNLI and adds generated atomic examples; fine-tuning on it improves atomic reasoning while maintaining sentence-level performance.The paper presents the dataset as a benchmark for granular NLI and a path toward transparent, robust reasoning systems.Source: §6 · p. 8

Limitations

The approach is limited by generated contradiction examples, the quality of atomic fact extraction, and an assumption that atomic facts are independent. The authors identify more robust generation and compositional reasoning as future directions.

  • LLM-generated contradiction examples may not fully capture the complexity of real-world contradictory relationships.Source: Limitations · p. 9
  • Atomic fact extraction depends on DECMODEL quality, which may introduce errors.Source: Limitations · p. 9
  • The inference rules assume independence between atomic facts, which may not hold in all cases.Source: Limitations · p. 9
  • Future work should pursue more robust generation methods and more sophisticated compositional reasoning frameworks.Source: Limitations · p. 9

A.1 Limitations of Simple Logical Rules

Simple logical aggregation rules yield imbalanced class performance, while probability summation is presented as a softer alternative that integrates information across atomic predictions.

A.1 Limitations of Simple Logical Rules

  • 100% precision but 20.69% recall in 3-fact entailment cases shows that the all-facts rule is overly strict.Source: §A.1 · p. 10
  • Contradiction has high recall but relatively low precision because any atomic contradiction triggers a sentence-level contradiction label.This one-vote veto makes the result sensitive to noise and errors.Source: §A.1 · p. 10
  • Neutral classification requires no contradiction and at least one neutral fact, producing performance instability across atomic-fact counts.Source: §A.1 · p. 10

A.2 Advantage Mechanisms of Probability Summation

  • Probability summation softens simple-rule strictness by tolerating individual errors, integrating confidence across atomic predictions, and smoothing borderline decisions.Source: §A.2 · p. 10

A.3 Comparison with Direct Sentence-Level Reasoning

Atomic-level and direct sentence-level reasoning have comparable performance on simpler hypotheses, while the atomic approach offers more interpretable decisions. The appendix also describes a prompt for generating minimally altered contradiction examples.

A.3 Comparison with Direct Sentence-Level Reasoning

  • For 2- and 3-fact cases, atomic and direct sentence-level methods perform comparably; 4-fact cases challenge both because of data sparsity.The atomic approach offers a transparent decision process that the authors connect to trust and debugging in practical applications.Source: §A.3 · p. 10

B Contradiction Generation Prompt

Contradiction Generation Prompt

Loading 2601.06528v1…