Source-linked AI summary
NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals
Neha Srikanth, Rachel Rudinger
TL;DR
The paper uses atomic decomposition to examine model consistency in traditional and defeasible NLI, where atomic labels relate to overall judgments in different ways. It identifies critical atoms in defeasible examples and finds that models perform better on those atoms than on all atomic problems, while some still predict full answers correctly despite errors on critical atoms.
Problem
LLMs can struggle to make logically consistent judgments between atomic inferences and an overall NLI judgment.
Source: §4 · p. 4Method
The paper decomposes hypotheses into atomic problems and identifies critical atoms to group defeasible examples for measuring consistency across contexts.
Source: §3 · p. 3, §5.2 · p. 7, §6 · p. 9Results
Models perform better on critical atomic sub-problems than on all atomic problems, yet some predict the full answer correctly with probability as high as 75% after incorrectly solving critical sub-problems.
Source: §5.2 · p. 8Takeaways & Limitations
Critical atoms enable grouping examples by shared facts and measuring whether model predictions about those facts remain consistently correct or incorrect across contexts.
Source: §7 · p. 9The authors note that annotation artifacts in update sentences may inflate the performance of finetuned RoBERTa-large and DeBERTa-v3-large.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Decomposition of text into atomic propositions is a flexible framework allowing for the closer inspection of input and output text. We use atomic decomposition of hypotheses in two natural language reasoning tasks, traditional NLI and defeasible NLI, to form atomic sub-problems, or granular inferences that models must weigh when solving the overall problem. These atomic sub-problems serve as a tool to further understand the structure of both NLI and defeasible reasoning, probe a model's consistency and understanding of different inferences, and measure the diversity of examples in benchmark datasets. Our results indicate that LLMs still struggle with logical consistency on atomic NLI and defeasible NLI sub-problems. Lastly, we identify critical atomic sub-problems of defeasible NLI examples, or those that most contribute to the overall label, and propose a method to measure the inferential consistency of a model, a metric designed to capture the degree to which a model makes consistently correct or incorrect predictions about the same fact under different contexts.
1 Introduction
The paper decomposes hypotheses into atomic sub-problems to examine traditional and defeasible NLI, including model consistency and the structure of benchmark examples. It reports that LLMs struggle with logical consistency and introduces a way to measure inferential consistency across contexts.
1 Introduction
- Decomposing hypotheses into granular inferences lets models’ judgments be probed and benchmark examples examined for repeated or diverse inferences.The paper applies this approach to traditional NLI and defeasible inference, where conclusions may change with new evidence.Source: §1 · p. 1
- Atomic sub-problems expose logical consistency problems in LLMs despite their high accuracy on traditional NLI.The paper also examines atomic sub-problems in defeasible NLI, where their relation to the overall label is more complex.Source: Figure 1 · p. 1, §1 · p. 2
- Critical atomic sub-problems identify the inference most relevant to a defeasible NLI example and support grouping examples by shared atoms.The paper uses these groups to measure whether a model predicts the same fact consistently correctly or incorrectly across contexts.Source: §1 · p. 2
2 Background
The paper situates its analysis in traditional natural language inference and defeasible inference, then distinguishes its focus on LLM consistency from related decomposition work.
2.1 Natural Language Inference
- Natural language inference classifies whether a premise entails, contradicts, or is neutral with respect to a hypothesis.SNLI pairs image-caption premises with crowdworker-written hypotheses and is used here as a resource for robustness and artifact studies.Source: §2.1 · p. 2
2.2 Defeasible Inference
- Defeasible NLI evaluates whether an update strengthens or weakens a neutral premise-hypothesis inference.The paper focuses on δ-SNLI, the SNLI-derived split of δ-NLI.Source: §2.2 · p. 2
2.3 Related Work
- Unlike prior span-level or premise-decomposition work, this study examines LLM logical consistency over atoms, including in defeasible inference.The authors identify their work as the first to explore atoms in defeasible inference.Source: §2.3 · p. 2
3 Atomic Sub-Problems in NLI
The paper represents hypotheses as atomic propositions and uses premise-atom or premise-atom-update pairs to study granular inferences. It generates and validates atoms differently for traditional and defeasible NLI.
3 Atomic Sub-Problems in NLI
- Atomic sub-problems capture constituent judgments that models must weigh consistently with their overall inference and help reveal repeated inferences in benchmarks.Each atom represents one piece of information; traditional NLI evaluates (P, ai), while defeasible NLI evaluates (P, ai, U).Source: §3 · p. 3
3.1 Generating Atomic Propositions
- Each conjunct in the semantic representation maps to a natural-language atom, keeping action predicates and their arguments as separate pieces.The authors use handconstructed exemplars to prompt llama-3-8b-instruct to generate atoms for δ-SNLI test examples and a sample of SNLI test examples.Source: §3.1 · p. 3
3.2 Validating Atomic Decompositions
- For δ-SNLI, valid atoms must be grammatical, entailed by H, and not entailed by P; atoms adding new or pragmatic information are excluded.Generated δ-SNLI atoms are pruned with a finetuned NLI model and then human-validated, while SNLI atoms are not independently validated and analyses use atoms models themselves accept as valid.Source: §3.2 · pp. 3–4
- 95.7% of pruned atoms were judged valid by the author annotator, and an external sample annotation reached κ = 0.82 agreement.The remaining analyses use the valid δ-SNLI atoms.Source: §3.2 · p. 4
4 Atoms in Traditional NLI
The paper tests whether LLM judgments on atomic NLI sub-problems align logically with full-example judgments. Models show consistency gaps that accuracy alone does not capture.
4.1 Atomic and Overall Label Consistency
- For an entailed hypothesis, every valid atom must be entailed; for a contradiction, at least one must contradict; for neutrality, at least one must be neutral and others neutral or entailed.These rules establish the expected relation between atomic sub-problem labels and the overall label.Source: §4.1 · p. 4
4.2 Experimental Setup
- The experiments evaluate six LLMs on SNLI test examples using 12 in-context examples, then elicit atom judgments with the same prompt and exemplars.For atoms a model predicts as entailed by H, it also predicts the relation between P and each atom.Source: §4.2 · pp. 4–5
- A model’s accuracy does not fully indicate its logical consistency, and gpt-4o outperformed other models on consistency even when its full prediction was incorrect.The table reports accuracy, overall consistency, consistency conditional on full-prediction correctness, and consistency by predicted label.Source: §4.2 · p. 5, Table 2 · p. 5
- All models exhibit logical consistency gaps across the three predicted labels, with different models struggling on different classes.Consistency offers another comparison when models have similar accuracy.Source: §4.2 · p. 5
5 Atoms in Defeasible Inference
Atomic decomposition treats defeasible inference as a softer weighing of update effects across hypothesis atoms. The paper uses critical atoms to identify the inference under discussion and examine model performance and consistency.
5 Atoms in Defeasible Inference
- Defeasible inference weighs the direction and magnitude of an update’s effects across atoms, unlike traditional NLI’s strict logical combination of atom relations.For example, an update can strengthen beliefs about two atoms while leaving another unaffected, compounding its overall strengthening effect.Source: §5 · p. 5
5.1 Understanding Defeasible Inference with Atomic Sub-Problems
- Many models surpassed the human performance benchmark, with gpt-4o the top-performing model at 92% accuracy.Source: §5.1 · p. 6
- A significant portion of δ-SNLI examples contain atomic labels with no effect or opposite polarity, illustrating nonmonotonicity.Some examples have atoms of only one polarity, while others mix positive and negative labels.Source: §5.1 · p. 6
- Across the board, models performed worse on atomic sub-problems than on full δ-SNLI examples.The authors hypothesize that updates acting on multiple parts may compound their effects and contribute to higher full-example performance.Source: Table 3 · p. 7, §5.1 · p. 7
5.2 Critical Atoms as Questions Under Discussion (QUD)
- Updates targeting particular atoms favor particular questions that the hypothesis answers, making critical atoms a way to identify the inference or knowledge an example tests.The paper uses this framework to interpret model behavior and represent the knowledge evaluated in δ-SNLI.Source: §5 · p. 6, §5.2 · p. 7
- Critical atoms are the valid atoms with the strongest labels matching the example’s overall polarity; most test examples have one, though some have multiple.Source: §5.2 · pp. 7–8
- Models performed better on critical atomic sub-problems than on all atomic sub-problems, while the authors hypothesize that weaker, indirect inferences remain difficult.Even larger models seem to struggle with the nuanced distinctions involved.Source: §5.2 · p. 8
- Some models had up to a 75% probability of predicting the full example correctly despite incorrectly solving all its critical atomic sub-problems.Correctly solving all critical atoms is a strong indicator of a correct full-example prediction, but the converse is not assured.Source: §5.2 · p. 8
6 Measuring Inferential Consistency in δ-NLI
The authors group defeasible NLI examples by the critical atoms they test and define inferential consistency as the likelihood of consistently correct or incorrect predictions for the same inference across contexts. All analyzed models have room to improve, with some inconsistencies arising when contexts require more implicit reasoning or background knowledge.
Measuring Inferential Consistency
- Grouping examples that share a critical atom reveals whether a model predicts the same fact consistently across strengthening and weakening contexts.Figure 4 contrasts two δ-NLI examples that evaluate the same atom under different premise-update contexts.Source: Figure 4 · p. 8, §6 · p. 8
Grouping Examples by Critical Atom
- The test set contains 1,761 δ-SNLI examples with at least one valid atom, yielding 429 unique atoms and 349 unique critical-atom groupings.The 1,761 examples come from 193 unique premise-hypothesis pairs; the 349 groupings result from clustering semantically equivalent atoms.Source: §6 · p. 8
Critical Atoms and Dataset Diversity
- Some critical atoms recur frequently: “The others are friends of the person” appears in 21 examples, while “The person is a man” appears in 20; examples sharing a critical atom form buckets.Source: §6 · p. 9
Measuring Inferential Consistency
- Inferential consistency (IC) measures the probability that two examples sharing a critical atom are both predicted correctly or both predicted incorrectly.The examples are grouped by shared critical atoms, and δ-NLI examples with multiple critical atoms share their weight across the corresponding buckets.Source: §6 · p. 9
Results
- All analyzed models have room to improve in inferential consistency, with some inconsistencies arising when contexts demand more implicit reasoning or background knowledge.The authors illustrate this with a shared “The people are tall” atom: several models struggle with a strengthening inference in one context, while all analyzed models succeed in another.Source: §6 · p. 9
Grouping Examples by Critical Atom
- The test set evaluates 223 unique questions under discussion, covering around 81% of the available atoms.The authors identify semantically equivalent critical atoms by embedding them, thresholding pairwise cosine similarity at 0.75, and requiring bidirectional entailment before finding maximal cliques.Source: §6 · pp. 8–9
7 Conclusion
Hypothesis decomposition supports measuring model consistency and examining non-monotonic reasoning. The paper introduces critical atoms to group defeasible NLI examples by the primary fact they evaluate and measure inferential consistency across contexts.
Conclusion
- Hypothesis decomposition measures model logical consistency and helps characterize non-monotonic reasoning, while critical atoms group defeasible NLI examples by the primary fact tested.The authors use these groups to measure whether LLM predictions remain consistently correct or incorrect across different contexts.Source: §7 · p. 9
Limitations
The paper reports limits related to decomposition coverage, dataset artifacts, and the neutral-update training data available for fine-tuned models.
Limitations
- The authors did not determine whether SNLI-TEST families were missing decompositions at particular levels of granularity.They manually validated generated atom validity for δ-SNLI, but did not make the same missing-decomposition assessment for SNLI-TEST.Source: Limitations · p. 10
- Annotation artifacts in δ-NLI and SNLI may inflate fine-tuned RoBERTa-large and DeBERTa-v3-large performance on update sentences.Prompt-based models are not directly trained on these datasets; the authors suggest collecting updates that indirectly or lightly affect hypotheses as an opportunity to avoid artifacts.Source: Limitations · p. 10
- Because the original δ-NLI training set lacks neutral updates, reported fine-tuned-model metrics cover only non-neutral atomic sub-problems.The authors suggest that adding contextual neutral atomic sub-problems to fine-tuning data could enable prediction of neutral updates.Source: Limitations · p. 10
A Atom Generation
The atom-generation prompt asks an expert linguist to produce independent, self-contained facts strictly supported by the input sentence.
Atom Generation
- The atom-generation prompt asks an expert linguist to produce independent, self-contained facts strictly supported by the input sentence.It directs the response to end with [END].Source: A Atom Generation · p. 13
A.1 Coverage of Generated Atoms
The authors assess whether generated atoms cover their source hypotheses, since logical-consistency checks depend on completeness in some label conditions. Coverage is high in sampled SNLI examples and most δ-SNLI examples, though some δ-SNLI omissions reflect flawed original examples or annotation choices.
A.1 Coverage of Generated Atoms
- Generated atoms must collectively cover the hypothesis, and pruning must preserve that coverage.The authors estimate coverage by checking whether information in the original hypothesis projects into the generated atoms.Source: §A.1 · p. 13
- Completeness matters for consistency checks when the overall label is neutral and all atoms are predicted entailed, or when it is contradiction.A neutral inconsistency involving an atom predicted contradiction does not require completeness; completeness estimates contextualize the consistency results as an upper bound.Source: §A.1 · p. 13
- 49/50 sampled SNLI hypotheses had all information represented in at least one atom.In a separate sample of 50 inconsistency cases, only two examples were actually missing an atom after review.Source: §A.1 · p. 13
- 28/70 flagged δ-SNLI examples had a missing atom, amounting to 1% of the full test set.The 70 examples were 3% of δ-SNLI-TEST; many remaining cases involved flawed source examples whose assumptions annotators did not propagate.Source: §A.1 · pp. 13–14
B.1 Inconsistency in SNLI
The authors identify several sources of inconsistency between whole-example and premise–atom judgments in SNLI, including atom entailment misjudgments and properties of the decomposed text. The supplied defeasible-NLI prompts ask models to judge whether evidence changes a hypothesis’s likelihood, including a no-change option for atom judgments.
B.1 Inconsistency in SNLI
- In SNLI consistency measurement, models first assess whether an atom follows from the hypothesis, then predict premise–atom relations; misjudging entailment can exclude atoms and produce inconsistent behavior.For example, gpt-4o-2024-08-06 predicted the hypothesis was neutral relative to the premise, but judged the last atom not entailed and excluded it from the atoms used to measure consistency.Source: §B.1 · pp. 14–15
- Hypernym atoms can be difficult when the premise uses a hyponym such as “man” and the atom uses its hypernym, “person.”The decomposition includes “there is a person,” “the person is a man,” and “the person dances” for “the man dances.”Source: §B.1 · p. 15
- Some generated atoms use syntactic constructions rare in SNLI, making them out of domain for encoder and prompt-based models.The authors give pseudo-clefts as an example and note the prompt-based models may have encountered SNLI training data in pretraining corpora.Source: §B.1 · p. 15
- Because crowdworkers write hypotheses and updates conditioned on labels, full sentences can express labels more strongly than atoms, making atomic judgments harder.The authors note that atom effects are lighter individually and can compound when taken together.Source: §B.1 · p. 15
- Removing implicit co-reference during atom generation can result in inconsistencies between whole-example and premise–atom judgments.Source: §B.1 · p. 15
C Defeasible NLI
- The defeasible-NLI atom prompt adds “none” for evidence that leaves the hypothesis’s likelihood unchanged; the whole-example prompt offers only “more” or “less.”Source: C Defeasible NLI · p. 15
D QUD Generation for Understanding Diversity
The authors convert critical atomic propositions into questions under discussion (QUDs) to examine diversity in δ-SNLI-TEST. An external annotator judged that the generated questions captured their critical atoms in 92% of cases.
D QUD Generation for Understanding Diversity
- Critical atoms are mapped to questions answerable by their proposition, with exemplars encouraging similar inferences to share QUDs.For example, “The dog is brown” maps to “What color is the dog?”; exemplars also map nongeneric terms such as “girl” to generic “person.”Source: D QUD Generation for Understanding… · p. 15
- Figure 5 shows the distribution of fine-grained labels across all δ-SNLI-TEST atoms.Source: Figure 5 · p. 16
- 92% of generated QUDs correctly captured the critical atom, according to an external annotator.QUDs were generated with gpt-4o-2024-08-06 using 15 exemplars and temperature 0.01; the caption identifies Table 9 as examples.Source: D QUD Generation for Understanding… · pp. 15–16, Table 9 · p. 16
E Validation Instructions
- The authors align validation instructions with those given to annotators for the original defeasible-inference task.Source: E Validation Instructions · p. 16
- Figure 6 presents the number of atoms and critical atoms per δ-SNLI-TEST example.Source: Figure 6 · p. 16
- 52% of updates target all valid possible atoms at once in a single update sentence.Figure 7 shows the proportion of valid atoms used as critical atoms per update.Source: Figure 7 · p. 16