Source-linked AI summary
Can We Do Interpretable NLI with Graphs Based on Atomic Propositions?
Younes Boufouss, Luc Pommeret, Thomas Gerald, Patrick Paroubek, Sophie Rosset
TL;DR
The paper investigates whether NLI can rely on interpretable graph structures instead of opaque text-based reasoning. It builds a text-free pipeline from atomic propositions, constrained ConceptNet triples, and retrieved knowledge graphs, then measures the resulting accuracy cost. Graphs nearly match text on SNLI but incur a larger gap on ANLI, which the paper attributes to representational limitations.
Problem
NLI systems achieve high accuracy, but the paper asks whether their decisions can instead rely on auditable graph structures without an inherent accuracy trade-off.
Method
The pipeline decomposes sentences into atomic propositions, converts them into constrained ConceptNet triples, adds a retrieved subgraph, and classifies only serialized graphs.
Results
The graph model is 1.9 points below its text counterpart on SNLI and 9 to 14 points below it on ANLI, with the gap attributed to representation rather than data constraints.
Takeaways & Limitations
Graphs can support interpretable NLI, but the fixed representation preserves performance better on simple SNLI examples than on adversarial ANLI examples.
Takeaways & Limitations
Upstream atomization and graph-construction errors propagate unchecked because the classifier has no text fallback, and some cleaning steps can drop source facts.
Abstract
from arXiv · showhide
While Large Language Model (LLM)-based Natural Language Inference (NLI) systems achieve high accuracy, their decision-making processes lack auditable structures. This paper explores whether NLI can be performed using only interpretable, graph-based representations of evidence. We introduce a fully graph-based pipeline where the classifier never directly processes the input text. Instead, sentences are decomposed into atomic propositions, converted into ConceptNet triples via constrained decoding, and represented as three graphs per pair: premise, hypothesis, and a retrieved ConceptNet subgraph. These graphs are then fed into a fine-tuned 0.8-billion-parameter language model. On the SNLI dataset, our pipeline achieves 89.7% accuracy, just 1.9 points below an identically trained text-based model. On ANLI, it matches the published performance of RoBERTa-large on rounds R2 and R3 (48.0% vs. 48.9% and 44.9% vs. 44.4%) but trails by 16 points on R1, resulting in an overall gap of 9 to 14 points compared to its text counterpart. We term this gap the price of interpretability and demonstrate that it stems from representational limitations rather than data constraints. Ablation studies further reveal that graphs and text are complementary: combining both modalities achieves 92.1% accuracy on SNLI.
1 Introduction
This paper asks whether NLI can use interpretable graph representations alone without sacrificing too much accuracy. It introduces a text-free ConceptNet pipeline and measures the resulting price of interpretability across standard and adversarial benchmarks.
- 1 Introduction: The paper asks whether structured graphs alone can achieve competitive NLI performance while preserving interpretability.The central comparison is between text-based models and models constrained to interpretable intermediate structures.
- 1 Introduction: The pipeline decomposes sentences into atomic propositions, converts them into ConceptNet triples, adds retrieved knowledge, and classifies serialized premise, hypothesis, and knowledge graphs without raw text.Its extraction schema uses 28 selected ConceptNet relations and constrained JSON decoding.
- 1 Introduction: The performance gap is attributed to representational limitations rather than data scarcity: adding 600,000 MNLI and FEVER-NLI training pairs improves ANLI by only 0.6 points.The paper also uses ablations to examine ConceptNet knowledge and graph–text combinations.
2 Related Work
Prior work uses external knowledge, explicit structures, and atomic decompositions to improve or explain NLI and related language-understanding systems. This paper differs by making a closed ConceptNet relation inventory and atomic-proposition pipeline decisive for classification.
- 2 Related Work: External knowledge graphs such as ConceptNet and WordNet are established tools for question answering and NLI systems.Prior systems use relations such as synonymy, antonymy, hyperonymy, and hyponymy to bridge semantic gaps.
- 2 Related Work: Atomic proposition decomposition improves interpretability and diagnosis, but does not inherently improve accuracy without model fine-tuning.The present pipeline makes those propositions inputs to constrained triplet extraction.
- 2 Related Work: Structure-only verification systems derive decisions from explicit proofs or semantic triples rather than directly from the original sentences.Examples include natural-logic proof sequences and knowledge-graph-augmented semantic triples.
- 2 Related Work: This work differs through atomic-proposition decomposition, a closed 28-relation ConceptNet inventory, constrained decoding, and end-to-end fine-tuning of the graph-based classifier.Unlike some prior work, it serializes graphs for a language model rather than using a dedicated graph encoder.
3 Proposed Approach
The proposed system converts premise and hypothesis sentences into interpretable graphs, augments them with retrieved ConceptNet knowledge, and classifies only the serialized graphs. Removing raw text makes the intermediate representation both decisive for prediction and directly auditable.
- 3 Proposed Approach: The pipeline maps each premise–hypothesis pair to entailment, neutral, or contradiction through four stages without using the original text at classification time.The stages are atomization, triplet extraction, knowledge enhancement, and classification.
- 3 Proposed Approach: Atomic propositions are converted into subject–relation–object triplets under a constrained JSON schema with 28 ConceptNet relations, forming premise graph P and hypothesis graph H.Processing one minimal semantic unit at a time simplifies extraction and reduces facts per extraction call.
- 3 Proposed Approach: A retrieved graph K augments P and H with ConceptNet bridge edges and one-hop neighborhoods around relevant entities.The retrieval uses the 3.3-million-edge ConceptNet base and connects entities occurring asymmetrically across the two graphs.
- 3 Proposed Approach: The classifier receives the graphs serialized as K, P, H and is fine-tuned on the training data of each NLI benchmark.The classifier is Qwen3.5-0.8B-Base, while triplet extraction uses Qwen3.5-9B under constrained decoding.
- 3 Proposed Approach: Because raw text is removed, graph errors directly affect predictions, while every decision can be traced to a finite list of triplets.This makes the intermediate structure decisive rather than merely an explanation inspected after prediction.
4 Experimental Protocol
The pipeline converts sentences into constrained ConceptNet graphs, retrieves external bridges, serializes three graph sections, and classifies without exposing the original text to the classifier.
- The pipeline constrains each stage: atomic propositions bound triplet extraction, and extracted entities bound ConceptNet retrieval.
- External knowledge retrieval: The retrieved graph K combines direct bridges, two-hop bridges, and informative neighborhoods connecting premise-only and hypothesis-only entities.
- Serialisation: Each pair is serialized as knowledge K, premise P, and hypothesis H, with truncation preferentially removing earlier sections before the hypothesis.
- Auditability: Auditing is finite because every prediction traces to triplets whose proposition, sentence, and retrieved bridge support can be checked without running the model.
5 Datasets and Benchmarks
The evaluation contrasts favorable SNLI image-caption premises with adversarial ANLI passages to expose representation limits, while supplementary datasets test whether more training data closes the gap.
- Benchmark selection: SNLI and ANLI span favorable and adversarial conditions: the gap grows from 1.9 to 14 points as inputs move from captions to adversarial passages.
- Benchmark selection: SNLI contains single-sentence image captions with concrete vocabulary, whereas ANLI uses increasingly difficult adversarial examples written to defeat strong models.
- Data scaling: MNLI and FEVER-NLI add about 600,000 training pairs for data scaling, but are used only as additional training data, never as test sets.
- Evaluation protocol: Accuracy is the evaluation metric, and ANLI test sets contain 1000 to 1200 examples depending on the round.
- Evaluation protocol: The price of interpretability compares raw-text and graph-only Qwen3.5-0.8B models with the same backbone, optimizer, and nominal epoch count.
6 Results and Analysis
The graph pipeline performs strongly on SNLI and on ANLI’s harder rounds, but its interpretability cost widens on adversarial data because triplet representations compress critical detail.
- Main results: 0.897 ± 0.006 SNLI accuracy is achieved by the graph model over three seeds, while the unfine-tuned backbone remains at chance.
- Main results: 0.480 on ANLI R2 and 0.449 on R3 are within or above published RoBERTa-large results, while R1 remains 16 points lower at 0.576 versus 0.738.
- Price of interpretability: The graph model retains 97.9% of text-model accuracy on SNLI but only 80% on ANLI R1, quantifying a larger interpretability cost for adversarial inputs.
- Error analysis: The gap reflects compressed numerical, temporal, and long-premise coreference information in the graph representation.
- Ablations: Removing ConceptNet retrieval changes SNLI accuracy from 0.892 to 0.895, whereas adding graphs to text raises accuracy from 0.916 to 0.921.
- Ablations: Adding MNLI and FEVER-NLI training data improves SNLI from 0.892 to 0.896 but does not eliminate the ANLI gap.
7 Conclusion
The paper demonstrates that graph-only NLI is feasible on SNLI but only partly successful on ANLI, while making predictions traceable to finite lists of triplets. It measures the accuracy cost of this interpretability guarantee and identifies richer target structures as the path toward closing the gap.
- 7 Conclusion: Graph-only NLI is feasible on SNLI but only partly on ANLI: the interpretability cost is 1.9 points on SNLI and 9 to 14 points across ANLI rounds.The same system matches published RoBERTa-large performance on ANLI R2 and R3 without taking text as input.
- 7 Conclusion: Atomic propositions, constrained ConceptNet extraction, and serialized three-graph inputs make each prediction traceable to a finite list of triplets.The classifier never receives a natural-language sentence, so the decision depends on explicitly represented intermediate structures.
- 7 Conclusion: The ConceptNet triplet vocabulary is lossy for numeral mentions, temporal granularity, and fine details of long premises.The paper therefore argues that closing the performance gap requires enriching the target structure rather than merely scaling the classifier.
Limitations
The method has important scope and measurement limitations: upstream extraction errors can propagate unchecked, interpretability and graph quality are not directly evaluated, and several benchmark comparisons remain constrained. These limitations are especially consequential for ANLI and for claims about broader applicability.
- Limitations: The pipeline’s interpretability and graph quality are evaluated only indirectly, without direct human readability or reference-based graph-quality measurements.The authors leave direct evaluation across other domains or through direct approaches for future work.
- Limitations: Upstream errors are unrecoverable because the classifier has no text fallback, allowing atomizer and coreference failures to propagate into predictions.Cleaning can also remove facts through length filtering, proposition caps, or fallback behavior, and the frequency of these losses is unmeasured.
- Limitations: The system is English-only, and ConceptNet’s culturally biased coverage limits the scope of its knowledge representation.The extraction vocabulary, ConceptNet base, and experiments are all in English.
- Limitations: Single-seed text baselines and unequal training budgets make the SNLI interpretability price an upper bound, while the larger ANLI gap is unaffected.The SNLI price is roughly two points and comparable to run-to-run variation; the ANLI gap is 9 to 14 points.
- Limitations: The usefulness of multi-hop ConceptNet retrieval for adversarial ANLI inference remains untested because the no-knowledge ablation was run only on SNLI.SNLI has substantial lexical overlap, unlike the low-overlap configuration targeted by the two-hop pass in ANLI.
- Limitations: The closed 28-relation inventory can lose predicates without faithful mappings, although this loss is nearly invisible on SNLI.Argument structure, adjunction, possession, class membership, and negation are covered, but several meaning families are not.
- Limitations: The representation flattens propositional attitudes and modality, while coreference errors and long-premise truncation account for much of the residual ANLI gap.These issues are frequent in ANLI but rare in SNLI captions.
A Extraction Prompt
The extraction prompt is documented across four figures, covering the task, vocabulary, normal form, examples, and constrained-decoding schema.
- A Extraction Prompt: The extraction prompt is specified by four figures covering the task and 28-relation vocabulary, node and argument normal form, few-shot examples, and JSON schema.Figures 3 through 6 present these components respectively.
B Reproduction Code
The reproduction materials specify the constrained ConceptNet extraction prompt, normalization rules, few-shot examples, JSON schema, datasets, and end-to-end pipeline commands.
- Reproduction resources: Code and atomic SNLI and ANLI datasets are released online, with commands assuming ConceptNet 5.7 English assertions and sequential fine-tuning from an SNLI checkpoint.The cited materials provide the repository and dataset links used to reproduce the pipeline.
- Extraction prompt: The pipeline converts each atomic English proposition into ConceptNet triples using a restricted relation vocabulary and JSON-only output.The prompt requires triples in the form {h, r, t} and permits only the listed ConceptNet relations.
- Node normalization: The prompt normalizes graph nodes by using lowercase lemmas, separating modifiers, and preserving only specified multi-word compounds or proper names.Verbs use bare lemmas, phrasal-verb particles are retained, and non-intersective adjectives remain attached.
- Extraction examples: Argument structure is illustrated with triples assigning properties, capabilities, actions received, locations, and possession to normalized nodes.Examples cover soccer players, chefs cutting bread, and a person training a horse for competition.
- Prompt assets: The reproduction package includes nine few-shot prompt pairs, a constrained-decoder JSON schema, and an end-to-end pipeline figure.The few-shot examples alternate user and assistant turns, while the schema enumerates the extraction relations.