Source-linked AI summary
e-SNLI: Natural Language Inference with Natural Language Explanations
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, Phil Blunsom
TL;DR
Machine-learning models need explanations that are interpretable and robust, but existing NLI resources lacked human-authored free-form justifications. The paper introduces e-SNLI, extends SNLI with such explanations, and tests models that generate or use them, finding a trade-off between label performance and explanation quality while supporting broader representation and transfer experiments.
Problem
Existing machine-learning datasets lacked free-form explanations that could support interpretable decisions and learning from human justifications.
Method
The paper augments SNLI with human-annotated natural-language explanations and develops models that incorporate explanations during training or generate them at test time.
Results
Explanation-first models reduced SNLI accuracy to 81.59% or 81.71% but produced 49.8% or 64.27% correct explanations, with attention achieving the higher explanation score.
Takeaways & Limitations
e-SNLI supports full-sentence justifications, explanation-informed universal sentence representations, and transfer to out-of-domain NLI datasets.
Takeaways & Limitations
The collected explanations had a 9.62% overall error rate, rising to 19.55% for entailment examples.
Abstract
from arXiv · showhide
In order for machine learning to garner widespread public adoption, models must be able to provide interpretable and robust explanations for their decisions, as well as learn from human-provided explanations at train time. In this work, we extend the Stanford Natural Language Inference dataset with an additional layer of human-annotated natural language explanations of the entailment relations. We further implement models that incorporate these explanations into their training process and output them at test time. We show how our corpus of explanations, which we call e-SNLI, can be used for various goals, such as obtaining full sentence justifications of a model's decisions, improving universal sentence representations and transferring to out-of-domain NLI datasets. Our dataset thus opens up a range of research directions for using natural language explanations, both for improving models and for asserting their trust.
1 Introduction
The paper argues that robust, interpretable NLI models need natural-language explanations and introduces e-SNLI to support learning and generating them. The dataset combines SNLI examples with human explanations and enables experiments on explanation quality, sentence representations, and transfer.
- Models trained only for accuracy can rely on shallow input statistics and become brittle under adversarial attacks.A cited classifier achieved 94% accuracy while spuriously using document-header words.
- Natural-language explanations are intended to make model decisions comprehensible to users and easier for humans to annotate than formal explanations.
- e-SNLI adds human-annotated explanations to the Stanford Natural Language Inference dataset for training and generating free-form textual justifications.
- The authors evaluate whether explanations resist spurious correlations, support joint label-and-explanation prediction, improve sentence representations, and transfer to out-of-domain NLI datasets.
2 Background
Textual entailment classifies the relation between a premise and hypothesis as entailment, contradiction, or neutral. Although SNLI enabled effective NLI modeling, prior work raised concerns that models could exploit hypothesis-specific artifacts rather than understand the premise.
- Textual entailment labels a premise–hypothesis pair as entailment, contradiction, or neutral.
- SNLI contains 570K human-generated premise, hypothesis, and label triples and has driven many neural NLI models.
- Training universal sentence representations on SNLI was reported as more efficient and accurate than training on much larger unsupervised datasets.
- Prior analysis found that SNLI models may exploit hypothesis-word artifacts, including words associated with neutral, entailment, or contradiction labels.
3 Collecting explanations
The authors collected e-SNLI explanations through structured crowdsourcing, asking annotators to identify essential words and formulate self-contained justifications. Quality checks and reannotation reduced templated or obviously invalid responses, but entailment explanations remained the most error-prone.
- e-SNLI explanations were collected from Amazon Mechanical Turk annotators.
- Annotators highlighted essential premise or hypothesis words before writing explanations focused on the elements inducing entailment, neutrality, or contradiction.
- Quality control required explanations to contain at least three tokens, avoid copying the premise or hypothesis, and pass template detection.
- The collection used one explanation per training example and three explanations per validation and test example, involving 6,325 workers.
- 9.62% of final explanations were erroneous overall, with the highest error rate for entailment at 19.55%.Neutral and contradiction error rates were 7.26% and 9.38%, respectively.
4 Experiments
The experiments test whether e-SNLI explanations can expose artifact reliance, justify predictions, improve sentence representations, and transfer across NLI datasets. Explanation-based models often preserve label performance or improve downstream representations, but explanation transfer remains difficult.
- 4.1 PREMISEAGNOSTIC: 6.834% of the first 100 hypothesis-only explanations were correct, versus 66% correct labels, making explanation prediction approximately 10 times harder.The comparison tests whether models can exploit SNLI artifacts to generate explanations as easily as labels.
- 4.2 PREDICTANDEXPLAIN: e-INFERSENT adds a one-layer LSTM explanation decoder to InferSent and conditions explanations on the predicted label at test time.The decoder receives the feature vector as its initial state and at every timestep; the label is prepended to the explanation.
- 4.3 EXPLAINTHENPREDICT: EXPLAINTHENPREDICT raises correct-explanation rates to 49.8% with seq2seq and 64.27% with attention, while test accuracy falls to 81.59% and 81.71%.The attention model also achieves lower perplexity and higher BLEU than the seq2seq model, while the authors associate the setup with greater trust in correct predictions.
- 4.4 REPRESENT: e-INFERSENT significantly outperforms InferSent on 4 downstream tasks and is significantly outperformed on only 1, whereas an autoencoder baseline loses on 6 tasks.The authors conclude that training with explanations helps learn overall better sentence representations.
- 4.5 TRANSFER: Without fine-tuning, explanation transfer is weak: correct explanations reach 30.64% on SICK-E and only 1.92% on MultiNLI.SICK-E explanations were generally on-topic even when wrong, while MultiNLI explanations were generally nonsense or off-topic.
5 Related work
Prior work mainly provides extractive or attention-based interpretability, while e-SNLI targets full-sentence natural-language justifications for textual entailment. The section situates this approach among explanation datasets and analyses of SNLI model behavior.
- Interpretability: Attention-based models can improve downstream performance, but soft attention often fails to isolate human-interpretable inputs.
- Interpretability: Extractive and attention-based methods cannot provide full-sentence explanations or capture fine-grained entailment relations and asymmetries.The example distinguishes learning that a woman is a person from incorrectly reversing that relation.
- Natural language explanations: Unlike the cited extractive approaches, e-SNLI combines human-readable explanations with textual-entailment modeling.
- Natural language explanations: e-SNLI builds a neural network that directly produces full-sentence natural-language justifications.The authors identify the lack of suitable datasets as a reason this direction has received little prior work.
- Natural language explanations: Prior datasets also collect natural-language explanations for visual question answering, activity recognition, and mathematical problem solving.
- Breaking natural language inference: Analyses of SNLI models examine whether their representations capture compositionality beyond word-level behavior.The cited work reports limited compositionality in InferSent sentence embeddings and reliance on shallow patterns in downstream models.
6 Conclusions and future work
The paper concludes that e-SNLI supports full-sentence justifications, explanation-guided sentence representations, and transfer studies, while also enabling further research using highlighted words.
- Conclusions: e-SNLI is a large natural-language explanation dataset for textual entailment.The authors establish recurrent-neural-network baselines for incorporating and generating explanations.
- Conclusions: The authors use e-SNLI to study human-interpretable justifications, explanation-guided universal sentence representations, and transfer to out-of-domain NLI datasets.
- Future work: The authors hope future models will outperform their straightforward recurrent-neural-network baselines and explore additional uses of the dataset.
- Future work: The dataset records highlighted words that can support supervision and evaluation for attention or post-hoc explanation models.These models use explanations consisting of subsets of the input.
General templates
The general templates express the relation between a premise and a hypothesis by explicitly identifying each sentence and its role.
- General templates: One template states the premise first and identifies the second sentence as the hypothesis.
- General templates: Together, the templates provide reciprocal wording for describing the premise–hypothesis relationship.
- General templates: A second template states the hypothesis first and identifies the first sentence as the premise.
Entailment templates
The appendix lists reusable explanation templates for entailment, neutrality, and contradiction. They express paraphrase, implication, uncertainty, and incompatibility between premise and hypothesis statements.
- Entailment templates: Entailment templates describe implication from the premise to the hypothesis.
- Entailment templates: Several templates frame entailment as saying one sentence in another way, including reciprocal premise–hypothesis formulations.
- Entailment templates: Other entailment templates characterize the hypothesis as a rephrasing, less-specific version, clarification, or synonym of the premise.
- Entailment templates: Additional entailment templates state that the two sentences express the same meaning or can be phrased in equivalent ways.
- Neutral templates: Neutral templates state that the premise does not imply the hypothesis or that the hypothesis cannot be inferred or assumed.
- Neutral templates: Further neutral templates express uncertainty by saying the hypothesis cannot be assumed, inferred, or established by the premise.
- Contradiction templates: Contradiction templates contrast premise and hypothesis statements as different, contradictory, or mutually exclusive.
B Architecture of EXPLAINTHENPREDICTATTENTION
EXPLAINTHENPREDICTATTENTION uses separate but identical attention modules for the premise and hypothesis, projecting encoder states and decoder context before computing attention. The resulting weights form weighted premise and hypothesis representations that, with the previous word embedding, drive decoding.
- Module structure: The model uses two identical but separate attention modules, one for the premise and one for the hypothesis, with 84 attended tokens each.The fixed length matches the maximum sentence length in SNLI.
- Attention computation: It projects each timestep of the premise and hypothesis encoder representations before calculating attention.The architecture applies three couples of linear projections followed by tanh nonlinearities.
- Attention computation: At each decoding timestep, the projected decoder context is compared by dot product with projected premise and hypothesis timesteps.The decoder hidden state serves as the attention context and is separately projected before comparison.
- Attention computation: Softmax converts the non-normalized comparison scores into the final attention weights.These weights determine how the model aggregates information from the premise and hypothesis.
- Decoder input: The model applies additional projections to premise and hypothesis token embeddings before computing their weighted sums.At each decoding step, the weighted premise and hypothesis vectors are concatenated with the previous word embedding and input to the decoder.