Source-linked AI summary
Reasoning about Entailment with Neural Attention
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Phil Blunsom
TL;DR
RTE systems had relied on engineered NLP pipelines, while earlier end-to-end neural models had not matched strong lexicalized approaches. The paper proposes LSTM models that process sentence pairs, extending them with word-by-word attention for word- and phrase-level entailment reasoning. On SNLI, the attention model achieves state-of-the-art accuracy and outperforms the benchmark LSTM and lexicalized classifier.
Problem
Existing RTE systems relied on engineered NLP pipelines, while earlier end-to-end neural architectures had failed to reach acceptable performance.
Method
The paper uses LSTMs to read sentence pairs and extends them with word-by-word attention over the premise while processing the hypothesis.
Results
83.5% accuracy on SNLI established a new state-of-the-art, exceeding the benchmark LSTM by 2.6 percentage points and a lexicalized classifier by 5.3 percentage points.
Takeaways & Limitations
End-to-end differentiable LSTM models with attention can outperform neural and hand-engineered-feature baselines for textual entailment on SNLI.
Takeaways & Limitations
Two-way attention did not improve RTE performance and may require separate LSTMs, doubling model parameters.
Abstract
from arXiv · showhide
While most approaches to automatically recognizing entailment relations have used classifiers employing hand engineered features derived from complex natural language processing pipelines, in practice their performance has been only slightly better than bag-of-word pair classifiers using only lexical similarity. The only attempt so far to build an end-to-end differentiable neural network for entailment failed to outperform such a simple similarity classifier. In this paper, we propose a neural model that reads two sentences to determine entailment using long short-term memory units. We extend this model with a word-by-word neural attention mechanism that encourages reasoning over entailments of pairs of words and phrases. Furthermore, we present a qualitative analysis of attention weights produced by this model, demonstrating such reasoning capabilities. On a large entailment dataset this model outperforms the previous best neural model and a classifier with engineered features by a substantial margin. It is the first generic end-to-end differentiable system that achieves state-of-the-art accuracy on a textual entailment dataset.
1 INTRODUCTION
Recognizing textual entailment is important for language understanding and downstream NLP tasks, yet existing systems relied heavily on engineered pipelines and neural approaches had not matched strong lexicalized baselines. The paper introduces an LSTM model with word-by-word attention and reports substantial gains on SNLI.
- Motivation: Textual entailment determines whether sentences contradict, are unrelated, or whether a premise entails a hypothesis.The task supports semantic relationship understanding between sentence pairs.
- Motivation: More accurate RTE systems could benefit information extraction, relation extraction, summarization, and machine translation.
- Research gap: Existing state-of-the-art RTE systems relied on engineered NLP pipelines, manually created features, external resources, and specialized components.End-to-end neural architectures had also struggled to reach acceptable RTE performance.
- Approach: The proposed model processes the hypothesis conditioned on the premise, enabling reasoning over entailments between pairs of words and phrases.This contrasts with encoding each sentence independently into a semantic space.
- Results: 83.5% accuracy on SNLI set a new state-of-the-art, while the benchmark LSTM reached 80.9% and exceeded a lexicalized classifier by 2.7 percentage points.Adding word-by-word attention improved the benchmark LSTM by a further 2.6 percentage points.
2 METHODS
The paper uses LSTMs to read sentence pairs jointly for textual entailment, then adds attention mechanisms that support fine-grained reasoning over premise and hypothesis words and phrases.
- Conditional encoding: The model processes the premise and hypothesis with two LSTMs, conditioning the hypothesis representation on the premise encoding.The second LSTM is initialized with the first LSTM’s final cell state.
- Attention: Attention over the premise combines its output vectors with the final sentence-pair representation to produce an attention-weighted premise representation.The attention weights are computed from nonlinear combinations of premise outputs and the transformed final output vector.
- Word-by-word attention: Word-by-word attention uses soft alignment between premise and hypothesis words and phrases to obtain a sentence-pair encoding.Unlike attention for word generation, this mechanism supports fine-grained reasoning for entailment classification.
- Two-way attention: Two-way attention applies the same model in both directions and concatenates the resulting sentence-pair representations for classification.The model attends over the premise conditioned on the hypothesis and over the hypothesis conditioned on the premise.
3 EXPERIMENTS
The experiments evaluate the models on SNLI, a large human-annotated textual entailment corpus, using validation-based hyperparameter selection before test evaluation.
- Dataset: SNLI is substantially larger than existing RTE corpora, and its sentence pairs were produced by human annotators.The paper contrasts SNLI with the smaller and partly heuristically generated SICK corpus.
- Optimization: The experiments optimize models with ADAM using momentum coefficients of 0.9 and 0.999.These settings are reported as the first and second momentum coefficients.
- Results: Table 1 summarizes the models’ results on the SNLI corpus.The supplied table passage provides the corpus-level scope but no numerical entries.
- Evaluation: The authors select the best configuration on the validation set and evaluate only that configuration on the test set.The grid search varies learning rate, dropout, and ℓ2 regularization strength.
4 RESULTS AND DISCUSSION
The attentive LSTM models improve textual-entailment accuracy over benchmark neural and lexicalized systems, while qualitative analyses show word- and phrase-level alignment capabilities with identifiable failure cases.
- Conditional Encoding: 3.3 percentage points: conditioning hypothesis processing on the premise improves accuracy over Bowman et al.’s independent-encoding LSTM.The authors attribute this to information flow from premise processing, enabling more focused checks for contradictions and entailments.
- Results: 2.7 percentage points: the benchmark LSTM outperforms a simple lexicalized classifier and achieves state-of-the-art performance among end-to-end differentiable models.The comparison is reported on the SNLI textual-entailment dataset.
- Attention: 0.9 percentage points and 1.4 percentage points: attention improves over a single LSTM and over conditional encoding with two LSTMs, respectively.The attention model summarizes premise context useful for selecting representations while processing the hypothesis.
- Word-by-word Attention: 1.2 percentage points: word-by-word attention improves over attention based only on the premise’s last output vector.The mechanism supports checking entailment or contradiction for individual hypothesis words and phrases.
- Limitations: Two-way attention does not improve RTE performance, and attention can fail for entirely unrelated sentence pairs.Attention based only on the last premise output may also become more uniformly distributed when longer premises require multiple words for classification.
- Qualitative Analysis: Word-by-word attention aligns reordered words, synonyms, multi-word expressions, irrelevant premise material, deeper semantic relations, and one-to-many relationships.Examples include “airplane”–“aircraft,” “garbage can”–“trashcan,” “snow”–“outside,” and “kids”–“boy” and “girl.”
5 CONCLUSION
General end-to-end differentiable sequence models improve textual entailment performance, with premise attention yielding further gains and new state-of-the-art accuracy on SNLI. Their generality also supports transfer learning, while extending them to larger text units and other sequential entailment problems remains future work.
- End-to-end differentiable LSTM sequence models outperform both a neural baseline and a classifier with hand-engineered features for textual entailment.
- Attention over the premise further improves predictive ability, producing new state-of-the-art accuracy on the Stanford Natural Language Inference corpus.
- The models require no natural-language-specific processing beyond tokenization, making them suitable for transfer learning across corpora and entailment tasks.
- Future work includes scaling the methods to paragraphs and documents with hierarchical attention and applying them to non-natural-language sequential entailment problems.