Source-linked AI summary
Learning Natural Language Inference with LSTM
Shuohang Wang, Jing Jiang
TL;DR
NLI asks whether a hypothesis can be inferred from a premise, a task supported by applications such as question answering and semantic search. The paper introduces match-LSTM for word-by-word hypothesis–premise matching rather than sentence-embedding matching. It achieves 86.1% accuracy on SNLI and identifies important mismatches as signals that the model remembers.
Problem
NLI determines whether a hypothesis can be inferred from a premise, but sentence-embedding matching can miss important word- or phrase-level mismatches.
Method
The match-LSTM processes the hypothesis word by word against attention-weighted premise representations, remembering critical matching results and forgetting less important ones.
Results
86.1% accuracy was achieved on the SNLI corpus, outperforming the state of the art.
Takeaways & Limitations
Analyses indicate that the model generally forgets good matches while remembering important mismatches associated with contradiction or neutral relationships.
Takeaways & Limitations
The model requires a large training dataset, and preliminary experiments on the smaller SICK corpus did not produce very good results.
Abstract
from arXiv · showhide
Natural language inference (NLI) is a fundamentally important task in natural language processing that has many applications. The recently released Stanford Natural Language Inference (SNLI) corpus has made it possible to develop and evaluate learning-centered methods such as deep neural networks for natural language inference (NLI). In this paper, we propose a special long short-term memory (LSTM) architecture for NLI. Our model builds on top of a recently proposed neural attention model for NLI but is based on a significantly different idea. Instead of deriving sentence embeddings for the premise and the hypothesis to be used for classification, our solution uses a match-LSTM to perform word-by-word matching of the hypothesis with the premise. This LSTM is able to place more emphasis on important word-level matching results. In particular, we observe that this LSTM remembers important mismatches that are critical for predicting the contradiction or the neutral relationship label. On the SNLI corpus, our model achieves an accuracy of 86.1%, outperforming the state of the art.
1 Introduction
NLI determines whether a hypothesis follows from a premise, and the SNLI corpus enabled large-scale neural approaches. The proposed match-LSTM performs word-by-word matching, emphasizing important mismatches, and reaches 86.1% accuracy on SNLI.
- NLI determines whether a hypothesis can be inferred from a premise and supports applications including question answering, semantic search, and text summarization.
- SNLI contains around 570K sentence pairs labeled entailment, contradiction, or neutral, enabling deep neural network training.
- Previous neural models matched premise and hypothesis sentence embeddings, limiting their ability to represent important word- or phrase-level mismatches.
- The match-LSTM sequentially matches each hypothesis word with an attention-weighted premise representation, remembering critical results and forgetting less important ones.
- 86.1% accuracy was achieved on SNLI, outperforming the state of the art; analyses found that important mismatches tend to be remembered.
2 Model
The model replaces sentence-level matching with sequential, word-by-word matching between the hypothesis and attention-weighted premise representations. An mLSTM retains important matching information while processing the hypothesis and uses the final state for classification.
- Attention: For each hypothesis word, attention weights identify aligned premise representations and produce an attention-weighted premise vector.The attention weight encodes the alignment between a hypothesis word and a premise word, while the weighted representation models premise content relevant to that hypothesis position.
- Matching LSTM: The mLSTM can remember important matching results and forget non-essential ones, allowing critical mismatches to influence prediction.The motivation includes mismatches such as differing sentence subjects, which may be sufficient to indicate contradiction.
- Model architecture: The model processes premise and hypothesis with separate LSTMs, without feeding the premise’s final cell state into the hypothesis LSTM.Matching is performed afterward using the hidden states from both sentence encoders.
- Matching LSTM: The mLSTM receives the concatenation of each attention-weighted premise vector and its corresponding hypothesis hidden state to model cumulative matching.Its hidden state represents the matching between the premise and hypothesis up to the current position.
- Prediction: The final mLSTM hidden state is used to predict the NLI label after the hypothesis has been processed.The resulting state summarizes the matching of the two sentences up to the complete hypothesis.
- Implementation: The implementation adds a NULL premise word for possible alignments, uses GloVe embeddings, and approximates unseen-word embeddings from nearby words within a window of 9.The NULL vector is fixed to zeros; the surrounding-word approximation uses four words on each side.
3 Experiments
Experiments evaluate mLSTM variants on SNLI and compare accuracy with prior and reimplemented attention models. Analyses examine alignment, gate behavior, and which word-level matches are retained for prediction.
- Experiment Settings: SNLI supplies 549,367 training, 9,842 development, and 9,824 test pairs after removing examples lacking annotator consensus.The task uses three-class classification with accuracy as the evaluation metric.
- Experiment Settings: The experiments compare word-by-word attention with mLSTM variants using standard or bidirectional sentence modeling, hidden-state dimensions of 150 or 300, and word embeddings.One variant directly uses pretrained word embeddings instead of LSTM-generated hidden states.
- Main Results: 86.1% accuracy is achieved by mLSTM with d = 300 on SNLI test data, reported as the highest performance on this dataset.Under d = 150, mLSTM reaches 85.7% versus 82.6% for the implemented word-by-word attention model, a statistically significant improvement at the 0.001 level.
- Main Results: 86.0% vs. 85.7% test accuracy shows only a small benefit from bi-LSTM sentence modeling over standard LSTM at d = 150.The paper notes that the bi-LSTM version has much higher complexity.
- Main Results: 85.3% test accuracy is obtained when mLSTM uses pretrained word embeddings instead of LSTM-generated hidden states, remaining better than previously reported state of the art.The result suggests the mLSTM attention architecture works with either type of initial sentence representation.
- Word Alignment: Alignment examples show sensible semantic correspondences, while mismatched alignments such as cat–dog can signal contradiction rather than indicating model failure.Content words aligned to NULL are associated with contradiction or neutral labels.
- Values of Gate Vectors: Input gates generally assign lower values to stop words than content words, with averages of 0.287 and 0.347, respectively.The analyses conclude that mLSTM differentiates important from less important word-level matching results.
- Values of Gate Vectors: Forget gates are lighter for entailment and generally darker for contradiction and neutral examples, consistent with retaining important mismatches for classification.The paper hypothesizes that good matches are forgotten, while mismatches useful for contradiction or neutral predictions are remembered.
4 Related Work
Prior NLI work spans lexical, syntactic, formal-logic, and neural approaches. The proposed method extends neural attention by using premise representations to sequentially match hypothesis words.
- NLI methods range from lexical classifiers to approaches using syntax, explicit alignment, formal logic, and neural sentence representations.
- Neural attention improved premise representations by incorporating alignment between the premise and hypothesis before sentence-level matching.
- The proposed approach uses neural attention to derive premise representations that sequentially match individual hypothesis words.
- Other SNLI systems reported 81.5% accuracy with Skip-Thought models and 82.1% with tree-based CNN sentence encoders.
5 Conclusions and Future Work
The paper introduces match-LSTM for word-by-word premise–hypothesis matching and reports state-of-the-art SNLI performance. Analyses indicate that the model retains mismatches associated with contradiction or neutrality, but its many parameters require large training data.
- The match-LSTM processes hypothesis words sequentially against attention-weighted premise representations, using its final hidden state for relationship prediction.
- The mLSTM outperformed the state-of-the-art SNLI performance reported at the time.
- Gate analyses found that important mismatches indicating contradiction or neutrality are remembered and passed on for prediction.
- The model requires a large training dataset because it learns many parameters, and preliminary SICK experiments did not perform well.