Source-linked AI summary
Enhanced LSTM for Natural Language Inference
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, Diana Inkpen
TL;DR
Natural language inference is difficult but important for natural language understanding, and large annotated datasets make neural approaches feasible. The paper enhances sequential and recursive inference models with syntactic information, achieving 88.6% accuracy on SNLI.
Problem
Modeling inference in human language is challenging, despite its importance as a condition for true natural language understanding.
Method
The paper enhances chain-based sequential inference models and incorporates recursive architectures with syntactic parsing information into local inference modeling and inference composition.
Results
88.6% accuracy is achieved on the SNLI benchmark, with syntactic tree-LSTMs improving the 88.0% ESIM model.
Takeaways & Limitations
Enhanced sequential models can outperform previous NLI models, while syntactic parsing further improves an already strong model.
Takeaways & Limitations
Future work identifies external lexical resources and closer modeling of negation as ways to increase word-level inference coverage and help contradiction detection.
Abstract
from arXiv · showhide
Reasoning and inference are central to human and artificial intelligence. Modeling inference in human language is very challenging. With the availability of large annotated data (Bowman et al., 2015), it has recently become feasible to train neural network based inference models, which have shown to be very effective. In this paper, we present a new state-of-the-art result, achieving the accuracy of 88.6% on the Stanford Natural Language Inference Dataset. Unlike the previous top models that use very complicated network architectures, we first demonstrate that carefully designing sequential inference models based on chain LSTMs can outperform all previous models. Based on this, we further show that by explicitly considering recursive architectures in both local inference modeling and inference composition, we achieve additional improvement. Particularly, incorporating syntactic parsing information contributes to our best result---it further improves the performance even when added to the already very strong model.
1 Introduction
Natural language inference is a difficult but foundational problem for language understanding, made more tractable by large annotated datasets and neural models. The paper improves sequential inference models and adds syntactic information, reaching 88.6% accuracy.
- Natural language inference is presented as a challenging problem central to human and artificial intelligence and necessary for true natural language understanding.
- The SNLI dataset contains 570,000 human-written English sentence pairs manually labeled by multiple subjects, enabling training of more complex inference models.
- 88.0% accuracy is achieved by the paper’s enhanced sequential inference model on the SNLI benchmark.
- 88.6% accuracy is achieved after explicitly encoding parsing information with recursive networks in local inference modeling and inference composition.
- The paper frames syntactic information as an additional improvement within neural inference models trained on relatively large data.
2 Related Work
Prior NLI research progressed from small-dataset conventional methods to neural and attention-based architectures. The paper argues that sequential inference models remain underexploited and can be strengthened with recursive architectures and syntax.
- Early natural language inference work used relatively small datasets and more conventional methods, including research on recognizing textual entailment.
- The SNLI dataset enabled neural models to use substantially more annotated sentence pairs, while attention-based models learned to focus on specific sentence regions.
- Munkhdalai and Yu’s networks combined sequential LSTM encoding, recursive networks, and complex attention combinations, gaining about 0.5% over Parikh et al.
- The paper identifies sequential inference networks as insufficiently exploited for NLI and proposes enhancing chain-based models before adding recursive syntactic architectures.
3 Hybrid Neural Inference Models
The model combines sequential and syntactic components to encode inputs, model local inference, and compose an overall inference decision. ESIM uses bidirectional sequential networks, while HIM adds tree-LSTM representations and averages predicted probabilities.
- The architecture has three components: input encoding, local inference modeling, and inference composition.
- Sequential inference: ESIM uses BiLSTMs to encode each sentence, represent words with context, and compose local inference information.BiLSTMs run in both directions, concatenating forward and backward hidden states for each time step.
- Syntactic inference: Tree-LSTMs extend chain LSTMs recursively, using child hidden states to compose phrase- and clause-level inference information.Tree-LSTM inputs differ by layer: leaf nodes encode word embeddings, whereas inference composition nodes encode local inference information.
- Local inference modeling: Soft alignment uses attention to associate relevant premise and hypothesis subcomponents during local inference modeling.The model computes attention from similarities between contextual hidden-state representations.
- More complicated multilayer-perceptron relationships and feedforward interaction modeling did not improve heldout inference accuracy.
- Inference composition: HIM incorporates parse-based tree-LSTM networks into ESIM and averages their predicted probabilities to produce the final label.The paper reports that parsing information complements ESIM and further improves performance.
4 Experimental Setup
Experiments use the SNLI corpus and evaluate three-way natural language inference classification with accuracy. Training uses Adam, dropout, pretrained GloVe embeddings, and 300-dimensional representations.
- Data: The SNLI corpus contains entailment, contradiction, and neutral premise-hypothesis relationships; the disagreement category is removed.
- Evaluation: Accuracy is the evaluation metric, and Stanford PCFG Parser 3.5.3 supplies the parse trees used in the models.
- Training: Training uses Adam with learning rate 0.0004, batch size 32, and 300-dimensional hidden states, tree-LSTM states, and word embeddings.
- Training: Dropout is 0.5, pretrained 300-D GloVe 840B vectors initialize embeddings, and all vectors are updated during training.
5 Results
On SNLI, ESIM achieves 88.0% accuracy and outperforms previous models, while ensembling it with syntactic tree-LSTMs raises accuracy to 88.6%. Ablations and analysis identify attention, inference-composition design, and syntactic structure as important contributors.
- Overall performance: 88.0% accuracy allows ESIM to outperform previous models, including substantially more complicated architectures.The enhanced sequential model uses sequential encoding based on chain networks.
- Overall performance: 88.6% accuracy is achieved by the final ensemble of ESIM and syntactic tree-LSTMs on SNLI.The ensemble significantly improves over ESIM alone and is reported as the best observed SNLI result.
- Ablation analysis: Replacing pooling with summation lowers accuracy to 87.1%, removing local-inference difference and product features lowers it to 87.0%, and replacing BiLSTMs yields 87.3% or 86.3%.These changes affect inference composition, local inference enhancement, and input encoding, respectively.
- Ablation analysis: Removing premise-based attention lowers accuracy to 87.2%, while removing hypothesis-based attention lowers it to 86.5%.The results indicate that both attention directions matter, with hypothesis-based attention having the larger effect.
- Overall performance: 87.8% accuracy is achieved by the stand-alone syntactic tree-LSTM, comparable to ESIM, while their oracle combination reaches 91.7%.The oracle selects the correct answer when either model is correct; training takes about 40 hours for tree-LSTM versus 6 hours for ESIM.
- Further analysis: Tree-LSTM analysis shows syntactic nodes can drive a correct contradiction decision when their aligned premise information receives larger input-gate values.In contrast, sequential BiLSTM attention does not assign comparable importance to the words sitting and down.
6 Conclusions and Future Work
The paper reports state-of-the-art SNLI results from neural inference models, first using an enhanced sequential model and then gaining further improvement from recursive architectures and syntactic parsing. Future work targets broader word-level inference coverage and closer modeling of negation.
- The proposed neural network models achieve the best results reported on the SNLI benchmark.
- The enhanced sequential inference model outperforms previous models, including those with more complicated network architectures.This suggests that sequential inference models had not been fully exploited.
- Recursive architectures improve performance in local inference modeling and inference composition, with syntactic parsing improving the already strong model further.
- Future Work: Future work includes external resources to increase coverage of word-level inference relations and closer neural modeling of negation for contradiction detection.