Source-linked AI summary
Shortcut-Stacked Sentence Encoders for Multi-Domain Inference
Yixin Nie, Mohit Bansal
TL;DR
NLI asks whether a hypothesis follows logically from a premise, and SNLI and MultiNLI also support evaluation of general language representation learning. The paper encodes sentences with shortcut-stacked biLSTMs and fine-tuned embeddings, reporting top single-model MultiNLI results and a new encoding-based SNLI state of the art.
Problem
NLI asks whether a hypothesis sentence can be logically inferred from a premise, and SNLI and MultiNLI also serve as resources for evaluating general language representation learning.
Source: §1 · p. 1Method
The model uses shortcut-stacked bidirectional LSTM sentence encoders with fine-tuned word embeddings, then classifies the relationship between the two sentence vectors.
Source: §1 · p. 1Results
The encoder was the top non-ensemble single-model result in the EMNLP RepEval 2017 Multi-NLI Shared Task and achieved a new state-of-the-art encoding-based result on SNLI.
Source: §1 · p. 2Takeaways & Limitations
The results suggest that simple shortcut-connected stacked layers are generally effective for sentence encoders.
Source: §4.2 · p. 5The authors’ reimplementation of the biLSTM-Max baseline scored 0.7% better, likely because of classifier and optimizer settings.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
We present a simple sequential sentence encoder for multi-domain natural language inference. Our encoder is based on stacked bidirectional LSTM-RNNs with shortcut connections and fine-tuning of word embeddings. The overall supervised model uses the above encoder to encode two input sentences into two vectors, and then uses a classifier over the vector combination to label the relationship between these two sentences as that of entailment, contradiction, or neural. Our Shortcut-Stacked sentence encoders achieve strong improvements over existing encoders on matched and mismatched multi-domain natural language inference (top non-ensemble single-model result in the EMNLP RepEval 2017 Shared Task (Nangia et al., 2017)). Moreover, they achieve the new state-of-the-art encoding result on the original SNLI dataset (Bowman et al., 2015).
1 Introduction and Background
The paper proposes a sequential sentence encoder for natural language inference and reports strong results on MultiNLI and SNLI. It situates the approach among sentence encoders that represent sentences individually, rather than jointly modeling sentence pairs.
- NLI asks whether a hypothesis sentence can be logically inferred from a premise, and SNLI and MultiNLI also serve as resources for evaluating general language representation learning.Source: §1 · p. 1
- Sentence encoders represent sentences without information from the other sentence, unlike joint pairwise models that use cross-features; encoders may be tree-based or sequential.Source: §1 · p. 1
- The proposed encoder stacks bidirectional LSTM-RNN layers with shortcut connections and fine-tuned word embeddings, then classifies the relationship between the two sentence vectors.Source: §1 · p. 1, Figure 1 · p. 2
- The encoder reports strong gains in matched and mismatched MultiNLI, top non-ensemble single-model performance in the EMNLP RepEval 2017 Shared Task, and a new encoding-based SNLI state of the art.Source: §1 · pp. 1–2
2 Model
The model encodes each premise and hypothesis separately into fixed-length vectors using a shared sentence encoder, then classifies their relationship. The encoder stacks bidirectional LSTM layers with shortcut connections and max pooling.
2 Model
- The same encoder maps each premise and hypothesis independently into a shared vector space, and a classifier predicts their three-way relationship.The encoding-based setup does not use information from the other sentence during sentence encoding.Source: §2 · p. 2
2.1 Sentence Encoder
- Each biLSTM layer receives the word embeddings and outputs from all preceding layers, rather than only the immediately preceding layer.The final layer’s outputs are row-max-pooled into a vector representation for the source sentence.Source: §2.1 · pp. 2–3
- Pre-trained word embeddings are fine-tuned end-to-end using NLI supervision.Source: §2.1 · p. 3
2.2 Entailment Classifier
- The classifier combines the premise and hypothesis vectors using concatenation, element-wise distance, and element-wise product, then applies an MLP and softmax.Source: §2.2 · p. 3
3 Experimental Setup
The experiments train on MultiNLI with a resampled SNLI supplement and use separate development sets for model selection; SNLI test results use SNLI-only training.
3.1 Datasets
- MultiNLI training was combined with 15% randomly selected SNLI training samples, resampled each epoch, for the shared-task models.Source: §3.1 · p. 3
- Matched and mismatched MultiNLI development sets guided model selection, while SNLI test results used only the SNLI training set.Source: §3.1 · p. 3
3.2 Parameter Settings
- Training minimized cross-entropy with Adam and batch size 32, using pretrained 300D GloVe 840B vectors to initialize word embeddings.The starting learning rate was 0.0002, halved every two epochs; the classifier MLP had 1600 hidden units and dropout rate 0.1.Source: §3.2 · pp. 3–4
4 Results and Analysis
The shortcut-stacked encoder leads the reported Multi-NLI single-model comparison and sets a new encoding-based SNLI result, while ablations identify gains associated with depth, shortcuts, and embedding fine-tuning.
4.1 Ablation Analysis Results
- Adding biLSTM layers improved accuracy, with around 2% gains over the single-layer Conneau et al. (2017) model on both matched and mismatched settings.Experiments covered up to three layers with dimensions of 512, 1024, and 2048.Source: §4.1 · p. 4
- Shortcut connections added around 1.5% accuracy over the full three-layer stacked-RNN model, indicating that stacking alone was insufficient.The higher layer connects to outputs and original inputs from prior layers.Source: §4.1 · p. 4
- Embedding fine-tuning improved in-domain and cross-domain results in a smaller 128+256 two-layer biLSTM, and a two-layer ReLU classifier was preferred for the strongest encoder.Source: §4.1 · p. 4
4.2 Multi-NLI and SNLI Test Results
- Around 3% improvement over the 1-layer biLSTM-Max Encoder on Multi-NLI, alongside the top single-model RepEval result and a new best encoding-based SNLI result.The reported biLSTM-Max baseline is a reimplementation that is 0.7% better than Conneau et al. (2017), likely due to classifier and optimizer settings.Source: §4.2 · pp. 4–5
5 Conclusion
The paper develops a Shortcut-Stacked Sentence Encoder for natural language inference and reports leading single-model Multi-NLI performance and a result surpassing prior SNLI encoders. The authors identify evaluation on other semantic tasks as future work.
5 Conclusion
- The paper develops a Shortcut-Stacked Sentence Encoder through exploration of biLSTM-RNN layer combinations and connections.Source: §5 · p. 5
- The model is the top single result in the EMNLP RepEval 2017 Multi-NLI Shared Task and surpasses state-of-the-art encoders on SNLI.Source: §5 · p. 5
- The authors are evaluating shortcut-stacked sentence encoders on other semantic tasks.Source: §5 · p. 5
6 Addendum: Shortcut vs. Residual
Additional SNLI experiments replaced shortcut connections with residual connections, which achieved similar accuracy with fewer parameters. The residual setup combined each layer’s inputs with the word embedding and summed outputs of prior layers.
6 Addendum: Shortcut vs. Residual
- Residual connections achieved similar accuracies with fewer parameters than shortcut connections in additional SNLI experiments.Table 6 reports SNLI results for residual-stacked models with three biLSTM layers and one MLP layer.Source: §6 · p. 5, Table 6 · p. 5
- Each next biLSTM layer received the word embedding concatenated with the sum of outputs from all previous layers.Source: §6 · p. 5
- The shortcut-model comparison used three stacked biLSTM layers and one 800-unit MLP layer, with settings tuned on the SNLI development set.Source: §6 · p. 5