Source-linked AI summary
Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
Yang Liu, Chengjie Sun, Lei Lin, Xiaolong Wang
TL;DR
The paper addresses sentence-based recognition of textual entailment and develops a biLSTM encoder with Inner-Attention. It first mean-pools word representations, then uses that sentence representation to attend within the same sentence. On SNLI, the model improved state-of-the-art sentence encoding performance by nearly 2% without parameter tuning.
Problem
RTE requires determining whether a hypothesis is entailed, contradicted, or neutral relative to premises, while sentence encoding remains central to one major class of deep-learning approaches.
Method
The model applies mean pooling over word-level biLSTM outputs, then uses the resulting sentence representation for Inner-Attention over words in that same sentence.
Results
Nearly 2% improvement over the state-of-the-art sentence encoding-based model was achieved without parameter tuning.
Takeaways & Limitations
Inner-Attention produces more focused and accurate sentence representations by re-weighting words according to their importance.
Abstract
from arXiv · showhide
In this paper, we proposed a sentence encoding-based model for recognizing text entailment. In our approach, the encoding of sentence is a two-stage process. Firstly, average pooling was used over word-level bidirectional LSTM (biLSTM) to generate a first-stage sentence representation. Secondly, attention mechanism was employed to replace average pooling on the same sentence for better representations. Instead of using target sentence to attend words in source sentence, we utilized the sentence's first-stage representation to attend words appeared in itself, which is called "Inner-Attention" in our paper . Experiments conducted on Stanford Natural Language Inference (SNLI) Corpus has proved the effectiveness of "Inner-Attention" mechanism. With less number of parameters, our model outperformed the existing best sentence encoding-based approach by a large margin.
1 Introduction
RTE determines whether a hypothesis is entailed by, contradicts, or remains neutral with respect to premises. The paper focuses on sentence encoding with biLSTM and Inner-Attention, reporting nearly a 2% improvement over the state of the art without parameter tuning.
- RTE classifies whether a hypothesis is inferred true, inferred false, or truth-unknown given premises.
- Deep-learning RTE methods use either sentence encoding or direct sentence matching, with sentence representations central to the former.
- Bidirectional LSTMs incorporate both previous and future token context, addressing limitations of unidirectional recurrent models.
- The proposed framework avoids feature engineering and external resources by encoding premises and hypotheses with biLSTMs and Inner-Attention.
- Nearly 2% improvement over the state-of-the-art sentence encoding-based model was achieved without parameter tuning.
2 Our approach
The approach uses Siamese biLSTM sentence encoders followed by matching and classification. Sentence representations are refined in two stages: mean pooling first, then Inner-Attention over the same sentence.
- The model treats RTE as supervised three-way classification using two identical, weight-sharing sentence encoders.
- Sentence encoding: Mean pooling over word-level biLSTM outputs produces an initial sentence vector before attention refinement.
- Sentence encoding: Inner-Attention uses the first-stage sentence representation to attend to words within that same sentence.
- Sentence encoding: The attention mechanism represents biLSTM outputs as Y, mean-pooled output as Rave, attention weights as α, and the weighted representation as Ratt.
- Sentence matching: Premise and hypothesis vectors are matched using concatenation, element-wise product, and element-wise difference before SoftMax classification.
3 Experiments
Experiments evaluated the model on the SNLI corpus, comparing input strategies and sentence-encoding baselines while analyzing Inner-Attention visualizations. Differentiating inputs and doubling hypotheses improved performance, and attention emphasized semantically rich words.
- Experimental Setup: SNLI contains 570K premise–hypothesis pairs with three labels: Entailment, Contradiction, and Neutral.The standard split used 550k training, 10k validation, and 10k test samples.
- Input Strategies: Four input modifications were investigated: inverting premises, doubling premises, doubling hypotheses, and differentiating inputs.Differentiating inputs removes words shared by the premises and hypotheses.
- Input Strategies: Doubling hypotheses and differentiating inputs both improved the model’s performance.The paper attributes these effects to compensating for sentence-length differences and focusing on sentence-specific content.
- Qualitative Analysis: Differentiating inputs helped address examples with many overlapping words, such as a contradiction pair that the model otherwise tended to label Neutral or Entailment.The example contrasts a pleasant conversation about a photograph with a heated discussion about Canon.
- Qualitative Analysis: Inner-Attention visualizations showed greater attention on nouns, verbs, and adjectives than on function words.Unlike mean pooling, attention re-weighted words according to their importance and produced more focused sentence representations.
4 Conclusion and Future work
The paper concludes that bidirectional LSTMs with Inner-Attention produce more accurate sentence representations for textual entailment. It also introduces a simple input strategy and identifies extensions to other sentence-matching tasks as future work.
- Conclusion: The proposed model uses Inner-Attention within each sentence, allowing the sentence to attend to its own words without another sentence’s information.The mechanism is presented as producing more accurate sentence representations through attention vectors.
- Conclusion: A simple effective input strategy further boosted the model’s results.The conclusion identifies this strategy as an additional contribution beyond the attention mechanism.
- Future Work: Future work includes applying the architecture to Question Answering, Paraphrase, and Sentence Text Similarity tasks.The paper also proposes trying additional heuristic matching methods with the sentence vectors.