Source-linked AI summary
Sentence-State LSTM for Text Representation
Yue Zhang, Qi Liu, Linfeng Song
TL;DR
BiLSTMs’ sequential computation limits parallelism and leaves local ngrams and long-range dependencies insufficiently modeled. S-LSTM instead maintains parallel word and sentence-level states that exchange local and global information, achieving strong results across classification and sequence labelling while using more memory.
Problem
BiLSTMs have non-parallel computation, do not explicitly model local ngrams, and are weaker at capturing long-range dependencies in longer sentences.
Method
S-LSTM models parallel word-level states and a global sentence-level state, recurrently exchanging local and non-local information.
Results
S-LSTM outperforms same-parameter BiLSTMs across classification and sequence-labelling tasks, with better accuracies and faster computation.
Takeaways & Limitations
S-LSTM is presented as a useful addition to the neural toolbox for encoding sentences.
Takeaways & Limitations
S-LSTM has relatively larger memory consumption than BiLSTM because its computation is performed in parallel.
Abstract
from arXiv · showhide
Bi-directional LSTMs are a powerful tool for text representation. On the other hand, they have been shown to suffer various limitations due to their sequential nature. We investigate an alternative LSTM structure for encoding text, which consists of a parallel state for each word. Recurrent steps are used to perform local and global information exchange between words simultaneously, rather than incremental reading of a sequence of words. Results on various classification and sequence labelling benchmarks show that the proposed model has strong representation power, giving highly competitive performances compared to stacked BiLSTM models with similar parameter numbers.
1 Introduction
The paper motivates S-LSTM as a parallel alternative to sequential BiLSTMs, exchanging local and global information across word states. It reports competitive classification and sequence-labelling results with faster computation.
- BiLSTMs process sentences sequentially, creating non-parallel computation that can become a computational bottleneck.
- BiLSTMs do not explicitly model local ngrams and have weaker power for capturing long-range dependencies in longer sentences.
- S-LSTM models all word hidden states simultaneously, together with an overall sentence-level state, rather than reading one word at a time.
- Word and sentence-level states recurrently exchange local and non-local information; the sentence-level vector can represent the sentence for classification.
- S-LSTM gives better accuracies than same-parameter BiLSTMs on classification and sequence-labelling tasks while being faster.
2 Related Work
Related work contrasts S-LSTM with CNNs, attention, hierarchical models, and graph-based recurrent networks. S-LSTM combines parallel processing with recurrent local and global information exchange for sentence encoding.
- CNNs parallelize sentence encoding efficiently but provide fixed-sized local ngram features, while pooling can lose information.
- Attention enables parallelization and can resemble S-LSTM’s sentence-level state, but S-LSTM additionally supports communication between neighboring words.
- S-LSTM incrementally refines sentence representations through recurrent state transitions, resembling hierarchical attention and stacked CNNs.
- S-LSTM uses parallel graph states to integrate word-level and sentence-level information, unlike the sequential DAG LSTM.
3 Model
The model represents a sentence with word-level and global sentence-level states, updating them recurrently through local and global information exchange. Unlike BiLSTM’s sequential state flow, S-LSTM updates all word states simultaneously and supports task-specific sentence or word representations.
- Sentence-State Representation: S-LSTM represents each word with a hidden vector and the whole sentence with a global sentence-level vector.Word-level vectors encode each word under sentential context, while the global vector encodes whole-sentence features.
- Baseline BiLSTM: BiLSTM processes words recurrently in forward and backward directions using one hidden state per direction, whereas stacked BiLSTMs add layers for greater representation power.The two directional components consume the input in opposite orders, and stacked layers use lower-layer hidden vectors as upper-layer inputs.
- State Transitions: S-LSTM exchanges information between neighboring word states and between every word state and the sentence-level state during each recurrent step.This structure combines local context exchange with global sentence-level communication.
- Information Flow: Unlike BiLSTM’s end-to-end information flow, S-LSTM enables simultaneous bidirectional word communication and global communication through the sentence-level state.Its word states capture increasingly larger n-gram contexts, while the global state communicates with all word states.
- Model Extensions and Tasks: S-LSTM can vary its communication window and number of sentence-level nodes, and supports classification through g or sequence labelling through each hi.The paper studies larger windows and multiple sentence-level nodes empirically.
4 Experiments
Experiments evaluate S-LSTM against BiLSTM and other encoders on classification and sequence labelling. Across benchmarks, S-LSTM combines competitive or better accuracy with faster computation, while using more memory.
- Experimental setup: Experiments compare S-LSTM and BiLSTM models on classification and sequence labelling benchmarks using a GTX 1080 GPU.Classification uses movie reviews and 16 additional datasets; sequence labelling uses Penn Treebank POS tagging and CoNLL NER.
- Configuration analysis: Removing the sentence-level node drops development accuracy to 81.76%, while increasing hidden size beyond 300 does not improve performance.The selected configuration uses one sentence-level node and hidden size 300.
- Classification development results: 82.64%: S-LSTM significantly outperforms 2-layer stacked BiLSTM on movie-review development data, with fewer parameters and 65-second training epochs.Three BiLSTM layers do not further improve the development results.
- Classification test results: S-LSTM significantly outperforms BiLSTM on the movie-review test set and remains highly competitive with existing methods.The test-set comparison also reports faster speed, while external attention improves both S-LSTM and BiLSTM without reversing S-LSTM’s advantage.
- Multi-dataset classification: 85.6% average accuracy across 16 datasets: S-LSTM exceeds 2-layer stacked BiLSTM at 84.9% and achieves the best result on 12 datasets.Its relative speed advantage is larger on these longer-input datasets.
- Sequence labelling results: On sequence labelling, S-LSTM significantly outperforms BiLSTM on WSJ POS tagging and achieves 91.57% F1 on the CoNLL NER test set.For CoNLL, the result is significantly better than BiLSTMs and is reported as the best under the same settings.
- Efficiency and robustness: Both models lose performance as sentences lengthen, but S-LSTM is more robust and gains a larger speed advantage on longer inputs.The paper attributes the speed advantage to S-LSTM’s fixed number of recurrent steps and links robustness to sentence-level communication.
- Efficiency and robustness: S-LSTM uses more GPU memory than BiLSTM because its computation is performed in parallel; measured usage is 252M versus 89M.Two-layer and four-layer stacked BiLSTMs use 146M and 253M, respectively, on the movie-review development set.
5 Conclusion
S-LSTM provides richer contextual information exchange with greater parallelism than BiLSTMs. Across classification and sequence labelling tasks, it outperforms BiLSTMs with the same parameter count, while its structure suggests extensions to tree LSTMs and other NLP tasks.
- S-LSTM offers richer contextual information exchange with more parallelism than BiLSTMs.
- Across classification and sequence labelling tasks, S-LSTM outperforms BiLSTMs using the same number of parameters.
- S-LSTM's structural states allow a straightforward extension to highly parallelisable tree LSTMs.
- Future work includes applying S-LSTM to tree structures and additional NLP tasks such as machine translation.