Source-linked AI summary
Semi-supervised Sequence Learning
Andrew M. Dai, Quoc V. Le
TL;DR
The paper tackles difficult RNN training for NLP sequence tasks by using unlabeled data to pretrain recurrent networks. It introduces recurrent language-model and sequence-autoencoder objectives for initializing supervised LSTMs, which usually improve performance and can match or surpass prior baselines across the tested benchmarks.
Problem
RNNs are powerful for sequential data, but difficult back-propagation-through-time training has limited their use in NLP tasks such as text classification.
Method
The paper pretrains LSTMs with recurrent language models or sequence autoencoders, then uses the learned weights to initialize supervised sequence classifiers.
Results
On five benchmarks, LSTMs reached or surpassed the performance levels of all previous baselines, while pretraining generally improved over random initialization.
Takeaways & Limitations
Additional unlabeled data can improve recurrent-network generalization, including when labeled data are limited.
Abstract
from arXiv · showhide
We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" step for a later supervised sequence learning algorithm. In other words, the parameters obtained from the unsupervised step can be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better. With pretraining, we are able to train long short term memory recurrent networks up to a few hundred timesteps, thereby achieving strong performance in many text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.
1 Introduction
The paper addresses difficult RNN training for NLP by pretraining LSTMs with unsupervised sequence objectives. Across several classification and sentiment tasks, this pretraining usually improves random-initialized LSTMs and can exploit additional unlabeled data.
- RNNs are powerful for sequential data, but back-propagation through time makes them difficult to train for NLP tasks such as text classification.
- The paper pretrains LSTMs with either recurrent language modeling or sequence autoencoding, then uses the learned weights to initialize supervised classifiers.
- LSTMs pretrained by either method are usually better than randomly initialized LSTMs across 20 Newsgroups, DBpedia, IMDB, and Rotten Tomatoes.
- 79.7% to 83.3%: Amazon-review pretraining improved Rotten Tomatoes classification accuracy, illustrating the benefit of related-task unlabeled data.
- The approach is positioned as easier to fine-tune than Paragraph Vectors and less purely unsupervised than Skip-Thought vectors.
2 Sequence autoencoders and recurrent language models
The sequence autoencoder adapts the encoder–decoder idea to unsupervised reconstruction: it encodes an input sequence and reconstructs that same sequence. Its learned weights initialize supervised sequence classifiers, while unlabeled data can improve generalization.
- Sequence-to-sequence learning uses an encoder recurrent network to map an input sequence to a hidden state consumed by a decoder.
- The sequence autoencoder replaces the usual output sequence with the input sequence, training the model to reconstruct what it read.
- Figure 1 illustrates encoding “WXYZ” into a hidden state and using that state to reconstruct the original sequence.
- Sequence-autoencoder weights initialize a supervised classifier; the authors hypothesize that memorization and shortcut gradients make this initialization stable.
- Because the sequence autoencoder is unsupervised, it can use large quantities of unlabeled data, which can improve recurrent-network generalization when labeled data are limited.
- Recurrent language modeling is another LSTM pretraining method and performs better than random initialization in the reported experiments.
3 Overview of methods
The methods overview compares standard LSTMs with pretrained variants and adds two ways to provide supervision across sequence positions. It also includes joint training of classification and sequence autoencoding.
- The experiments compare basic LSTMs with sequence-autoencoder initialization, called SA-LSTMs, and language-model initialization, called LM-LSTMs.
- Most experiments predict the document label from the LSTM output at the last timestep.
- Linear label gain places the label at every timestep and increases objective weights from 0 to 1 to inject gradients into earlier recurrent steps.
- Joint training simultaneously trains the supervised task and the sequence autoencoder.
4 Experiments
Across sentiment and document-classification benchmarks, pretrained LSTMs were generally more stable and competitive than randomly initialized models, including on long documents and when using additional unlabeled data.
- Overall results: SA-LSTMs matched or surpassed reported results across all evaluated datasets, providing one strong model across multiple tasks.The compared previous best results came from different methods.
- Optimization stability: LSTM training on long documents was unstable, whereas sequence-autoencoder pretraining improved optimization stability.Increasing hidden units or backpropagation steps could make randomly initialized training break down.
- Sentiment analysis: On Rotten Tomatoes, tuning SA-LSTMs reduced test error to 19.3%, while additional unlabeled pretraining further reduced it to 16.7%.Pretraining used related IMDB and Amazon-review data; the Amazon corpus contained 7.9 million movie reviews.
- Document classification: On 20 Newsgroups, SA-LSTM achieved 15.6% test error with input-embedding and word dropout, outperforming previous classifiers.The result used 70% input-embedding dropout and 75% word dropout.
- Document classification: On DBpedia, linear label gain achieved 1.32% test error, while combining it with SA-LSTM achieved 1.19%.The combined method improved over convolutional-network results.
- Object classification: On CIFAR-10, pretrained LM-LSTM exceeded the baseline convolutional DBN and the non-pretrained LSTM, but not state-of-the-art convolutional networks.The experiment treated each image row as an LSTM timestep and fine-tuned after unsupervised learning.
5 Discussion
The paper demonstrates that LSTM recurrent networks can be used for NLP document classification and that unsupervised pretraining can stabilize their learning. Across five benchmarks, the LSTMs reached or surpassed previous baseline performance.
- LSTM recurrent networks can be used for NLP tasks such as document classification.
- Language-model or sequence-autoencoder pretraining can help stabilize learning in LSTM recurrent networks.
- On five benchmarks, LSTMs reached or surpassed the performance levels of all previous baselines.