Source-linked AI summary
Recurrent Neural Network Regularization
Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals
TL;DR
RNNs need effective regularization, but standard dropout performs poorly with recurrent models. The paper introduces a way to apply dropout only to non-recurrent LSTM connections and reports reduced overfitting and large performance increases across multiple domains.
Problem
Standard dropout is highly successful for feed-forward networks but does not work well with RNNs, which can overfit when scaled to larger models.
Method
The paper applies dropout only to non-recurrent LSTM connections, using an operator that randomly sets a subset of its input to zero.
Results
The method produces large performance increases across language modeling, speech recognition, machine translation, and image caption generation.
Takeaways & Limitations
Dropout can be useful for RNNs when applied correctly, producing a single model that can match an ensemble in the image-captioning setting.
Abstract
from arXiv · showhide
We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.
1 INTRODUCTION
RNNs perform strongly across important sequence tasks, but standard dropout does not work well with RNNs, contributing to overfitting and undersized practical models. The paper shows that correctly applied dropout greatly reduces overfitting in LSTMs.
- RNNs achieve state-of-the-art performance on language modeling, speech recognition, and machine translation.
- Standard dropout, despite its success in feed-forward networks, does not work well with RNNs.
- Large RNNs often overfit, so practical applications commonly use models that are too small.
- Correctly applied dropout greatly reduces overfitting in LSTMs.
- The work's implementation code is publicly available on GitHub.Repository: https://github.com/wojzaremba/lstm.
2 RELATED WORK
Prior work found limited success applying dropout to RNNs, including claims that recurrence amplifies dropout noise. This paper fixes the problem by applying dropout to a subset of connections and evaluates the approach across several RNN tasks.
- Conventional dropout was reported to hurt RNN learning because recurrence amplifies its noise.
- The paper applies dropout only to a certain subset of RNN connections to address this problem.
- An independently developed version of the same regularization method was applied to handwriting recognition.
- The method targets LSTMs, the most commonly used RNN variant, and may also work with other RNN architectures.
- The evaluation considers language modeling, speech recognition, and machine translation.
3 REGULARIZING RNNS WITH LSTM CELLS
The paper describes deep LSTMs and a dropout scheme that regularizes only non-recurrent connections. This preserves long-term information flow while applying dropout a fixed number of times determined by network depth.
- 3.1 LONG-SHORT TERM MEMORY UNITS: The deep LSTM uses the top-layer activations to predict the output at each timestep.States are indexed by timestep and layer, with L denoting the number of layers.
- 3.1 LONG-SHORT TERM MEMORY UNITS: LSTM memory cells can overwrite, retrieve, or retain information across timesteps.The architecture stores long-term memory in explicit cell states and uses gated dynamics to control that information.
- 3.2 REGULARIZATION WITH DROPOUT: The method applies dropout only to non-recurrent LSTM connections.The regularized multilayer RNN uses dropout on selected connections while leaving recurrent connections unperturbed.
- 3.2 REGULARIZATION WITH DROPOUT: Information carried through the LSTM is corrupted exactly L + 1 times, independent of the number of timesteps traversed.The fixed corruption count follows the network depth rather than the temporal distance of the information path.
- 3.2 REGULARIZATION WITH DROPOUT: Avoiding dropout on recurrent connections lets the LSTM use regularization without sacrificing its memorization ability.The paper contrasts this with standard dropout, which perturbs recurrent connections and makes long-term storage difficult to learn.
4 EXPERIMENTS
The experiments evaluate the dropout scheme across language modeling, speech recognition, machine translation, and image caption generation. Across these settings, dropout improves LSTM performance, while image captioning shows a single dropout model can match an ensemble.
- Experimental scope: The study evaluates LSTMs on language modeling, speech recognition, machine translation, and image caption generation.The experiments include Penn Tree Bank language modeling, Icelandic speech recognition, English-to-French translation, and image caption generation.
- Language modeling: The language-modeling experiments compare regularized medium and large LSTMs with a smaller non-regularized LSTM.The medium and large models use dropout on non-recurrent connections, whereas the non-regularized comparison is constrained to a smaller network because larger networks overfit.
- Speech recognition: Dropout improves frame accuracy for LSTMs on the relatively small Icelandic Speech Dataset, where overfitting is a major concern.The training set contains 93k utterances; training accuracy decreases under dropout while unseen-data accuracy improves.
- Machine translation: Dropout improves LSTM translation performance on the WMT’14 English-to-French task, although the LSTM does not outperform the phrase-based LIUM system.The optimal dropout probability in this experiment is 0.2.
- Image caption generation: In image caption generation, dropout improves performance relative to no dropout, but an ensemble removes the gain.The reported practical effect is a single model approximately as good as an ensemble.
5 CONCLUSION
The paper presents a simple LSTM dropout method that produces large performance increases across several domains. It concludes that this implementation makes dropout useful for RNNs and may improve many applications.
- The proposed LSTM dropout method produces large performance increases across problems in different domains.
- The work makes dropout useful for RNNs.
- The results suggest that the implementation could improve performance on a wide variety of applications.