Source-linked AI summary
Recurrent Dropout without Memory Loss
Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth
TL;DR
RNNs are prone to overfitting, while conventional dropout is usually applied to forward rather than recurrent connections. The paper proposes recurrent dropout for gated networks, placing masks to avoid long-term-memory loss, and reports improvements across three NLP tasks, including with forward dropout.
Problem
RNN architectures are prone to overfitting, while conventional dropout has been applied mainly to forward connections rather than recurrent connections.
Method
The paper proposes recurrent dropout for gated architectures, applying dropout to hidden-state update vectors and examining forward-dropout combinations and mask sampling.
Results
Recurrent dropout improves regularization across word-level and character-level language modeling and named entity recognition, including when combined with conventional forward dropout.
Takeaways & Limitations
The approach provides a recurrent regularizer for LSTMs and GRUs that targets dependency-learning weights without affecting the ability to capture long-term relationships.
Takeaways & Limitations
Under the paper’s analysis, dropout can make distant-past contributions effectively vanish during testing, limiting exploitation of learned long-term dependencies.
Abstract
from arXiv · showhide
This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to \textit{forward} connections of feed-forward architectures or RNNs, we propose to drop neurons directly in \textit{recurrent} connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.
1 Introduction
RNNs achieve strong results on sequential NLP tasks but are prone to overfitting, while conventional dropout is mainly applied outside recurrent connections. This paper proposes and evaluates recurrent dropout designed to preserve long-term memory and improve regularization across NLP tasks.
- RNNs, including LSTMs, are widely used for sequential NLP tasks ranging from classification and tagging to language modeling and sequence-to-sequence prediction.
- RNN architectures are prone to overfitting despite their ability to capture complex linguistic phenomena.
- Conventional dropout is typically applied to input-to-hidden and hidden-to-output connections, not recurrent connections along the time axis.
- The paper proposes recurrent dropout for LSTMs and investigates how to prevent long-term-memory corruption, combine recurrent and forward dropout, and sample masks across time.
- Across word-level and character-level language modeling and named entity recognition, recurrent dropout improves regularization, including when combined with conventional forward dropout.
- The paper also compares its method with alternative recurrent-dropout approaches and reports superiority in almost all cases.
2 Related Work
Prior work established that forward dropout can regularize LSTMs and explored dropping hidden or cell states. This paper introduces a recurrent dropout technique for gated architectures that aims to regularize dependency-learning weights without losing long-term information.
- Neural network models often overfit when they have many parameters and limited training data, motivating research on improved generalization.
- Prior studies showed that dropout in forward connections can effectively regularize LSTMs.
- Other work extended dropout to cell states, while variational Bayesian methods proposed LSTM dropout variants with improvements over no-dropout baselines.
- This paper introduces recurrent dropout for gated architectures such as LSTMs and GRUs, where dropping arbitrary cell vectors may cause memory loss.
- The proposed technique targets weights responsible for short- and long-term dependencies while preserving the network’s ability to capture long-term relationships.
- Compared with recently introduced recurrent-dropout methods, the paper reports better results for its approach.
3 Recurrent Dropout
The section develops recurrent dropout for vanilla RNNs and gated networks, focusing on LSTMs where directly dropping hidden states can impair long-term memory. The proposed method drops cell update vectors instead, preserving the hidden state while permitting per-step masks.
- Recurrent dropout: Recurrent dropout extends conventional dropout from input-to-hidden and hidden-to-output connections to recurrent connections in RNNs.The section contrasts this approach with feed-forward dropout and discusses its application to vanilla RNNs, LSTMs, and GRUs.
- Dropout in LSTM networks: The proposed method applies dropout to the cell update vector g_t rather than scaling the hidden state directly.Because only newly added differences are dropped, the hidden state itself is not directly scaled.
- Dropout in LSTM networks: Previous hidden-state dropout produces mixed results, helping without forward dropout but usually losing its benefit when combined with it.The section emphasizes the interaction between recurrent dropout schemes and conventional forward dropout.
- Dropout in LSTM networks: In LSTMs, dropping previous hidden states directly can attenuate information from distant timesteps during inference.Repeated multiplication by the keep probability causes components from the distant past to become effectively negligible.
- Dropout in LSTM networks: Per-step mask sampling is compatible with learning long-term dependencies under the proposed update-vector dropout scheme.This differs from Gal and Moon, whose methods use masks sampled once per training sequence.
- Dropout in LSTM networks: The discussed scaling problem does not affect vanilla RNNs because they overwrite their hidden state at every timestep.The section also notes that Gal's approach is not affected by this issue.
4 Experiments
Experiments evaluate recurrent dropout on a synthetic memory task and three NLP benchmarks, comparing sampling schemes and recurrent versus forward dropout. The proposed approach solves the synthetic task across cases and improves language-modeling and tagging results under several settings.
- Synthetic Task: The synthetic Temporal Order task tests whether LSTMs retain and use two symbols' order after processing an entire sequence.Sequences contain two meaningful symbols among distractors, and the model predicts one of four order classes after the full sequence.
- Synthetic Task: Dropping hidden states can corrupt LSTM memory: per-sequence sampling fails on the test set in Medium cases, while per-step sampling requires 10-20 times more iterations in Short cases.The authors attribute these behaviors to scaling issues and memory-cell corruption that hinders long-term memory.
- Synthetic Task: The proposed recurrent dropout solves the synthetic task in all tested cases, including longer sequences whose results match the Medium case.The alternative approach of Gal (2015) does not appear to exhibit the memory-loss problem in these experiments.
- Word Level Language Modeling: On word-level language modeling, dropping hidden-state updates outperforms dropping hidden states, and can reduce perplexity by more than 30 points in some cases.Per-step sampling is better for direct hidden-state dropout, while the proposed update-based method performs similarly with per-step and per-sequence sampling.
- Word Level Language Modeling: With forward dropout, the LSTM validation perplexity decreases from 130 to 91.6 using the proposed recurrent dropout.The experiments also report that the proposed approach remains effective when combined with conventional forward dropout.
- Character Level Language Modeling: Learning curves show slower convergence, larger training error, and lower validation error with 0.25 per-step recurrent dropout, consistent with regularization.Figure 2 distinguishes training errors with solid lines from validation errors with dashed lines.
- Character Level Language Modeling: Character-level results favor recurrent-only regularization over forward-only dropout, with the best result achieved by the proposed method at dropout rate 0.25 and per-step sampling.The proposed method is reported as the only approach improving over the baseline LSTM with forward dropout.
- Named Entity Recognition: On NER, recurrent-dropout gains are larger for LSTMs and are larger after training on shorter 5-word sequences than on longer 15-word sequences.The authors suggest recurrent dropout may affect generalization to sequences longer than those used during training, while higher rates caused excessive regularization.
5 Conclusions
The paper introduces a recurrent dropout method for gated recurrent networks and finds it effective across three public NLP benchmarks, especially alongside conventional forward dropout. Its benefits beyond these NLP tasks and domains remain unexplored.
- The proposed recurrent dropout is tailored to gated recurrent neural networks and is easy to combine with conventional forward dropout.The paper reports that the combined approach is more effective than recurrent dropout alone.
- The method was verified on three different public NLP benchmarks, with improvements reported across various NLP benchmarks.The evaluated tasks include word- and character-level language modeling and named entity recognition.
- Recurrent dropout is most effective in LSTMs when applied to hidden state update vectors rather than arbitrary cell vectors or hidden states.Applying dropout to arbitrary cell vectors produces suboptimal performance in both LSTMs and GRUs.
- Per-step and per-sequence mask sampling produce similar results with the proposed recurrent dropout, and both outperform Moon et al.'s dropout scheme.The paper notes that this result was contrary to the authors' expectations.
- The method's benefits for sequence-to-sequence prediction and domains such as speech recognition remain unexplored.The authors leave these settings for future work.