Source-linked AI summary
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, Chris Pal
TL;DR
RNN transition dynamics can be sensitive to hidden-state perturbations, motivating a regularizer that improves robustness without disrupting temporal information flow. Zoneout injects training noise by randomly preserving previous hidden activations, and experiments report improvements across language modeling and permuted sequential MNIST. Combined with recurrent batch normalization, it achieves state-of-the-art pMNIST results.
Problem
RNN transition dynamics can magnify minor hidden-state perturbations, creating a need for recurrent regularization that improves robustness while preserving temporal information and gradients.
Method
Zoneout regularizes RNNs by randomly replacing selected hidden or cell activations with their previous-timestep values during training.
Results
Zoneout improves performance across tasks, achieves competitive results on Penn Treebank and Text8, and reaches state-of-the-art results on permuted sequential MNIST when combined with recurrent batch normalization.
Takeaways & Limitations
Low state-zoneout probabilities of 0.05–0.2 reliably improve existing models, while zoneout also works in combination with other regularizers.
Abstract
from arXiv · showhide
We propose zoneout, a novel method for regularizing RNNs. At each timestep, zoneout stochastically forces some hidden units to maintain their previous values. Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization. But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks. We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks. We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization yields state-of-the-art results on permuted sequential MNIST.
1 INTRODUCTION
Zoneout regularizes recurrent neural networks by randomly preserving previous hidden activations instead of zeroing units. The method targets sensitivity to hidden-state perturbations while preserving information and gradient flow through time, and is evaluated across language-modeling and classification tasks.
- Zoneout is introduced as a novel regularizer for recurrent neural networks.
- RNN transition dynamics can exponentially magnify minor hidden-state perturbations, motivating zoneout’s robustness objective.
- During training, zoneout randomly replaces selected activations with their previous-timestep values rather than setting them to zero.
- Preserving activations helps maintain forward information flow and backward gradient flow, addressing vanishing gradients.
- The study evaluates zoneout on Penn Treebank and Text8 language modeling and permuted sequential MNIST classification, reporting competitive or state-of-the-art performance across tasks.
2 RELATED WORK
Zoneout is framed as identity-masked dropout for recurrent models and as a per-unit form of stochastic depth. Related recurrent regularizers differ in what they mask, how masks persist through time, and whether previous activations are preserved exactly.
- Zoneout injects stochastic identity masks that preserve previous activations, making it easier to carry information forward and gradients backward.
- Several alternatives reuse masks across timesteps or mask weights, gates, or updates rather than directly preserving activations.
- Unlike recurrent dropout on LSTM updates, zoneout preserves hidden-state activations exactly and supports hidden-state zoning without an analogue in that method.
- Zoneout corresponds to independently zoning out units, whereas stochastic depth zones out all units in a layer simultaneously.
- The paper reports that zoneout outperforms stochastic depth and recurrent dropout in its RNN experiments, while residual recurrent connections were unstable in their experiments.
- Unlike periodic clockwork and hierarchical RNN updates, zoneout uses stochastic update decisions.
3 ZONEOUT AND PRELIMINARIES
Zoneout modifies recurrent transitions by mixing the usual update with an identity connection selected by Bernoulli masks. In LSTMs, it can independently preserve memory cells or hidden states, contrasting with zero-masking approaches that alter update contributions or expose stale memory.
- 3.1 RECURRENT NEURAL NETWORKS: An RNN maps each previous hidden state and current input to a new hidden state through a repeated transition operator.
- 3.1 RECURRENT NEURAL NETWORKS: Zoneout mixes the ordinary transition with the identity operator, whereas dropout mixes it with a null operator.
- 3.2 LONG SHORT-TERM MEMORY: In an LSTM, the cell c_t stores long-term information while the hidden state h_t provides a transient representation, with the forget gate connecting c_t to c_t−1.
- 3.2 LONG SHORT-TERM MEMORY: The standard LSTM computes candidate updates, cell states, and hidden states using gates, the cell recurrence, and an output-gated tanh transformation.
- 3.2 LONG SHORT-TERM MEMORY: Naive recurrent dropout zero-masks cells or hidden states without changing gate computation, while masking the input gate removes additive input and hidden-state contributions.
- 3.2 LONG SHORT-TERM MEMORY: Zoneout randomly either preserves a cell or hidden state from the previous timestep or updates it normally, creating stochastic identity connections.
- 3.2 LONG SHORT-TERM MEMORY: The implementation usually uses separate zoneout masks for cells and hidden states, with an additional variant coupling recurrent dropout masks to output gates.
4 EXPERIMENTS AND DISCUSSION
Experiments evaluate zoneout across language modeling, permuted sequential MNIST, and gradient-flow analyses, comparing it with recurrent regularizers and unregularized LSTMs. Zoneout improves several benchmarks, with strongest reported results from combined or task-specific settings.
- Experimental setup: Zoneout was evaluated on character- and word-level Penn Treebank, character-level Text8, and permuted sequential MNIST, alongside gradient-flow measurements.The language-modeling metrics are BPC for character tasks and perplexity for Word-PTB.
- Character-level language modeling: 1.27 BPC was achieved on character-level Penn Treebank by combining cell zoneout zc = 0.5 with hidden-state zoneout zh = 0.05.This setting outperformed the best-performing recurrent dropout configuration in the reported exploration.
- Character-level language modeling: 1.53 to 1.41 for GRU and 1.67 to 1.52 for tanh-RNN were the reported BPC reductions from low zoneout probabilities.For LSTMs, lower probabilities worked best on hidden states, while cell probabilities around 0.5 worked well.
- Word-level language modeling: 78.4 to 77.4 was the test-perplexity improvement when zoneout was added to a word-level Penn Treebank model already optimized for non-recurrent dropout.Zoneout alone did not produce competitive results in that setting, and no model search was performed.
- Permuted sequential MNIST: Zoneout significantly improved over the LSTM baseline and outperformed recurrent dropout on permuted sequential MNIST, while recurrent batch normalization outperformed both individually.Combining zoneout with recurrent batch normalization achieved state-of-the-art performance, using shared masks and probabilities of 0.15.
- Gradient flow: Zoneout propagated gradient information to early timesteps more effectively than recurrent dropout and an unregularized LSTM on permuted sequential MNIST.The same effect was observed for hidden states; gradients were measured with normalized average norms across timesteps.
5 CONCLUSION
Zoneout is a simple regularizer that stochastically preserves hidden activations, improving existing RNN models across tasks. Low state-zoneout probabilities reliably improve performance, while task-specific tuning can further help.
- Zoneout improves performance across tasks and achieves results competitive with state of the art on Penn Treebank and Text8, plus state-of-the-art results on pMNIST.
- The authors conjecture that zoneout benefits arise from hidden-state robustness to stochastic changes and improved forward and backward information flow through identity connections.
6 APPENDIX
The appendix separates zoneout’s noise injection from its identity connections and tests static identity connections against zoneout and vanilla LSTM training. Noise injection is essential, while static identity connections fail to reproduce zoneout’s validation benefits.
- STATIC IDENTITY CONNECTIONS EXPERIMENT: Noise injection is essential for obtaining zoneout’s regularization benefits.
- STATIC IDENTITY CONNECTIONS EXPERIMENT: Static identity connections produced slightly lower training error than zoneout but not lower validation error, and underperformed an unregularized LSTM on both sets.
- STATIC IDENTITY CONNECTIONS EXPERIMENT: Figure 7 compares training and validation curves for static identity connections, zoneout with Zc = 0.5 and Zh = 0.05, and a vanilla LSTM.