PaperlayerView exact revision on arXiv

Source-linked AI summary

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Gal, Zoubin Ghahramani

arXiv:1512.05287v5stat.ML

TL;DR

RNNs are powerful but prone to overfitting, and standard dropout has failed when applied to recurrent layers. The paper derives a Bayesian variational dropout method for LSTM and GRU models, finding improved performance across language modelling and sentiment analysis, including 73.4 test perplexity on Penn Treebank.

  • Problem

    RNNs overfit readily, while dropout has not been successfully applied to recurrent layers, leaving a need for effective recurrent regularisation.

  • Method

    The paper interprets LSTM and GRU weights probabilistically and derives variational inference with a Gaussian-mixture posterior approximation, implemented as shared-mask dropout across inputs, outputs, and recurrent layers.

  • Results

    The method outperforms existing techniques across language modelling and sentiment analysis, achieving 73.4 test perplexity on Penn Treebank.

  • Takeaways & Limitations

    Variational inference provides a theoretically grounded way to regularise recurrent parameters and extend dropout to recurrent neural networks.

Abstract

from arXiv · show

Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bayesian interpretation of common deep learning techniques such as dropout. This grounding of dropout in approximate Bayesian inference suggests an extension of the theoretical results, offering insights into the use of dropout with RNN models. We apply this new variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks. The new approach outperforms existing techniques, and to the best of our knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity). This extends our arsenal of variational tools in deep learning.

1 Introduction

RNNs are powerful sequence models but overfit readily, while standard dropout has not worked well in recurrent layers. The paper develops a Bayesian, variational interpretation that yields a dropout variant using shared masks across time and recurrent connections.

  • 1 Introduction: RNNs support sequence-based tasks including language understanding, language generation, and video processing, but overfit quickly.Limited regularisation makes small datasets difficult to handle, encouraging early stopping or small, under-specified models.
  • 1 Introduction: Standard dropout randomly masks network units, but applying it to recurrent layers has not succeeded empirically.Existing practice therefore applies dropout only to RNN inputs and outputs, which still leads to overfitting.
  • 1 Introduction: Bayesian interpretations of dropout motivate extending approximate variational inference to recurrent neural networks.In this view, dropout approximates the posterior of a Bayesian neural network using a mixture of Gaussians, including a component fixed at zero.
  • 1 Introduction: The paper treats LSTM and GRU models as probabilistic RNNs and derives a tractable variational objective whose optimisation is equivalent to a new dropout variant.Network weights are treated as random variables with suitably defined likelihood functions.
  • 1 Introduction: The proposed method repeats one dropout mask at every time step for inputs, outputs, and recurrent layers, unlike existing ad hoc techniques.Existing techniques resample masks across time for inputs and outputs and omit recurrent-connection dropout.

2 Related Research

Earlier RNN-dropout approaches were largely empirical and often avoided recurrent connections because naive noise was thought to destabilize dynamics. The paper instead derives recurrent dropout through variational inference, enabling regularisation of recurrent parameters.

  • 2 Related Research: Prior work reported that naive dropout noise in recurrent connections can destabilise RNN dynamics or be amplified over long sequences.Researchers consequently restricted dropout to decoding or other non-dynamic parts, with mixed empirical results.
  • 2 Related Research: Existing approaches selected dropout variants through trial-and-error and generally left large recurrent parameter sets unregularised.These conclusions led many researchers to believe dropout could not be extended effectively within recurrent layers.
  • 2 Related Research: The paper contrasts this empirical tradition by deriving a variational-inference-based dropout variant that successfully regularises recurrent parameters.The theoretical grounding is presented as the basis for revisiting dropout inside recurrent layers.

3 Background

The background frames Bayesian neural networks as models with distributions over weights and variational inference as an approximation to otherwise intractable posteriors. These ideas are then extended to probabilistic RNNs to derive recurrent dropout.

  • 3 Background: Bayesian neural networks place prior distributions over weights and infer a posterior distribution conditioned on observed inputs and outputs.The posterior captures how likely different function parameters are given the data.
  • 3 Background: Because posterior inference over neural-network weights is generally intractable, variational inference introduces an approximating distribution q(ω) and minimises its KL divergence from the posterior.The approximation provides a tractable route to estimating the weight distribution that generated the data.
  • 3 Background: The paper extends approximate variational inference to probabilistic RNNs using a q(ω) distribution that produces a new RNN dropout variant.This connects the Bayesian approximation directly to the recurrent regularisation method developed later.

4 Variational Inference in Recurrent Neural Networks

The paper formulates RNNs as probabilistic models and uses variational inference to derive dropout with weight distributions and time-consistent masks. This framework extends to LSTMs, GRUs, input embeddings, and several test-time prediction schemes.

  • A simple RNN applies the same recurrent function across an input sequence to generate hidden states, with model weights treated as random variables in the probabilistic formulation.The probabilistic model includes recurrent, bias, and output parameters, with a likelihood defined for the sequence model.
  • Variational inference approximates the complex posterior over RNN weights with q(ω), and Monte Carlo integration supplies an unbiased estimator for the optimisation objective.The estimator uses a single Monte Carlo sample and is plugged into the minimisation objective.
  • The approximating distribution factorises over weight matrices and rows, using a zero-centred Gaussian mixture whose variational parameters correspond to standard RNN weights.The KL term can be approximated as L2 regularisation over the variational parameters mk.
  • Implementation and relation to dropout: Sampling from q(ω) is equivalent to randomly masking weight-matrix rows, with each sequence using one mask that remains fixed across all time steps.This yields dropout on recurrent connections as well as inputs and outputs, unlike techniques that resample masks across time and omit recurrent dropout.
  • LSTM and GRU extensions: The same variational dropout construction extends to LSTMs and GRUs, although different parameterisations can induce different approximating distributions and gate-specific masks.The LSTM formulation includes additive cell-state updates, and tied- and untied-weight parameterisations are deterministic-equivalent but variationally distinct.
  • Word embeddings dropout: For discrete word inputs, dropping embedding-matrix rows is equivalent to dropping word types consistently throughout a sequence rather than individual word tokens.When sequence length T is much smaller than vocabulary size V, embeddings can be zeroed after lookup for more efficient implementation.

5 Experimental Evaluation

Experiments evaluate the variational dropout method on Penn Treebank language modelling and sentiment analysis, comparing LSTM and GRU variants with existing regularisation techniques. The method improves perplexity and error outcomes while reducing overfitting across the reported settings.

  • Language Modelling: The Penn Treebank experiments use medium and large LSTM models, with the variational dropout variant replacing naive dropout while other benchmark hyperparameters remain largely unchanged.The study also tunes weight decay and compares tied and untied weight parametrisations, standard dropout approximation, and MC dropout.
  • Language Modelling: 73.4 test perplexity is achieved with MC dropout and untied weights, improving the prior 78.4 result and reported as the best single-model Penn Treebank perplexity.Validation perplexity improves from 82.2 to 77.3 with tied weights and 77.9 without weight tying.
  • Additional Analysis: Tied weights provide a more time-efficient implementation under the dropout approximation because one matrix product uses a single GPU kernel.The tied and untied models have very close performance under this approximation.
  • Language Modelling: 10 Variational LSTMs improve test perplexity from 69.5 to 68.7, matching the perplexity obtained in the comparison using 38 models.This result comes from independently trained large models whose outputs are averaged.
  • Language Modelling: Without learning-rate decay, Variational LSTM achieves lower validation perplexity than naive dropout LSTM and standard LSTM, even with early stopping.Learning-rate decay scheduling lowers perplexity for all models, so this comparison isolates dropout more directly.
  • Sentiment Analysis: On sentiment analysis, Variational LSTM and Variational GRU avoid the overfitting seen in standard techniques and achieve the lowest reported test error in their respective comparisons.The non-regularised GRU reaches a lower test error with early stopping at the second epoch, while standard techniques show repeated decreases and increases in test error.
  • Sentiment Analysis: Embedding dropout changes the effect of recurrent-layer dropout: with pE = 0.5, higher recurrent dropout improves robustness, whereas without embedding dropout it is associated with overfitting.The authors therefore suggest that embedding dropout can be crucial in some tasks.
  • Additional Analysis: Weight decay remains important with the variational dropout variant because it corresponds to the prior distribution over weights.This differs from the common practice of removing weight decay with naive dropout.

6 Conclusions

The paper presents a theoretically motivated recurrent-neural-network regularisation technique and demonstrates its effectiveness empirically. It identifies model uncertainty as a future direction for Variational LSTMs.

  • Conclusions: The paper presents a new theoretically motivated technique for recurrent neural network regularisation and demonstrates its effectiveness empirically.The authors describe the method as an RNN dropout variant.
  • Conclusions: Future work will assess model uncertainty in Variational LSTMs, with potential implications for language ambiguity and dynamics in control tasks.These implications are stated as future research directions rather than demonstrated results.

A Bayesian versus ensembling interpretation of dropout

The paper contrasts Bayesian and ensembling interpretations of dropout in deep networks. It argues that the Bayesian approximation determines weight tying in the recurrent setting, whereas ensembling alone does not.

  • Comparison: The ensembling interpretation supports MC dropout but does not determine whether the ensemble is over network units or weights.For RNNs, it therefore does not by itself specify the paper’s dropout variant.
  • Comparison: The Bayesian approximation view forces weight tying through its probabilistic interpretation, whereas the ensembling view would require tying recurrent weights ad hoc.This distinction explains why the two interpretations lead to different recurrent-dropout designs.

B Sentiment analysis – further experiments

Further experiments examine how dropout probabilities, weight decay, sequence length, and dropout approximations affect Variational LSTM and GRU performance on sentiment analysis. They also compare recurrent-mask strategies and report stable convergence across evaluated models.

  • Dropout approximation: The deterministic dropout approximation gives test errors similar to MC dropout for both Variational LSTM and GRU models.For LSTM, the approximation propagates the mean weights through a deterministic pass; the GRU comparison is shown in Figure 12.
  • Hyper-parameter effects: Higher weight decay produces lower Variational LSTM test error, indicating that weight decay remains important alongside dropout.Weight decay is applied to all layers, including embeddings, and different values yield significant error differences.
  • Sequence length: Longer sequences improve Variational LSTM performance but require longer convergence times.The comparison uses sequence lengths T = 20, 50, 200, and 400; follow-up tests show that lower embedding dropout does not explain the shorter-sequence degradation.
  • Mask comparison: Using the same dropout mask across time steps matches the performance of input/output dropout with different masks in the compared LSTM configurations.Both compared models apply dropout only to inputs and outputs; the figure reports identical performance.
  • Convergence: All evaluated models have converged with stable training loss in the reported train-loss plots.The plots cover various models from the main experiments, including the LSTM and GRU evaluations.

C Code

The implementation computes recurrent updates using dropout masks for input and recurrent terms within the LSTM cell.

  • Mask construction: The Keras implementation constructs input and recurrent retention probabilities and samples masks for the LSTM computation.The code initializes retain_prob_W and retain_prob_U, then creates binomial masks for the input and recurrent pathways.
  • Input transformations: Separate input masks are applied to the input transformations for the input, forget, cell, and output gates.The gate-specific preactivations multiply X by B_W before applying the corresponding weight matrices.
  • Recurrent transformations: Recurrent masks are applied to the previous hidden state in each gated recurrent transformation.The recurrent update receives B_U and multiplies h_tm1 by gate-specific masks before the recurrent matrix products.
  • State updates: The masked gate activations update the cell state and hidden state through the standard LSTM recurrence.The implementation combines forget and input contributions for c_t, then computes h_t from the output gate and activated cell state.
Loading 1512.05287v5…