Source-linked AI summary
Survey of Dropout Methods for Deep Neural Networks
Alex Labach, Hojjat Salehinejad, Shahrokh Valaee
TL;DR
Deep neural networks need regularization, while practical systems also require compression, uncertainty estimates, and methods applicable beyond dense layers. This survey synthesizes dropout methods, their theoretical interpretations, applications, and extensions to convolutional and recurrent networks. It presents dropout as a widely used family supporting regularization, compression, and uncertainty estimation across these settings.
Problem
Deep neural networks are prone to overfitting, and practical applications require methods for regularization, model compression, uncertainty estimation, and broader layer applicability.
Method
The paper surveys the history, applications, theoretical explanations, layer-specific variants, compression methods, uncertainty methods, and research directions of dropout.
Results
Dropout methods support neural-network regularization, compression, uncertainty estimation, and applications in convolutional and recurrent layers.
Takeaways & Limitations
Dropout is a widely used family of stochastic techniques spanning regularization, model compression, uncertainty measurement, and multiple neural-network architectures.
Abstract
from arXiv · showhide
Dropout methods are a family of stochastic techniques used in neural network training or inference that have generated significant research interest and are widely used in practice. They have been successfully applied in neural network regularization, model compression, and in measuring the uncertainty of neural network outputs. While original formulated for dense neural network layers, recent advances have made dropout methods also applicable to convolutional and recurrent neural network layers. This paper summarizes the history of dropout methods, their various applications, and current areas of research interest. Important proposed methods are described in additional detail.
1 Introduction
Dropout began as a stochastic technique for reducing overfitting in deep neural networks and grew into a broad family of methods for training or inference. This survey covers its applications, extensions to convolutional and recurrent layers, and ongoing research directions.
- Motivation: Dropout omits each neuron with probability 0.5 during training and includes all neurons during testing, improving results across varied tasks.Its original use addressed overfitting in parameter-rich deep neural networks.
- Scope: The term dropout methods encompasses stochastic techniques that randomly modify neural-network parameters or activations during training or inference, or approximate that process.Examples include dropconnect, standout, fast dropout, variational dropout, and Monte Carlo dropout.
- Applications: Dropout methods have expanded beyond regularization to compress deep neural networks and measure model uncertainty during inference.Monte Carlo dropout provides uncertainty estimates, while other methods target network compression.
- Architectural extensions: Research has developed dropout methods for convolutional and recurrent neural-network layers, with recurrent-layer methods contributing to state-of-the-art natural-language-processing results.These extensions broaden dropout beyond its original use in dense layers.
- Survey scope: The survey reviews influential and representative dropout methods, theoretical explanations, layer-specific approaches, compression, uncertainty estimation, and future research directions.The treatment is broad but explicitly not exhaustive.
2 Standard dropout
Standard dropout randomly removes neurons during training and rescales the full network during testing. This simple procedure regularizes neural networks, allowing longer training without overfitting and improving test accuracy.
- Method: During each training iteration, standard dropout omits neurons with probability p, while the full network is used after training.The output layer is not dropped, and recommended probabilities are p = 0.2 for inputs and p = 0.5 for hidden layers.
- Formulation: Training-layer outputs are computed as y = f(Wx) ◦m, where m is a Bernoulli dropout mask with each element equal to zero with probability p.The mask is applied elementwise to the activated layer output.
- Testing: At testing, standard dropout rescales outputs by 1 −p; equivalent formulations instead rescale weights or training outputs without changing the resulting effect.These variants implement the same overall behavior through different scaling conventions.
- Effect: Standard dropout regularizes neural networks, enabling longer training without overfitting and improving test accuracy.The method subsequently became widely used in practice.
3 Theoretical understandings of dropout
Theoretical work explains dropout through ensemble averaging and Bayesian interpretations, while also examining regularization, data augmentation, and information-theoretic views. Empirical studies connect dropout’s effects to reduced feature co-adaptation and increased weight sparsity.
- Theoretical perspectives: Two major theoretical directions interpret dropout as implicit averaging over an ensemble of networks and connect it to Bayesian machine learning.These perspectives seek to explain how dropout functions and how it relates to established learning techniques.
- Ensemble interpretation: For a single-hidden-layer network with N units and softmax activation, dropout corresponds to the geometric mean of outputs from 2^N possible subnetworks.This differs from bagging because one model is trained and the ensemble output is approximated geometrically.
- Bayesian interpretation: Bayesian interpretations treat dropout training as optimizing a variational objective between an approximate distribution and the posterior of a deep Gaussian process.This provides a probability-theoretic interpretation of standard dropout.
- Alternative interpretations: Alternative analyses describe dropout as weight regularization, data augmentation, or an information-theoretic method.Neither ensemble nor Bayesian interpretations has completely dominated research.
- Empirical properties: Standard dropout reduces feature co-adaptation and promotes weight sparsity, motivating research into sparsifying and compressing neural networks.Feature co-adaptation occurs when neurons provide useful information mainly in combination with other neuron outputs.
4 Dropout methods for training
Dropout methods for dense layers extend standard dropout through weight masking, Bayesian approximations, adaptive probabilities, and training schedules. These variants target improved regularization, faster convergence, or more flexible stochastic behavior.
- Weight-level methods: Dropconnect generalizes dropout by setting individual weights and biases to zero probabilistically instead of omitting neuron outputs.At test time, it uses a Gaussian approximation, retaining stochasticity beyond training.
- Bayesian and speed-oriented methods: Fast dropout approximates dropout outputs with a Gaussian distribution and exposes all neurons to each training sample to avoid training slowdown.It can also be applied directly at test time.
- Adaptive methods: Adaptive methods tune dropout probabilities from weights or activations, including Standout’s belief network, variational dropout, evolutionary dropout, and concrete dropout.Variational dropout can determine effective probabilities for an entire network, layers, or individual neurons.
- Scheduled methods: Annealed dropout gradually reduces dropout during training, whereas curriculum dropout increases it to apply stronger regularization later.The two schedules reflect opposing assumptions about the preferred timing of regularization.
5 Convolutional layers
Convolutional layers require dropout designs that account for correlated feature-map activations. Proposed methods drop larger regions, modify pooling or operation order, and randomly omit residual-network layers or paths.
- Motivation and normalization: Correlated pixels in convolutional feature maps make randomly dropping individual activations weak, motivating CNN-specific regularization methods.Adapted dropout methods have often outperformed batch normalization alone, although some studies report that batch normalization reduces standard dropout’s benefits.
- Region-level dropout: Spatial dropout removes entire feature maps, while cutout masks a random square in each input image to eliminate larger visual regions.Spatial dropout prevents nearby pixels from recovering information removed by dropout; cutout operates directly on input images.
- Pooling and topology: Max-pooling dropout masks feature values before pooling, allowing smaller activations to influence later layers instead of always selecting only the maximum.This reduces the contribution of saturated activations during training and becomes a linear sum over activations at test time.
- Residual and normalization methods: Reordering dropout and batch normalization addresses variance from random deactivation, while stochastic depth randomly drops whole residual layers and passes values unchanged.Swapout instead randomly selects among neuron outputs, inputs, residual sums, and zero.
6 Recurrent layers
Dropout in recurrent networks must preserve long-term memory, because independently sampled noise at each time step can disrupt recurrent state. The surveyed methods therefore use sequence-level masks or selectively apply dropout to recurrent updates and weights.
- Motivation: Standard dropout on recurrent connections performs poorly because new noise at each time step prevents long-term memory retention.The problem motivates dropout methods specialized for recurrent layers.
- Per-sequence masks: RNNdrop samples one dropout mask per input sequence and keeps it fixed across time steps, allowing surviving hidden-state elements to persist.The paper contrasts this with per-step mask sampling, illustrated in Figure 5.
- Per-sequence masks: Variational RNN dropout uses a Bayesian variational Monte Carlo interpretation to mask feedforward and recurrent connections for each training sequence.Its sequence-level mask design is motivated as the natural recurrent-layer application under that interpretation.
- Other recurrent methods: Weight-dropped LSTMs apply dropconnect-style dropout to weights and achieved state-of-the-art language-modeling results at the time.Zoneout offers another memory-preserving alternative by replacing activations with those from the previous time step.
- Selective recurrent dropout: Recurrent dropout preserves LSTM memory by applying dropout to hidden-state updates rather than erasing the cell state.Its cell-state equation modifies the update term with a sampled mask while leaving the remaining LSTM equations unchanged.
7 Dropout methods for model compression
Dropout methods can support model compression because their induced sparsity reduces the parameters needed for effective performance. Recent approaches combine variational sparsification, adaptive neuron selection, or structured probabilistic selection with pruning and compression pipelines.
- Compression motivation: Dropout-induced sparsity enables model compression by reducing the number of parameters needed for effective performance.Several dropout-based compression approaches have been proposed since 2017.
- Compression methods: Variational dropout sparsifies fully connected and convolutional layers, achieving large parameter reductions in standard convolutional networks with minimal performance change.The resulting sparse representation can be passed to existing sparse-network compression methods.
- Compression methods: Targeted dropout adaptively selects neurons so the network can adapt to pruning and shrink considerably without much accuracy loss.Ising-dropout instead overlays a graphical Ising model to identify less useful neurons.
8 Monte Carlo dropout
Monte Carlo dropout extends dropout from regularization to uncertainty estimation, addressing the need to know how certain a model’s output is. It repeatedly samples dropout masks at inference to estimate the implicit Bayesian model’s output and uncertainty, without changing training.
- Motivation: Monte Carlo dropout provides model uncertainty estimates analogous to Bayesian methods, which ordinarily produce probability-distribution outputs.This addresses the limitation that most neural networks do not directly provide output certainty information.
- Bayesian interpretation: A Bayesian interpretation views dropout as a variational approximation of a deep Gaussian process, with dropout retained at test time.The method treats stochastic forward passes as samples from the implicit Bayesian model.
- Inference procedure: Monte Carlo dropout runs the trained network T times on the same input using independently generated dropout masks.The resulting outputs support estimators for the model’s mean prediction and uncertainty.
- Applications: Monte Carlo dropout has been applied in time-series prediction and medical imaging, while requiring no changes to the model training procedure.Bayesian neural networks and ensembles instead incur substantially greater training complexity according to the surveyed comparison.
9 Discussion
The discussion identifies regularization as the dominant research direction while emphasizing expansion into compression, uncertainty estimation, adversarial methods, and theoretical analysis. It presents dropout research as continuing alongside the increasing scale and use of deep neural networks.
- Regularization: Improving dropout regularization remains the most common research direction, especially to accelerate convergence or improve final performance.Dropout can slow training because it reduces each neuron’s exposure to individual training samples.
- Broader applications: Research also seeks broader applications, including model compression and improved methods specialized for network types or neuron selection.These efforts extend dropout beyond its original regularization role.
- Emerging directions: Adversarial dropout methods combine dropout with adversarial learning or use adversarial ideas to guide which elements are dropped.The discussion identifies this as a promising research line.
- Theory: Theoretical work increasingly uses Bayesian interpretations to justify dropout and related stochastic methods.This follows a broader movement toward Bayesian and variational techniques in deep learning.
- Outlook: The paper expects dropout methods to remain useful as deep neural networks become more advanced and widely used.This is presented as a broad concluding outlook rather than a specific empirical result.