Source-linked AI summary
Dropout as data augmentation
Xavier Bouthillier, Kishore Konda, Pascal Vincent, Roland Memisevic
TL;DR
The paper asks whether dropout can be understood as data augmentation in addition to shared-parameter bagging. It back-projects dropout noise into input-space samples and evaluates deterministic training on them. The authors report good results for this approach and introduce random dropout, which improves performance without additional computational cost.
Problem
The paper asks whether dropout's noise can be understood as implicit data augmentation, beyond its shared-parameter bagging interpretation.
Source: §1 · p. 1Method
The authors back-project dropout noise into the input space to generate training samples and train a deterministic network on those samples.
Source: §3.1 · p. 3, §7 · p. 8Results
Takeaways & Limitations
Random dropout improves performance over dropout without additional computational cost and yields competitive MNIST permutation-invariant classification results across network architectures.
Source: §1 · p. 1, §5.3 · p. 7The back-projection evidence is limited to relatively small networks; larger-network experiments and analysis of co-adaptation under per-layer noise remain needed.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Dropout is typically interpreted as bagging a large number of models sharing parameters. We show that using dropout in a network can also be interpreted as a kind of data augmentation in the input space without domain knowledge. We present an approach to projecting the dropout noise within a network back into the input space, thereby generating augmented versions of the training data, and we show that training a deterministic network on the augmented samples yields similar results. Finally, we propose a new dropout noise scheme based on our observations and show that it improves dropout results without adding significant computational cost.
1 INTRODUCTION
The paper presents noise as a potentially beneficial ingredient in learning and reframes dropout as a form of data augmentation alongside its shared-parameter bagging interpretation. It proposes back-projecting dropout noise into the input space and extending dropout with random noise levels.
1 INTRODUCTION
- Successful noise-based learning methods motivate studying whether neuronal noise can play a beneficial role in learning.The introduction names denoising autoencoders, dropout, and DropConnect as examples.Source: §1 · p. 1
- Dropout can be viewed as implicit data augmentation, with back-projected samples allowing deterministic training without significant accuracy loss.The paper contrasts this view with the interpretation of dropout as shared-parameter bagging.Source: §1 · p. 1
- Random noise levels extend dropout to increase sample variety and improve classification across architectures, with competitive MNIST permutation-invariant results.The extension is presented as a way to improve the variety of augmented samples.Source: §1 · p. 1
2 DROPOUT
Dropout randomly removes neurons to regularize a network, and is commonly interpreted as bagging shared-parameter networks. The section introduces a linear projection and contrasts noisy training activations with scaled test-time activations.
2 DROPOUT
- Dropout randomly drops neurons with probability p to discourage co-adaptation, and is commonly interpreted as test-time bagging of shared-parameter networks.The proposed co-adaptation rationale is that successive neurons cannot rely solely on one another.Source: §2 · p. 2
- The section models h(x) as a linear projection from a d_i-dimensional input into a d_h-dimensional space.Source: §2 · p. 2
- During training, the noisy activation uses a Bernoulli mask M ~ B(p_h) and a rectifier; at test time, activations are scaled by p to approximate their average.Source: §2 · p. 2
3 FROM A DATA AUGMENTATION PERSPECTIVE
The paper frames noise-based regularization as increasing training-data coverage at hidden layers and describes dropout as input-space augmentation through back-projected samples. The projection fits input samples to noisy hidden activations, with separate samples for different layers.
3 FROM A DATA AUGMENTATION PERSPECTIVE
- Noise-based regularization is hypothesized to increase training-data coverage at every hidden layer, complementing domain-specific augmentation that expands coverage of the natural distribution.The paper reports multiple experimental observations in support of this hypothesis.Source: §3 · p. 2
3.1 PROJECTING NOISE BACK INTO THE INPUT SPACE
- Stochastic gradient descent minimizes squared activation error to find an input x* that approximates a noisy hidden-layer activation.The equation is generalized to networks with n hidden layers, and back-projections for all hidden activations can be computed together.Source: §3.1 · pp. 2–3
- A single shared x* is unlikely to reduce the loss substantially in a multilayer network, so the method uses a separate back-projected input for each hidden layer.Each x(i)* represents a transformation at the corresponding layer's representation level.Source: §3.1 · p. 3
- With these inputs, dropout stochasticity occurs at only one layer for each x(i)*, which may be insufficient to prevent co-adaptation.Source: §3.1 · p. 3
- The deterministic training version updates projection weights using different gradients from dropout training, whose weight gradients equal the input transformation.Source: §3.1 · p. 3
2. The gradients of the linear projections will differ greatly. In the case of dropout,
The authors report that back-projecting dropout noise into input space can replicate dropout without significant accuracy loss, though the demonstrated approximation is limited to relatively small two-hidden-layer networks.
2. The gradients of the linear projections will differ greatly. In the case of dropout,
- Back-projection replicates dropout without significant accuracy loss in experiments, with reasonable approximation demonstrated for a relatively small two-hidden-layer network.Source: §2 · p. 3
4 IMPROVING THE SET OF AUGMENTATIONS
The section asks how to enrich dropout’s implicit input-space transformations and focuses on varying the noise-level distribution. Random dropout produces more diverse transformations than standard dropout.
4 IMPROVING THE SET OF AUGMENTATIONS
- The section focuses on changing the noise distribution to enrich transformations, while noting that the network features used to encode h(x) also influence them.The motivating question is which noise scheme yields the richest set of implicit input-space transformations.Source: §4 · p. 3
4.1 RANDOM NOISE LEVELS
- Standard dropout keeps transformation magnitude relatively constant because the dropped-neuron proportion stays close to p, although transformation directions vary.The resulting displacements are at relatively constant distances in a high-dimensional space.Source: §4.1 · pp. 3–4
- Random dropout samples a level ρ_h ∼ U(0, p_h) per layer and uses that same level for all neurons in the layer.For random dropout, the training scaling replaces p with ρ; no scaling is needed at test time.Source: §4.1 · p. 4
- Random dropout produces more diverse transformations than standard dropout, with a uniform dropout fraction making near-identity transformations more likely.Figure 1 contrasts the fraction dropped, concentrated near p for standard dropout, with the uniform random level ρ ∼ U(0, p).Source: §4.1 · p. 4, Figure 1 · p. 4
5 EXPERIMENTS
The experiments visualize dropout noise projected into the input space, test deterministic training on generated samples, and evaluate a dropout variant with randomized noise levels.
5.1 VISUALIZATIONS OF NOISE PROJECTED BACK INTO THE INPUT SPACE
- Autoencoders with hidden-layer dropout generate projected samples; Figure 2 aligns each noisy sample with its source and varies the mask across columns.Source: §5.1 · p. 4, Figure 2 · p. 5
- Projected samples stay near natural inputs with recognizable classes, while shutting off a selected feature distorts its input region but preserves dependent features’ activation.Source: §5.1 · p. 5, Figure 3 · p. 5
5.2 EQUIVALENCE OF DROPOUT AND NOISY SAMPLES
- Deterministic training on back-projected noisy inputs replicates dropout with no significant accuracy drop, and is slightly better on CIFAR-10.Source: Figure 4 · p. 6, §5.2 · pp. 6–7
5.3 RICHER NOISE SCHEMES
- In the evaluated random-dropout scheme, input and hidden layers share a noise level sampled over a range [0, x].Source: §5.3 · p. 7
- Random dropout improves performance over dropout without additional computational cost.Source: §5.3 · p. 7
6 RELATED WORK
The cited work treats dropout through regularization, uncertainty, and bagging, while this paper identifies a gap in analyzing it as data augmentation. Related studies also explore noise schemes and noise-level variation.
6 RELATED WORK
- The authors report no prior analysis of dropout from a data-augmentation perspective.They note that prior dropout work instead examines regularization properties or develops noise schemes from other intuitions.Source: §6 · p. 7
- Bishop (1995) relates noise-induced regularization to generalized Tikhonov regularizers, while Baldi & Sadowski (2013) connect dropout to a term similar to weight decay.Gal & Ghahramani (2015) offer a Bayesian-approximation interpretation of dropout.Source: §6 · pp. 7–8
- Figure 6 organizes error percentages by dataset and corruption setting, contrasting input-noise sweeps, hidden-noise sweeps, and random-dropout ranges.Its rows are CIFAR-10 and MNIST; its columns vary input noise, vary hidden noise, or vary a shared input-and-hidden noise range.Source: Figure 6 · p. 8
- Poole et al. (2014) report that Gaussian noise outperforms dropout on MNIST, while Bachman et al. (2014) develop pseudo-ensembles and a related semi-supervised regularizer.Poole et al. apply noise at input, pre-activation, and post-activation positions in autoencoders.Source: §6 · p. 8
- Prior work reports benefits from scheduled noise levels; Geras & Sutton (2014) also sample levels for an autoencoder, where the approach was not very useful.The paper instead proposes random levels to avoid adaptation to a slowly changing noise distribution.Source: §6 · p. 8
- Graham et al. (2015) relate submatrix multiplication and random noise levels to inducing independence between neurons in a layer.They report that this independence is not too damaging when enough different submatrix patterns are used.Source: §6 · p. 8
7 CONCLUSION
The paper presents dropout as prior-knowledge-free data augmentation by back-projecting dropout noise into input-space samples. It reports useful deterministic-network results on those samples, while identifying open scope questions and further possible substitutes.
7 CONCLUSION
- Dropout noise can be back-projected into input-space samples, and deterministic networks trained on those samples can still yield good results.The paper frames this as a novel, prior-knowledge-free data-augmentation interpretation of dropout.Source: §7 · p. 8
- Whether the observation generalizes beyond relatively small networks, and whether deep per-layer back-projection still avoids co-adaptation, remains to be investigated.The authors call for experiments on larger networks and analysis of trained networks.Source: §7 · p. 9
- The proposed substitutes are not exhaustive: feature choices and noise schemes can vary, and the authors also suggest semi-supervised costs or samples generated by another network.The passage presents these as possible avenues for modifying induced augmentation or generating training samples.Source: §7 · p. 9
8 APPENDIX
The appendix argues that a single input-space sample minimizing the loss is unlikely for a large network, using a contradiction proof and a probability estimate.
8.1 PROOF OF x∗UNLIKELINESS
- The proof argues by contradiction that a single x∗ minimizing L is unlikely to exist for a large network.It supposes such an x∗ exists, rewrites a composition, and derives an equality involving the transformed representations.Source: §8.1 · p. 11
- The derived equation can hold only when the layer’s mask leaves rect unmodified; the passage states that this event occurs with probability p > 0.This condition is the proof’s concluding constraint on the mask.Source: §8.1 · p. 11
- 10^-47 is the stated probability for p(i) = 0.5, d(i) = 1000, and s(i) = 0.15, illustrating how low the probability is under standard hyperparameters.Here p(i) is the Bernoulli success probability, d(i) the hidden-unit count, and s(i) the mean percentage of active units in layer i.Source: §8.1 · p. 11