Source-linked AI summary
Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, Bohyung Han
TL;DR
Deep networks need regularization because noise injection must balance fitting the data distribution against preventing overfitting. The paper interprets noisy training as lower-bound optimization and introduces importance weighted stochastic gradient descent with multiple noise samples. Across several visual recognition tasks, the method consistently improves performance and achieves near-state-of-the-art CIFAR results.
Problem
Noise-based regularization such as dropout is effective, but its operation and optimization remain insufficiently understood under the competing goals of data fitting and overfitting prevention.
Method
The paper treats noise-injected hidden units as stochastic activations and derives importance weighted stochastic gradient descent using multiple noise samples per training example.
Results
The method consistently improves performance across image classification, visual question answering, image captioning, and action classification, with near-state-of-the-art performance on CIFAR datasets.
Takeaways & Limitations
Using multiple noise samples can provide tighter lower bounds and a better balance between model fitting and regularization in noisy training.
Takeaways & Limitations
The analysis assumes that injecting noise into deterministic hidden units constructs stochastic hidden units, and the paper mainly evaluates the approach through dropout.
Abstract
from arXiv · showhide
Overfitting is one of the most critical challenges in deep neural networks, and there are various types of regularization methods to improve generalization performance. Injecting noises to hidden units during training, e.g., dropout, is known as a successful regularizer, but it is still not clear enough why such training techniques work well in practice and how we can maximize their benefit in the presence of two conflicting objectives---optimizing to true data distribution and preventing overfitting by regularization. This paper addresses the above issues by 1) interpreting that the conventional training methods with regularization by noise injection optimize the lower bound of the true objective and 2) proposing a technique to achieve a tighter lower bound using multiple noise samples per training example in a stochastic gradient descent iteration. We demonstrate the effectiveness of our idea in several computer vision applications.
1 Introduction
The paper frames noise injection as a regularizer that creates a tradeoff between fitting the data distribution and preventing overfitting. It interprets conventional noisy training as lower-bound optimization and proposes importance weighted stochastic gradient descent using multiple noise samples.
- Deep networks are prone to overfitting because they are often over-parametrized, making regularization important for generalization.
- Noise injection, including dropout, regularizes hidden units during training but creates a tradeoff between data fitting and regularization strength.More noise makes fitting harder, while less noise weakens regularization.
- Conventional noisy training can be interpreted as optimizing a lower bound of the marginal likelihood over stochastic hidden-unit values.The interpretation treats noise-injected hidden-unit outputs as stochastic activations sampled with a reparametrization trick.
- Importance weighted stochastic gradient descent uses multiple noise samples per training example to obtain tighter lower bounds during optimization.The method is motivated by importance weighted autoencoders and is mainly instantiated for dropout.
2 Related Work
Related work presents noise regularization as a broad family of methods, with dropout as a prominent example and several variants targeting generalization. Prior interpretations explain dropout through ensembles, adaptive regularization, optimization, and Bayesian uncertainty, while importance weighting supplies the closest optimization precedent.
- Regularization by noise is widely used to improve deep-network generalization, with implementations varying by architecture and application.
- Dropout randomly turns off hidden units by multiplying them with Bernoulli-distributed noise.
- Dropout variants modify the dropout rate or noise distribution to further improve generalization.Examples include adaptive dropout and multinomial dropout, which selects hidden units from multiple subsets.
- Prior work interprets dropout as model ensembling, adaptive regularization, a way to escape local optima, or Bayesian uncertainty estimation.
- Importance weighted autoencoders provide a closely related precedent by using multiple importance-weighted samples to compute gradients and tighten lower bounds.The related objective has also been applied to generative modeling and stochastic neural networks.
3 Proposed Method
The method interprets noise-injected hidden units probabilistically and uses multiple noise samples to tighten the lower bound optimized during stochastic gradient descent. IWSGD computes importance-weighted gradient averages while retaining standard dropout inference.
- 3.1 Main Idea: Noise injection is modeled as stochastic hidden-unit activation, enabling probabilistic analysis of regularized neural-network training.The reparameterization trick expresses stochastic activations as z = g(hφ(x), ϵ), with dropout noise representing random hidden-unit selections.
- 3.1 Main Idea: The marginal likelihood is optimized over noise variables after replacing stochastic hidden units with their reparameterized form.The noise distribution p(ϵ) defines the marginalization used for training the noise-injected network.
- 3.2.1 Objective: The conventional SGD objective is a Jensen lower bound of the marginal likelihood, with tightness determined by the number of noise samples.For each example, S denotes the number of noise samples and E = {ϵ1, ϵ2, ..., ϵS} is their set.
- 3.2.1 Objective: IWSGD uses S > 1 noise samples per training example to optimize a tighter lower bound than conventional noise-regularized training.The conventional procedure corresponds to S = 1, whereas increasing the sample count tightens the bound.
- 3.2.2 Training: Training averages gradients from multiple noise samples using importance weights based on each sample’s normalized likelihood.The procedure draws noise samples, performs forward and backward propagation, computes weights, and updates parameters with the weighted gradients.
- 3.2.3 Inference: IWSGD uses standard dropout-style inference, requiring neither additional sampling nor computation at inference time.Input activations are scaled according to dropout probability instead of sampled stochastically.
- 3.3 Discussion: The authors note that multiple samples are not straightforwardly equivalent to multiple iterations because parameters change after every iteration.They report that single-sample performance saturates easily and is unlikely to match IWSGD through additional iterations.
4 Importance Weighted Stochastic Gradient Descent for Dropout
The section applies importance-weighted stochastic gradient descent to dropout by sampling multiple dropout masks per training example. It reports comparisons on CIFAR datasets and notes that implementation requires only minor modifications.
- Dropout setup: Dropout samples Bernoulli masks that randomly keep or discard hidden activations during training.The post-dropout activations are written as g(hφ(x), ϵ).
- Lower-bound interpretation: Conventional dropout optimizes a lower bound of the ideal marginal likelihood, corresponding to IWSGD with S = 1.Increasing the number of dropout masks improves this lower bound.
- Multi-sample training: IWSGD realizes multi-sample dropout training by sampling multiple dropout masks for each training example in an iteration.This implements multiple noise sampling within the dropout training procedure.
- Experimental setup: Figure 2 evaluates multi-sample training on CIFAR datasets with a wide residual network using variable dropout rates.The network has widening factor 10 and depth 28; points and error bars use 3 trials with different seeds.
- Experimental results: Table 1 compares wide residual network models trained with the multi-sample objective against other CIFAR models.The table reports test errors, standard deviations, and a ×4-iterations comparison; the caption states that the method achieves near-state-of-the-art performance.
- Implementation: IWSGD reuses standard dropout gradient computation, adding importance-weighted gradient aggregation as its main implementation change.This makes the method easy to incorporate into applications using dropout.
5 Experiments
Experiments across object recognition, visual question answering, image captioning, and action recognition evaluate IWSGD with dropout-based models. Multiple noise samples generally improve performance, though gains are inconsistent when fine-tuning only a saturated fusion unit.
- Experimental setup: IWSGD was evaluated on object recognition, visual question answering, image captioning, and action recognition using models that actively employ dropout.Experiments fixed hyperparameters and avoided additional performance heuristics to isolate the training method's effect.
- Object Recognition: IWSGD improves wide residual network performance on CIFAR-10 and CIFAR-100, with test errors decreasing as sample count increases regardless of dropout rate.The experiments use a widening factor of 10 and depth 28, with results averaged over three seeds.
- Object Recognition: With S = 8 samples, the wide residual network reaches accuracy very close to state-of-the-art CIFAR performance, while 4 times more iterations do not improve performance.The authors suggest that simply increasing iterations is likely to overfit the trained model.
- Visual Question Answering: Increasing the number of IWSGD noise samples consistently improves VQA performance.The evaluation uses a baseline with standard dropout on the VQA test-dev dataset.
- Image Captioning: IWSGD with 5 samples consistently improves image-captioning performance on BLEU, METEOR, and CIDEr.BLEU is evaluated using BLEU-4 on MSCOCO.
- Action Recognition: IWSGD improves action-recognition performance, but performance does not consistently increase with sample count when only the fusion unit is fine-tuned.The authors suspect the model is already saturated, leaving little room for improvement through fusion-unit fine-tuning.
6 Conclusion
The paper proposes IWSGD for dropout and other noise-based regularization by interpreting noisy hidden units as stochastic hidden units. Across visual recognition tasks, the method consistently improves performance and achieves near-state-of-the-art CIFAR results through tighter lower-bound optimization.
- 6 Conclusion: The paper proposes IWSGD, an optimization method for noise-based regularization, especially dropout, in deep neural networks.The method is based on interpreting noise-injected deterministic hidden units as stochastic hidden ones.
- 6 Conclusion: IWSGD achieves tighter lower bounds as the number of noise samples increases.The method is applied to dropout, a special case of regularization by noise.
- 6 Conclusion: Across image classification, visual question answering, image captioning, and action classification, the authors observe consistent improvement with IWSGD.The paper reports near-state-of-the-art performance on CIFAR datasets through better optimization.