Source-linked AI summary
Adding Gradient Noise Improves Learning for Very Deep Networks
Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, James Martens
TL;DR
Complex neural architectures create optimization challenges that standard techniques do not always robustly solve. The paper adds annealed Gaussian noise to gradients during training and reports consistent improvements across complex models, including a 72% relative error reduction in question answering and more than twice as many low-error Neural GPU models.
Problem
More complicated neural architectures for tasks such as question answering and general computation create new optimization challenges.
Method
The paper trains deep neural networks with time-dependent annealed Gaussian noise added to the gradient during each training step.
Results
72% relative error reduction was achieved on a challenging question-answering task, while Neural GPU runs produced more than twice as many models with < 1% error as training without noise.
Takeaways & Limitations
Gradient noise is a simple, low-overhead technique that the authors found broadly effective for training and generalizing complicated neural networks.
Abstract
from arXiv · showhide
Deep feedforward and recurrent networks have achieved impressive results in many perception and language processing applications. This success is partially attributed to architectural innovations such as convolutional and long short-term memory networks. The main motivation for these architectural innovations is that they capture better domain knowledge, and importantly are easier to optimize than more basic architectures. Recently, more complex architectures such as Neural Turing Machines and Memory Networks have been proposed for tasks including question answering and general computation, creating a new set of optimization challenges. In this paper, we discuss a low-overhead and easy-to-implement technique of adding gradient noise which we find to be surprisingly effective when training these very deep architectures. The technique not only helps to avoid overfitting, but also can result in lower training loss. This method alone allows a fully-connected 20-layer deep network to be trained with standard gradient descent, even starting from a poor initialization. We see consistent improvements for many complex models, including a 72% relative reduction in error rate over a carefully-tuned baseline on a challenging question-answering task, and a doubling of the number of accurate binary multiplication models learned across 7,000 random restarts. We encourage further application of this technique to additional complex modern architectures.
1 INTRODUCTION
More complex neural architectures create optimization challenges that standard techniques may not robustly solve. The paper studies annealed Gaussian gradient noise as a simple, broadly applicable remedy for training such models.
- Motivation: Complex tasks such as question answering and program induction require architectures that pose new optimization challenges.Researchers have used intermediate supervision, warmstarts, random restarts, and altered activation functions to obtain good performance.
- Proposed approach: Annealed Gaussian noise added to gradients improves training of deep neural networks with stochastic gradient descent.The authors contrast gradient noise with random weight noise and report that it can lower training loss by encouraging parameter-space exploration.
- Scope: The method is presented as broadly applicable to complex modern neural architectures and implementable in a single line of code.The authors report improvements for very deep fully connected networks, question-answering systems, and algorithm-learning architectures.
2 RELATED WORK
Prior work has used noise, momentum, and adaptive learning rates to improve neural-network training, but gradient noise has been less documented in modern deep networks. The related-work discussion emphasizes its suitability for nonconvex optimization through added stochasticity.
- Noise-based methods: Gradient noise has been less commonly used and less fully documented than weight noise, hidden-unit noise, or dropout in modern deep networks.Weight-noise methods may retain nonzero noise at convergence, while adaptive weight noise also requires variance parameters.
- Prior gradient-noise uses: Annealed Gaussian gradient noise has appeared in stochastic neighbor embedding and resembles stochastic gradient Langevin dynamics.Those related methods use gradient noise for nonconvex optimization or MCMC inference rather than establishing the present paper's training results.
- Optimization context: Momentum and adaptive learning rates are prominent neural-network optimization techniques commonly developed from convex-optimization settings and applied heuristically to nonconvex problems.The paper contrasts these approaches with gradient-noise injection.
- Motivated mechanism: Adding stochasticity to gradients can give nonconvex models more chances to escape local minima or move through early learning plateaus.The paper presents these as proposed explanations for why gradient noise may help optimization.
3 METHOD
The method adds time-dependent Gaussian noise to the gradient during every training step. Its variance is annealed over time, with a schedule chosen to provide stronger perturbations early in training.
- Gradient perturbation: At each training step, the method adds time-dependent Gaussian noise to the gradient g.The operation is applied during stochastic optimization.
- Noise schedule: The preferred schedule uses decaying variance, because annealed Gaussian noise works better than fixed Gaussian noise in the experiments.The schedule is inspired by Welling and Teh (2011).
- Noise schedule: The schedule uses η selected from {0.01, 0.3, 1.0} and γ = 0.55, with higher noise at the beginning of training.The initial perturbation forces the gradient away from 0 during early training.
4 EXPERIMENTS
Experiments across deep and specialized architectures show that annealed gradient noise can improve optimization, especially under difficult initialization or training conditions. Benefits include higher accuracy, lower errors, and greater robustness across random restarts, though gains depend on the model and setup.
- Experimental scope: The experiments evaluate gradient noise on deep MNIST networks, End-To-End Memory Networks, Neural Programmer, NRAM, and Neural GPUs.These models cover digit classification, question answering, and algorithm learning.
- Deep fully-connected networks: Gradient noise improved average and best MNIST accuracy from Simple Init, but did not improve the five-layer network.The comparison used 40 runs across fixed learning rates of 0.1 and 0.01.
- Deep fully-connected networks: 94.5% accuracy was reached with gradient noise from zero initialization, whereas networks without noise failed to learn.The result demonstrates sensitivity to poor initialization in the 20-layer fully-connected MNIST network.
- Deep fully-connected networks: With carefully chosen initialization and gradient clipping, comparable no-noise accuracy was possible; under poor initialization, noise helped overcome optimization difficulty.Noise was insensitive to the tested clipping thresholds of 100 and 10, while carefully tuned Good Init removed its benefit.
- End-To-End Memory Networks: Added gradient noise helped End-To-End Memory Networks in both training and validation errors, especially without warmstarting.Warmstarting also helped, but noise provided an additional improvement.
- Neural Programmer: Neural Programmer results improved with added gradient noise across tasks, while its interaction with dropout was mixed and problem-dependent.In the single-column experiment, 9/216 noisy runs reached 100% test accuracy versus 1/216 without noise.
- Neural GPUs: Across 7290 Neural GPU runs, more than twice as many noisy models achieved less than 1% error, with improved robustness across initializations and parameter settings.Noise also produced more models at lower error rates throughout the experiment.
5 CONCLUSION
Experiments show that adding noise to the gradient helps train and generalize complicated neural networks. The authors recommend the simple technique for practitioners and suggest it warrants further formal analysis.
- Gradient noise improved both training and generalization in experiments with complicated neural networks.
- The authors describe gradient noise as a simple, effectively single-line addition for practitioners facing neural-network training problems.
- The authors suggest the empirical results could motivate formal analysis of why gradient noise benefits very deep neural networks.