Source-linked AI summary
Noisy Activation Functions
Caglar Gulcehre, Marcin Moczulski, Misha Denil, Yoshua Bengio
TL;DR
Saturating activation functions can create vanishing gradients in gated networks while soft saturation prevents hard decisions. The paper injects noise into saturated regimes and anneals it during training, then removes it at test time. Noisy variants improve training across tasks and datasets, producing state-of-the-art or competitive results, while practical use includes a bias choice in the noise design.
Problem
Soft-saturating gates can have vanishing gradient information and cannot make fully open or closed decisions.
Method
The paper injects learnable noise into strongly saturating activation functions, using hard-saturating nonlinearities during training and removing or replacing noise at test time.
Results
Noisy activation variants improve training across different tasks and datasets, yielding state-of-the-art or competitive results, including in PennTreebank experiments.
Takeaways & Limitations
The technique enables hard decisions and deterministic test-time networks while serving as a simple drop-in replacement for existing nonlinearities.
Takeaways & Limitations
The noise design may be slightly biased in practice, despite the desired expectation property being satisfied directly only for zero-mean noise with µ = 0.
Abstract
from arXiv · showhide
Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that the gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent toexplore more. By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and task, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.
1. Introduction
Soft-saturating gates can lose gradient information while preventing fully open or closed decisions. The paper replaces them with hard-saturating activations plus noise, enabling exploration during training and deterministic inference afterward.
- The proposed technique trains neural networks with strongly saturating activation functions by injecting and learning noise in their saturated regime.
- Gated architectures use sigmoid or softmax nonlinearities to emulate hard digital decisions, but these functions can saturate and impede gradient flow.
- Hard-saturating nonlinearities allow gates to make perfectly on or off decisions, avoiding information loss from soft-gating leakiness.
- Noise that grows with saturation encourages random exploration where hard activations otherwise have exactly zero gradients.
- At test time, noise can be removed or replaced by its expectation, while resulting deterministic networks outperform soft-saturating counterparts across varied tasks.
- The approach also addresses optimization difficulty through simulated annealing and supports piecewise-linear activations in gated recurrent networks such as LSTM and GRU.
2. Saturating Activation Functions
Saturating activations can make optimization difficult because their gradients vanish, especially for hard-saturating functions. The paper constructs hard versions from linearizations and motivates noise as a way to preserve crisp decisions while improving gradient flow.
- An activation function saturates when its derivative tends to zero at either or both input extremes.
- Soft-saturating functions reach zero gradient only in the limit, whereas hard-saturating functions have zero gradients beyond finite thresholds.
- Hard-saturating activations are built by clipping first-order Taylor linearizations to an appropriate range.
- Linear behavior near zero supports gradient flow, while saturation supplies crisp decisions.
- Exactly zero saturated gradients can hide small pre-activation changes that would reduce the objective, leaving units stuck and potentially requiring many examples and substantial computation to recover.
- Hard-sigmoid saturates for x ≤ −2 or x ≥ 2, while hard-tanh saturates for x ≤ −1 or x ≥ 1.
3. Annealing with Noisy Activation Functions
Noisy activations make large gradients available in saturated regimes, allowing SGD to explore difficult non-convex objectives. Annealing the noise shifts optimization from broad exploration toward convergence near minima of the noise-free objective.
- The noisy activation framework injects iid noise into saturating nonlinearities and analyzes training while gradually reducing its variance.
- The method assumes that the activation derivative also grows as the noise level becomes large.
- With large noise, SGD sees predominantly noisy gradients and can move broadly through parameter space without following a clear trend.
- Annealing raises the signal-to-noise ratio; when activation noise vanishes, remaining exploration comes from stochastic-gradient Monte Carlo noise.
- As noise decreases, SGD increasingly favors regions where the signal is visible and eventually fine-tunes near a minimum of the noise-free objective.
4. Adding Noise when the Unit Saturate
The proposed noisy activation adds noise only in saturated regimes, with noise magnitude tied to saturation, so gradients can flow and optimization can explore toward the linear regime. The method uses signed noise variants and multiple gradient paths while producing deterministic test-time units.
- Selective noise injection: Noise is added only when hard-sigmoid or hard-tanh saturates, unlike noisy rectifiers that inject noise independently of the input regime.The noise magnitude is proportional to the nonlinearity’s saturation magnitude.
- Selective noise injection: The noisy activation aims to preserve a clean training signal in the non-saturating regime while supplying noisy gradients in the saturating regime.For unbiased noise, the expected activation can match the hard-saturating activation; biased noise may require adjusting the location parameter.
- Parameterization: The noise scale can be learned through p or controlled by c, while α adjusts the mean of the added term and affects approximation to the hard nonlinearity.The stochastic activation was illustrated for different α values and Gaussian noise when approximating hard-tanh.
- Gradient flow: The linearized, nonlinear, and stochastic paths jointly carry gradients, making optimization easier across different layers.The linear path is used in the non-saturated regime, while stochasticity drives exploration in the saturated regime.
- Gradient flow: In saturation, the noiseless derivative is zero, but conditioning on the sampled noise yields a derivative that is non-zero almost surely.Because the noise scale depends on x, gradients flow through the stochastic path even where h′(x)=0.
- Directional noise: Half-normal noise uses a manually chosen direction toward h(x), avoiding sign cancellation and helping push saturated activations toward the non-saturated regime.The direction depends on d(x), the sign of x, and the bias introduced by α.
- Test-time behavior: At test time, the expectation can replace the stochastic activation to obtain deterministic units.The paper uses the expectation of the noisy activation for deterministic inference.
5. Adding Noise to Input of the Function
The paper also considers injecting Gaussian noise into the input of the activation function, with either learned or fixed noise scale. This alternative satisfies the desired expectation condition only when the scale is learned, and small fixed noise is empirically preferable.
- Input-noise formulation: Input noise for ReLU has been investigated previously using a fixed standard deviation before the activation function.The cited prior work concerns ReLU activation functions.
- Input-noise formulation: The input-noise formulation is φ(x, ξ) = h(x + σξ), with σ either learned or fixed as a hyperparameter.The paper presents this as a separate parametrization of noisy activation functions.
- Input-noise behavior: The expectation condition holds only when σ is learned; experimentally, small values of σ work better.With small fixed σ, noise becomes less likely to return activations from far saturation to the linear regime.
6. Experimental Results
Experiments across MLPs, recurrent language models, program execution, translation, and Neural Turing Machines evaluate noisy activation variants. The results generally show faster convergence or improved performance, with noise annealing and curriculum learning especially useful on difficult tasks.
- 6.1. Exploratory Analysis: Noisy activations converged faster than tanh and hard-tanh and reached lower negative log-likelihood in single-layer MNIST MLPs.
- 6.1. Exploratory Analysis: A three-layer MLP with tanh and noisy-tanh activations solved the three-Gaussian classification task almost perfectly.
- 6.1. Exploratory Analysis: NAN and NANI had similar PTB character-level training curves, while NANIS progressed better initially before its curve stopped improving.
- 6.2. Learning to Execute: About 2.5% higher accuracy was obtained on Learning to Execute by replacing the reference model’s activation functions with noisy variants.
- 6.3. Penntreebank Experiments: Noisy activations produced substantial validation and test improvements on the Penn Treebank word-level LSTM task, establishing a new state-of-the-art result according to the authors.
- 6.4. Neural Machine Translation Experiments: Noisy activations outperformed hard-tanh and hard-sigmoid models, improving English-to-French translation by more than 2 BLEU points over the reference.
- 6.6. Experiments with Continuation: With annealed noise, the Neural Turing Machine converged much faster and solved associative recall, whereas the regular model failed to approach low error.
7. Conclusion
The paper formalizes noisy activations to make difficult nonlinearities easier to optimize by injecting noise where saturation would impede gradients. Experiments report improved performance across tasks, with further gains possible when noise is annealed.
- 7. Conclusion: Noisy activations inject noise into nonlinear functions so large noise makes SGD more exploratory and allows gradients to flow in saturating regions.Noise can be added at the function input or output where units would otherwise saturate.
- 7. Conclusion: The framework supports a wider family of activation functions than previously trainable, including hard versions of sigmoid and tanh.The experiments found that harder sigmoid and tanh nonlinearities often improved results.
- 7. Conclusion: Noisy activations outperform sigmoid or tanh counterparts on different tasks and datasets, yielding state-of-the-art or competitive results with simple modifications.PennTreebank is cited as an example of this outcome.
- 7. Conclusion: Annealing the noise level can act as a continuation method and further improve performance.