Source-linked AI summary
On the Impact of the Activation Function on Deep Neural Networks Training
Soufiane Hayou, Arnaud Doucet, Judith Rousseau
TL;DR
Deep-network training depends strongly on initialization and activation choices because poor choices can lose forward information or produce vanishing or exploding gradients. The paper analyzes random networks through Gaussian-process approximations, characterizes the Edge of Chaos, and studies activation-function choices. It reports that suitable Edge-of-Chaos initialization and smooth activations can deepen signal propagation, accelerate training, and improve performance, while the analysis has scope boundaries for ReLU-like functions and idealized infinite-width networks.
Problem
Initialization and activation choices can cause forward information loss and vanishing or exploding gradients, motivating theory for identifying trainable deep networks.
Method
The paper analyzes random neural networks using Gaussian-process approximations, studying Edge-of-Chaos initialization and activation-function regularity.
Results
Suitable Edge-of-Chaos initialization deepens signal propagation and accelerates training, while smooth activations can propagate signals deeper than ReLU.
Takeaways & Limitations
Choosing both Edge-of-Chaos prior parameters and an activation satisfying the paper’s smoothness condition is relevant for obtaining non-degenerate Bayesian-neural-network function priors.
Takeaways & Limitations
The analysis uses an idealized infinite-width Gaussian-process setting, and the paper identifies a non-existence result for the Edge of Chaos for ReLU-like activations in a stated regime.
Abstract
from arXiv · showhide
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Samuel et al (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the `Edge of Chaos' can lead to good performance. While the work by Samuel et al (2017) discuss trainability issues, we focus here on training acceleration and overall performance. We give a comprehensive theoretical analysis of the Edge of Chaos and show that we can indeed tune the initialization parameters and the activation function in order to accelerate the training and improve the performance.
1. Introduction
The paper studies how initialization and activation choices shape the optimization landscape of deep neural networks. It analyzes the Edge of Chaos to connect these choices with information propagation, training acceleration, and performance.
- Weight initialization and activation functions determine the functional subspace explored by optimization in non-convex deep-network training.
- The paper comprehensively analyzes the Edge of Chaos and argues that initialization on this curve deepens information propagation and accelerates training.
- Feedforward ReLU networks initialized on the Edge of Chaos propagate correlations as if they had residual connections.
- The authors introduce smooth activation functions that permit deeper signal propagation than ReLU and analyze optimal points on the Edge of Chaos.
2. On Gaussian process approximations of neural networks and their stability
The section models infinitely wide fully connected networks as Gaussian processes and characterizes how variances and correlations propagate across layers. It then defines convergence domains and the Edge of Chaos, relating initialization and activation functions to ordered, chaotic, and constant-output behavior.
- Gaussian process approximation: Infinite-width feedforward networks are approximated by centered Gaussian processes whose covariance kernels satisfy layer-wise recursions determined by the activation function.The approximation corresponds to taking preceding-layer widths to infinity, invoking the central limit theorem.
- Stability and convergence domains: The variance and correlation dynamics are studied through recursive functions, with separate convergence domains defined for variance and correlation propagation.The framework tracks whether layer-wise quantities converge uniformly for individual inputs and input pairs.
- Stability and convergence domains: Under sufficient derivative-moment conditions, variance and correlation convergence can occur exponentially fast because the associated maps are contractions.The sufficient conditions include a bounded activation-dependent quantity Mφ and constraints on σw and σb.
- Examples: For ReLU, the activation-dependent bound satisfies MReLU = 1/2, and the correlation-related quantity obeys CReLU,δ ≤ 1 for every δ > 0.These values provide an explicit example within the sufficient conditions used to analyze convergence.
- Phase behavior: In the ordered convergence regime, network outputs become constant functions, whereas in the chaotic infinite-width, infinite-depth regime, the output function is non-continuous everywhere.The paper illustrates constant outputs for ReLU and Tanh in ordered settings and highly differing outputs for close inputs in the chaotic Tanh setting.
- Edge of Chaos: The Edge of Chaos is characterized by χ1 = 1, separating an ordered phase with correlations converging to 1 from a chaotic phase where they do not.The correlation depth scale is infinite at χ1 = 1, while χ1 < 1 and χ1 > 1 define the ordered and chaotic regimes, respectively.
3. Edge of Chaos
The Edge of Chaos (EOC) is characterized by correlation preservation at a critical initialization, with its structure and benefits depending on the activation function. ReLU has a weak EOC, while smooth activations support deeper signal propagation and yield a broader EOC analysis.
- EOC behavior: On the EOC, correlations converge to 1 more slowly than exponentially, preserving variability through deeper layers.For ReLU, the convergence is polynomial at rate 1/ℓ^2 rather than exponential in the ordered phase.
- ReLU-like activations: For ReLU-like activations, the ordinary EOC can be empty, so a weak EOC is defined where correlations remain critical and variances bounded.For ReLU, this weak EOC reduces to a singleton initialization, while ordered and divergent regimes occur on either side.
- Smooth activation functions: Smooth activation functions such as Tanh and ELU have EOC curves, whereas ReLU has an EOC reduced to a single point.Tanh and ELU satisfy the conditions used to establish EOC existence; the curves can be determined from the activation-specific analysis.
- Smooth activation functions: Sufficiently smooth activations improve correlation convergence to O(1/ℓ), compared with O(1/ℓ^2) for ReLU-like activations.The paper includes Tanh, SiLU, and ELU in the relevant regularity class and links the faster rate to deeper information propagation.
- Initialization choice: Small σb can deepen correlation and gradient propagation, but choosing σb too small drives layer outputs toward zero, creating an initialization trade-off.For a network of depth L, the paper suggests choosing an EOC point with βq ≈ L.
4. Experiments
Experiments show that EOC initialization accelerates training, while smooth activations and depth-matched EOC points improve performance in deep networks.
- Initialization on the EOC performs better than Batch Normalization for deep networks.
- Smooth activation functions perform better than ReLU-like activations, especially for very deep networks.
- Choosing the EOC point so that L ∼ βq further accelerates training across depths 30–90.For depths 30–90, σb = 0.05 produced the best approximate performance among the tested choices because βq was closest to L.
- Experiments used MNIST and CIFAR10 with width 300, depths 10–200, SGD or RMSProp, and tuned learning rates.
- EOC initialization dramatically accelerates training for ELU, ReLU, and Tanh networks.
5. Discussion
The discussion connects the theoretical analysis to deeper signal propagation and Bayesian neural networks. It also identifies depth-dependent initialization and activation choices as important for maintaining useful propagation.
- As depth increases, the optimal EOC point requires smaller σb, but the limiting variance q approaches zero as σb approaches zero.
- An activation satisfying Proposition 3 can keep βq large independently of σb, addressing the limiting-variance trade-off.
- For Bayesian neural networks, non-degenerate induced function-space priors require EOC prior parameters and an activation satisfying Proposition 3.
A. Proofs
The supplementary material supplies proofs of the theoretical results and additional theoretical and experimental results.
- The supplementary material contains proofs of the main theoretical results and additive theoretical and experimental results.
A.1. Convergence to the fixed point: Proposition 1
For suitable activations and initialization parameters, the variance converges to a fixed point, while EOC correlations approach one and gradient covariance remains controlled.
- Variance convergence: Under bounded activation conditions, the variance recursion is a contraction, yielding a unique fixed point independent of the input.The limiting variance therefore converges across layers for the analyzed parameter domain.
- Correlation convergence: On the EOC, the correlation sequence is Cauchy and converges to a limit in [0, 1].For the contraction argument, the limiting fixed point is one, so correlations converge to one.
- ReLU-like activations: A feedforward ReLU network initialized on the EOC acts as a simple residual ReLU network for information propagation.The proposition establishes the connection between the EOC regime and residual connections.
- Gradient propagation: On the EOC, |φ|EOC controls the vanishing of gradient covariance during back-propagation.Linear activations do not change gradient covariance, providing the stated interpretation of this quantity.
B.1. Results on the Edge of Chaos
The EOC structure depends strongly on the activation function: some smooth activations have a nontrivial EOC, while others reduce it to a single zero-bias point.
- EOC existence: For activations satisfying the stated positivity condition, sufficiently small positive σb admits no EOC parameters.The limiting variance fails the EOC equation, so no σw exists in that small-bias interval.
- Practical boundary: These activations cannot use small σb > 0, because σb = 0 yields q = 0, which is impractical for training.Consequently, the result concerning small |φ|EOC does not apply in this case.
- Smooth activations: Shifted Softplus has EOC = {(0, 2)}, while SiLU has the same singleton EOC in a sufficiently small-bias region.The SiLU conclusion follows from positivity of the relevant expectation over a neighborhood of zero.
B.2. Beyond the Edge of Chaos
Making an activation function close to linear on the EOC brings the correlation dynamics close to the identity and improves signal propagation, but reduces expressiveness.
- EOC semi-norm: As |φ|EOC approaches zero, the limiting variance q diverges for fixed positive σb when EOC parameters exist.This links near-linearity in the activation to increasingly large limiting variance.
- Signal propagation: Small |φ|EOC makes the correlation function close to the identity, allowing signal to propagate deeper through the network.The paper proposes activation functions φα(x) = x + αTanh(x), whose semi-norm tends to zero as α approaches zero.
- Trade-off: Reducing |φ|EOC creates a trade-off: signal propagation improves while expressiveness decreases because φ becomes nearly linear.Adding more layers can compensate for the expressiveness loss within the paper’s stated scope.
C.1. Training with RMSProp
RMSProp experiments on deep MNIST networks show that ELU and Tanh outperform ReLU, with ELU converging faster than Tanh.
- RMSProp results: For RMSProp, learning rate 10^-5 is nearly optimal for networks with depth L ≤200, while 10^-6 performs better for deeper networks.The learning rate was selected by an exponential grid search.
- RMSProp results: On MNIST with depth 200 and width 300, ELU and Tanh perform better than ReLU.The result supports the stated training-acceleration analysis across training algorithms.
- RMSProp results: ELU converges faster than Tanh in the reported RMSProp experiment.
C.2. Training with activation φα(x) = x + αTanh(x)
The experiments compare activation functions in very deep MNIST networks, finding that φ0.5 performs similarly to ELU and better than Tanh and ReLU, while MSiLU outperforms SiLU early in training.
- φa activation: φ0.5 has approximately similar performance to ELU and better performance than Tanh and ReLU in depth-200, width-300 MNIST networks trained with SGD.The comparison is shown in Figure 8.
- φa activation: The authors suggest that φa’s advantage over Tanh may relate to its avoiding gradient saturation.
- SiLU and MSiLU: MSiLU performs better than SiLU, especially at the beginning of MNIST training with SGD.Figure 9 plots their test-accuracy training curves over 50 epochs.