PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Selection of Initialization and Activation Function for Deep Neural Networks

Soufiane Hayou, Arnaud Doucet, Judith Rousseau

arXiv:1805.08266v2stat.MLcs.LG

TL;DR

The paper examines how initialization and activation shape information propagation in deep random networks, using an infinite-width Gaussian-process framework. It shows that edge-of-chaos initialization improves propagation depth and identifies activations including Swish that further improve information flow, with implications for richer Bayesian neural-network priors.

  • Problem

    The paper examines how initialization and activation functions affect information propagation in deep random networks, complementing earlier analyses of this behavior.

    Source: §1 · p. 1

  • Method

    The authors analyze random networks through an infinite-width Gaussian-process approximation and derive sufficient activation conditions for improved information flow at the edge of chaos.

    Source: §1 · pp. 1–2

  • Results

    On the edge of chaos, sufficient-condition activations improve information propagation beyond ReLU-like functions, and the class includes Swish and ELU.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Richer Bayesian neural-network priors require both edge-of-chaos parameters and an activation satisfying Proposition 4.

    Source: §5 · p. 9

    The analysis assumes independent centered Gaussian-process layer outputs, an idealization corresponding to infinite network width.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propagation. Understanding the theoretical properties of untrained random networks is key to identifying which deep networks may be trained successfully as recently demonstrated by Schoenholz et al. (2017) who showed that for deep feedforward neural networks only a specific choice of hyperparameters known as the `edge of chaos' can lead to good performance. We complete this analysis by providing quantitative results showing that, for a class of ReLU-like activation functions, the information propagates indeed deeper for an initialization at the edge of chaos. By further extending this analysis, we identify a class of activation functions that improve the information propagation over ReLU-like functions. This class includes the Swish activation, $\phi_{swish}(x) = x \cdot \text{sigmoid}(x)$, used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and Ramachandran et al. (2017). This provides a theoretical grounding for the excellent empirical performance of $\phi_{swish}$ observed in these contributions. We complement those previous results by illustrating the benefit of using a random initialization on the edge of chaos in this context.

1 INTRODUCTION

The paper analyzes how initialization and activation affect information propagation in deep random networks, complementing earlier theoretical work. It shows that edge-of-chaos initialization supports deeper propagation and identifies activation functions, including Swish, with improved information flow.

1 INTRODUCTION

  • The analysis follows prior work by studying infinite-width random networks with finite-variance i.i.d. weights, approximated by Gaussian processes.This framework is used to investigate the behavior of untrained networks.Source: §1 · p. 1
  • For ReLU-like activations, edge-of-chaos initialization replaces an exponential depth scale with a polynomial one, allowing information to propagate deeper.The paper complements results by Poole et al. (2016) and Schoenholz et al. (2017).Source: §1 · p. 1
  • The paper gives sufficient conditions for good information flow beyond ReLU-like activations, and Swish satisfies those conditions.The conditions also require avoiding exploding or vanishing gradients; Swish is used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and Ramachandran et al. (2017).Source: §1 · p. 2

2 ON GAUSSIAN PROCESS APPROXIMATIONS OF NEURAL NETWORKS AND THEIR STABILITY

This section sets up a Gaussian-process approximation for random neural networks and studies the limiting behavior of their variance and correlations. It characterizes convergence domains, constant-output behavior, and the edge-of-chaos condition governing correlation depth scales.

2.1 SETUP AND NOTATIONS

  • The analysis models layer outputs as independent, centered Gaussian processes, an idealization of the true processes when widths tend to infinity.The processes are described by covariance kernels κl.Source: §2.1 · p. 2
  • The covariance kernel is recursively calculated using a function determined by the activation, with variance and correlation updated across layers.The text denotes the variance by q and the correlation by c.Source: §2.1 · p. 2

2.2 LIMITING BEHAVIOUR OF THE VARIANCE AND COVARIANCE OPERATORS

  • If Mφ is finite, σw² < 1/Mφ is sufficient for variance convergence for any σb.Under this condition, the maximal convergence bound Kφ,var is infinite.Source: §2.2 · p. 3
  • For ReLU, outputs in the domains of variance and correlation convergence become constant functions almost surely; Figure 1 shows nearly constant ReLU and Tanh outputs.The figure illustrates two draws for each activation at (σb, σw) = (1, 1).Source: §2.2 · p. 3, Figure 1 · p. 3
  • When fixed points exist, Schoenholz et al. (2017) describe variance and correlation depth scales through exponential convergence rates; the edge of chaos gives an infinite correlation depth scale.The scales indicate how deeply variance and correlation propagate without being exponentially close to their limits.Source: §2.2 · pp. 3–4
  • The edge of chaos is defined by χ1 = 1, the boundary between ordered and chaotic phases and a point of infinite correlation depth scale.Below this condition correlations converge to 1; above it, they do not converge to 1.Source: §2.2 · p. 4
  • The correlation update is approximated by a limiting function f, with the edge-of-chaos condition equivalently expressed as f′(1) = 1.The approximation converges uniformly over [0, 1] under the stated conditions on the activation function.Source: §2.2 · p. 4

3 EDGE OF CHAOS

At the edge of chaos, ReLU-like networks preserve variance while their correlations approach 1 more slowly; suitable activation functions can further slow that convergence. Swish meets the paper’s conditions for this improved information flow, while comparison among qualifying activations remains open.

3 EDGE OF CHAOS

  • At (σ_b^2, σ_w^2) = (0, 2), a 20-layer, 100-neuron ReLU network’s output shows more variability, though its correlations still converge to 1 more slowly.The figure depicts one output-function draw at the edge of chaos.Source: §3 · p. 4, Figure 2 · p. 4

3.1 RELU-LIKE ACTIVATION FUNCTIONS

  • For ReLU-like activations, edge-of-chaos initialization preserves variance, and the correlation analysis gives a slower convergence rate than exponential decay.The paper presents ReLU’s rate as 1/l^2 and says the result generalizes straightforwardly to the broader class.Source: §3.1 · pp. 4–5
  • For ReLU on the edge of chaos, correlations approach 1 at polynomial rate 1/l^2 rather than exponentially, while variance is preserved across layers.The stated edge-of-chaos initialization is (σ_b^2, σ_w^2) = (0, 2), matching He et al. (2015)’s recommendation for constant variance.Source: §3.1 · pp. 4–5

3.2 A BETTER CLASS OF ACTIVATION FUNCTIONS

  • Proposition 4 gives sufficient conditions for tuning (σ_b, σ_w) so the correlation function f approaches the identity and correlations converge more slowly.The result requires conditions including σ_b approaching zero while the limiting variance q also approaches zero; ReLU does not satisfy condition (ii).Source: §3.2 · p. 5
  • Tanh-like activations provide better information flow than ReLU-like functions but suffer from vanishing gradients; Tanh and Arctan satisfy the stated sufficient conditions.The paper identifies bounded activations satisfying Proposition 4’s conditions through Proposition 5.Source: §3.2 · p. 6
  • Swish satisfies Proposition 4’s conditions; at small σ_b its correlation function approaches the identity, and its depth-30 outputs show more variability than same-architecture ReLU outputs.The paper also notes that Swish has a gradient close to 1 for large inputs, avoiding the vanishing-gradient problem described for Tanh-like functions.Source: §3.2 · p. 6, Figure 4 · p. 6
  • ELU, SELU, and Softplus also satisfy Proposition 4’s conditions, but comparing qualifying activation functions remains an open question.Source: §3.2 · p. 7

4 EXPERIMENTAL RESULTS

The experiments compare initialization choices and ReLU, Tanh, and Swish across network depths. Swish performs better than ReLU, especially in deeper networks, while Tanh’s advantage in shallow networks is limited by vanishing gradients at greater depths.

Initialization on the Edge of Chaos

  • As Swish-network depth increases, initialization away from the edge of chaos eventually leaves optimization stuck near test accuracy ∼0.1.In the ordered phase, outputs converge exponentially to the same fixed value across inputs, making the gradient approximately zero.Source: §4 · p. 7, Figure 5 · p. 7

ReLU versus Tanh

  • For depth 5, Tanh converges faster than ReLU, but for L ≥40 Tanh is stuck at very low test accuracy as small gradients leave many parameters essentially unchanged.The validation-accuracy curves use 90% confidence intervals from 30 simulations.Source: §4 · p. 8, Figure 6 · p. 8

ReLU versus Swish

  • Swish performs clearly better than ReLU, especially at depth 40, and the final-accuracy comparison reports a clear advantage for Swish at large depths.The supplied Table 2 caption identifies test accuracy across width and depth, but provides no cell values.Source: §4 · p. 8, Figure 7 · p. 8, Table 2 · p. 8

5 CONCLUSION AND DISCUSSION

The paper extends information-propagation analysis at the edge of chaos and identifies activation functions that improve propagation beyond ReLU-like functions. Its conclusions also point to activation choice, alongside edge-of-chaos parameters, as relevant to the richness of Bayesian neural network priors.

5 CONCLUSION AND DISCUSSION

  • At the edge of chaos, ReLU networks still have correlations converging to 1 at a polynomial rate, while sufficient activation-function conditions can further improve information propagation.The ReLU edge-of-chaos initialization corresponds to the approach proposed in He et al. (2015).Source: §5 · pp. 8–9
  • Tanh, Swish, and ELU satisfy the stated sufficient conditions, and Swish and ELU also avoid vanishing or exploding gradients.Swish is used in Hendrycks & Gimpel (2016), Elfwing et al. (2017) and promoted in Ramachandran et al. (2017); ELU is cited to Clevert et al. (2016).Source: §5 · p. 9
  • For richer Bayesian neural network priors, the paper indicates choosing both edge-of-chaos parameters and an activation function satisfying Proposition 4.With i.i.d. Gaussian weight and bias priors, the resulting prior is concentrated on close to constant functions even at the edge of chaos for ReLU-like activations.Source: §5 · p. 9

A PROOFS

The supplementary material provides proofs of the main-document propositions and additional theoretical and experimental results.

A PROOFS

  • The supplementary material includes proofs of the main-text propositions and additional theoretical and experimental results.The authors recall each proposition before presenting its proof.Source: A PROOFS · p. 11

A.1 CONVERGENCE TO THE FIXED POINT: PROPOSITION 1

Under stated bounds on the activation function and weight variance, the proof establishes convergence of variances and, under additional conditions, covariances. A contraction argument yields the fixed point, with an illustration comparing variance convergence for Tanh and ReLU.

A.1 CONVERGENCE TO THE FIXED POINT: PROPOSITION 1

  • If Mφ < ∞ and σw^2 < 1/Mφ, then for any σb the parameters lie in Dφ,var and Kφ,var(σb, σw) = ∞.The proof uses contraction of the variance map F and the Banach fixed-point theorem to establish convergence to a unique fixed point, independent of the input.Source: §A.1 · p. 11
  • Under the additional stated conditions, parameters lie in both variance and correlation convergence domains, with Kφ,var(σb, σw) = Kφ,corr(σb, σw) = ∞.The passage states the condition using Mφ and Cφ, alongside any σb.Source: §A.1 · p. 11
  • For covariance convergence, the limiting map has derivative bounded below 1, making it a contraction with unique fixed point c = 1.The proof concludes that both variance and correlation convergence scales are infinite.Source: §A.1 · p. 11
  • With (σb, σw) = (1, 1), the illustrated variance converges faster for Tanh than for ReLU.The passage describes variance across three different inputs as a function of layer l.Source: §A.1 · p. 12, Figure 8 · p. 12

A.2 RESULTS FOR RELU-LIKE ACTIVATION FUNCTIONS: PROOF OF PROPOSITIONS 2 AND 3

The ReLU-like analysis gives conditions for variance convergence and examines the ReLU correlation map at the edge of chaos. There, correlations move toward 1 through an increasing sequence.

A.2 RESULTS FOR RELU-LIKE ACTIVATION FUNCTIONS: PROOF OF PROPOSITIONS 2 AND 3

  • For ReLU-like activations, Proposition 2 gives conditions under which the variance convergence depth scale is infinite.The stated conditions include σ_b ≥ 0 and an upper bound on σ_w; the passage’s bound is partially truncated.Source: §A.2 · p. 12
  • The variance recursion is used to identify the parameter frontier, which reduces to a single point when σ_b = 0.For σ_b > 0, the passage says q_l diverges to infinity.Source: §A.2 · pp. 12–13
  • At the edge of chaos, the ReLU correlation sequence increases and converges to the fixed point 1.The proof establishes this by showing the correlation map is increasing and that the sequence is bounded.Source: §A.2 · p. 13

A.3 A BETTER CLASS OF ACTIVATION FUNCTIONS: PROOFS OF PROPOSITIONS 4, 5, 6 AND LEMMA 2

The section gives conditions under which edge-of-chaos initialization slows correlation convergence, then shows those conditions include Swish. It also notes that the theoretical analysis uses an equivalent Gaussian-process approximation.

A.3 A BETTER CLASS OF ACTIVATION FUNCTIONS: PROOFS OF PROPOSITIONS 4, 5, 6 AND LEMMA 2

  • For σ_b > 0 at the edge of chaos, correlations converge to 1, but the result is best understood as slowing that convergence.This holds for any inputs under the stated edge-of-chaos condition.Source: §A.3 · p. 14
  • If the minimal variance fixed point stays positive as σ_b approaches zero, the stated result holds only for linear activations.The lemma contrasts this case with Proposition 4’s limit in which q approaches zero.Source: §A.3 · pp. 14–15
  • Swish, φ_swish(x) = x · sigmoid(x), satisfies the conditions of Proposition 4.Proposition 5 supplies sufficient conditions for bounded activation functions, and Proposition 6 applies the result to Swish.Source: §A.3 · p. 15
  • For large q, f′′ has very small values because φ′′ is concentrated around zero.The accompanying discussion says E[φ′′(√qZ)φ(√qZ)] is about 0.01, so F′(q) is approximately 1.Source: §A.3 · p. 17
  • The analysis concerns the equivalent Gaussian-process approximation of the neural network.The authors state that using parameters close to the edge of chaos should not essentially change the quality of the result.Source: §A.3 · p. 17

B SUPPLEMENTARY THEORETICAL RESULTS

The supplementary result gives sufficient conditions for a bounded activation function to satisfy Proposition 4’s conditions. The stated sufficient condition is not satisfied by Tanh.

B.1 SUFFICIENT CONDITIONS FOR BOUNDED ACTIVATION FUNCTIONS

  • The alternative sufficient condition is not satisfied by Tanh.The authors state that this condition can replace condition (ii) in Proposition 5.Source: §B.1 · p. 17
  • A bounded activation function satisfying the listed shape conditions and |Eφ′(xZ)²| ≳ |x|^-2β for some β in (0, 1) satisfies Proposition 4’s conditions.The passage specifies φ(0) = 0, φ′(0) > 0, φ′(x) ≥ 0, odd symmetry, xφ(x) > 0, and xφ′′(x) < 0 for x ≠ 0.Source: §B.1 · p. 17

B.2 IMPACT OF SMOOTHNESS

For small σ_b, Hard-Tanh satisfies Proposition 4's conditions despite its undefined second derivative. In the comparison, Tanh approximates the identity function better and provides greater output variance at equal approximation quality.

B.2 IMPACT OF SMOOTHNESS

  • Hard-Tanh satisfies Proposition 4 because f′′ > 0 and its other required conditions hold, although φ′′ is undefined.The φ′′ conditions are excluded because they were used only to establish f′′ > 0.Source: §B.2 · p. 19
  • For small σ_b, Tanh outperforms Hard-Tanh in approximating f by the identity function and gives greater output variance at equal approximation quality.The comparison uses the ratio σ_b^2/q, where a smaller ratio indicates a better approximation.Source: §B.2 · p. 19
  • Hard-Tanh's nonsmoothness may slow dominated convergence in the proof of Proposition 4, potentially explaining Tanh's advantage.Source: §B.2 · p. 19

C SUPPLEMENTARY EXPERIMENTAL RESULTS

The supplementary experiments numerically examine whether ELU satisfies Proposition 4's conditions and compare ReLU and Swish when depth exceeds width. ELU's displayed properties hold on the edge of chaos, while Swish retains an advantage in deeper architectures despite the comparison's limited-width caveat.

C.1 ELU ACTIVATION

  • Numerical results indicate ELU satisfies Proposition 4's conditions, with its edge-of-chaos curves and convex f properties holding for any edge-of-chaos parameter values.The displayed F and f curves use one parameter pair, while the stated results apply to any pair on the edge of chaos.Source: §C.1 · pp. 19–20, Figure 12 · p. 20

C.2 WHAT HAPPENS WHEN THE DEPTH IS LARGER THAN THE WIDTH?

  • Swish retains a clear validation-accuracy advantage for deeper architectures when depth exceeds width, although ReLU tends to outperform it when both width and depth are small.In this regime, the Gaussian-process approximation is not accurate.Source: Table 3 · p. 20, §C.2 · p. 20

Related papers

Loading 1805.08266v2…