PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Normalizing Neural Networks

Günter Klambauer, Thomas Unterthiner, Andreas Mayr, Sepp Hochreiter

arXiv:1706.02515v5cs.LGstat.ML

TL;DR

Deep feed-forward networks rarely achieve strong results at depth, motivating a way to stabilize their activations without explicit normalization. The paper introduces SELU-based self-normalizing networks and proves convergence and variance bounds, then reports strong results across broad benchmarks.

  • Problem

    Deep feed-forward networks are typically shallow and lack success stories on non-vision and non-sequential tasks, limiting their use for multiple abstract representation levels.

  • Method

    The paper constructs SNNs with SELU activations and analyzes their mean-and-variance mapping using stable fixed points and contraction arguments.

  • Results

    SNNs outperformed other FNNs on 121 UCI datasets and achieved the best reported results on the cited drug-discovery and astronomy evaluations.

  • Takeaways & Limitations

    SNNs support deep, robust FNN learning by keeping activations near normalized values and controlling variance across layers.

  • Takeaways & Limitations

    The theoretical derivation assumes independent lower-layer activations, although the paper notes this assumption is not generally fulfilled.

Abstract

from arXiv · show

Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.

Introduction

Standard deep learning excels in vision and sequential tasks, but deep feed-forward networks have few success stories and are usually shallow. The paper attributes this gap partly to normalization sensitivity and introduces SNNs to stabilize deep FNN training.

  • Deep learning success is concentrated in CNN-based vision and RNN-based sequential tasks.
  • Non-vision and non-sequential benchmarks are mostly won by gradient boosting, random forests, or SVMs rather than deep learning.
  • FNN success stories such as HIGGS, Merck Molecular Activity, and Tox21 used at most four hidden layers.
  • Normalization makes FNN training sensitive to SGD, stochastic regularization, and normalization-parameter estimation, producing high training-error variance.
  • SNNs use SELUs to push activations toward zero mean and unit variance, reducing perturbation-related variance and supporting robust learning of many layers.

Self-normalizing Neural Networks (SNNs)

SNNs construct a mean-and-variance mapping whose stable fixed points keep activations normalized across layers. SELUs, suitable initialization, and contraction-based analysis provide perturbation robustness and bounds that control gradient behavior.

  • Normalization and SNNs: The mapping g propagates activation means and variances from one layer to the next.
  • Normalization and SNNs: A self-normalizing network keeps mean and variance in a domain and iteratively converges toward a stable, attracting fixed point.
  • Normalization Across Layers: Across layers and learning steps, SNNs damp perturbations by drawing activations toward fixed points or bounded intervals.
  • Constructing Self-Normalizing Neural Networks: SELUs are proposed as the activation function that makes an FNN self-normalizing, while weight initialization uses ω = 0 and τ = 1.
  • Constructing Self-Normalizing Neural Networks: SELUs require both negative and positive outputs, saturation, a slope above one, and continuity to regulate mean and variance.
  • Stable and Attracting Fixed Point (0, 1) for Normalized Weights: The fixed point (0, 1) is stable because the Jacobian spectral norm is 0.7877 < 1, making g a contraction around it.
  • Stable and Attracting Fixed Points for Unnormalized Weights: For unnormalized weights, stable attracting fixed points remain within bounded mean and variance intervals under the stated domain restrictions.
  • Variance Bounds: Theorems bound variance from above and below, preventing the activation variance from sustaining exploding or vanishing gradient behavior.

Experiments

Experiments compare SNNs with normalized and unnormalized FNNs, residual and highway networks, and classical methods across UCI, drug-discovery, and astronomy benchmarks. SNNs perform best on larger UCI datasets, Tox21, and HTRU2, with selected architectures typically deeper than competing FNNs.

  • UCI benchmarks: On 46 UCI datasets with at least 1,000 data points, SNNs achieved the highest performance, while random forests and SVMs led on 75 smaller datasets.Overall, SNNs outperformed state-of-the-art methods on UCI datasets with more than 1,000 data points.
  • Architecture depth: SNNs selected substantially deeper architectures, averaging 10.8 layers versus 3.8–7.1 layers for the other listed FNNs.SNNs with more than four layers often provided the best predictive accuracies among neural networks.
  • Drug discovery: On Tox21, single SNNs reached an average AUC of 0.845±0.003, close to the 0.846 AUC of the winning shallow ReLU ensemble.The best-performing SNN used eight layers, whereas competing normalized FNNs typically performed best at two to four layers.
  • Astronomy: On HTRU2 pulsar prediction, SNNs outperformed all other methods and raised the reported state of the art to an AUC of 0.98.Table 3 compares FNNs with Naive Bayes, C4.5, and SVM using AUC averaged over 10 cross-validation folds.

Conclusion

The paper introduces SNNs and proves that their activations move toward zero mean and unit variance through network layers. Across UCI, drug-discovery, and astronomy tasks, SNNs outperform competing methods, with the strongest architectures typically very deep.

  • Theoretical contribution: SNNs push neuron activations toward zero mean and unit variance during propagation through the network.The paper also proves upper and lower bounds on variance mappings away from unit variance.
  • Theoretical contribution: The variance bounds imply that SNNs avoid vanishing and exploding gradient problems.The paper connects this property with deep architectures, strong regularization, and robust learning.
  • Empirical findings: SNNs outperform other FNNs on 121 UCI datasets and achieve the best reported results on drug-discovery and astronomy tasks.The winning SNN architectures are typically very deep compared with other FNNs.

Appendix

The appendix documents the paper’s organization and the proof strategy for its main theorem. The proof uses Banach’s fixed-point theorem together with contraction and domain-invariance arguments, including a computer-assisted component.

  • Appendix organization: The appendix covers background and formulations, main theorems, proofs, computational experiments, and numerical checks of theoretical bounds.Its final section uses numerical methods as a sanity check.
  • Proof strategy: Theorem 1 is proved with Banach’s fixed-point theorem by establishing that the mapping is contractive and remains within its domain.The contraction argument relies on a computer-assisted proof whose validity is checked through error analysis and implementation precision.

A1 Background

The background formulates how SELU activations transform layer-wise means and variances through a mapping g. For normalized weights and selected SELU parameters, (0, 1) is shown to be an attracting fixed point.

  • Setup: For SELU networks, the lower-layer activations, weighted inputs, and higher-layer activations are modeled as random variables with means and variances.The analysis focuses on the fixed point (µ, ν) = (0, 1) using parameters α01 and λ01.
  • Layer-wise mapping: The mapping g sends the mean µ and variance ν of one layer to the next layer’s mean ˜µ and variance ˜ν.The variance is obtained from the second moment and squared mean through ˜ν = ˜ξ − (˜µ)^2.
  • Fixed-point behavior: Figure 2 shows that (0, 1) is an attracting fixed point of g for normalized weights and the selected SELU parameters.The visualization evaluates g at several pre-selected points.

A2.1 Theorem 1: Stable and Attracting Fixed Points Close to (0,1)

Theorem 1 establishes a stable, attracting fixed point for the mean and variance mapping near (0, 1), including perturbations in weight parameters. All points in the stated domain converge to this fixed point.

  • The mapping has a stable and attracting fixed point near zero mean and unit variance.
  • For perturbed parameters, the fixed point lies within µ ∈[−0.03106, 0.06773] and ν ∈[0.80009, 1.48617].
  • Every point in the stated (µ, ν)-domain converges to the fixed point under iterative application of the mapping.

A2.2 Theorem 2: Decreasing Variance from Above

Theorem 2 shows that SNNs reduce excessive activation variance across layers. For input variances from 3 to 16, the next-layer variance is always smaller.

  • Theorem 2 proves that activation variance decreases when propagated through the high-variance domain.
  • For ν ∈[3, 16], the next-layer variance is smaller than the original variance under the stated parameter ranges.
  • The variance decreases across the interval [3, 16], preventing variance explosion through consecutive layers.

A2.3 Theorem 3: Increasing Variance from Below

Theorem 3 shows that SNNs increase insufficient activation variance in low-variance domains. This provides a lower variance bound and prevents variance from vanishing across layers.

  • Increasing variance through layers ensures that vanishing gradients will not be observed.
  • In the low-variance domains, the next-layer variance is always larger than the original variance.
  • Theorem 3 considers two domains with input variance as low as 0.05 and parameter ranges differing from the original domain.
  • For 0.8 ≤ τ, the next-layer variance exceeds 0.16, while for 0.9 ≤ τ, it exceeds 0.24.

A3.1 Proof of Theorem 1

The proof applies the Banach fixed-point theorem by showing that the mean-variance mapping is a contraction and remains within its domain. These properties establish convergence to the fixed point.

  • The proof uses the Banach fixed-point theorem, requiring a contraction mapping that remains within its domain.
  • A contraction mapping decreases distances between points, with a factor δ satisfying 0 ≤ δ < 1.
  • The proof bounds the Jacobian’s largest singular value below one using grid evaluation and mean-value-theorem bounds.
  • With contraction and domain invariance established, the theorem yields a stable fixed point and convergence from every point in the domain.
  • The mapping is shown to remain within the domain µ ∈[−0.03106, 0.06773] and ν ∈[0.80009, 1.48617].

A3.2 Proof of Theorem 2

Theorem 2 proves that the variance mapping decreases over the domain 3 ≤ ν ≤ 16, supporting an upper-bound argument for the propagated variance.

  • The variance mapping decreases for 3 ≤ ν ≤ 16.The proof establishes this behavior for λ = λ01 and α = α01 over the stated domain.
  • Monotonicity of g is established by examining its derivatives with respect to ν, τ, µ, and ω across partitioned domains.The argument uses derivative signs and boundary values to locate the maximum.
  • The proof applies Lemma 24 to show a relevant sub-function increases in y, while its derivative analysis establishes increase in x.These derivative properties support the required monotonicity inequalities.
  • Boundary minimization and algebraic factorization reduce the remaining inequalities to checks over the domain endpoints.The proof evaluates minimal x and y values and factors the resulting expressions before solving a quadratic equation.
  • The proof analyzes g = ˜ξ − ν and shows that its maximum over the domain is −0.0180173.Because the maximum is negative, g remains below zero throughout the domain.

A3.3 Proof of Theorem 3

Theorem 3 establishes that the variance mapping increases in low-variance domains, yielding lower bounds that push the mapped variance above the domain thresholds.

  • The variance mapping increases over the analyzed low-variance domains.The proof treats domains with ν ranges [0.05, 0.16] and [0.05, 0.24].
  • Together, the fixed-point mapping ensures ˜ν > 0.16 when τ ≥ 0.8 and ˜ν > 0.24 when τ ≥ 0.9.Thus the variance mapping provides a lower bound on variance in the considered regions.
  • For 0.05 ≤ ν ≤ 0.16, the mapped variance satisfies ˜ν ≥ 1.049301ν > ν.The bound uses the minimum mapped second moment and a bound on the squared mapped mean.
  • For 0.05 ≤ ν ≤ 0.24, the mapped variance satisfies ˜ν ≥ 1.060255ν > ν.This second case uses τ ≥ 0.9 and the corresponding minimum of ˜ξ.

A3.4.1 Lemmata for proofing Theorem 1 (part 1): Jacobian norm smaller than one

The proof establishes that the mean-and-variance mapping has Jacobian norm below one on the specified domain, making it contractive and invariant within a suitable domain.

  • Jacobian analysis: The analysis studies the Jacobians of mappings from activation mean and variance to their next-layer values.It distinguishes J for (µ, ν) 7→ (˜µ, ˜ξ) from H for (µ, ν) 7→ (˜µ, ˜ν).
  • Contraction proof: A grid evaluation combined with the Mean Value Theorem bounds the largest singular value between grid points.The proof remains below one after adding gradient-based deviations from evaluated grid points.
  • Jacobian analysis: The proof bounds Jacobian entries and their derivatives over µ, ω ∈ [−0.1, 0.1], ν ∈ [0.8, 1.5], and τ ∈ [0.8, 1.25].For example, |J11| ⩽ 0.104497 and |J12| ⩽ 0.194145 in this domain.
  • Mapping bounds: The mean, variance, and second-moment mappings remain bounded, with −0.041160 < ˜µ < 0.087653 and 0.695574 < ˜ν < 1.636023.These bounds are obtained from monotonicity and boundary evaluations.
  • Contraction proof: S(µ + ∆µ, ω + ∆ω, ν + ∆ν, τ + ∆τ, λ01, α01) ⩽ 0.9912524171058772 + 0.008747 < 1.Therefore, the mapping is a contraction mapping on the analyzed domain.
  • Domain invariance: The mapping also maps the specified domain into a smaller interval, supporting the fixed-point argument used for Theorem 1.The image lies within µ ∈ [−0.03106, 0.06773] and ν ∈ [0.80009, 1.48617].
Loading 1706.02515v5…