Self-Normalizing Neural Networks

Background

• CNNs and RNNs have achieved major benchmark and application successes in vision, speech, language, and sequential tasks, contrasting with the limited success of deep FNNs.Examples include ImageNet, skin-cancer detection, self-driving vision, TIMIT, language translation, and handwriting recognition.Source: Introduction · pp. 1–2

Problem

• Deep FNNs rarely succeed on non-vision or non-sequential benchmarks, where gradient boosting, random forests, and SVMs usually win; successful FNNs are typically shallow.The cited HIGGS, Merck Molecular Activity, and Tox21 challenges were won by FNNs with at most four hidden layers, limiting hierarchical abstraction.Source: Introduction · p. 2
• Normalization-based training makes FNNs sensitive to SGD, stochastic regularization, and normalization-parameter estimation, producing high training-error variance that hinders learning.Strong regularization such as dropout can further increase variance and cause divergence, unlike weight-sharing CNNs and RNNs.Source: Introduction · p. 2

Contribution

• SNNs use SELUs to push neuron activations toward zero mean and unit variance, stabilizing variance and avoiding exploding or vanishing gradients.This self-normalization enables robust learning of many layers without the high training-error variance observed in normalized FNNs.Source: Introduction · p. 2

Normalization and SNNs

• SELU activations keep means and variances normalized across layers by damping excessive variance, increasing insufficient variance, and drawing activations toward shared or parameter-dependent fixed points.Perturbations in lower-layer activations or weights are damped in higher layers, while differing weight statistics yield fixed points within bounded mean and variance intervals.Source: Self-normalizing Neural Networks (SNNs) · pp. 3, 5

Constructing Self-Normalizing Neural Networks

• SELU is designed with negative and positive outputs, saturation regions, slope greater than one, and continuity, properties that jointly regulate mean and variance.The construction adjusts the activation function and weight initialization; normalized initialization uses ω = 0 and τ = 1, while self-normalization can persist after learning changes the weights.Source: Self-normalizing Neural Networks (SNNs) · pp. 3–4

Applicability of the central limit theorem and independence assumption

• The mean-variance mapping is justified by approximating weighted sums as Gaussian using the central limit theorem, with convergence improving for broad layers and remaining applicable under non-identical or weakly dependent inputs.The Berry–Esseen convergence rate is n^-1/2, and the Lyapunov and weak-dependence extensions address limitations of independence and identical-distribution assumptions.Source: Self-normalizing Neural Networks (SNNs) · pp. 4, 7

Stable and Attracting Fixed Points

• SELU networks have stable, attracting fixed points: (0, 1) for normalized weights and points with µ ∈ [−0.03106, 0.06773] and ν ∈ [0.80009, 1.48617] for specified unnormalized weights.All points in the stated domain converge to the corresponding fixed point under repeated application of the mean-variance mapping.Source: Self-normalizing Neural Networks (SNNs) · p. 5

Variance Bounds

• Variance cannot remain excessively high or low: for ν ∈ [3, 16] it decreases, while specified low-variance domains increase, preventing exploding and vanishing gradients.The mapping eventually forces fixed-point variance below 3 and above 0.16 or 0.24 under the stated τ conditions; experiments observed means and variances entering the self-normalizing domain.Source: Self-normalizing Neural Networks (SNNs) · p. 6

New Dropout Technique

• Alpha dropout preserves SELU self-normalization by replacing dropped activations with the negative saturation value and applying an affine correction to retain the original mean and variance.Dropout rates of 0.05 or 0.10 were empirically found to produce good performance.Source: Self-normalizing Neural Networks (SNNs) · p. 6

Experimental setup

• The experiments tune architecture and hyperparameters by grid search on separate validation sets before evaluating accuracy or AUC on held-out test data.Compared methods include MSRAinit, BatchNorm, LayerNorm, WeightNorm, Highway, ResNet, and SNNs.Source: Experiments · pp. 7–8

121 UCI Machine Learning Repository

• Selected SNN architectures average 10.8 layers, compared with 3.8–7.1 layers for other FNNs and 6.35 residual blocks for ResNets.SNNs with more than 4 layers often provide the best predictive accuracies across neural networks.Source: Experiments · p. 7
• SNNs significantly outperform competing FNNs across 121 UCI classification tasks and outperform all competing methods on datasets with more than 1,000 data points.On 75 small datasets, random forests and SVMs outperform SNNs; on 46 larger datasets, SNNs show the highest performance.Source: Experiments · p. 7, Table 1 · p. 8

Drug discovery: The Tox21 challenge dataset

• SNNs achieve an AUC of 0.845±0.003 on Tox21, close to the challenge-winning ensemble’s 0.846, with the best method using 8 layers.The best performing SNNs have 8 layers, whereas runner-up normalized ReLU networks use 2 or 3 layers; SNNs’ advantage increases with depth.Source: Table 2 · p. 8, Experiments · p. 8

Astronomy: Prediction of pulsars in the HTRU2 dataset

• SNNs outperform all other methods on HTRU2 and set a new state-of-the-art AUC of 0.98.The evaluation uses 10-fold nested cross-validation and compares FNNs with Naive Bayes, C4.5, and SVM references.Source: Experiments · pp. 8–9, Table 3 · p. 9

Conclusion

• SNNs push neuron activations toward zero mean and unit variance, while bounding variance away from this regime to prevent vanishing and exploding gradients.These properties enable deep architectures, novel regularization, and robust learning.Source: Conclusion · p. 9
• SNNs outperform other FNNs across 121 UCI benchmark datasets, including models using normalization techniques or specialized architectures.The comparisons include batch, layer, and weight normalization, as well as Highway and Residual networks.Source: Conclusion · p. 9
• SNNs achieve the best results on drug-discovery and astronomy tasks, with their strongest architectures typically being very deep.This depth contrasts with other feed-forward neural networks.Source: Conclusion · p. 9

• The appendix covers background and formulations, main theorems, theorem proofs, computational experiments, and numerical checks of the theoretical bounds.The computational section includes details such as hyperparameter selection, while the final section provides a numerical sanity check.Source: Appendix · p. 10
• Theorem 1’s proof uses Banach’s fixed-point theorem, requiring a contraction mapping and proof that the mapping remains within its domain.These requirements are established in Subsections A3.4.1 and A3.4.2.Source: Appendix · p. 10
• The contraction proof relies on computer-assisted Lemma 12, whose validity is assessed through error analysis and implementation-precision checks.Subsection A3.4.6 compiles intermediate lemmata supporting the main lemmata and theorems.Source: Appendix · p. 10

• The framework treats lower-layer activations, weighted inputs, and higher-layer SELU activations as random variables characterized by their means and variances.For a higher-layer unit, the net input is z = w^T x and the activation is y = selu(z).Source: A1 Background · p. 11
• The mapping g sends a layer’s mean µ and variance ν to the next layer’s mean ˜µ and variance ˜ν.The next-layer variance is obtained from its second moment minus the squared mean.Source: A1 Background · p. 11
• The analysis fixes SELU parameters α01 and λ01 so that the normalized state (µ, ν) = (0, 1) is a fixed point for normalized weights with ω = 0 and τ = 1.The functions are analyzed over µ, ω ∈ [−0.1, 0.1], ν ∈ [0.8, 1.5], and τ ∈ [0.95, 1.1].Source: A1 Background · pp. 11–12
• Figure 2 shows that (0, 1) is an attracting fixed point of g when ω = 0 and τ = 1 using α01 and λ01.Source: A1 Background · p. 12

A2 Theorems

• Theorem 1 establishes SNN self-normalization through a stable and attracting fixed point near zero mean and unit variance.For ω = 0 and τ = 1, the fixed point is exactly (µ, ν) = (0, 1).Source: A2.1 Theorem 1: Stable and Attracting… · p. 12

A2.1 Theorem 1: Stable and Attracting Fixed Points Close to (0,1)

• Under α = α01 and λ = λ01, the theorem restricts µ, ω, ν, and τ to specified domains around the self-normalizing regime.The ranges are µ, ω ∈ [−0.1, 0.1], ν ∈ [0.8, 1.5], and τ ∈ [0.95, 1.1].Source: A2.1 Theorem 1: Stable and Attracting… · p. 12
• For other ω and τ values, the mapping has a fixed point with µ ∈ [−0.03106, 0.06773] and ν ∈ [0.80009, 1.48617].All points in this (µ, ν)-domain converge to the fixed point under iterative application of the mapping.Source: A2.1 Theorem 1: Stable and Attracting… · p. 12

Theorem 2: Decreasing Variance from Above

• In the high-variance range ν ∈ [3, 16], the next-layer variance ˜ν is always smaller than ν, so exploding gradients cannot occur.The theorem applies on Ω++ with −1 ≤ µ ≤ 1, −0.1 ≤ ω ≤ 0.1, 3 ≤ ν ≤ 16, and 0.8 ≤ τ ≤ 1.25, for λ = λ01 and α = α01.Source: A2.2 Theorem 2: Decreasing Variance… · p. 12

Theorem 3: Increasing Variance from Below

• Below the target variance range, activation variance increases across layers, so it does not vanish and vanishing gradients cannot occur.The result concerns the variance mapping for consecutive layers of self-normalizing networks.Source: A2.3 Theorem 3: Increasing Variance… · pp. 12–13
• Theorem 3 considers Ω−1 with 0.05 ⩽ν ⩽0.16 and Ω−2 with 0.05 ⩽ν ⩽0.24, alongside the stated µ, ω, and τ ranges.The domains use −0.1 ⩽µ ⩽0.1 and −0.1 ⩽ω ⩽0.1; τ ranges are 0.8 ⩽τ ⩽1.25 for Ω−1 and 0.9 ⩽τ ⩽1.25 for Ω−2.Source: A2.3 Theorem 3: Increasing Variance… · p. 13
• For fixed points of the variance mappings, ˜ν > 0.16 when 0.8 ⩽τ and ˜ν > 0.24 when 0.9 ⩽τ, establishing lower variance bounds.These bounds follow from the variance mappings in Eq. (5) and Eq. (4).Source: A2.3 Theorem 3: Increasing Variance… · p. 13

A3.1 Proof of Theorem 1

• The proof applies Banach’s fixed-point theorem by establishing that g is contractive and maps the function domain into itself.These are the two conditions required to obtain a unique fixed point and convergence of iterates.Source: A3.1 Proof of Theorem 1 · pp. 13–14
• The contraction condition uses the Euclidean distance and follows from the definition that distances decrease by a factor δ with 0 ⩽δ < 1.This supplies the Lipschitz condition invoked for g in the proof.Source: A3.1 Proof of Theorem 1 · pp. 13–14
• The mapping g is shown to contract because the largest singular value of its Jacobian is smaller than 1 throughout the domain.The singular value is evaluated on a grid and bounded between grid points using the Mean Value Theorem and gradient bounds.Source: A3.1 Proof of Theorem 1 · p. 13
• The theorem restricts µ and ω to [−0.1, 0.1], ν to [0.8, 1.5], and τ to [0.95, 1.1], with α = α01 and λ = λ01.Within these restrictions, (µ, ν) = (0, 1) is stable when ω = 0 and τ = 1.Source: A3.1 Proof of Theorem 1 · p. 14
• All points in the specified (µ, ν)-domain converge under iteration to a stable and attracting fixed point, with µ ∈[−0.03106, 0.06773] and ν ∈[0.80009, 1.48617].For ω = 0 and τ = 1, the fixed point is exactly (µ, ν) = (0, 1); other ω and τ yield a stable attracting fixed point depending on those parameters.Source: A3.1 Proof of Theorem 1 · p. 14
• The mapping remains inside the required domain because bounds on its mean and variance establish an image contained in µ ∈[−0.03106, 0.06773] and ν ∈[0.80009, 1.48617].The domain is compact, hence a complete metric space, satisfying the setting needed for Banach’s theorem.Source: A3.1 Proof of Theorem 1 · pp. 13–14

A3.2 Proof of Theorem 2

• The variance decreases for ν ∈ [3, 16] on Ω++, and the fixed points of the mappings satisfy the theorem’s stated condition.The domain is −1 ⩽ µ ⩽ 1, −0.1 ⩽ ω ⩽ 0.1, 3 ⩽ ν ⩽ 16, and 0.8 ⩽ τ ⩽ 1.25.Source: A3.2 Proof of Theorem 2 · p. 14
• The proof defines g as the next-layer second moment ˜ξ minus the lower-layer variance ν and shows that g < 0 implies variance contraction.This reduces the theorem to bounding g over the domain.Source: A3.2 Proof of Theorem 2 · p. 14
• Monotonicity in ν and τ, together with sign-based monotonicity in µ and ω, reduces the maximum search to boundary values.The proof sets ν = 3 and τ = 1.25, then evaluates the relevant µ,ω domain combinations.Source: A3.2 Proof of Theorem 2 · pp. 14, 15, 17
• −0.0180173 is the maximal value of g, proving ˜ν ⩽ ˜ξ < ν throughout Ω++.The argument restricts to 3 ⩽ ν ⩽ 16 after analyzing a larger domain.Source: A3.2 Proof of Theorem 2 · pp. 14, 18

Theorem 3

• The variance mapping increases on both domains, ensuring ˜ν > 0.16 for 0.8 ⩽τ and ˜ν > 0.24 for 0.9 ⩽τ.These bounds establish a lower bound on the variance ν for the stated domains.Source: A3.3 Proof of Theorem 3 · p. 18

First domain

• For 0.05 ⩽ν ⩽0.16 and 0.8 ⩽τ ⩽1.25, the derivative exceeds 0.969231 and the proof obtains ˜ν ⩾1.049301ν > ν.The minimum transformed ξ-value is 0.0662727, producing the bound ˜ν > 0.01281115 + 0.969231ν.Source: A3.3 Proof of Theorem 3 · p. 19

Second domain

• For 0.05 ⩽ν ⩽0.24 and 0.9 ⩽τ ⩽1.25, the derivative exceeds 0.976952 and the proof obtains ˜ν ⩾1.060255ν > ν.The minimum transformed ξ-value is 0.0738404, yielding ˜ν > 0.0199928 + 0.976952ν.Source: A3.3 Proof of Theorem 3 · p. 19

A3.4.1 Lemmata for proofing Theorem 1 (part 1): Jacobian norm smaller than one

• The largest singular value of the Jacobian is smaller than 1, making the mean–variance mapping a contraction over the stated parameter domain.The proof combines derivative bounds, a mean-value-theorem deviation bound, and grid evaluation yielding an upper bound of 0.9912524171058772 + 0.008747 < 1.Source: A3.4.1 Lemmata for proofing Theorem 1… · p. 28

A3.4.2 Lemmata for proofing Theorem 1 (part 2): Mapping within domain

• The mapping sends the prescribed parameter region into µ ∈[−0.03106, 0.06773] and ν ∈[0.80009, 1.48617], so its image remains within the domain.Because |˜ξ −˜ν| = |˜µ^2| < 0.004597, the variance remains in [0.8, 1.5], and the image g(Ω′) is a subset of Ω′.Source: A3.4.2 Lemmata for proofing Theorem 1… · p. 29

A3.4.3 Lemmata for proofing Theorem 2: The variance is contracting

• The variance mapping ˜ν is contracting for λ = λ01 and α = α01 over Ω+ with −0.1 ⩽µ,ω⩽0.1, 1.5 ⩽ν⩽16, and 0.8 ⩽τ⩽1.25.Its main subfunction is negative, increases with x = ντ, and decreases with y = µω at the minimum x = 1.2; these monotonicities support the contraction proof.Source: A3.4.3 Lemmata for proofing Theorem 2:… · p. 30, Figure A3 · p. 30
• 0.995063 < 1 bounds the derivative used to prove variance contraction, while supporting positivity arguments establish the required monotonicity conditions.The proof also shows ˜µ(0.01, 0.01, 1.5, 0.8) = 0.008293 > 0 and J12 > 0 throughout Ω+.Source: A3.4.3 Lemmata for proofing Theorem 2:… · pp. 31–32

A3.4.4 Lemmata for proofing Theorem 3: The variance is expanding

• Lemma 17 shows the main subfunction governing Theorem 3 is negative and strictly increasing in both variables over the specified domain.The domain is 0.007 ⩽ x ⩽ 0.875 and −0.01 ⩽ y ⩽ 0.01, with λ = λ01 and α = α01.Source: A3.4.4 Lemmata for proofing Theorem 3:… · p. 32

A3.4.5 Computer-assisted proof details for main Lemma 12 in Section A3.4.1.

• The singular-value evaluation has precision better than 292ϵ when µ, ω, ν, and τ each have precision ϵ, with Jacobian-error bounds below 78ϵ, 189ϵ, 405ϵ, and 52ϵ.For typical machine precision ϵ ≈ 10^-16, the evaluation precision is better than 10^-13; the implementation targets correctness up to 3 ulps.Source: A3.4.5 Computer-assisted proof details… · pp. 34–35

A3.4.6 Intermediate Lemmata and Proofs

• Intermediate lemmas provide monotonicity and curvature properties for exp(x), erfc(x), and exp(x^2)erfc(x), while Abramowitz bounds on erfc are displayed in Figure A4.The function exp(x^2)erfc(x) is strictly decreasing for x > 0 with positive curvature, and Figure A5 displays its graph.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 37–38
• The auxiliary function e^(x^2)erfc(x) is strictly decreasing for x > 0, while xe^(x^2)erfc(x) has x’s sign and increases toward 1/√π.The limiting upper and lower bounds for xe^(x^2)erfc(x) both approach 1/√π as x→∞.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 38–39
• The lemmas bound auxiliary functions h11(µ,ω) and h22(ν,τ), including ranges t11 = −0.01 to T11 = 0.01 and t22 = 0.64 to T22 = 1.875.Related expressions receive bounds t1 = 0.5568, T1 = 0.9734; t2 = 1.1225, T2 = 1.9417; and t3 = −0.0088388, T3 = 0.0088388.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 39–40
• Further auxiliary expressions are shown to be nonnegative or positive, monotone in ντ and µω, and bounded over the specified parameter ranges.One expression ranges from t4 = 0 to T4 = 0.000078126, while another is positive and decreasing in ντ.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 40, 43, 44
• Lemma 39 derives absolute-value bounds for derivatives of all four Jacobian entries with respect to µ, ω, ν, and τ.Each bound is obtained by minimizing and maximizing constituent terms, combining terms when this produces tighter bounds, and applying earlier lemmata.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 44–45
• The derivative-bound calculations evaluate lower and upper extrema of the component expressions and combine them into final bounds for the relevant Jacobian derivatives.Examples include a maximal absolute-value contribution of 1.53644 and term sums ranging from −0.154133 to 1.61749.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 45, 46, 48, 49
• The mapping satisfies derivative-sign properties on µ, ω ∈ [−0.1, 0.1], ν ∈ [0.8, 1.5], and τ ∈ [0.8, 1.25]: derivatives with respect to µ follow the sign of ω, while derivatives with respect to ν are positive.These properties are established for both ˜µ and ˜ξ under α = α01 and λ = λ01.Source: A3.4.6 Intermediate Lemmata and Proofs · p. 51
• At low variances, the mean mapping is bounded by |˜µ| < 0.289324.The bound applies over the stated low-variance domain with 0.02 ⩽ ντ ⩽ 0.5.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 55–56
• Lemma 42 gives bounds for derivatives of ˜µ with respect to µ, ω, ν, and τ in Ω−; symmetry in µ and ω makes the bounds apply to ω derivatives.Source: A3.4.6 Intermediate Lemmata and Proofs · p. 56
• The function ˜µ^2 is tightly bounded in Ω−, where its grid-evaluated maximum is 0.00451457.The proof combines derivative bounds with the mean value theorem and evaluates ˜µ^2 on a grid.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 57–58
• The numerical error of the ˜µ^2 evaluation is smaller than 10^-13 in the worst case.This error estimate uses the error-propagation rules derived earlier in the appendix.Source: A3.4.6 Intermediate Lemmata and Proofs · p. 58
• For the first subfunction, the approximation error satisfies 0.0113556 ≤ E(x, y) ≤ 0.0169551 on 1.2 ≤ x ≤ 20 and −0.1 ≤ y ≤ 0.1.Subtracting a safety gap of 0.0131259 makes the approximation suitable for proving the required inequality, including possible negative x.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 59–60
• The first subfunction is strictly increasing in x, remains below zero, and is strictly decreasing in y at the minimum x = 1.2.The x-monotonicity follows from a positive derivative; the y result is established only at x = 1.2 for the needed minimum.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 62, 65
• Lemma 45 shows that the main subfunction is below zero and strictly increasing in x and, at minimal x, in y over the stated parameter settings.The settings include 0.007 ≤ x ≤ 0.875 and −0.01 ≤ y ≤ 0.01, with minimal x values 0.007, 0.56, 0.128, and 0.216.Source: A3.4.6 Intermediate Lemmata and Proofs · p. 65
• For the second subfunction, the approximation error satisfies −0.000228141 ≤ E(x, y) ≤ 0.00495688 on 0.08 ≤ x ≤ 0.875 and −0.01 ≤ y ≤ 0.01.Because the approximation is smaller than the target function, an error gap of −0.0003 is introduced before proving the derivative inequality.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 66, 68
• The second subfunction’s derivative with respect to x is proven positive independently of y over 0.007 ≤ x ≤ 0.875, establishing x-monotonicity.The proof bounds the final polynomial expression using x ≥ 0.007 and x ≤ 0.875.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 66, 69
• The sub-function is strictly monotonically increasing in y for x = 0.007, 0.56, 0.128, and 0.216.The four values correspond to the parameter products ντ examined in the intermediate checks.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 70–72
• Lemma 46 analyzes the derivatives on −0.1 ≤ µ, ω ≤ 0.1, 0.00875 ≤ ν ≤ 0.7, and 0.8 ≤ τ ≤ 1.25.The proof uses Lemma 17 and examines the derivative with respect to ν, y = µω, and τ.Source: A3.4.6 Intermediate Lemmata and Proofs · p. 72
• The derivative with respect to τ is smaller than zero for maximal ν = 0.7 and ν = 0.16, and also for ν = 0.24 when 0.9 ≤ τ.For ν = 0.7, the associated function is strictly decreasing in τ because its second derivative is negative and its derivative is already negative at τ = 0.8.Source: A3.4.6 Intermediate Lemmata and Proofs · pp. 72, 76, 77, 80
• The derivative with respect to ν is larger than zero, and the derivative with respect to y = µω is larger than zero at ντ = 0.007, 0.56, 0.128, and 0.216.These positivity results follow directly from Lemma 17 and the chain rule.Source: A3.4.6 Intermediate Lemmata and Proofs · p. 72
• The proof appendix establishes that the relevant function decreases with τ for maximal ν = 0.24 on 0.9 ≤τ ≤1.25 and identifies global extrema of a related function.The related function has a global maximum at y = 0.64, x = −0.01 and a global minimum at y = 1.875, x = 0.01.Source: A3.4.6 Intermediate Lemmata and Proofs · p. 84

A4.1 121 UCI Machine Learning Repository data sets: Hyperparameters

• UCI experiments selected hyperparameters by grid search with 15% validation data, testing rectangular and conic layers while allowing every method dataset-specific adjustment.Early stopping used smoothed 100-epoch validation curves, and ties favored deeper networks, lower learning rates, and higher dropout.Source: A4.1 121 UCI Machine Learning… · p. 85

A4.2 121 UCI Machine Learning Repository data sets: detailed results

• SNNs ranked first on 46 large UCI classification tasks, whereas SVMs and Random Forests ranked ahead of SNNs on 75 small tasks.The comparison included 24 methods; SNNs ranked third on small data sets and first on large data sets.Source: A4.2 121 UCI Machine Learning… · p. 87, Table A12 · p. 90, Table A13 · p. 91

A4.3 Tox21 challenge data set: Hyperparameters

• For Tox21, grid-search tuning used the challenge winners’ validation split, and network-input densities were close to normal before and during learning.The density check is illustrated for initialization and after 40 epochs, with all methods allowed dataset-specific hyperparameter adjustment.Source: A4.3 Tox21 challenge data set:… · pp. 92, 94, Figure A8 · p. 94

A4.4 HTRU2 data set: Hyperparameters

• HTRU2 hyperparameters were selected through nested cross-validation using one non-testing fold for validation, with smoothed learning curves determining early stopping.Rectangular and conic architectures were tested, and all methods could adjust hyperparameters to the data set.Source: A4.4 HTRU2 data set: Hyperparameters · p. 95

A5 Other fixed points

• A5 gives an alternative fixed point with μ = μ̃ = 0 and ν = ν̃ = 2, yielding SELU parameters α02 = 1.97126 and λ02 = 1.06071.A6 additionally reports numerically determined bounds for derivatives of Jacobian entries over the parameter domain.Source: A5 Other fixed points · p. 97, A6 Bounds determined by numerical… · p. 97