Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

I. INTRODUCTION

• Residual networks can achieve dynamical isometry for many activation functions when initialization variance scales inversely with the number of skip connections.This contrasts with feedforward networks, where orthogonal weights and antisymmetric sigmoidal activations such as tanh are required.Source: §I · p. 1

A. Related work

• Free Probability and Random Matrix Theory extend prior feedforward Jacobian analyses to residual networks, where singular values must be analyzed directly because the Jacobian is non-normal.The paper studies the singular spectrum, equivalently the eigenvalue spectrum of JJT, rather than relying on ordinary eigenvalues.Source: A. Related work · p. 2, A. Relevance of the input-output… · p. 4

B. Our results

• The paper derives a universal Jacobian singular-spectrum formula whose activation dependence is compressed into one parameter and validates it with random-matrix and neural-network simulations.The authors report that singular values concentrate around 1 for a wide parameter range, with numerical agreement already at L = 10 and N = 500.Source: B. Our results · p. 2

II. THE MODEL

• The model uses residual propagation xl = φ(hl) + axl−1 with hl = Wlxl−1 + bl, and studies the input-output Jacobian from x0 to xL.The skip-connection parameter a is introduced to track residual influence but is not studied as a separate spectral or learning variable.Source: §II · p. 3

A. Relevance of the input-output Jacobian

• Dynamical isometry keeps Jacobian singular values near 1, preventing large values from dominating learning and small values from suppressing it.The Jacobian spectrum should remain concentrated around 1 across network depth so learning is driven by data rather than initialization.Source: A. Relevance of the input-output… · p. 3

A. Spectral analysis

• Free Probability uses R- and S-transforms to analyze sums and products of large random matrices, enabling recursive treatment of the residual Jacobian.The S-transform is especially useful for products because it factorizes for freely independent matrices when one factor is positive definite.Source: A. Spectral analysis · p. 4
• Biunitarily invariant weight matrices, including Gaussian and scaled orthogonal initializations, are treated as R-diagonal operators so the single-layer Jacobian can be analyzed in a common framework.The product DlWl remains R-diagonal under the stated assumptions.Source: A. Spectral analysis · p. 5
• The exact spectral equation is simplified in the deep-network regime by rescaling weights by 1/L and retaining the leading term in 1/L.This produces a quadratic equation for the S-transform after selecting the appropriate solution branch.Source: A. Spectral analysis · p. 5
• The Jacobian spectrum is obtained by deriving a Green’s-function equation for the squared singular values and solving it numerically across activation functions.The theoretical spectra are compared with numerical simulations in Fig. 1.Source: A. Spectral analysis · pp. 5–6
• The framework also applies when skip connections bypass multiple fully connected blocks, provided L is replaced by the number of skip connections.The paper states that the qualitative spectral results remain unchanged under this modification.Source: A. Spectral analysis · p. 6

B. Signal propagation

• The effective cumulant c is the layer-averaged squared activation derivative and is determined by how pre-activation variance propagates through the network.Calculating c requires tracking the variance of pre-activations across consecutive layers.Source: B. Signal propagation · p. 6
• Signal propagation is analyzed for iid Gaussian weights and extended to scaled orthogonal matrices, with the same qualitative results under the stated scaling assumptions.The Gaussian and scaled-orthogonal cases are considered within the initialization framework used for the spectral analysis.Source: B. Signal propagation · p. 6
• The analysis assumes Gaussian pre-activations, zero means, and ergodicity, allowing layer averages to be replaced by averages over a normal distribution with variance ql.These assumptions motivate the dynamical mean field treatment of signal propagation.Source: B. Signal propagation · p. 6
• For tanh networks, the theoretical singular-value density agrees remarkably well with simulations even at depth L = 10.The figure compares the asymptotic density with finite-depth numerical results.Source: Figure 1 · p. 7
• Smaller c produces a narrower singular-value spectrum and brings it closer to ideal dynamical isometry.The figure compares asymptotic distributions for several c values with ReLU simulations from a single initialization.Source: Figure 1 · p. 7

• Numerical simulations corroborate the theoretical predictions for Jacobian singular-value spectra across the studied activation functions.The simulations propagate an N = 500 Gaussian input through randomly initialized networks and compare empirical spectra with theory.Source: C. Random matrix simulations · p. 7
• For tanh, the Jacobian spectrum is evaluated across network depths despite dependence between the Jacobian and signal propagation.Source: C. Random matrix simulations · p. 7

• CIFAR-10 experiments use a single-representation fully connected residual network to test the theoretical predictions on real data.The model omits multiple hidden-representation stages and the two-layer residual block, which are left for future work.Source: §IV · p. 8

A. Achieving dynamical isometry for any activation function

• With c = 0.125, Jacobian eigenvalues concentrate around one, enabling dynamical isometry across activation functions and depths.Source: A. Achieving dynamical isometry for… · p. 8
• Matching the effective cumulant makes the empirical Jacobian spectrum independent of activation function and residual-block count.Source: A. Achieving dynamical isometry for… · p. 8, Figure 3 · p. 9

B. Learning dynamics are more similar at universality under dynamical isometry

• Most activations show similar training-accuracy evolution under matched dynamical-isometry initialization, but sigmoid optimization is significantly slower.The paper attributes sigmoid’s slower optimization to faster growth of pre- and post-activation variances, worsening neuron saturation.Source: B. Learning dynamics are more similar… · p. 8
• Equalizing weight variances instead of effective cumulants produces more divergent learning dynamics, especially early in training.Source: B. Learning dynamics are more similar… · pp. 8–9
• Treating the initial Jacobian spectrum as a confounding variable enables more meaningful comparisons of activation-function effects on learning dynamics.Source: B. Learning dynamics are more similar… · p. 9

V. COMMENT ON BATCH NORMALIZATION

• With batch normalization, the universal large-L equation for the Jacobian Green’s function still holds.Source: §V · p. 9
• The batch-normalization analysis is limited because the layerwise evolution of the normalization statistics’ probability densities is more complicated and beyond the paper’s scope.Source: §V · p. 9

VI. SYNOPSIS AND DISCUSSION

• The paper reports that the universal Jacobian spectrum depends on one effective cumulant and is validated with random matrices and CIFAR-10 ResNets.Source: §VI · p. 10
• Matching weight variances can produce identical Jacobian spectra across activations, reducing initialization’s confounding effect in learning comparisons.Source: §VI · p. 11

Appendix A: Spectrum of the Jacobian

• The Jacobian spectral edges are obtained by solving the derivative condition G′(z*) = 0 together with the Green’s-function equation.Eliminating the exponent and solving the resulting equations determines both edge locations and the Green’s function values.Source: Appendix A · p. 13
• c = 0 yields perfect dynamical isometry, while small c keeps the spectral support near 1 and large c creates exponentially large and small eigenvalues.The spectral support grows approximately as z± ≈ 1 ± 2√c for small c.Source: Appendix A · p. 13

Appendix B: Detailed derivation of the signal propagation

• In the large-width limit, factorization is justified because each preactivation aggregates contributions from many elements of the preceding layer.This assumption supports the recursive signal-propagation derivation.Source: Appendix B · p. 13
• The signal-propagation derivation produces a closed recursion for q_l, recovering feed-forward behavior at a = 0 and showing additive growth across residual layers at a = 1.For the studied examples, the recursion’s positive and non-negative terms imply increasing preactivation variance and no non-trivial fixed points.Source: Appendix B · p. 14
• The 1/L scaling of weight variance is essential because without it q_l grows uncontrollably with depth.The recursion uses this scaling to maintain controlled signal propagation.Source: Appendix B · pp. 13–14

Appendix C: Results for various activation functions

• The Green’s function equation can be solved numerically to obtain the Jacobian spectral density for the activation-function cases studied.The parameter α interpolates between linear activation at α = 1 and ReLU at α = 0.Source: Appendix C · p. 14
• For linear activation, the closed-form solution explicitly shows why the 1/L rescaling is required.The linear case provides the analytically available reference solution for the recurrence.Source: Appendix C · p. 14
• The analysis examines hard tanh, ReLU, and SELU activation examples, including a preactivation-variance recurrence for hard tanh and a ql recursion for SELU, with SELU recovering ReLU when β = 0 and λ = 1.Source: Appendix C · pp. 15–16
• For tanh, the variance of preactivations grows linearly with depth, and the resulting cumulants can be obtained from the recurrence.The same analysis applies to other antisymmetric activations such as arctan.Source: Appendix C · p. 15
• Sigmoid’s non-zero first moment makes post-activation means and second moments grow with depth, increasing preactivation variance and causing rapid saturation.The paper states that this saturation precludes training of deep networks.Source: Appendix C · pp. 15–16

Appendix D: Numerical verification of the recurrence relations

• Numerical simulations validate the theoretical recurrence relations and provide c as a function of weight and bias variances.The resulting parameter dependence is presented in Figure 6.Source: Appendix D · p. 16

Appendix E: Baseline

• The baseline keeps weight-matrix entry variance equal to 1/NL across activation functions instead of matching the effective cumulant and Jacobian spectrum.The paper advocates matching the effective cumulant when comparing activation functions, but uses fixed weight variance for this baseline.Source: Appendix E · p. 16
• Figure 5 compares recurrence solutions with simulations for post-activations, preactivations, and layerwise effective cumulants in 200-layer, width-800 residual networks.The small variation of c_l supports the approximation used in deriving equation (22).Source: Figure 5 · p. 17
• Figure 6 relates c to bias and weight variances; smaller c corresponds to closer dynamical isometry, while sigmoid’s low c reflects saturation.The plots use different scales.Source: Figure 6 · p. 18
• Figure 7 reports training and validation accuracy over the first 200 epochs and 100 iterations for residual networks using several activation functions.The initialization uses Gaussian weights with variance 1/NL, with α = 0.05 and α = 0.25 for leaky ReLU.Source: Figure 7 · p. 18