Random Walk Initialization for Training Very Deep Feedforward Networks

• Feedforward back-propagated error signals typically decrease or increase exponentially with distance from the final layer, limiting the usefulness of adding depth.Source: §1 · p. 1
• Recurrent-network analyses repeatedly apply similar matrices, so eigenvalue magnitudes above or below one produce exponential growth or decay.Source: §1 · p. 1
• Appropriate initialization can constrain gradient behavior initially, but optimization almost surely destroys this fine-tuning.Source: §1 · p. 2
• Feedforward networks differ because back-propagation applies a different random matrix at every layer, yielding a gradient magnitude that scales only with the square root of depth when correctly initialized.Source: §1 · p. 2
• The paper studies successive random-matrix applications mathematically and experimentally, introducing Random Walk Initialization from the random walk of back-propagated log-norms.Source: §1 · p. 2

2.1 THE MAGNITUDE OF THE ERROR GRADIENT IN FFNS

• The network assumes width N, Gaussian weights with variance 1/N, zero biases, and a scale factor g normalized by f′(0) = 1.These assumptions define the initialization regime analyzed for linear, ReLU, and tanh nonlinearities.Source: §2.1 · p. 2
• Different random matrices at successive FFN layers make the gradient-norm analysis a random-walk problem rather than repeated-matrix dynamics.The across-layer ratio Z is kept near order 1 by adjusting the matrix scale g before learning changes the weights.Source: §2.1 · p. 3, §2.2 · p. 3
• Taking ln(Z) converts the product of layerwise gradient factors into a sum whose steps are ln(z_d), and Random Walk Initialization chooses g to make it unbiased.This logarithmic formulation avoids relying on the long-tailed extremes of the approximately log-normal Z distribution.Source: §2.1 · p. 3

2.2 CALCULATION OF THE OPTIMAL g VALUES

• Gaussian matrix projections make each normalized back-propagated vector Gaussian and z chi-squared distributed, enabling layer-independent calculations of the optimal g.For linear and ReLU networks, the analysis uses the Gaussian structure of the matrix projections; tanh is treated numerically.Source: §2.2 · pp. 3–4
• The derived optimal scaling and random-walk variance are accurate across tested widths, with variance slope inversely proportional to N.The formulas are lowest-order 1/N approximations, but numerical studies found them more accurate than expressions retaining 1/N^2 terms across the tested range.Source: §2.2 · p. 4
• For ReLU, the derivative randomly removes about half the matrix rows, while tanh lacks an analytic calculation and is expected to require an intermediate g.ReLU analysis averages over binomially distributed active-row counts and excludes zero-activity cases in small layers; tanh uses numerical results.Source: §2.2 · pp. 4–5

2.3 COMPUTATIONAL VERIFICATION

• Simulations confirm unbiased linear-network walks and linear variance growth, while predicted optimal g values match numerical data across linear, ReLU, and tanh networks.For tanh, saturation produces a wider serviceable range of g than in the linear or ReLU cases.Source: §2.3 · p. 5, Figure 1 · p. 5, Figure 2 · p. 6

3.1 RANDOM WALK INITIALIZATION

• g is set using analytical values for linear and ReLU networks, while tanh works well in practice for g between 1.1 and 1.3; input and output scaling may also require adjustment.The procedure tunes input scaling, final-layer scaling, and the network-wide g, with g identified as the most important parameter.Source: §3.1 · p. 6

3.2 EXPERIMENTAL METHODS

• Experiments evaluate MNIST and TIMIT classification plus MNIST auto-encoder reconstruction, focusing on training error rather than generalization.Networks use a standard feedforward architecture, with parameter counts controlled across depths.Source: §3.2 · pp. 6–7
• Learning rates are varied by depth through an exponential schedule so shallow and deep networks receive a fair comparison.The schedule is anchored at λout in the output layer and uses λin at the first layer only when D equals Dmax.Source: §3.2 · p. 7

3.3 PERFORMANCE RESULTS

• For MNIST, minimum training error occurs near g = 1.1–1.4 with tanh and g = 1.4–1.55 with ReLU, matching analytical predictions.Figure 3 reports these results across depths and learning-rate combinations under a 4e6 parameter limit.Source: §3.3 · p. 7, Figure 3 · p. 8
• Depth 2 gives the best MNIST classification training error, while depths 16 and 32 tie for second; the shallowest auto-encoder is also best, but depth 128 is only roughly 2x worse.A 1000-layer MNIST network achieved about 50 training mistakes, and TIMIT performed best at depth 16 with depth 32 essentially tied.Source: §3.3 · p. 8, Figure 4 · p. 9
• Across the examined tasks, increasing depth did not improve training error, but the results provide evidence that Random Walk Initialization is reasonable for very deep nonlinear feedforward networks.The experiments compare networks under equal parameter limits and varied learning-rate and g settings.Source: §3.3 · p. 8, Figure 4 · p. 9

Experimental results

• The initialization scheme allowed training to proceed for networks deeper than 200 layers, although depth did not clearly improve training error.Source: §4 · pp. 9–10

Practical implications

• Increasing layer width N decreases fluctuations in the norm of back-propagated errors.Source: §4 · p. 9

Optimization limits

• For extremely deep networks around 1000 layers, curvature can make effective optimization with a first-order method impossible.The authors set g = 1.05 in Figure 4C to address this issue.Source: §4 · pp. 9–10

Experimental results

• Correctly chosen g values enabled nearly all non-broken models to reach zero or near-zero training classification error, or extremely low reconstruction error, regardless of depth.Models could fail when learning-rate hyperparameters mismatched the architecture.Source: §4 · p. 10

Practical implications

• The authors recommend Random Walk Initialization with g from Figure 2 or the stated equations instead of g = 1 as an easily implemented default.Source: §4 · pp. 9–10