Correlated initialization of deep residual networks

Felix Benning; Ivan Nourdin; Giovanni Peccati

Source-linked AI summary

Correlated initialization of deep residual networks

Felix Benning, Ivan Nourdin, Giovanni Peccati

arXiv:2609.03589v1Sep 3, 2026math.PRcs.LGstat.ML

TL;DR

The paper asks whether correlated layerwise initialization interpolates between ODE and Brownian SDE limits. It develops a generalized correlated-initialization framework and identifies scaling-dependent depth behavior, while noting unresolved conditions for proving super-critical blow-up.

Problem
Correlated initialization raises whether intermediate correlation structures can interpolate between the ODE and Brownian SDE limits.
Method
The paper analyzes depth-dependent residual scaling for correlated weights using a Banach-space formulation of residual updates.
Results
Scalings larger than L^-1/2 lead to explosion, whereas smaller scalings suppress random fluctuations and converge to the identity map.
Takeaways & Limitations
The proposed phase diagram suggests that initialization criticality need not coincide with maximal local updates during training.
Takeaways & Limitations
The conjectured super-critical blow-up remains unproved because it requires suitable lower bounds on the activation function.

Abstract

from arXiv · show

We study the large-depth behavior of residual networks whose weights are correlated across layers at initialization. Our results confirm and extend a conjecture of Marion et al. [2025], according to which correlated initializations should interpolate continuously between the Brownian stochastic differential equation arising from independent initialization and the ordinary differential equation arising from perfectly correlated initialization. When the initialization is obtained from the application of a feature function to a stationary Gaussian sequence with regularly varying correlation, we prove that there exists a unique critical scaling such that the infinite-depth limit is the solution of a Young differential equation driven by a Hermite process. Hermite processes reduce to the fractional Brownian motion if the feature function generating the initialization has Hermite rank one, which is the case for the identity function, for example. We show that the critical scaling and asymptotic limit are uniquely determined by the decay of correlations together with the Hermite rank of the feature function. Consequently, the correlation structure and Hermite rank of the initialization represent meaningful hyperparameters in the asymptotic regime. By contrast, under finite-variance iid initialization, the asymptotic driver is universally Brownian up to normalization regardless of the choice of distribution. Our proofs rely on a collection of novel results establishing a robust stability theory for Young differential equations in Banach spaces.

1 Introduction

The paper analyzes how correlated layer initialization changes the large-depth limits of residual networks, extending the Brownian and ODE endpoint regimes to intermediate Hermite-process-driven Young differential equations. It identifies the critical scaling and limiting dynamics from correlation decay and Hermite rank, supported by a stability theory for Young differential equations.

The residual update is scaled by λ_L, which controls each layer’s contribution to the network transformation and is distinct from batch-normalization-based scale control.
Correlated initialization interpolates between Brownian SDE limits for independent weights and ODE limits for smoothly varying weights, motivating a rigorous treatment of intermediate dependence.
For stationary Gaussian sequences with regularly varying covariance, a unique critical scaling yields convergence of interpolated hidden states to a Young differential equation driven by fractional Brownian motion.
For nonlinear feature functions of Hermite rank q, the limiting driver is a Hermite process, with fractional Brownian motion recovered at q = 1 and the scaling and limit jointly determined by correlation decay and q.
The results make correlation decay, Hermite rank, one-layer distribution, and depth scaling meaningful initialization hyperparameters, unlike finite-variance iid initialization’s Brownian universality.
The convergence proof relies on Banach-space stability and Euler-approximation results for parameter-dependent Young differential equations.

2 Depth Limit with correlated weights at initialization

The paper characterizes large-depth limits of ResNets with layer-correlated initialization, showing that suitably normalized correlated updates converge to Hermite processes and hidden states to Young differential equations.

Correlated residual updates converge to Hermite processes, which drive the limiting Young differential equation for the interpolated hidden states.The identity feature yields fractional Brownian motion, while general Hermite rank q yields a rank-q Hermite process.
The generalized ResNet uses h_l+1 = h_l + λ_L σ(w_l,h_l)v_l, with λ_L controlling each layer’s residual-update magnitude.The architecture maps an input through hidden layers and then applies output processing to h_L.
For identity features, the correlated Gaussian random walk converges to fractional Brownian motion, the rank-one Hermite-process case.The identity feature has Hermite rank one.
For a centered feature function of Hermite rank q with αq < 1, the normalization and limit change, producing a generally non-Gaussian Hermite-process driver.The Hermite rank is the first nonzero coefficient index in the feature function’s Hermite expansion.
The limiting argument separates v_l, which provide increments of the driver, from w_l, which approximate a limiting parameter path.Convergence of both interpolated processes is transferred to hidden-state convergence through Young-equation stability.
At critical scaling, the interpolated hidden states converge in Hölder topology to the unique Young differential-equation solution, while sub-critical scaling gives a trivial limit.The paper states that larger super-critical scaling may amplify the driver, but proving hidden-state blow-up requires lower bounds on the activation.

3 Discussion and experiments

The discussion distinguishes rigorously established initialization limits from conjectural training behavior and uses experiments to examine scaling choices. The phase diagram and MNIST results suggest that initialization criticality need not identify the scaling most favorable for training.

3.1.1 Overview of Chizat [2026]: For independent initialization, prior analyses associate λL = L^-1/2 with stochastic criticality, while structured training changes can accumulate an order-one first-order effect despite vanishing layerwise displacements.The quadratic Taylor remainder is reported to vanish asymptotically, supporting a locally linearized description.
3.2 Experiments: The experiments do not establish whether the critical initialization scaling or λL = L^-1 is more favorable for training.The authors note that empirical evidence comparing L^-1/2 and L^-1 remains inconclusive.
3.1.2 A conjectural phase diagram: Theorem 2.11 proves a non-trivial Hermite-process limit at critical initialization scaling, while the surrounding training phase diagram remains conjectural.The conjectured regions include blow-up and locally linearized training regimes extrapolated from independent-initialization analysis.
3.1.2 A conjectural phase diagram: Criticality at initialization need not coincide with maximal local parameter updates during training.The proposed phase diagram separates the initialization-critical curve from the scaling associated with maximal local updates.
3.2 Experiments: MNIST accuracy experiments plot ten-epoch performance against correlation index α and scaling exponent γ, with the dashed red line marking the critical exponent.For Hermite ranks one and two, low accuracy lies predominantly below the theoretical critical curve, but the empirical transition slope differs from that curve.

4 Young integral equation solution theory

This section develops a Banach-space solution theory for non-homogeneous Young differential equations. It establishes existence, uniqueness, stability, and Euler approximation results needed for the residual-network large-depth limit.

Assumptions: The theory permits time-dependent coefficients that are locally Hölder continuous in time and locally Lipschitz in parameters and states.The optional differentiability condition in the parameters yields stability with respect to parameter paths.
Young integration: The framework treats non-differentiable drivers that are sufficiently Hölder regular for Young integration, whereas rougher drivers would require rough path theory.The integral is defined through limits over partitions when the relevant Hölder exponents sum to more than one.
Solution theory: Theorem 4.6 gives unique Young solutions for equations driven by sufficiently regular Hölder paths under an α-nice coefficient assumption.The driving and parameter paths satisfy complementary Hölder regularity with exponents whose sum exceeds one.
Stability: The solution flow is locally Lipschitz in the initial condition, driving signal, and parameter path.This stability extends standard homogeneous Young-equation theory to non-homogeneous coefficients and supports continuity of the solution map.
Euler approximation: Piecewise-linear Euler interpolations converge to the Young differential-equation solution in Hölder topology and hence in supremum norm.This approximation theorem is identified as a key ingredient in proving large-depth convergence of ResNets.

5.1 Proof of Theorem 2.11

The proof of Theorem 2.11 combines functional convergence of correlated initialization paths, continuity of the Young solution map, and Euler-scheme convergence. Tightness and boundedness arguments complete convergence for the interpolated ResNet states.

Technical conditions: The proof selects auxiliary Hölder exponents below the limiting regularity so that the continuity and Euler estimates apply.The exponent constraints include γ1 < H - 1/2 and γ2 < βw.
Proof strategy: The correlated initialization paths converge in Hölder space to the Hermite-process driver, with joint convergence for the relevant independent components.The proof invokes functional limit theorems and product-space convergence.
Proof strategy: Continuity of the Young differential-equation solution map transfers convergence of initialization paths and parameters to convergence of the limiting solutions.The argument uses the stability assumptions and the continuous mapping theorem.
Proof strategy: Euler discretization convergence identifies the finite-depth ResNet interpolation with the limiting differential equation.The proof applies the Euler method after conditioning on bounded driver and parameter norms.
Proof strategy: Tightness of Hölder norms controls the probability of leaving bounded sets, allowing the conditioned convergence argument to pass to the full sequence.Convergence in Hölder space implies tightness of the corresponding norms, and the parameter-path bound is handled analogously.

5.2 Proof of Example 2.7

The proof of Example 2.7 verifies the regularity assumptions required by the Young-equation theory for the neural-network coefficient map. It establishes boundedness and continuity controls using properties of the activation function and matrix norms.

Regularity verification: Boundedness of the activation function yields boundedness of the coefficient map.The proof explicitly links boundedness of σ to boundedness of ψ.
Regularity verification: The coefficient map satisfies a continuous local Lipschitz bound in the state and parameter variables.The proof decomposes activation differences row by row and uses the Lipschitz constant of ψ together with norm equivalence.
Regularity verification: The same argument verifies the required Fréchet-derivative regularity, with the extra matrix-row factor handled by applying the preceding bound to ψ'.The proof uses the triangle inequality to reduce derivative estimates to the established activation estimates.
Regularity verification: Collecting the estimates produces continuous control functions for the assumptions needed by the solution theory.The argument defines the final control function from the bounds obtained for the coefficient and its derivative.

5.3 Proof of Theorem 4.6

The proof establishes existence, uniqueness, and Hölder regularity by solving the equation on finitely many small intervals and gluing the local solutions. Banach’s fixed point theorem applies because the local operator preserves a suitable ball and is contractive.

Existence and uniqueness: The operator F_k maps the ball B_k into itself under the stated boundedness conditions.This is the content of Lemma 5.3 and is used to ensure the fixed-point iteration remains in the admissible ball.
Existence and uniqueness: F_k is a contraction on B_k, providing the second hypothesis required by Banach’s fixed point theorem.The contraction estimate uses continuity of the Young integral and bounds on the coefficient function.
Existence and uniqueness: Banach’s fixed point theorem yields a unique local fixed point on each small interval, which the proof concatenates into a solution on the full interval.The construction uses intervals [t_k,t_{k+1}] and takes each fixed point’s endpoint as the next initial condition.
Regularity and uniqueness: Local α-Hölder estimates glue to a global α-Hölder solution, with the resulting constant controlled by the finite number of intervals.Lemma 5.5 supplies the gluing step, and the interval count is uniformly bounded.
Regularity and uniqueness: Uniqueness extends from each fixed-point ball to any pair of α-Hölder solutions by choosing η as the maximum of their Hölder seminorms.Induction over the interval partition places both solutions in the same balls.

5.4 Proof of Theorem 4.8

The proof of the Euler approximation theorem obtains uniform sup-norm and Hölder bounds through incremental flow restarts, local Lipschitz estimates, and discrete sewing. These bounds enable convergence in a lower Hölder space.

Sup-norm convergence: The proof controls the Euler scheme in sup-norm by incrementally restarting the flow at Euler points inside a sufficiently large ball.Local Lipschitz continuity in the initial condition is combined with induction to control both the ODE solution and its discretization.
Uniform Hölder bounds: Uniform Hölder bounds for the Euler discretization are obtained by controlling increments on anchor intervals and then gluing those bounds globally.The anchor construction keeps the number of intervals uniformly bounded, while discrete sewing controls increments within each interval.
Hölder convergence: Uniform convergence is converted into convergence in a lower Hölder norm using uniform Hölder bounds and an interpolation inequality.Lemma 5.8 applies with an exponent gap ε between the available and target Hölder regularities.
Discrete sewing: A discrete sewing lemma bounds the discrepancy between stepwise integral approximations and endpoint increments, supplying the key local Hölder estimate.The argument controls the addition defect before iterating the discrete sewing estimate.