PaperlayerView exact revision on arXiv

Source-linked AI summary

Width and Depth Limits Commute During Training

Ruijia Zhang, Soufiane Hayou

arXiv:2610.30729v1cs.LGstat.ML

TL;DR

This paper asks whether width and depth limits that commute at initialization still commute during training, when gradient descent couples residual layers. It analyzes linear residual networks with clipped and ordinary gradient descent, proving joint-path L2 convergence for clipped training and weaker convergence in probability without clipping.

  • Problem

    It is unclear whether width and depth limits continue to commute during training, because gradient descent couples residual weights across layers.

    Source: §1 · p. 1

  • Method

    The authors analyze linear residual networks by taking the depth limit at finite width and then the width limit of a continuous-depth system, with width-uniform estimates.

    Source: §4 · p. 4

  • Results

    With clipping, outputs converge in L2 to the same deterministic limit along every joint width–depth path for any fixed training horizon; without clipping, commutativity holds in probability but not in L2.

    Source: §1 · p. 1, §6 · p. 10

  • Takeaways & Limitations

    For linear residual networks with L^-1/2 branch scaling, clipping yields width–depth commutativity in L2 throughout any fixed finite training horizon.

    Source: §4 · p. 4, §6 · p. 10

    The depth convergence rate L^-1/2 is an upper bound from the discrete-to-continuous proof estimates and is not claimed to be tight.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by $1/\sqrt{depth}$. In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same $1/\sqrt{depth}$ residual-branch scaling (Depth-$\mu$P) and the $\mu$P width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in $L^2$ along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding $L^2$ convergence does not hold in general, but we establish the weaker convergence in probability.

1 Introduction

The paper asks whether width–depth commutativity at initialization persists during training, when gradient descent couples residual weights across layers. It establishes the answer for linear residual networks under L^-1/2 branch scaling, distinguishing clipped from ordinary gradient descent.

1 Introduction

  • Whether commutativity persists during training is open because gradient descent couples residual weights across layers, unlike the independent layer weights at initialization.Prior initialization arguments therefore do not directly extend to trained outputs.Source: §1 · p. 1
  • With clipped gradient descent, width and depth limits commute in L2 for any fixed training horizon, with errors of order n^-1/2 in width and L^-1/2 in depth.The proof takes the depth limit at finite width, then the width limit of a continuous-depth model, using estimates uniform in width.Source: §1 · p. 1
  • Without clipping, the limits commute in probability after one step, but not in L2; the depth-first L2 limit fails at every fixed finite width.The second moment diverges as depth grows, despite convergence in probability.Source: §1 · p. 1
  • Earlier work established commutativity for L^-1/2 branches at initialization [15] and during training for L^-1 branches [21]; this paper addresses the L^-1/2 training regime.The paper distinguishes the L^-1 scaling's ODE-style dynamics from the SDE-style dynamics associated with L^-1/2 scaling.Source: §1 · p. 2

2 Setup and Notation

The setup uses a linear residual network with Depth-µP and µP scaling, defines commutativity through a deterministic output limit, and specifies the gradient-descent training setting.

2.1 Model and parameterization

  • The linear residual network uses an L^-1/2 branch multiplier and µP initialization: input weights have variance d^-1, residual weights n^-1, and output weights 1.The input weights and output weights are fixed for the main analysis; only residual weights are trained, and randomness comes from initialization.Source: §2.1 · p. 2

2.2 Width–Depth Commutativity

  • L2 width–depth commutativity requires a deterministic output limit as min{n, L} grows, independent of the relative growth of width and depth.The paper also considers the weaker notion of convergence in probability; L2 commutativity implies convergence in probability.Source: §2.2 · pp. 2–3

2.3 Training dynamics

  • Training uses gradient descent on a quadratic loss for a single input–target pair, with a learning rate fixed independently of width and depth.The setup can be extended to finite training datasets; the weights at initialization follow the stated scaling rules.Source: §2.3 · p. 3

3 Width and Depth Limits at k = 1

After one gradient step, width and depth limits commute in probability but not in L2: at every fixed finite width, the depth limit already fails in L2. The unbounded first update as depth increases motivates clipping.

3 Width and Depth Limits at k = 1

  • At every fixed finite width, the network output diverges in L2 as depth tends to infinity after the first gradient step.The proposition states this for n ≥ 1, η > 0, and y⋆ ∈ R.Source: §3 · p. 4
  • After one gradient step, width and depth limits commute in probability, but not in L2.The depth-first L2 limit fails at every fixed finite width.Source: §3 · p. 4
  • The depth-wise L2 failure rules out an unrestricted joint L2 limit under ordinary gradient descent.For each n, a sufficiently large depth can be chosen so the second moment is at least n while both dimensions grow.Source: §3 · p. 4
  • The L2 failure arises from an unbounded first gradient-descent update as depth increases, motivating clipped gradient descent.Clipping uniformly controls each update independently of width and depth.Source: §3 · p. 4

4 Width–Depth Commutativity during Training

For any fixed training horizon, clipped gradient descent yields a deterministic output limit in L2 along every joint width–depth growth path. The proof handles training-induced layer coupling with width-uniform depth estimates and a separate width-limit comparison.

4 Width–Depth Commutativity during Training

  • For any fixed number of training steps, clipped gradient descent gives the same deterministic output limit in L2 along every joint width–depth growth path.The bound holds along any sequence n, L → ∞, establishing commutativity as min{n, L} → ∞.Source: §4 · p. 4
  • The theorem's O(L^-1/2) depth rate is an upper bound, not a claimed tight rate; empirical results suggest it may be sharpened.By contrast, the O(n^-1/2) width term is stated to be un-improvable.Source: §4 · pp. 4–5

4.1 Understanding the complexity of commutativity during training

  • After a gradient-descent update, residual blocks are coupled across layers, so the initialization argument in [15] does not extend directly to training.The updated blocks depend on initial weights from other layers, removing the independent-increment Brownian coupling used at initialization.Source: §4.1 · p. 5

4.2 Proof Overview

  • The depth-discretization error remains O(L^-1/2) uniformly in width through any fixed number of clipped training steps.The proof preserves this rate by induction over training updates.Source: §4.2 · pp. 6–7
  • For each fixed training step, the continuous-depth model approaches its deterministic width limit with an O(n^-1/2) error.The comparison uses an intermediate surrogate and controls the finite-width-to-surrogate discrepancy inductively.Source: §4.2 · p. 7
  • The proof combines the width and depth error bounds by the triangle inequality to establish the joint limit.The two comparisons are controlled separately before being combined.Source: §4.2 · p. 7

5 Experiments

The experiments study width–depth commutativity during training on synthetic regression and HIGGS, then examine convergence rates and nonlinear activations. Across five joint scaling paths, outputs approach a common value as scale grows; the nonlinear experiments show the same pattern for five activations.

5.1 Commutativity during Training

  • Across five width–depth scaling paths, outputs on synthetic regression and HIGGS approach a common value as the common scale grows.At small scales, outputs differ across paths.Source: §5.1 · p. 8

5.2 Width–Depth Convergence Rate

  • Width error decreases steadily, while depth error drops rapidly at small L before finite-width effects produce a plateau.The sweeps vary one dimension while fixing the other at 12,288 and cover training iterations 1, 5, 10, and 15.Source: §5.2 · p. 9, Figure 3 · p. 9
  • The width convergence rate stays close to n^-1/2, while the depth-dependent error approaches an empirical L^-1 rate after initial iterations.The depth exponent is estimated over the decreasing regime of the RMS error curve.Source: §5.2 · p. 9, Figure 3 · p. 9
  • The theorem’s L^-1/2 depth rate is an upper bound from the discrete-to-continuous proof estimates, not a claimed tight rate.The faster empirical depth rate suggests the finite-depth bound may be sharpened.Source: §5.2 · p. 9

5.3 Nonlinear Activations

  • Outputs along different width–depth paths converge with increasing scale for ReLU, tanh, GELU, SiLU, and sigmoid across training iterations.The experiments train for 10 updates; theoretical guarantees remain restricted to linear residual networks.Source: §5.3 · p. 9, Figure 4 · p. 10

6 Conclusion

The paper concludes that width and depth limits commute during training under the stated settings, with the convergence mode depending on gradient clipping. It also reports its AI-assistance process and points to the locations of theoretical proofs and experimental details.

6 Conclusion

  • With ordinary gradient descent, width and depth limits commute in probability for any fixed number of iterations, while the depth-first L2 limit fails after one step at every fixed finite width.With clipped gradient descent, the paper proves L2 commutativity for any fixed number of iterations, with errors of order n^-1/2 and L^-1/2.Source: §6 · p. 10

AI use statement

  • OpenAI Codex assisted with proof auditing, consistency checking, manuscript revision, and LaTeX formatting; the authors reviewed that work and take responsibility for the final content.Source: AI use statement · p. 11

Reproducibility statement

  • Theoretical results are proved in the appendices, while experimental results use synthetic data and the publicly available HIGGS dataset.Experimental settings, hyperparameters, and implementation details are in Appendix G.Source: Reproducibility statement · p. 11

A Proof of the L2 Divergence at k = 1

This appendix proves Proposition 3.2 and develops depth-discretization comparisons used for the analysis. Its width-uniform approximation couples residual matrices to matrix Brownian motion and establishes estimates uniform in width and depth.

A Proof of the L2 Divergence at k = 1

B Depth Discretization

  • The depth-discretization proof compares the trained residual network with a continuous-depth model.The comparison is used to establish the depth limit at finite width before taking the width limit.Source: B Depth Discretization · p. 14

B.1 Width-uniform base approximation

  • The width-uniform approximation couples Gaussian residual matrices to matrix Brownian motion, whose diffusion coefficient is globally Lipschitz with a constant independent of width.The stated discrete-to-continuous comparison uses constants independent of both width n and depth L.Source: §B.1 · pp. 14–15

B.2 Finite-training depth comparison

The finite-training depth comparison shows that the depth-discretization rate persists through every fixed training step. The subsequent width-limit argument compares finite-width states with surrogate states and then compares those surrogates with deterministic limits.

B.2 Finite-training depth comparison

  • The depth-discretization rate O(L^-1/2) is preserved through every fixed training step, with the cumulative update bounded uniformly over width and depth.Source: §B.2 · p. 17
  • Reversing depth and transposing matrices makes the backward equations take the same form as the forward equations, so the corresponding estimates also apply backward.Source: §B.2 · pp. 17–18

C Width Limit of the Continuous-Depth Model

  • The width-limit proof introduces surrogate states, compares them with the deterministic limiting states, and separately compares the finite-width states with the surrogates.At the output level, the two comparisons meet through the output identity.Source: C Width Limit of the Continuous-Depth… · p. 20

C.1 Surrogate construction

The construction compares finite-width training states with a surrogate whose matrix-flow components remain finite-width while its integral coefficients come from the limiting model. The approximation is controlled uniformly across width and, by induction, across any fixed training horizon.

C.1 Surrogate construction

  • The surrogate keeps the matrix-Brownian flow Φ, W0x, and ξ finite-width while taking its integral coefficients from the limiting model.Source: §C.1 · p. 21
  • For fixed T and even p ≥2, the surrogate-to-limit comparison is bounded by a constant independent of width n.The comparison is stated in Proposition C.1.Source: §C.1 · p. 21

C.2 Approximation by the surrogate

  • The finite-width-to-surrogate induction controls its error at order n^-1/2, yielding fixed-horizon approximation estimates.The listed terms in the induction are each of order n^-1/2, and the induction is carried through fixed T.Source: §C.2 · pp. 24–25

D Commutativity in Probability

The section establishes commutativity of width and depth limits in probability during training, then outlines the random-matrix comparisons and limiting system used to describe the dynamics. The limiting construction represents finite-width states through a noncommutative flow acting on width-limit state components.

D Commutativity in Probability

  • For fixed T, the width and depth limits commute in probability during training and have deterministic output limits.The result is stated in Proposition D.1.Source: D Commutativity in Probability · p. 25

E Random-Matrix Estimates

  • The random-matrix estimates compare normalized traces of finite-width matrix-flow products with their infinite-width limits, supporting an n^-1/2 surrogate-output approximation.Source: E Random-Matrix Estimates · p. 27

E.1 Deterministic limiting system

  • In the width limit, matrix coefficients of finite-width states are represented in H0, and the limiting flow uses Us,t to act on those state components.Finite-width states are linear combinations of W0x and ξ, while the limiting action is induced by the noncommutative limit us,t.Source: §E.1 · p. 27

E.2 Matrix-Brownian trace comparison

This section develops uniform trace and moment comparisons for matrix and circular flows, including surrogate and clipped coefficients, alongside stability bounds for trained residual-network states and discrete and continuous error equations.

E.2 Matrix-Brownian trace comparison

  • For fixed-length matrix-flow words, Proposition E.1 gives trace comparison with circular-flow words uniformly over depth intervals, with constants independent of width and the intervals.A matrix-flow word may include transposed factors and overlapping or coincident intervals.Source: §E.2 · p. 28

E.3 Trace comparison for surrogate expressions

  • For fixed T and even p, Proposition E.2 establishes n^-1 trace estimates for the surrogate coefficient matrices with constants independent of width.The coefficient comparisons follow by expressing surrogate coefficients as finite sums of depth integrals of matrix-flow words and applying Proposition E.1.Source: §E.3 · pp. 30–31

F Stability under Clipped Training

  • Global clipping bounds the cumulative training update by kητ in L2, enabling control of trained states and preservation of the O(L^-1/2) and O(n^-1/2) comparison rates at the next update.The backward state is not measurable with respect to the forward filtration, so the predictability condition used at initialization is unavailable during training.Source: F Stability under Clipped Training · p. 32

F.1 Moment control during training

  • When the trained update process has bounded L2 norm, the trained propagator is uniquely defined and has width-uniform moments of every order p ≥2; corresponding bounds hold for reversed and transposed flows and discrete states.The proposition assumes a finite bound Λ on the update-process L2 norm.Source: §F.1 · pp. 32, 34
  • Gaussian initialization and reversal with transposition yield moment bounds for the forward and backward states, and taking the maximum over k ≤ T gives the discrete state bound.Source: §F.1 · p. 35

F.2 Error propagation during training

  • Under a uniformly bounded L2-in-time coefficient V, discrete and continuous error equations have stability constants independent of width n, depth L, and starting index a; the bounds also hold for reversed, transposed equations.The proposition assumes p ≥ 2 and ∥V∥L2 t (F) ≤ Λ almost surely.Source: §F.2 · p. 35
  • Lemma F.3 gives a clipped-coefficient comparison with a constant independent of width n.Source: §F.2 · pp. 37–38

F.3 Extension to Finite Training Sets and Vector-Valued Outputs

The paper extends its commutativity result from a single training pair and scalar output to fixed finite training sets and fixed finite output dimensions. Its proof extends the state and surrogate comparisons to finite collections of Gaussian vectors and uses fixed-size sums in the estimates.

F.3 Extension to Finite Training Sets and Vector-Valued Outputs

  • The commutativity result holds for any fixed finite training set and any fixed finite output dimension.The main theory uses a single training pair and scalar output to simplify notation.Source: §F.3 · p. 38
  • For fixed T, τ, Ntr, and dout, Proposition F.4 gives deterministic output limits for each training input and training step.The stated scope fixes the number of training examples and output dimensions independently of width and depth.Source: §F.3 · p. 39
  • The extension represents states using a finite family of Gaussian vectors, and fixed Ntr and dout make the finite sums change only the constants.Source: §F.3 · p. 40

G.1 Experimental and Training Details

The appendix specifies the training setups and rate-fitting procedure, then extends width–depth comparisons across trajectories, examples, and a single-datum task. It also reports empirical error-rate behavior and a memory-saving implementation for the largest synthetic models.

G.1 Experimental and Training Details

  • The synthetic experiments use a fixed dataset with d = 16 and N = 128, while HIGGS uses fixed training and test splits and minibatch SGD.The synthetic runs use full-batch gradient descent; HIGGS training runs for 5,126 updates over two epochs, using five model seeds.Source: §G.1 · p. 43
  • Width rates are fit at fixed reference depth for n ≥ 1536, while depth rates use the decreasing error regime and estimate the large-depth error level separately.The empirical reference is the coordinatewise average of 25 output vectors at m = 3072.Source: §G.1 · p. 43
  • Width error stays close to n^-1/2 throughout training, while depth error approaches an empirical L^-1 rate after the initial iterations.Table 1 reports the full trajectories.Source: §G.1 · p. 43, Table 1 · p. 43

G.2 Additional Results on Width–Depth Commutativity

  • Synthetic results include all 16 saved iterations for one training input and four additional fixed training inputs.Figure 5 shows means over five seeds with ±1 standard deviation; Figure 6 covers the additional inputs.Source: §G.2 · p. 44, Figure 5 · p. 44, Figure 6 · p. 45
  • HIGGS results extend the fixed-example comparison to all saved checkpoints and three additional test examples.The logit margin is tracked from initialization through two training epochs on common scales m ∈ {16, 32, 64, 128, 256}.Source: §G.2 · p. 44, Figure 7 · p. 46, Figure 8 · p. 47

G.3 Single-Datum Experiment

  • In the single-datum experiment, finite-scale path differences diminish as width and depth increase, matching the transition seen in the multi-sample experiment.The experiment uses one input–target pair and evaluates five scaling paths across m = min(n, L) values from 128 to 3072.Source: §G.3 · pp. 44, 46, Figure 9 · p. 48

G.4 Experimental Details

  • The largest synthetic model, with (n, L) = (61,440, 3,072), would require approximately 46.4 TB to store residual matrices in float32.An exact-in-law lazy Gaussian implementation reduces peak GPU memory to 46.9 GB.Source: §G.4 · p. 46, Table 2 · p. 46
  • Additional HIGGS plots organize three fixed test examples by rows and representative checkpoints by columns, showing means over five seeds and ±1 standard deviation.Source: Figure 8 · p. 47
Loading 2610.30729v1…