PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae

arXiv:2610.35514v1stat.MLcs.LG

TL;DR

Whether network depth changes training-time scaling under standard squared loss has remained unsettled. The paper refines the analysis of deep linear networks using power-law task spectra and predictor-norm time. It finds depth improves the asymptotic exponent for difficult tasks, while the reparameterized-time exponent is depth-independent.

  • Problem

    Although scaling exponents are linked to data and task structure, whether network depth changes training-time scaling under standard squared loss remained unresolved.

    Source: §1 · p. 2

  • Method

    The paper analyzes end-to-end predictor dynamics in deep linear networks on power-law spectral tasks and reparameterizes time through the predictor norm.

    Source: §1 · p. 2

  • Results

    Depth improves the asymptotic training-time scaling exponent for spectrally difficult tasks, while predictor-norm time yields a depth-independent spectral exponent and depth does not improve the power-law exponent for easy tasks.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results provide a basis for studying how task structure and architecture jointly shape scaling laws in broader classes of neural networks.

    Source: §7 · p. 10

    The theory assumes a linear teacher and student and population gradient flow, and standard-initialization results are limited to difficult tasks.

    Source: §7 · p. 9

Abstract

from arXiv · show

Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easy tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.

1 INTRODUCTION

Neural scaling exponents depend on data and task structure, but whether depth changes training-time scaling under standard squared loss remained unresolved. The paper analyzes deep linear networks on power-law spectral tasks and finds that depth’s effect depends on task difficulty.

1 INTRODUCTION

  • Neural scaling laws are empirical power laws in training time, dataset size, and model scale that can guide performance extrapolation and resource allocation.The paper frames identifying their exponents as important for theoretically understanding learning in large neural networks.Source: §1 · p. 1
  • Prior work linked scaling exponents to data and task structure, while architectural effects on those exponents remained less characterized.A two-layer power-law analysis found task-dependent training-time scaling, leaving depth’s role unresolved.Source: §1 · pp. 1–2
  • For spectrally difficult tasks, increasing depth strictly improves the asymptotic training-time loss-decay exponent under sufficiently small random initialization.The same asymptotic scaling extends to standard random initialization, though finite-dimensional effects can delay its onset.Source: §1 · p. 2
  • Predictor-norm time reparameterization yields a common loss-scaling exponent across depths, determined solely by the task spectrum.The reparameterization separates the common spectral scaling from the depth-dependent conversion back to training time.Source: §1 · p. 2
  • Predictor-norm growth is polynomial, logarithmic, or bounded depending on task spectrum; through the relation between reparameterized and training time, it determines depth dependence of the training-time scaling law.Source: §1 · p. 2
  • Experiments on a regression task derived from benchmark image data and on nonlinear ReLU networks show qualitatively similar depth dependence across task regimes.The paper presents these results as a basis for studying architecture and task structure jointly in more complex networks.Source: §1 · p. 2

2 RELATED WORK

Prior scaling-law research studied spectral learning curves, resource scaling, and nonlinear models, while deep linear work examined depth’s effects on learning dynamics. The closest prior scaling analysis predicted no depth-dependent exponent, but its general-depth argument did not derive the multilayer kernel-growth relation.

Theory of Scaling Laws and Spectral Learning Curves

  • Analytical scaling-law work covers spectral and target-regularity conditions, kernel eigenspectra, and target alignment in sample-size learning curves.The cited literature also studies training-time, dataset- and model-size, and compute-optimal scaling.Source: §2 · p. 2
  • Nonlinear-model scaling studies include structured or anisotropic spectra and single- or multi-index tasks with sequential acquisition of target components.Bordelon et al. (2025) specifically found task-dependent training-time exponents for a two-layer linear model on power-law tasks.Source: §2 · p. 3

Deep Linear Networks and the Role of Depth

  • Deep linear networks preserve linear predictors while their multilayer factorization induces nonlinear parameter dynamics studied for convergence, generalization, and implicit bias.Later work also examined how initialization scale and layer balance shape learning regimes.Source: §2 · p. 3
  • Arora et al. (2018) identified depth-induced optimization acceleration for ℓp loss with p > 2, but its mechanism does not directly extend to squared loss.Their mechanism combines depth-induced predictor preconditioning with error-dependent weighting; squared loss lacks that error-dependent factor.Source: §2 · p. 3
  • Bordelon & Pehlevan (2025) predicted task-dependent but depth-independent training-time exponents using a two-layer kernel-growth relation without deriving its multilayer counterpart.The authors note that the analyses concern different settings and are not directly comparable, leaving open whether depth changes scaling under standard squared loss.Source: §2 · p. 3

3 PROBLEM SETUP

The setup defines a power-law covariance and linear teacher, then uses the task exponent p to distinguish spectral regimes. It studies a depth-L linear factorization trained by population gradient flow on squared loss with Gaussian layer initialization.

3 PROBLEM SETUP

  • The notation f ≍ g means each quantity bounds the other up to positive uniform constants, while f ≲ g denotes an upper bound up to a constant.The setup also defines f ≳ g analogously and assigns zero to the maximum of an empty family of nonnegative quantities.Source: §3 · p. 3
  • Inputs are i.i.d. zero-mean vectors in R^d with diagonal covariance eigenvalues λ_i = c_λ i^-a, and labels follow the linear teacher y_i = β⊤x_i.The target coordinates follow β_i = c_β i^-b.Source: §3 · p. 3
  • The task exponent p = (a + 2b − 1)/a is assumed positive and distinguishes task regimes, with p = 1 marking logarithmic growth of the target norm.At p = 1, the target norm grows logarithmically; the supplied Figure 1 caption identifies p < 1 as difficult and p > 1 as easy, with depth effects varying across these regimes.Source: §3 · p. 4, Figure 1 · p. 4
  • The predictor is parameterized by an L-layer linear network with hidden widths proportional to d; depth preserves the linear function class but induces nonlinear dynamics for L ≥ 2.The scale associated with the depth-L factorization is denoted σ = ||θ||^(1/L).Source: §3 · p. 4
  • All layers are jointly trained by population gradient flow to minimize squared loss, with the layer dynamics specified for the factorized predictor.The initialization draws entries independently from Gaussians with variance s²/n_(k−1).Source: §3 · p. 4

4 DEPTH-DEPENDENT TRAINING-TIME SCALING UNDER SMALL INITIALIZATION

The section derives how task spectrum governs depth-dependent training-time loss decay, connecting predictor-norm growth to a common depth-independent loss law in reparameterized time.

4.1 DEPTH-DEPENDENT PRECONDITIONING AND RESCALED PREDICTOR DYNAMICS

  • Rescaling the end-to-end predictor as z = σ^−(L−1)θ cancels the leading depth-dependent anisotropy in its dynamics.The leading preconditioner anisotropy contains the rank-one term (L−1)Pθ.Source: §4.1 · p. 5

4.2 ESCAPE FROM THE ORIGIN

  • The analysis separates the initial escape phase by defining te as the time when predictor scale σ reaches O(1), then uses shifted time t̄ = t − te.The initial escape time depends on the initialization scale.Source: §4.2 · p. 5

4.3 PREDICTOR NORM AND LOSS SCALING

  • Reparameterized time τ puts the leading spectral dynamics in a common form across depths, with depth entering through predictor scale and perturbation terms.The relation between τ and training time is controlled by the predictor scale.Source: §4.3 · pp. 5–6, §4.3.1 · p. 6

4.3.1 SCALING OF THE PREDICTOR NORM

  • The squared predictor norm grows as τ^(1−p) for p<1, log τ for p=1, and remains bounded for p>1.The stated scaling holds under sufficiently small initialization in the analyzed time window.Source: §4.3.1 · p. 6

4.3.2 COMMON SPECTRAL LAW FOR THE LOSS

  • In reparameterized time, the loss has a common power-law exponent across depths, determined by the task spectrum.Modes with τλ_i ≫ 1 are essentially learned, while modes with τλ_i ≪ 1 remain largely unlearned.Source: §4.3.2 · pp. 6–7

4.3.3 TRAINING-TIME LOSS SCALING

  • For p<1, the decay exponent pL/[1 + p(L−1)] increases with depth; for p>1 it is depth-independent, while at p=1 depth changes only the logarithmic correction.The result is stated under sufficiently small initialization.Source: §4.3.3 · p. 7

5 STANDARD INITIALIZATION AND FINITE-SIZE EFFECTS FOR DIFFICULT TASKS

Under standard initialization, difficult tasks retain depth-dependent training-time scaling, but finite dimensions can delay its visibility. That delay grows more pronounced with depth and as task difficulty approaches the boundary p = 1.

5 STANDARD INITIALIZATION AND FINITE-SIZE EFFECTS FOR DIFFICULT TASKS

  • For p < 1 under standard initialization, loss decays faster with increasing depth, consistent with the predicted depth dependence.The result is stated for sufficiently large dimensions and holds with high probability over a time interval beginning at a dimension-independent t0.Source: §5 · p. 8
  • As input dimension increases, fitted decay exponents approach their asymptotic predictions, but convergence slows with increasing depth.The finite-dimensional power-law window can limit how clearly the asymptotic scaling appears.Source: §5 · p. 8
  • Larger dimensions may be needed to observe the asymptotic power law, especially for deeper networks or p near 1.The per-layer scale separates more slowly from its O(1) initialization scale as depth increases or p approaches 1.Source: §5 · p. 8

6 EXPERIMENTS ON REAL DATA AND NONLINEAR NETWORKS

Experiments test whether task-dependent depth effects extend to a benchmark-derived task and nonlinear networks. The benchmark-derived task shows depth-related acceleration and collapse in predictor-norm time, while ReLU results show a qualitative contrast between difficult and easy tasks.

6 EXPERIMENTS ON REAL DATA AND NONLINEAR NETWORKS

  • The experiments examine whether task-dependent depth effects extend to real data and nonlinear networks.Source: §6 · p. 8
  • On the CIFAR-5M-derived task with p_eff ≈ 0.478, loss decay steepens with depth, trajectories nearly collapse in predictor-norm time, and squared predictor norm grows approximately as predicted.The task’s empirical spectrum is approximately power-law and places it in the p < 1 regime.Source: §6 · pp. 8–9, Figure 6 · p. 9
  • In ReLU MLPs, deeper networks attain lower late-time loss for p = 0.6, while p = 1.4 shows no systematic depth advantage.The task-dependent difference is qualitatively consistent with the theory but less pronounced than in deep linear networks.Source: §6 · p. 9, Figure 7 · p. 9

7 CONCLUSION

The conclusion attributes depth’s effect on training-time scaling to task difficulty and distinguishes difficult from easy tasks. The supplied appendix passages also state global existence of the gradient-flow solution and identify assumptions limiting the analysis.

7 CONCLUSION

  • For difficult tasks, increasing depth improves the asymptotic training-time scaling exponent; predictor-norm time reveals a depth-independent spectral loss exponent.The conclusion says depth changes the relation between reparameterized time and training time, while easy tasks do not gain a power-law exponent improvement.Source: §7 · p. 9
  • The theory assumes a linear teacher and student and population gradient flow, and standard-initialization results are limited to difficult tasks.Extension to more general task structures, nonlinear networks, and finite-sample discrete-time stochastic optimization remains open.Source: §7 · p. 9
  • The authors describe task structure as capable of qualitatively changing how depth affects training-time scaling, providing a basis for studying broader network classes.Source: §7 · p. 10

A.1 EXACT GRADIENT-FLOW DYNAMICS

  • For finite dimension and any initialization, the gradient-flow system is stated to have a unique solution for all t ≥ 0.The proof passage says finite-time boundedness of every weight matrix would contradict the ODE continuation criterion.Source: §A.1 · pp. 15–16

A.2 LAYER BALANCE AND PRECONDITIONER GEOMETRY

The supplied appendix passages cover layer-balance and preconditioner geometry, rescaled predictor dynamics, Gaussian initialization bounds, and the setup for analyzing the onset of power-law scaling.

A.2 LAYER BALANCE AND PRECONDITIONER GEOMETRY

  • The layer-balance argument uses a recursively defined decomposition and an invariant whose derivative is zero, with its claims fixed by evaluating at initialization.The supplied passages do not specify the resulting claims in full.Source: §A.2 · p. 16

A.3 RESCALED PREDICTOR DYNAMICS AND TIME REPARAMETERIZATION

  • The rescaled predictor dynamics use a time change satisfying dτ/dt = σ^{2(L−1)} to express predictor and loss identities in τ.Source: §A.3 · pp. 18–19

A.4 GAUSSIAN INITIALIZATION

  • Under the stated width and Gaussian-initialization conditions, layer-product norms scale as q_k ≍ s^{L−k+1} with probability at least 1 − Ce^{−cd}.In particular, the end-to-end predictor norm is stated as r(0) ≍ s^L.Source: §A.4 · pp. 19–20

B FROM INITIALIZATION TO THE ONSET OF THE POWER-LAW SCALING PHASE

B.1 ESCAPE FROM THE ORIGIN UNDER SMALL INITIALIZATION

The small-initialization analysis establishes predictor and loss bounds after an initial escape phase, then uses spectral estimates to characterize loss decay and its eventual exponential regime.

B.1 ESCAPE FROM THE ORIGIN UNDER SMALL INITIALIZATION

  • Under small initialization, σ(u) ≍ σ(0) + u and r(τ) ≍ s^L + τ during escape from the origin.The time change is strictly increasing, with τ(u) ≍ (σ(0) + u)^L − σ(0)^L.Source: §B.1 · pp. 22–23

B.2 STANDARD INITIALIZATION

  • For standard initialization, σ(τ)^2 stays bounded below and K_L(τ) remains uniformly comparable to the identity for all τ ≥ 0.These bounds hold for sufficiently large d and allow the estimates to be expressed in τ-parameterization.Source: §B.2 · pp. 24–25

C SPECTRAL ESTIMATES AND AUXILIARY LOSS BOUNDS

  • The appendix develops the spectral and loss bounds used to control the target’s spectral components and the resulting loss.The analysis combines target-vector bounds above and below a spectral cutoff with bounds on the low-eigenvalue contribution.Source: C SPECTRAL ESTIMATES AND AUXILIARY… · p. 25, §C.2 · p. 27

C.1 SPECTRAL SCALING AND TAIL ESTIMATES

  • The reported τ-dependent values are (1 + τ)^(1−p) for p < 1, 1 + log(1 + τ) for p = 1, and 1 for p > 1.Source: §C.1 · p. 26

C.2 LOSS BOUNDS VIA SPECTRAL CUTOFFS

  • Beyond τ+ = c0d^a, the loss decays exponentially in reparameterized time under both sufficiently small and standard initialization.The small-initialization result applies from τe, while for s = 1 it applies from τ0 = 0.Source: §C.2 · p. 28

D PREDICTOR-NORM SCALING IN REPARAMETERIZED TIME

The analysis relates predictor norm to the task’s spectral scale in reparameterized time, then uses those bounds to derive loss scaling for difficult tasks.

D PREDICTOR-NORM SCALING IN REPARAMETERIZED TIME

  • Under small initialization, the squared predictor norm scales as τ^(1−p) for p < 1, log τ for p = 1, and 1 for p > 1.The corollary holds with high probability uniformly over the stated τ interval.Source: D PREDICTOR-NORM SCALING IN… · p. 33

E.1 DIFFICULT TASKS

  • For Gaussian standard initialization and p < 1, L(τ) ≍ (1 + τ)^−p over the stated interval.The result holds with probability at least 1 − Ce^−cd for sufficiently large d.Source: D PREDICTOR-NORM SCALING IN… · p. 34
  • For p < 1, the analysis derives loss scaling from predictor-norm and loss bounds; the predictor norm squared is bounded above by a scale growing as (1 + τ)^(1−p).The section’s proposition is stated for conditions of Theorem D.2.Source: §E.1 · pp. 34–35

E.2 BOUNDARY AND EASY TASKS

For boundary and easy tasks, the loss has a common spectral law under small initialization, with the supplied result specifying the p ≥ 1 regime.

E.2 BOUNDARY AND EASY TASKS

  • For p ≥ 1, L(τ) ≍ (1 + τ)^−p under Gaussian initialization satisfying the stated small-initialization condition.The comparison holds uniformly over τe ≤ τ ≤ τ+ with probability at least 1 − Ce^−cd.Source: §E.2 · p. 37

E.3 COMMON SPECTRAL LAW UNDER SMALL INITIALIZATION

  • The p < 1 and p ≥ 1 cases combine into a common spectral loss law under small initialization.The proposition states the result uniformly over τe ≤ τ ≤ τ+ with high probability.Source: §E.3 · p. 40

F TRAINING-TIME LOSS SCALING

The section converts reparameterized-time spectral laws into training-time scaling. For difficult tasks, the standard-initialization result gives a depth-dependent decay exponent.

F TRAINING-TIME LOSS SCALING

  • The relation between τ and training time grows as τ^[1+p(L−1)]/L for p < 1, as τ(log τ)^−(L−1)/L for p = 1, and as τ for p > 1.These cases describe the conversion used to express the spectral loss laws in training time.Source: F TRAINING-TIME LOSS SCALING · pp. 40–41
  • For p < 1 under Gaussian standard initialization, L(t) ≍ t^(−pL/[1+p(L−1)]) over the stated training-time interval.The result holds with probability at least 1 − Ce^−cd for sufficiently large d.Source: F TRAINING-TIME LOSS SCALING · p. 41

G.1 SYNTHETIC POWER-LAW EXPERIMENTS

The experiments describe fitting procedures and finite-dimensional controls for synthetic power-law tasks, then extend the analysis to a CIFAR-5M regression task and synthetic ReLU MLPs.

G.1 SYNTHETIC POWER-LAW EXPERIMENTS

  • Figure 8 shows the Figure 1 trajectories in unshifted training time t, with dotted lines marking t_e where τ(t_e)=0.1 for L=1, 2, 3, 4.The synthetic experiments use p∈{0.6, 1.0, 1.4} and four depths, with hidden-layer width d and initialization scale s=d^−1/2.Source: §G.1 · p. 41, Figure 8 · p. 42
  • For p≠1, the loss exponent q is fit over 50≤τ≤5000; at p=1, the fit uses a logarithmically corrected loss, with maximum seed standard deviation approximately 0.004.The reported means and sample standard deviations use five seeds.Source: §G.1 · p. 42
  • The finite-dimensional fitting window is selected before the spectral cutoff using a proxy that approximates each residual mode as β_i e^(-λ_iτ).The proxy accounts for modes i>d that are absent from the dimension-d experiment.Source: §G.1 · p. 42

G.2 EXTENSIONS BEYOND THE THEORETICAL SETTING

  • The CIFAR-5M experiment uses all 5,002,240 examples, centered 3072-dimensional inputs, and empirical covariance and input–target cross moments.The target is airplane-versus-rest, and the inputs are rescaled so ∥Σ∥op = 1.Source: Figure 6 · p. 42, §G.2 · p. 43
  • p_eff≈0.478 is the fitted effective task exponent for the CIFAR-5M airplane-versus-rest regression task.The fit uses rounded logarithmically spaced ranks from 10 to 2765.Source: Figure 9 · p. 43, §G.2 · p. 43
  • The ReLU MLP extension uses synthetic tasks with p=0.6 and p=1.4, depths L∈{2, 3, 4}, and width 2048.The networks are trained by online SGD for 100,000 steps using fresh batches of 4096.Source: §G.2 · p. 43

Related papers

Loading 2610.35514v1…