PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons

Mana Sakai, Masaaki Imaizumi

arXiv:2609.35029v1cs.LGstat.ML

TL;DR

As width and training horizon grow together, the conditions for fast learning-rate transfer remain incompletely understood. The paper analyzes transfer in a shallow linear network using finite-width perturbations and spectral structure, and proves fast transfer when T=o(√n) under additional spectral assumptions.

  • Problem

    The joint growth of width and training horizon makes fast learning-rate transfer less understood because the limiting objective changes with the horizon.

    Source: §1.1 · p. 2

  • Method

    The paper studies a shallow linear network with one trainable hidden matrix and characterizes transfer using finite-width perturbations, loss sensitivities, and local curvature.

    Source: §1.2 · p. 2, §3.2 · p. 7

  • Results

    Under additional spectral assumptions, fast learning-rate transfer holds as n,T grow when T=o(√n).

    Source: §1.2 · p. 2

  • Takeaways & Limitations

    Transfer rates depend on spectral structure and on the loss-value response, loss-gradient response, and local curvature.

    Source: §6 · p. 13

    The analysis is restricted to linear networks with one trainable linear layer and relies on nondegenerate first-order sensitivities.

    Source: §6 · p. 13

Abstract

from arXiv · show

Hyperparameter transfer across model width can substantially reduce the cost of tuning large neural networks, but its behavior when the training horizon grows with width is not fully understood. Building on the framework of fast hyperparameter transfer (Ghosh et al., 2026), which formalizes when transfer is effective, we investigate conditions that ensure fast transfer in the growing-horizon regime. Specifically, we study learning-rate transfer in a shallow linear network with a single trainable hidden matrix, trained by full-batch gradient descent. Under additional spectral assumptions, our main results are threefold. (i) We prove fast learning-rate transfer as $n,T\to\infty$ whenever $T=o(\sqrt{n})$. (ii) We characterize the transfer rates through the finite-width perturbation scale, the first-order sensitivities of the loss and its learning-rate derivative to finite-width perturbations, and the local loss curvature. (iii) We derive limiting distributions for the optimal learning rate and optimized loss, governed by fluctuations associated with the extreme eigenvalues of the data Gram matrix. These results clarify how spectral structure and local loss sensitivities govern learning-rate transfer at growing horizons.

1 Introduction

Hyperparameter transfer can reduce the cost of tuning larger models, but its theoretical behavior when width and training horizon grow together remains limited. This paper studies fast learning-rate transfer in a shallow linear network and relates its rates to finite-width perturbations and spectral structure.

1.1 Background and Motivation

  • The joint growth of width and training horizon makes transfer harder to analyze because the limiting objective changes with the horizon.The paper identifies this regime as theoretically less understood than fixed-horizon transfer.Source: §1.1 · p. 2
  • The study asks which conditions ensure fast learning-rate transfer as model width and training horizon grow jointly, and what quantities govern it.Source: §1.1 · p. 2

1.2 This Study

  • The analysis considers a shallow linear network with one trainable hidden matrix, trained by full-batch gradient descent, comparing finite-width and infinite-width optima at each horizon.Source: §1.2 · p. 2
  • Under additional spectral assumptions, fast transfer holds when T→∞ and T=o(√n) for fixed sample size and input dimension.Source: §1.2 · p. 2
  • Transfer-error rates are organized by the finite-width perturbation scale, loss and learning-rate-derivative sensitivities, and local loss curvature.The paper also derives limiting distributions for the optimal learning rate and optimized loss, whose limits depend on perturbation components associated with the largest and smallest positive Gram-matrix eigenvalues.Source: §1.2 · pp. 2–3
  • The leading loss-value and slope responses to the same perturbation need not align, so optimized loss can change at leading order without a leading-order shift in the optimal learning rate.The analysis concerns local finite-width perturbations, rather than a decomposition of the optimization trajectory.Source: §1.2 · p. 3

2 Problem Setup and Transfer Framework

The paper reduces learning-rate selection in a shallow linear network to a scalar loss curve whose finite-width randomness is summarized by a fixed-dimensional pair. It defines transfer by comparing finite-width and infinite-width optima and losses at the same training horizon.

2.1 Model and exact loss dynamics

  • The model has a single trainable hidden matrix, frozen random input and output maps, µP initialization, and full-batch gradient descent with a constant learning rate.Source: §2.1 · p. 3
  • Exact loss dynamics reduce learning-rate selection to minimizing a scalar random loss curve, with initialization dependence captured by a fixed-dimensional pair.This separates the spectral geometry of the reference curve from finite-width fluctuations.Source: §2.1 · p. 4

2.2 Infinite-width objectives and spectral geometry

  • The infinite-width objective is expressed through active spectral modes, each contributing a learning-rate-dependent contraction term, while the null-space component is independent of learning rate.Source: §2.2 · p. 4

2.3 Optimal learning rates and transfer quantities

  • The loss gap, hyperparameter gap, and transfer suboptimality gap compare finite-width and infinite-width optima at a common horizon; fast transfer means the transfer penalty is negligible relative to the optimized-loss discrepancy.The finite-width optimizer is defined on a compact learning-rate interval.Source: §2.3 · pp. 4–5

3 Main results

The results identify the long-horizon optimizer through active spectral endpoints and give conditions for fast learning-rate transfer as width and horizon grow. They also show that transfer can fail at fixed horizons when a leading loss sensitivity degenerates.

3.1 Long-horizon structure of the infinite-width objective

  • The long-horizon infinite-width optimizer balances the largest and smallest active spectral modes, while the finite-horizon optimizer approaches its limiting location at rate O(T^-1).At the limiting optimizer, the two endpoint contraction factors have equal magnitude and opposite signs.Source: §3.1 · pp. 5–6

3.2 Fast learning-rate transfer at growing horizons

  • Fast learning-rate transfer persists for growing horizons when T=o(√n), under Theorem 3.4’s assumptions.The analysis tracks finite-width fluctuation scale, first-order loss and loss-gradient sensitivities, and local curvature; the theorem also gives limiting fluctuations for the optimal learning rate and optimized loss.Source: §3.2 · p. 7

3.3 A fixed-horizon benchmark

  • At every fixed horizon, fast transfer holds under Theorem 3.6’s nondegeneracy conditions, with the transfer suboptimality gap smaller than the loss gap by Θp(n^-1/2).The fixed-horizon rates provide a baseline for the growing-horizon analysis.Source: §3.3 · pp. 7–8

3.4 Counterexample to fast transfer

  • With a single positive eigenvalue, fast transfer fails at fixed T: the optimizer gap remains Θp(n^-1/2), but the loss and transfer suboptimality gaps are both Θp(n^-T).The stated mechanism is degeneracy of the leading loss sensitivity, so convergence of the optimal learning rate alone does not ensure fast transfer.Source: §3.4 · p. 8

4 Perturbation structure of hyperparameter transfer

The section derives first-order finite-width perturbation limits for the shallow linear model, then organizes their transfer-rate consequences using loss response, gradient response, curvature, and localization. Under its spectral assumptions, the formulation recovers fast transfer when T = o(√n).

4.1 Model-specific perturbation limits for Theorem 3.4

  • For the shallow linear model, the leading finite-width perturbations at the infinite-width optimizer come from A_n − K rather than the initial prediction.The section derives limiting distributions for the optimizer displacement and optimized-loss fluctuation from these perturbations.Source: §4.1 · p. 9
  • The normalized optimized-loss fluctuation converges to a linear functional of the limiting matrix perturbation Ξ.The optimizer displacement also has a distributional limit, and reoptimization leaves the leading loss limit unchanged.Source: §4.1 · p. 9

4.2 A local perturbation formulation for horizon-dependent transfer

  • The local formulation tracks loss-value response, loss-gradient response, curvature, and optimizer localization to characterize horizon-dependent transfer rates.The loss perturbation sets the leading scale of a_n,T, while curvature converts the gradient perturbation into optimizer displacement.Source: §4.2 · p. 10, Figure 1 · p. 11
  • When the optimization correction is negligible relative to the first-order loss fluctuation, the transfer penalty is governed by the loss and gradient responses relative to local curvature.A sufficient condition given for this comparison is ϵ_nρ_G,T^2/(κ_Tρ_L,T) → 0.Source: §4.2 · p. 11
  • For this model’s scales, T = o(√n) makes the required ratios vanish and recovers the fast-transfer conclusion.The section notes that its finite-dimensional perturbation estimates differ from the spectral-tail-dependent localization used by Wen et al. (2026).Source: §4.2 · pp. 11–12

5 Implications for mechanisms of learning-rate transfer

The section distinguishes structural concentration of the training trajectory from the local perturbation behavior that determines transfer. It also relates transfer to spectral endpoints and to the alignment of loss and gradient responses.

Relation to the top-k loss decomposition

  • The rank-one gradient–update alignment gives an exact top-1 representation, but occurs both in a fast-transfer regime and where c_n,T/a_n,T = Θ_p(1).Thus, low-rank concentration alone does not determine the transfer rate.Source: §5 · p. 12
  • The framework requires a nonvanishing first-order loss response: in the T = 1 counterexample it vanishes while the gradient response remains nondegenerate, and a_n,1 and c_n,1 are both Θ_p(n^-1).For fixed T ≥ 2, the first-order responses and curvature also degenerate, placing the example outside the framework’s first-order regime.Source: §5 · p. 12

Directional dependence of local responses

  • The leading loss-value and loss-gradient responses to the same perturbation need not align, so some directions change optimized loss without a leading-order optimizer shift.The responses are distinct linear functionals of the perturbation.Source: §5 · pp. 12–13

Relation to spectral explanations of transfer

  • The largest eigenvalue sets the stability boundary, whereas the long-horizon optimal learning rate depends on both active spectral endpoints.Under Assumption 3.2, η_∞,T → 2m/(λ_1 + λ_r) < 2m/λ_1 as T → ∞.Source: §5 · p. 13

6 Conclusion

The paper concludes that fast learning-rate transfer is supported in its shallow linear setting when endpoint activity holds and T = o(√n). Its scope is limited to one trainable linear layer and first-order sensitivities that do not vanish.

6 Conclusion

  • Under endpoint activity and T = o(√n), the three transfer quantities satisfy c_n,T = o_p(a_n,T).The results concern learning-rate transfer across width in a shallow linear network under µP.Source: §6 · p. 13
  • The local perturbation formulation explains transfer rates through loss-value response, loss-gradient response, and local curvature.These quantities also capture how rates depend on the problem’s spectral structure.Source: §6 · p. 13
  • The analysis is restricted to linear networks with a single trainable linear layer; nonlinear or deep networks and joint width–depth transfer remain outside its scope.The conclusion also identifies vanishing first-order sensitivities as requiring higher-order analysis.Source: §6 · p. 13

A Additional related work

  • In this model, the largest eigenvalue determines the stability boundary, while the long-horizon optimal constant learning rate balances two active spectral endpoints.The paper distinguishes its spectral conclusion from work focused on leading Hessian eigenvalue consistency.Source: A Additional related work · p. 14

B Numerical experiments

The numerical experiments find transfer-error slopes near the predicted exponents and a declining transfer-penalty ratio along the tested growing-horizon schedule. The supplied proof passages also establish Gaussian perturbation limits, uniform fixed-horizon loss convergence, and a range identity.

B Numerical experiments

  • The experiments use m = d = 4, K = diag(7, 4, 2, 1), and 256 independent initializations at each width.They compare fixed T = 10 with T_n = ⌈2n^0.15⌉, ranging from 5 to 16.Source: B Numerical experiments · p. 15
  • All six empirical slopes are close to (−1/2, −1/2, −1), and the transfer-penalty ratio declines overall along the tested schedule.Means are noisy and nonmonotone, while medians decline more smoothly; the theoretical rates in probability do not imply matching expectation rates.Source: B Numerical experiments · p. 15

C.2 Proof of Lemma 3.3

  • The joint initialization limit includes a Gaussian matrix G, independent of the scalar Gaussian component α and the Gaussian vector Z.The stated joint central limit theorem specifies the covariance of G and the distribution of α and Z.Source: §C.2 · pp. 16–17

C.3 Infinite-width loss for fixed horizon

  • For fixed T, the infinite-width loss is the stated spectral expression and converges uniformly on every deterministic compact learning-rate interval.The fixed-horizon losses are polynomials in η of degree at most 2T, with coefficients converging in probability.Source: §C.3 · p. 18

C.4 Range and null-space geometry

  • When n ≥ d, the proof establishes that range(A_n) equals range(K) almost surely.Source: §C.4 · p. 18

C.5 Optimization landscape

The optimization landscape gives conditions for a unique positive learning-rate optimum and characterizes the degenerate flat case. First-order sensitivities of both the loss and its learning-rate score describe their response to perturbations in the residual vector and matrix.

C.5 Optimization landscape

  • When Aχ ≠ 0, ΦT is strictly convex and coercive, so its unique global minimizer is positive and solves FT = 0; on [a, b], the optimizer is clipped to the appropriate boundary if needed.If Aχ = 0, ΦT is constant in η and every η is a global minimizer.Source: §C.5 · p. 19
  • For K ≠ 0 and n ≥ d, the finite-width objective is almost surely strictly convex and has a unique global minimizer.The finite-width nondegeneracy result supplies the condition under which this optimization problem is well posed.Source: §C.5 · pp. 19–20

C.6 First-order sensitivities of the score and the loss

  • The first-order sensitivity results treat both ΦT and its score FT as functions of the residual vector and matrix, with perturbations measured in a product norm.The section establishes derivatives with respect to these inputs at fixed T.Source: §C.6 · p. 22

D Proof of the fixed-horizon transfer theorem

For fixed T, the proof establishes limiting distributions for the optimal learning rate and optimized loss. It also localizes the finite-width optimizer to the interior of the search interval with probability tending to one.

D Proof of the fixed-horizon transfer theorem

  • The finite-width optimizer is interior with probability tending to one, enabling the proof to use its score equation at the optimizer.The proof first establishes that the optimizer is the unique root of the finite-width score.Source: D Proof of the fixed-horizon transfer… · pp. 24–25
  • The fixed-T optimal learning rate has a limiting distribution with fluctuation term ΩF,T = g⊤F,TZ + ⟨QF,T, Ξ⟩F.The theorem gives the limit under the assumptions of Theorem 3.6.Source: D Proof of the fixed-horizon transfer… · p. 25
  • The optimized loss also has a limiting distribution for fixed T under the assumptions of Theorem 3.6.The proof obtains it by replacing the loss at the infinite-width optimum with the loss at the finite-width optimizer.Source: D Proof of the fixed-horizon transfer… · pp. 25–26

E.1 Proof of Theorem 3.1

The appendix proof develops large-horizon optimizer and loss asymptotics, endpoint contraction behavior, and uniform power-perturbation tools. These results support the spectral analysis of the finite-width problem.

E.1 Proof of Theorem 3.1

  • When the active spectrum has a single eigenvalue λ, the loss is a positive multiple of (1 − ηλ/m)^(2T) plus a constant, with unique minimizer m/λ.The irreducible training loss is unaffected by learning-rate or step-count choices.Source: §E.1 · pp. 28–29

E.2 Expansion of the infinite-width optimizer and asymptotic loss

  • The infinite-width optimizer has a first-order large-T expansion around η*, while the optimal loss scales as Θ(q*^(2T)).The optimizer expansion includes an O(T^-1) correction; the loss asymptotic is stated in terms of q*.Source: §E.2 · pp. 29, 31, 32

E.3 Endpoint contractions and spectral separation

  • Endpoint contractions have large-horizon asymptotics, and when r > 2 an interior spectral-separation bound holds for all sufficiently large T.The result supplies a contraction factor strictly below one.Source: §E.3 · p. 33

F.1 Fréchet expansion of matrix powers

  • The matrix-power map B ↦ B^k has a first-order Fréchet expansion that does not require B and its perturbation C to commute.The lemma applies to any submultiplicative matrix norm.Source: §F.1 · p. 34

F.2 Uniform noncommutative power bounds

  • Under the stated perturbation and learning-rate bounds, power perturbations are controlled uniformly over η and k ≤ 2T.The bound includes k∥Δ∥op/qT uniformly bounded; the setting assumes T∥E∥op is bounded.Source: §F.2 · pp. 35–36

F.3 Application to the finite-width kernel

The finite-width kernel application gives uniform contraction control near the optimal learning rate when the horizon grows more slowly than √n. The required perturbation event has probability tending to one.

F.3 Application to the finite-width kernel

  • When T → ∞ and T/√n → 0, finite-width powers contract uniformly near η∞,T for 0 ≤ k ≤ 2T with probability tending to one.The proof uses T∥An − K∥op = op(1) to establish the high-probability perturbation condition.Source: §F.3 · pp. 37–38

G Proof of the joint width-horizon transfer theorem

This section proves growing-horizon transfer results under Theorem 3.4’s assumptions, using score and derivative approximations to localize the finite-width optimizer and derive perturbation and limit-distribution results. It also derives Corollary 3.5’s convergence via Theorems G.8 and G.10 and the continuous mapping theorem, relying on T/√n → 0 and fixed q* > 0.

G.1 Uniform score approximation

  • The proof establishes uniform approximation of the finite-width score and its derivative near the infinite-width optimizer.The neighborhood has radius C/T around η∞,T.Source: §G.1 · p. 39

G.2 Localization of the finite-width optimizer

  • With probability tending to one, the finite-width optimizer is interior and differs from the infinite-width optimizer by Op(n^-1/2).The result is established when T/√n → 0.Source: §G.2 · pp. 42–43

G.3 First-order perturbation expansion

  • The score admits a first-order triangular-array expansion in the finite-width perturbation, with an explicit remainder.The expansion supports the subsequent limit law for the optimal learning rate.Source: §G.3 · p. 44, §G.5 · p. 47

G.4 Limits of the first-order sensitivities of the loss and score

  • The local Fréchet coefficients for the loss and score have asymptotics in which off-diagonal and interior blocks are negligible relative to endpoint blocks.The cited results state the former are o(Tq2T−1*).Source: §G.4 · pp. 45–46

G.5 Score linearization and weak convergence of the optimal learning rate

  • The optimal learning rate is interior with probability tending to one and has a growing-T limit distribution determined by Ξ.Ξ is defined in Lemma 3.3.Source: §G.5 · p. 49

G.6 Fluctuations and weak convergence of the optimal loss

  • The optimized loss has a growing-T limit distribution under the assumptions of Theorem 3.4.The proof obtains the result by replacing the infinite-width optimizer with the finite-width optimizer and applying Slutsky’s theorem.Source: §G.6 · pp. 51, 53

G.7 Convergence rates of the optimal loss and the optimal learning rate

  • The transfer gap has a quadratic expansion, and Theorem G.10 establishes convergence rates for a_n,T, b_n,T, and c_n,T.The supplied passage does not specify the rates’ values.Source: §G.7 · pp. 53–54
  • Theorem G.8, Theorem G.10, and the continuous mapping theorem yield the scaled convergence used to prove Corollary 3.5.The supplied passage begins the expression with √n but does not include the full convergence statement.Source: §G.7 · p. 55
  • Because ΩL,* is nondegenerate Gaussian and P(|ΩL,*| = 0) = 0, the proof applies the continuous mapping theorem again; its final convergence uses T/√n → 0 and fixed q* > 0.The supplied passages do not include the full expression for the final convergence.Source: §G.7 · p. 55

H Counterexample to fast transfer

This counterexample shows that vanishing absolute transfer gaps do not guarantee fast transfer under the relative criterion. The rescaled local objective has a unique minimizer, but the first-order loss sensitivity can vanish, removing the fluctuation scale used for that criterion.

H Counterexample to fast transfer

  • The rescaled finite-width objective converges through its polynomial coefficients to H_T(s; M), providing a local objective for analyzing the counterexample.H_T(s; D) is defined as (2m)^-1 y^T(D + sI_m)^2T y.Source: H Counterexample to fast transfer · p. 56
  • Strict convexity and coercivity give the local objective a unique minimizer, and the continuous mapping theorem transfers this minimizer to the finite-width optimizer.The finite-width optimizer coincides with the global minimizer of the rescaled objective with probability tending to one.Source: H Counterexample to fast transfer · p. 57
  • For m ≥ 2, the optimized-loss fluctuation is positive almost surely, while the other fluctuation is determined by the Gaussian matrix component rather than the scalar shift.The argument uses that a fixed nonzero y is an eigenvector of the Gaussian matrix with probability zero.Source: H Counterexample to fast transfer · pp. 57–58
  • When the first-order loss sensitivity vanishes, both loss gaps are O(n^-T), so the relative fast-transfer criterion fails even though absolute transfer accuracy remains.The absolute gaps still converge to zero in probability, while cn,T = op(an,T) does not hold.Source: H Counterexample to fast transfer · p. 58

Related papers

Loading 2609.35029v1…