Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective

1 Introduction

• Large-width limits and random-feature models often freeze features, while mean-field parameterizations allow feature evolution but have harder-to-describe nonlinear dynamics.Existing feature-learning analyses also include low-dimensional single- and multi-index settings and finite-width models described by low-dimensional order parameters.Source: §1 · p. 3
• Characterizing both alignment and learning from random initialization, with a macroscopic number of directions involved, remains a challenge in the extensive-width regime.The paper focuses on this proportional-dimension setting, where low-dimensional descriptions do not capture the whole dynamics.Source: §1 · pp. 3–4
• The silent-alignment phase was previously analyzed in related settings, including by Atanasov et al. (2022); this work characterizes the whole dynamics from random initialization in the extensive-width regime.Earlier high-dimensional work described spectral learning from initialization already aligned with the data.Source: §1 · p. 4
• Combining a known closed-form solution with prior random matrix theory yields a high-dimensional training trajectory and spectral and learned-subspace descriptions at all times.The analysis also applies to factorization of a matrix of extensive rank.Source: §1 · p. 4
• In the small-initialization limit, singular directions align on an order-one timescale while singular values and loss remain at their initial values; later, singular values escape zero and loss decreases.The finite-dimensional dynamics becomes continuous spectral learning in the high-dimensional limit.Source: §1 · p. 4

2.2 Model

• The network maps x to UV^⊤x and is trained by gradient flow to minimize a loss based on the prediction and the input-output cross-moment matrix C.The model uses U and V as the two factor matrices.Source: §2.2 · p. 5

2.3 Main Assumptions

• The inputs are whitened, and the input, inner, and output dimensions grow proportionally as d diverges.The cross-moment matrix has a macroscopic number of nonzero singular directions to learn.Source: §1.1 · p. 6, §1.2 · p. 6, §1.4 · p. 6

1.4. Initialization.

• Initialization sets U0 to zero and V0 to √2ρR, with R uniform on the Stiefel manifold; the resulting imbalance matrix is 2ρI.The isotropic imbalance is constant along the flow.Source: §2.2 · p. 5, §1.3 · p. 6, §1.4 · p. 7
• The squared Frobenius prediction error is twice the excess empirical loss, and the analysis studies learning through C rather than generalization.Generalization would require an additional data model beyond the empirical cross-moment matrix.Source: §1.4 · p. 6

2.4 Overview of the Results

• At t = Θ(1) as ρ → 0, the loss stays at its initial value while the prediction remains near zero and its singular subspaces become correlated with those of C.The high-dimensional result describes the alignment curves on this timescale.Source: §2.4 · p. 7
• Later, at t = s log(ρ−1), singular values escape zero and the loss decreases as the prediction recovers singular values of C above a threshold, subject to prediction rank.The loss decay tracks progressive learning starting from the largest singular values.Source: §2.4 · p. 7
• Combining a closed-form solution with random matrix theory gives high-dimensional spectral distributions and projector-distance limits for the learned subspace.These limits describe the dynamics throughout training.Source: §2.4 · p. 7

3 Finite-Dimensional Dynamics

• Figure 2 contrasts alignment and loss: alignment changes on t = Θ(1), while loss decay occurs later as singular values are learned from largest to smallest.The curves are normalized by their initial values.Source: §3 · p. 9, Figure 2 · p. 9

3.1 Closed-Form Solution

• The gradient flow admits a closed-form solution that reduces the dynamics to an Oja flow.This expression underlies the high-dimensional analysis by turning the dynamics into a random matrix problem involving W0 and Aρ.Source: §3.1 · pp. 9–10

3.2 The Prediction Matrix

• For t > 0, rank U(t) and rank Z(t) equal min(m, m*), while rank V(t) equals m; when m < d, the prediction’s singular vectors evolve.This evolving orientation produces a genuine alignment phase, unlike the frozen singular vectors in the related m ≥ d setting.Source: §3.2 · pp. 10–11

3.3 Limit of the Dynamics

• In the overparameterized limit, initialization selects a projection within Ker(C), but that choice leaves the prediction and loss unchanged.The projection has rank m − m* and represents an uninformative component.Source: §3.3 · p. 11
• At convergence, the underparameterized dynamics selects the top part of C’s spectrum, while the overparameterized dynamics recovers its whole non-zero spectrum.The prediction converges to the limit characterized by Proposition 3.Source: §3.3 · p. 12

4 High-Dimensional Limit

• Because ordinary Stieltjes transforms contain no eigenvector information, the analysis separately uses range-projector distances and a generalized transform to track learned-subspace alignment.The generalized transform compares learned Gram matrices with CC⊤ and C⊤C and encodes eigenvector overlaps.Source: §4 · p. 12
• Stieltjes transforms encode the empirical eigenvalue distributions of the symmetric Gram matrices and the empirical singular-value distribution of the rectangular prediction.The transform is scalar-valued and uniquely determines the empirical spectral distribution.Source: §4 · p. 12, §4.1 · p. 13

4.1 Stieltjes Transforms

• At every fixed training time, the high-dimensional limit gives deterministic Stieltjes-transform limits for the Gram matrices and prediction spectrum.The transform is determined through a unique holomorphic solution of a self-consistent equation.Source: §4.1 · p. 13

The limiting measures µU

• The predicted singular-value density matches finite-dimensional gradient-descent simulations throughout the dynamics as spectral mass moves toward the selected part of C’s spectrum.Figure 4 compares empirical singular-value histograms with the high-dimensional density at different training times.Source: The limiting measures µU · p. 14, Figure 4 · p. 15
• The high-dimensional and long-time limits commute: the limiting spectral measures agree with the long-time spectral density.The long-time measures have a rank-selection threshold: underparameterization selects a fraction κ of the target measure, whereas overparameterization selects all positive singular values.Source: The limiting measures µU · pp. 14–15

4.2 Range-Projector Distances

• Range-projector distances quantify alignment because the learned subspaces converge to the selected singular spaces of C.The right range can evolve when m < d, and the left range can evolve when m < m*.Source: §4.2 · pp. 15–16
• At fixed positive initialization and training time, the high-dimensional range-projector distances have deterministic limits under the stated singular-value gap condition.The theorem gives a simple expression for these distances.Source: §4.2 · p. 16
• With zero initialization of U, its range is generated at leading order by CRR⊤ as soon as training begins; for m > m*, it is the full non-zero left singular space of C.This explains why directional alignment can begin immediately in the overparameterized case.Source: §4.2 · p. 16

5.1.1 Finite-Dimensional Linearized Dynamics

• At fixed times, the linearized dynamics select singular directions while prediction eigenvalues remain O(ρ); those eigenvalues grow to order one only around log(ρ^-1).The cubic term is subleading in this regime.Source: §5.1.1 · p. 17

5.1.2 Range Projectors

• Range-projector distances decrease toward the selected singular spaces; convergence is exponential in the overparameterized case but can relax as 1/t under the rank-selected threshold.In the overparameterized case the rate is controlled by the spectrum’s left edge; in the underparameterized case it is controlled by singular values near the threshold.Source: §5.1.2 · p. 18

5.1.3 Top Singular Subspaces

• A spectral gap below the top-k cutoff yields exponential alignment, whereas a continuous positive density at the cutoff produces power-law convergence.At finite initialization, decreasing ρ extends the early alignment envelope before the projector distances converge to zero.Source: §5.1.3 · p. 20

5.2.1 Finite-Dimensional Saddle-to-Saddle Dynamics

• At logarithmic times, each singular mode activates when s exceeds (2σ_i)^-1; between activations, the prediction is a truncated SVD near a saddle of the loss.It waits near each saddle for order log(ρ^-1), then moves to the next in order-one time.Source: §5.2.1 · p. 21

5.2.2 Spectra at Logarithmic Times

• After taking the high-dimensional limit first, finite-dimensional transitions merge into a continuous spectral front: modes above x_s=(2s)^-1 are learned, while the front stops at the rank cutoff x*.For an underparameterized prediction, the rank constraint prevents learning more modes once the front reaches x*.Source: §5.2.2 · p. 22

5.2.3 Loss Function and Spectral Dynamics

• Loss remains at its initial value during order-one-time alignment, then decreases on the logarithmic scale as singular values above x_s are learned.The remaining loss is carried by singular values not yet learned.Source: §5.2.3 · pp. 23–24

6 Conclusion and Perspectives

• At small initialization, singular spaces align while prediction and loss remain at their initial values; later, a descending spectral threshold governs singular-value growth.The paper says this makes loss alone a poor clock for feature learning.Source: §6 · p. 24
• In the extensive-width regime, explicit limits track prediction spectra and learned-to-target subspace distances, with finite-dimensional saddle transitions merging into a continuous spectral front.These limits describe both successive learning phases.Source: §6 · p. 24
• At time t = s log(ρ^-1), the predictor retains singular values of C above (2s)^-1 until width limits its rank, making training time an effective-rank control.The resulting factorized-dynamics filter is sharp: each direction is either learned or not learned.Source: §6 · p. 24
• In a spiked model, silent alignment could orient the network toward informative directions before the spectral front reaches the random bulk.The authors suggest this setting could help determine when to stop training and how much noise enters the predictor.Source: §6 · p. 24
• The analysis assumes whitened inputs, two layers, and isotropic imbalance; with general covariance, the fixed spectral structure is lost and learning order may depend on the trajectory.Source: §6 · p. 24
• For deeper networks, amplitude-growth timescales differ and rigorous ordering of alignment and growth remains open; without a closed form in the discussed generalizations, persistence of the two phases at extensive width is also unknown.The paper also identifies loss-based learning schedules as a practical open issue: stepsizes may perturb alignment that loss monitoring does not reveal.Source: §6 · p. 25

A.2 Complex Analysis

• Stieltjes transforms are holomorphic away from their measure's support and bounded on compact sets separated from that support.The appendix then invokes the Vitali–Porter theorem for locally bounded holomorphic-function sequences.Source: §A.2 · p. 33
• For locally bounded holomorphic sequences, convergence on a set with an accumulation point yields uniform convergence on compact subsets of the domain.Source: §A.2 · p. 33
• The paper uses the identity theorem to extend pointwise Stieltjes-transform convergence from a smaller set, typically the negative real axis, across the domain.This establishes the transform convergence needed for the underlying spectral-measure limits.Source: §A.2 · pp. 33–34

A.3 Self-Consistent Equations

• The self-consistent equation (34) has a unique positive solution hλ under Proposition 19's stated setup.Source: §A.3 · p. 34
• As λ approaches its limit, the rescaled solutions converge to the unique positive root when it exists, or to zero or infinity under the stated alternatives.Monotonicity of Φλ establishes convergence in each case.Source: §A.3 · p. 34

B.1 Closed-Form Solution

• The finite-dimensional derivation uses constant imbalance under the stated initialization to obtain a differential equation for W(t) and a closed-form solution for Y(t) = W(t)W(t)⊤.Source: §B.1 · pp. 35–36

B.2 Block Expressions for the Solution

• The block expressions recover U(t)U(t)⊤, V(t)V(t)⊤, and U(t)V(t)⊤ from the closed-form solution using the block structure of Aρ and W0.Source: §B.2 · pp. 36–37

B.3 SVD of the Prediction Matrix

• Almost surely, the prediction has rank min(m, m*); the weight ranks and compatible singular-value ordering follow from the conserved imbalance identity.Source: §B.3 · pp. 38–39

B.4.1 Convergence of the Oja Flow

• Under the stated spectral gap and initialization conditions, the Oja flow converges to the subspace associated with the M largest positive eigenvalues.The proof diagonalizes A and analyzes the flow in its eigenbasis.Source: §B.4.1 · pp. 39–40

B.4.2 Spectral Structure of Aρ

• The spectral decomposition of Aρ is built from the singular-value decomposition of C and supplies the eigenstructure used to apply the Oja-flow result.Source: §B.4.2 · p. 41

B.4.3 Reduction in the Overparameterized Case

• In the overparameterized case, E0 has dimension max(m − m*, 0), and W(t)W(t)⊤ acts on it as multiplication by 2ρ for all times.The proof reduces the remaining dynamics to an m*-dimensional Oja-flow problem.Source: §B.4.3 · p. 42, §B.4.4 · p. 43

B.4.4 Convergence in the Overparameterized Case

• The overparameterized convergence proof removes the invariant subspace and shows the remaining top m* eigenvalues exceed ρ, enabling the Oja-flow convergence argument.Source: §B.4.4 · pp. 43–44

B.4.5 Convergence in the Underparameterized Case

• In the underparameterized case, the top eigenvalues satisfy the required gap and the projected singular vectors form a basis almost surely, allowing the Oja-flow limit to apply.Source: §B.4.5 · pp. 44–45

B.4.6 Identification of the Block Matrices

• The limiting block matrices are identified from the positive eigenvectors and the block structure of the projection Π.Source: §B.4.6 · p. 45

C High-Dimensional Limit

• The high-dimensional appendix proves Section 4's results through random-matrix convergences, Stieltjes-transform limits, a two-point transform result, and a range-projector formula.Source: C High-Dimensional Limit · p. 45

C.1 Random Matrix Results

• Under the stated spectral and dimensional assumptions, Theorem 22 establishes almost-sure high-dimensional limits for trace quantities with real positive weights.The limits are derived using deterministic equivalents for sketched pseudoinverses.Source: §C.1 · pp. 45–47
• Theorem 23 extends the trace limit to complex weights through a unique holomorphic function and gives almost-sure convergence uniformly on compact subsets of C \ R+.The result assumes continuous q and r with q bounded below by a positive constant.Source: §C.1 · p. 48
• For r = 0, Theorem 24 derives the high-dimensional limit of a second-order trace quantity from the positive solution defining the preceding trace limit.Its result is obtained by combining deterministic limits with the relationship between the parameters ν and h.Source: §C.1 · pp. 50–51

C.2 Stieltjes Transforms

• The Stieltjes-transform section develops the transforms mU_t defined in equations (10) and (11) to prove the results stated in Section 4.1.Source: §C.2 · p. 51

C.2.1 High-Dimensional Limit

• As d→∞, the empirical spectral distribution of (C⊤C)^(1/2) converges to ˜µC=(1−β)δ0+βµC, where β=min(1,τ) and τ=lim_{d→∞}p/d.Source: §C.2.1 · p. 52

C.2.2 Initialization

• At initialization, the derived spectral distributions are consistent with U initialized at zero and V(0)V(0)⊤ equal to 2ρ times a rank-m orthogonal projector.The projector’s rank is asymptotically κd.Source: §C.2.2 · p. 54

C.2.3 Limiting Stieltjes Transforms

• In the overparameterized case, the limiting integral covers the full support of ϕC; in the underparameterized case, it begins at any threshold x* with TC(x*)=κ.The density ϕC vanishes almost everywhere between qualifying threshold choices, so they give the same limiting integrals.Source: §C.2.3 · p. 55
• The limiting Stieltjes transforms identify the high-dimensional spectral law of U(t)U(t)⊤, whose pushforward yields the singular-value distribution and recovers Corollary 5.The derivation uses the finite-dimensional spectral relation and the high-dimensional limits of the model dimensions.Source: §C.2.3 · p. 56
• In the high-dimensional limit, selecting the min(m,m*) largest singular values of C is equivalent to thresholding at x* under Assumption 1.3 and the absence of an atom at x*.Here x* is the threshold that selects the top singular values in the finite-dimensional limit.Source: §C.2.3 · p. 56

C.3 Two-Point Stieltjes Transforms

• The two-point Stieltjes transforms have an almost-sure high-dimensional limit when U(t)U(t)⊤ and V(t)V(t)⊤ serve as reference matrices.These references are relevant because, in finite dimension, the two matrices converge to spectral functions of CC⊤ and C⊤C, respectively, with a projection Π for V.Source: §C.3 · p. 57

C.4 Range-Projector Distances

• When m > m∗, V(t)'s projector contains a time-independent component Π that cancels in the difference from its long-time projector.Π projects onto Im(R) ∩ Ker(C) and has dimension m − m∗ with probability one.Source: §C.4 · p. 58
• The range-projector distance limits hold for V when κ ∈ (0, 1), and for U when κ ∈ (0, βκ∗).The overparameterized U case is treated separately, where its left distance is zero for all positive times.Source: §C.4 · p. 60
• For κ > βκ∗, the left range-projector distance is identically zero for every t > 0 because the evolving range equals its long-time limit.In this regime, U(t) has rank m∗ and Im U(t) = Im C almost surely.Source: §C.4 · p. 60
• The finite-dimensional projectors select singular directions of C above x∗, while the comparison projectors select the top rd directions.The normalized ranks of both choices converge to min(κ, βκ∗).Source: §C.4 · p. 60

D.1.2 High-Dimensional Limit

• In the small-initialization limit, the rescaled weight-matrix spectral distributions have high-dimensional limits characterized by Stieltjes transforms and a unique holomorphic function.The limits are stated under Assumption 1.Source: §D.1.2 · p. 62

D.1.3 Small-Initialization Limit

• The limiting eigenvalue distributions agree whether initialization tends to zero before or after dimension tends to infinity, at fixed t ≥ 0.Source: §D.1.3 · p. 64

D.2.1 Small Initialization

• The small-initialization range-projector analysis gives a limiting indicator f(x) = 1_{x≥x∗}.The cited result applies this limit in the proof of Theorem 8.Source: §D.2.1 · p. 65

D.2.2 Monotonicity in Time

• The squared small-initialization distance is strictly decreasing in time for R, and for L when κ < βκ∗.Source: §D.2.2 · p. 65

D.2.3 Long Times

• The functions b^a_t converge to zero exponentially fast in the stated setting of Theorem 9.Source: §D.2.3 · p. 67
• For κ > βκ∗, the left range-projector distance is zero, so the long-time analysis focuses on R in that regime.Source: §D.2.3 · p. 68
• The long-time proof treats the underparameterized case separately and uses x∗ as the unique point satisfying T_C(x∗) = κ.Source: §D.2.3 · p. 68

• The section replaces an unavailable exact high-dimensional limit for top singular-vector distances with direct, non-sharp convergence rates.The authors note that two-point Stieltjes transforms could encode the needed spectral overlaps, but do not pursue that technically involved route.Source: §D.3 · p. 69
• In the small-initialization limit, the rescaled Gram matrices share their nonzero eigenvalues with 2M_a(t), connecting the subspace dynamics to filtered subspace iteration.The section identifies this as the continuous version of filtered subspace iteration.Source: §D.3 · p. 70
• For proportional-rank subspaces, the bound has a non-exponential asymptotic because the spectral gap between adjacent singular values vanishes as dimension grows.The proof lower-bounds random-projection quantities away from zero when the projector and subspace ranks grow proportionally with 0 < α < κ < 1.Source: §D.3 · pp. 70, 72, 73
• A fixed spectral cutoff selects an asymptotic fraction of singular directions, avoiding assumptions about individual singular values near the cutoff.Only o(d) singular values can remain between the kth singular value and the cutoff.Source: §D.3 · pp. 73–74

D.4 Saddle-to-Saddle Dynamics

• Every intermediate limiting point is a strict saddle; the selected index set stays fixed between thresholds, and crossing a threshold moves the limit to the next critical point.The limiting points are critical points of the loss.Source: §D.4 · pp. 77–78

D.5.1 Eigenvalues

• For the eigenvalue limits, scalar integrands converge pointwise away from x_s; dominated convergence yields equations (28) and (29) for w<0.The limits extend to w∈C\R+ using holomorphicity and uniform boundedness.Source: §D.5.1 · p. 79

D.5.2 Singular Values

• The prediction singular-value empirical measure converges weakly, and its Stieltjes transforms converge to the expression leading to equation (30).The proof obtains this by relating the prediction measure to the spectral measure of U(t)U(t)⊤.Source: §D.5.2 · p. 80

D.5.3 Wasserstein Distance

• The optimal transport from μ_C to μ_s maps x≤x_s to 0 and leaves x>x_s unchanged.The mass removed from [0,x_s] is added to the atom at zero; the monotone map is optimal for quadratic cost on the real line.Source: §D.5.3 · p. 80

D.6.1 High-Dimensional Limit for the Distance

• Under Assumption 1, the normalized distance has an almost-sure high-dimensional limit expressed using ϑ_t, the unique positive solution of a self-consistent equation.The limit expression is assembled from terms involving ϑ_t, Δ_t, and J_t.Source: §D.6.1 · pp. 80–81

D.6.2 Small-Initialization Limit at Fixed Times

• As ρ→0 at fixed times, ϑ_t,ρ converges to κ and the normalized loss-distance limit is the one obtained by replacing Z(t) with 0.The proof also establishes J_t→1 before taking this limit.Source: §D.6.2 · p. 82

D.6.3 Small-Initialization Limit at Log-Timescale

• In the small-initialization limit, at timescale tρ = s log(ρ−1) with xs = (2s)−1 > x*, the estimates are 0 ≤ rL_t(γx)qt(γx) ≤ γx − ρ and 0 ≤ rR_t(γx)qt(γx) ≤ γx + ρ.Source: §D.6.3 · p. 82
• Dominated convergence, using that ˜µC has no atom at xs, yields the integral limits that recover equation (33) after multiplication by β^-1.Source: §D.6.3 · pp. 83–84

D.6.4 Comparison with the Wasserstein Distance

• On t=s log(ρ^-1), the quantities in (31) and (33) have the same limit, implying that singular-vector misalignment contributes vanishingly in the asymptotic limit.The second quantity is the squared Wasserstein distance between the empirical singular-value distributions.Source: §D.6.4 · p. 84

• Finite-dimensional simulations sample nonzero singular values of C from ϕ_C and use canonical-basis singular vectors, justified by rotational invariance for the observables studied.The simulations evolve U(t) and V(t) directly using explicit Euler steps.Source: E Numerical Simulations · p. 85
• The high-dimensional spectral curves use Gauss–Legendre quadrature and scalar self-consistent equations solved by damped Newton iterations.The density is approximated through the Stieltjes–Perron formula.Source: E Numerical Simulations · p. 85
• The small-initialization and logarithmic-time curves are evaluated directly from theorem formulas, and no parameters are fitted to finite-dimensional simulations.Doubling quadrature nodes changes theoretical densities by less than 0.2% of their maximum.Source: E Numerical Simulations · p. 86
• At d = 800, simulation distances from the formulas of Theorems 4 and 6 are at most about 10^-3 for τ ∈ {0.7, 1, 1.4}.At d = 40, Euler iterates are within 1.2 · 10^-2 in relative Frobenius norm of a fourth-order Runge–Kutta solution, which matches Proposition 1's closed form up to 10^-8.Source: E Numerical Simulations · p. 86