Learning reshapes power-law anisotropy in internal representations

• Power-law covariance spectra have been reported across biological and artificial neural systems [1] [7] [8] and linked to coding efficiency, robustness, generalization, and neural scaling laws [7] [11] [12] [13] [14].Source: §1 · p. 1
• Prior theories often treat representation spectra as fixed, leaving unclear how power laws arise, whether learning creates different local exponents across modes, and how this depends on learning regime.Earlier studies assumed power-law input spectra and derived representation dynamics, but had not characterized exponent evolution during learning [19] [20].Source: §1 · pp. 1–2
• In a wide two-layer linear teacher–student model, feature learning produces up to four asymptotic local power-law exponents, while lazy learning leaves the exponent nearly unchanged from the input spectrum.In the lazy regime, the learning-induced component never exceeds the initialization component.Source: §1 · p. 2
• The representation spectrum combines an input-inherited initialization component with a teacher-dependent learned component, shaped by teacher anisotropy and mode-dependent learning timescales.The model derives the temporal evolution of the hidden-layer representation kernel under population gradient flow.Source: §1 · pp. 2–3
• Numerical experiments with nonlinear networks show spectral transitions like those predicted by the linear model.Source: §1 · p. 2

Power-law spectra of internal representations

• Mouse V1 and other biological systems, vision models, and large language models exhibit power-law representation spectra, with broken-power-law fits motivating local, mode-dependent exponents [26] [7] [8].Mouse V1 eigenvalues decay approximately as k^-1; power-law feature-covariance spectra and their evolution during training have also been observed in vision models and large language models.Source: Related Works · p. 3

Input structure and spectral theories of fixed representations

• V1 power laws persist under spatially whitened inputs, while random-feature models inherit input exponents up to logarithmic corrections; how evolving kernels deform those exponents remains unknown [30].Fixed-kernel and fixed-feature theories use eigenspectra and target alignment to derive learning order, generalization curves, and neural scaling laws [14] [12] [19] [13].Source: Related Works · p. 3

Learning dynamics of deep linear networks

• Deep linear networks permit tractable study of nonlinear factorized-weight dynamics, with prior work deriving aligned-mode learning curves and hidden-representation and neural-tangent-kernel evolution [21] [23].Building on these results, the paper derives population-limit hidden-representation covariance for simultaneously diagonalizable power-law input and teacher structures, emphasizing its eigenspectrum decay.Source: Related Works · p. 3

2 Setting

• The model is a mean-field two-layer linear network in a teacher–student setting, with input dimension D, hidden width N, and output dimension P.The variables x and w_i are in R^D, while a_i, f, and y are in R^P; the teacher matrix Θ is in R^(D×P).Source: §2 · p. 3
• The setting takes P = D and N to infinity; P ≥ D is needed for spectrum evolution across a broad range of modes.The text motivates P ≥ D as natural for language models, diffusion models, and autoencoders, and focuses on P = D as the simplest such case.Source: §2 · p. 3
• Following previous work [31] [32] [33], γ0 controls the learning regime.Source: §2 · p. 3
• The input covariance and teacher Gram matrix are assumed diagonal in the same orthonormal basis, giving the input and teacher aligned structure.Source: §2 · p. 3
• Weights w_i and a_i are independently Gaussian-initialized and trained by gradient flow on mean-squared error, with learning rate ν.The analysis derives the infinite-data temporal evolution of internal-representation covariance eigenvalues and characterizes their power-law exponents; prior work [21] had studied parameter and loss evolution.Source: §2 · p. 4

3.1 Exact solution for the time evolution of representation spectra

• The exact spectrum separates an input-derived initialization component from a teacher-induced component, which is incorporated from low to high mode index during training.At initialization, the spectrum has the input variance's power-law exponent; analytical predictions closely match finite-width network simulations.Source: §3.1 · p. 5

3.2 Representation spectra exhibit four asymptotic exponent domains during learning

• Three boundaries distinguish input- from teacher-dominated modes, unlearned from learned modes, and first-layer-only from two-layer learning; near boundaries, slopes vary continuously.The boundaries are defined by R_k(t) = 1, ℓ_k(t) = 0.9, and u_k(t) = 1, respectively.Source: §3.2 · p. 6

3.2.1 Region I: Initialization-dominated regime

• In Region I, the initialization component dominates, including at sufficiently high modes, so exponent changes occur only over a finite mode range.This includes modes not yet affected by learning and modes with teacher signals too weak to displace initialization.Source: §3.2.1 · p. 7

3.2.2 Region II: The 3α + β regime at early learning times

• When ℓk ≪ 1 and uk ≫ 1, the nonlinearity of ξk(t) dynamics becomes relevant, and the eigenvalue spectrum ρk(t) has no single power-law exponent, interpolating between adjacent exponents.Source: §3.2.2 · p. 7

3.2.3 Region III: Learned modes dominated by the learned component

• Region IIIa requires σ_a ≫ σ_w and corresponds to learning primarily in the first layer, while Region IIIb corresponds to learning in both layers.These regions describe learned modes dominated by the learned component.Source: §3.2.3 · p. 7

3.2.4 Four asymptotic exponent regions

• Feature learning yields as many as four asymptotic exponent domains and nonmonotonic exponent evolution; in the lazy regime, the exponent remains close to its initial value.The four domains coexist under suitable hyperparameters; near domain boundaries, the spectrum may not follow a single power law.Source: Figure 3 · p. 5, §3.2 · p. 6, §3.2.4 · pp. 8–9

3.3 Similar exponent dynamics appear in nonlinear networks

• In nonlinear networks, feature-learning spectra evolve from α to α + β across a finite mode range, whereas lazy-regime spectra remain at α.The nonlinear-network results agree quantitatively with the linear-theory predictions and suggest that the mechanism may extend beyond linear networks.Source: §3.3 · p. 9, Figure 4 · p. 9

4 Conclusion

• The analysis derives exact population-limit eigenvalue dynamics and separates the representation spectrum into an initialization baseline and a training-formed increment.It concludes that exponents are reshaped over time and across modes through the interaction between input statistics and task learning.Source: §4 · pp. 9–10
• In feature-learning, local exponents take four asymptotic values: initialization-dominated, early mode-dependent, post-learning first-layer-driven, and deformation distributed across both layers.These regimes occur over finite mode ranges sufficiently far from the boundaries.Source: §4 · pp. 9–10
• Early learning rates set by input eigenvalues transiently steepen the spectrum, whereas in the lazy regime its exponent remains nearly unchanged from the input exponent α.In the lazy regime, the training-induced component never exceeds the initialization component.Source: §4 · p. 10
• Numerical experiments show that the predicted exponent transitions and the feature-learning versus lazy-regime difference persist in nonlinear networks.The results suggest that input anisotropy, teacher-signal modal structure, and mode-dependent learning rates may govern representation formation in more general networks.Source: §4 · p. 10
• The analysis assumes aligned input covariance and teacher structure, infinite width, and population-limit gradient flow; future work includes misalignment, finite-width and sample effects, deeper networks, and general nonlinear activations.The paper also identifies testing the predicted transitions and boundaries in real data and large-scale models as an open question.Source: §4 · p. 10

Appendix

• The appendices define three Gram matrices used in the subsequent analysis.Source: Appendix · p. 13

A The anisotropy of the internal representation corresponds to the eigenvalues of M(t)

• In the population limit, the kernel eigenvalue problem becomes an integral-operator problem whose nonzero eigenvalues coincide with those of M(t), characterizing representation anisotropy.Every eigenfunction associated with a nonzero eigenvalue is linear in x.Source: A The anisotropy of the internal… · pp. 13–14

B Warm-up: Derivation of the eigenvalues when the second layer is fixed

• With the second-layer weights fixed, the derivation solves the time evolution of the first-layer weights and yields an analytical solution for M(t) under infinite-data gradient flow.The learning rate is set to ν = N so parameter dynamics remain O(1) as N → ∞.Source: B Warm-up: Derivation of the… · pp. 14–15
• Assuming ΘΘ⊤ is diagonal, M(t) remains diagonal through training, and its eigenvalue dynamics decouple across k.Source: B Warm-up: Derivation of the… · p. 14
• Figure 5 illustrates eigenvalue dynamics for the model with a fixed second layer.Source: Figure 5 · p. 16, B Warm-up: Derivation of the… · p. 16

C Warm-up: Derivation of the power-law exponent boundaries when the second layer is fixed

• The local exponent is derived analytically from ρk(t), yielding its time evolution as a function of mode k.The derivation expresses the exponent using Q(k, t) and its derivative.Source: C Warm-up: Derivation of the power-law… · pp. 16–17

C.1 Learned modes

• For learned modes, the spectrum has exponent α + β at finite k, but ultimately decays with exponent α at sufficiently large k.The α term reflects input-induced anisotropy, while the learning-induced term decays more rapidly with k.Source: §C.1 · p. 17

C.2 Unlearned modes

• For unlearned modes, finite-k spectra have exponent 3α + β, while the asymptotic exponent at sufficiently large k is α.The mode is unlearned when k is much greater than klearn(t); the learned/unlearned boundary follows klearn(t) = (cλ0t)^(1/α).Source: C Warm-up: Derivation of the power-law… · p. 17, §C.2 · p. 18
• The fixed-second-layer spectrum has three asymptotic regions: α for k > kcross(t), 3α + β for klearn(t) < k < kcross(t), and α + β otherwise.These regions summarize the learned and unlearned mode boundaries.Source: §C.2 · p. 18

D Analytical eigenvalue solution when both layers are trained

• When Λ, Θ, Gwa, Gw, and Ga are diagonal initially, their Gram-matrix dynamics preserve diagonality for all time.This reduces the analysis to independent dynamics of the diagonal entries.Source: D Analytical eigenvalue solution when… · p. 20
• Two conserved quantities reduce the eigenvalue dynamics to a sequence of analytical steps from ξk(t) to ηk(t) to ρk(t).The solution uses a change of variables and separation of variables; the resulting eigenvalues are ρk(t) = λkηk(t).Source: D Analytical eigenvalue solution when… · pp. 20–22

E Derivation of the power-law exponent boundaries when both layers are trained

• Three relations suffice to derive the power-law exponents.Source: E Derivation of the power-law exponent… · p. 22

E.1 Learning boundary tlearn(k)

• The learning boundary tlearn(k) is when ξ_k(t) reaches 90% of its limiting value, separating unlearned from learned modes.ξ_k(t) increases monotonically from 0 to γ_0θ_k.Source: §E.1 · p. 23, E Derivation of the power-law exponent… · p. 23
• klearn(t) is defined as the inverse of tlearn(k).Source: §E.1 · p. 23

E.2 Boundary tcross(k) at which learning becomes dominant

• At tcross(k), the learning contribution in η_k(t) equals the input contribution present at initialization.tcross(k) is the time required for the learning contribution to overtake the input contribution.Source: §E.2 · p. 23
• For larger k, tcross(k) diverges.Source: §E.2 · p. 23

E.3 Boundary tlayer(k) governing the allocation of learning

• The layer-allocation boundary tlayer(k) is defined by u_k(t) = 1, equivalently ξ_k(t) = S.Source: §E.3 · p. 23
• For larger k, tlayer(k) diverges.Source: §E.3 · p. 24

E.4 Conditions for the existence of the asymptotic regions

• Region I always exists because Rk(0) = 0 < 1.Source: §E.4 · p. 24
• Region II exists when the intersection of tcross(k) and tlearn(k) occurs at a k-coordinate much larger than 1.The intersection coordinate is determined using the monotonic increase of t∗k(C).Source: §E.4 · p. 24
• For Region IIIa to exist over a broad range, the second-layer initialization scale must be sufficiently larger than the first-layer scale.Source: §E.4 · p. 24
• Region IIIb requires k < ku and k < kl.Source: §E.4 · p. 24
• The four asymptotic regions coexist under combined parameter conditions.The supplied passage does not specify those conditions.Source: §E.4 · p. 24
• Large second-layer initialization relative to the first layer and large γ0 generally correspond to feature-learning; in the lazy regime, the full range 1 ≤ k ≤ D is Region I and the exponent remains nearly unchanged.In feature-learning, the second layer changes significantly.Source: §E.4 · p. 25

F Proof that the ordering of the eigenvalues ρk(t) is preserved

• For every t, k > l implies ρ_k(t) ≤ ρ_l(t), preserving the eigenvalue ordering throughout training.The proof concludes that the input-dimension index k coincides with eigenvalue rank.Source: F Proof that the ordering of the… · pp. 25–26
• The proof shows that ξ(t; λ, θ) is monotonically increasing in t, λ, and θ.Because λ_k and θ_k are ordered in descending order, their monotonicity supports the preserved ordering across modes.Source: F Proof that the ordering of the… · pp. 25–26

G Numerical settings

• The plots use λ_k = k^-1 and μ_k = ζ(2)^-1 k^-1, with α = β = λ_0 = 1 and μ_0 = 1/ζ(2).These choices give Σ_k λ_kμ_k = 1.Source: G Numerical settings · p. 26
• Inputs and targets are generated as x ~ N(0, Λ) and y = Θ^T x, with Λ = diag(λ_1, …, λ_D), Θ = diag(√μ_1, …, √μ_D), and σ_a = 1.0.Source: G Numerical settings · p. 26
• For Figure 2, the finite-width linear network uses D = P = 512, N = 8192, and M = 4096 fixed i.i.d. training samples, trained by full-batch gradient descent on empirical mean-squared loss.Source: Figure 2 · p. 26
• Figure 4 uses the same data model and teacher with a replacement student, and records preactivation spectra as in Figure 2.Source: Figure 4 · p. 26, G Numerical settings · p. 26