Learning curves of generic features maps for realistic datasets with a teacher-student model

Model definition

• The Gaussian covariate model lets teacher and student use different Gaussian-correlated spaces with arbitrary covariance, generalizing the classical i.i.d. Gaussian teacher-student framework.The teacher generates labels from u, while empirical-risk-minimizing students access only v through generic convex losses and regularization.Source: §1 · p. 3

Related work

• The framework extends rigorous teacher-student analyses to generic convex losses and regularization for regression and classification, while its Gaussian-equivalence scope remains subject to explicit applicability limits.The Gaussian covariate model contains the vanilla case and connects to kernel ridge regression, margin classification, random features, and related universality results [32] [33] [35] [46].Source: §1 · pp. 3, 5

Technical contributions

• Theorems 1 and 2 rigorously characterize the estimator and its asymptotic training and generalization errors in closed form.The proof uses Gaussian comparison inequalities.Source: §1 · p. 3
• The same asymptotic expression is also derived through the replica method, placing previously heuristic machine-learning results on a rigorous basis.This establishes a connection to the broad applications of replica methods in machine learning and computer science.Source: §1 · p. 4

Towards realistic data

• The Gaussian equivalent-model conjecture proposes that, for broad data distributions and teacher-student feature maps, realistic-data training and generalization errors asymptotically match the jointly Gaussian model.The feature maps may be deterministic, random, or learned from data, with covariance matrices encoding the data-generating process and learning model.Source: §1 · p. 4
• The theory captures nontrivial learning curves, including GAN-generated data with neural-network feature maps and ridge-regression curves on real datasets despite violated Gaussian assumptions.The latter universality-like behavior is illustrated for multiple feature maps on the odd-versus-even MNIST task.Source: §1 · pp. 4–5, Figure 1 · p. 4

• The relevant model statistics are the student covariance eigenvalues and rotated, rescaled projections of the teacher weights, encoded by their joint empirical density.The projection Φ^⊤θ0 onto the student space determines the teacher coordinates entering the joint distribution with the eigenvalues ωi.Source: §2 · p. 5
• The asymptotic theorem assumes proportional growth of n, p, and d, positive-definite covariance conditions, convergent spectral distributions, bounded singular values, and regularity ensuring feasible minimization.The cost r + g is coercive and must satisfy the stated Gaussian-integrability and scaling conditions.Source: §2 · p. 6
• Theorem 1 gives closed-form asymptotic training and generalisation errors for the ℓ2-regularized estimator through a unique self-consistent fixed point.The model parameters enter through ρ and the limiting joint distribution µ; solving the fixed-point equations yields the overlap parameters used in the error expressions.Source: §2 · pp. 5–7
• The fixed-point parameters have estimator-level interpretations: m⋆ measures teacher-space correlation, q⋆ is a covariance-weighted norm, and V⋆ controls loss-induced output nonlinearity.The regularization counterpart V̂⋆ describes deformation through the proximal operator.Source: §2 · p. 7
• Theorem 2 extends the analysis to generic non-separable losses and regularizations, providing finite-dimensional concentration and distributional characterizations that recover simpler asymptotic formulas under ℓ2 regularization.The framework also yields specialized expressions for ridge regression and binary classification with logistic training loss.Source: §2 · pp. 7–8

3.1 Random kitchen sink with Gaussian data

• Random kitchen sink features reproduce established Gaussian-data analyses [11] [12] [43] [46], while the framework additionally accommodates random-feature or multilayer teacher architectures.For these models, the required averages can be expressed using the Stieltjes transform of FF^⊤’s spectral density.Source: §3.1 · p. 8

3.2 Kernel methods with Gaussian data

• Kernel methods map directly to the Gaussian teacher-student framework through Mercer eigenfeatures, and Theorem 1 rigorously proves the kernel-ridge formula conjectured in [32] while extending replica analyses [57].The framework also covers losses beyond ridge regression, including logistic and hinge losses, and regularization beyond the hard-margin limit.Source: §3.2 · p. 8

3.3 GAN-generated data and learned teachers

• For GAN-generated CIFAR10-like data, Gaussian-model predictions closely match logistic-regression learning curves, while random features at initialization outperform learned features early and epoch-50 features separate at lower sample complexity than epoch-200 features.The experiment uses a dcGAN generator, neural-network teacher labels, and independently trained student feature maps.Source: §3.3 · p. 9, Figure 3 · p. 10

3.4 Learning from real data sets

• For real datasets, all interpolating linear teachers are asymptotically equivalent because predictions depend on the labels through shared covariance-projection quantities.Theorem 3 establishes this universality for any teacher feature map and interpolating teacher weights.Source: §3.4 · pp. 10–11
• For real datasets, the model accurately predicts ridge-regression learning curves, including double-descent interpolation peaks, but fails for classification error and real CIFAR10 logistic regression.Agreement deteriorates near the full dataset size, while learned Fashion-MNIST features progressively become low-rank, reaching rank 380 after 200 epochs.Source: §3.4 · pp. 11–12

A Main result from the replica method

• The Gaussian covariate model uses independently drawn labelled data with covariance determined by positive-semidefinite matrices and a feature-coupling matrix, with deterministic nonlinear labels as a principal case.The likelihood can be P0(y|x) = δ(y − f0(x)), including binary labels generated by f0(x) = sign(x).Source: A Main result from the replica method · p. 17
• The estimator is a regularized linear classifier characterized by sample complexity α = n/d and aspect ratio γ = p/d.Its asymptotic training and generalization performance is analyzed through the regularized empirical-risk minimization problem.Source: A Main result from the replica method · p. 17

A.1 Replica computation of the free energy; Averaging

• The replica method replaces the logarithmic free-energy calculation with an averaged replicated partition function, introducing overlap parameters linking teacher and student weights.A Gibbs measure concentrates around minimizers as β →∞, while the logarithm is handled by the replica trick.Source: A Main result from the replica method · p. 17, §A.1 · p. 18, Averaging · p. 18

Rewriting as a saddle-point problem

• In the high-dimensional limit with fixed α and γ, the replicated integral concentrates at overlap values that extremize a potential, yielding a saddle-point formulation.Delta functions free the overlaps and rewrite the averaged partition function as an extremization problem.Source: Rewriting as a saddle-point problem · p. 19

Replica symmetric ansatz; Summary

• The replica-symmetric ansatz sets replica-indexed overlaps and conjugate variables to common values, enabling the r → 0+ limit and producing the replica-symmetric free-energy density.The consistency conditions fix ρ = Eθ0 and ˆρ = 0.Source: Replica symmetric ansatz · pp. 19–20, Summary · p. 20

A.2 Ridge regression and fixed weights

• For ℓ2 regularization with fixed weights, the prior contribution to the free energy is evaluated explicitly using logarithmic determinant and trace identities.The calculation proceeds by taking logarithms and averaging over the Gaussian variables ξ.Source: §A.2 · p. 20

A.3 Taking the β →∞ limit

• The β →∞ zero-temperature limit is obtained by rescaling the order parameters; the prior term becomes elementary, while the likelihood term is expressed through the Moreau envelope of the loss.The Moreau-envelope representation is used to take the zero-temperature limit explicitly.Source: §A.3 · p. 21

A.4 Saddle-point equations

• Setting the potential’s gradients to zero yields self-consistent saddle-point equations, with the loss-dependent function defined as the negative derivative of the Moreau envelope and obtainable from a proximal operator.A Python implementation is provided for the losses discussed in the appendix.Source: §A.4 · p. 21

A.5 Examples

• The ridge-regression setting is closely related to mismatched models where the teacher acts in a higher-dimensional space or on an orthogonal feature subset [12].The generalized model additionally incorporates fixed feature layers, such as random projections or pre-trained feature maps, through the covariances.Source: §A.5 · p. 22
• For ridge regression, the asymptotic training and generalisation errors are given by the model’s closed-form equations and self-consistent fixed-point conditions.The fixed point is specified by (V⋆, q⋆, m⋆), with ρ defined from the teacher covariance structure.Source: §A.5 · p. 22
• The variance gap between ridge-regression training and generalisation error is parametrized by V⋆, which depends only on the population-covariance spectrum.V⋆ is characterized through an equation involving the spectral density νΩ.Source: §A.5 · p. 22

Binary classification

• For binary classification with f0(x)=f̂(x)=sign(x), the asymptotic classification error is expressed explicitly in terms of the overlaps q⋆ and m⋆.These overlaps solve the self-consistent saddle-point equations, and the formulation generalizes the d=p, Ψ=Id, Gaussian-teacher case derived in [33].Source: Binary classification · p. 22
• For the squared loss g(y,x)=1/2(y−x)^2, the binary-classification saddle-point equations simplify considerably, and the asymptotic training error also has a simple expression.The explicit equation depends on the selected loss function.Source: Binary classification · pp. 22–23
• With logistic loss g(y,x)=log(1+e^−yx), the equations for (V̂,q̂,m̂) cannot be integrated explicitly because the proximal operator lacks a closed-form solution.The corresponding function must instead be obtained by solving a self-consistent equation.Source: Binary classification · p. 23
• For hinge loss g(y,x)=max(0,1−yx), the proximal operator has a closed-form solution, although the resulting equations still cannot be integrated explicitly.As λ→0, both logistic and soft-margin solutions converge to the max-margin estimator.Source: Binary classification · p. 23

Random features

• The model encompasses random-features analyses for Gaussian data, with covariance statistics explicitly related asymptotically to the projection matrix and nonlinearity via the Gaussian equivalence theorem [16].These relations use constants determined by the activation function and rely on the Gaussian equivalence theorem [45].Source: §A.6 · p. 23

Generative models

• Assuming the Gaussian Equivalence Principle, the framework generalizes from generative models to structured teachers, including fixed or learnable teacher weights and composite pretrained maps.The teacher map may combine generative models with random features, scattering transforms, or pretrained neural networks.Source: §A.6 · p. 24

Kernel methods

• Kernel regression is equivalent to ridge regression in Mercer-feature space when the feature expansion is finite or truncated, with p = d and covariance matrices determined by the kernel features.The target may lie outside the RKHS while remaining square-integrable, provided its coefficients satisfy the stated decay condition.Source: §A.6 · pp. 24–25
• Rescaling the model parameters recovers the self-consistent equations of [32] for kernel ridge regression and extends the discussion to kernel logistic regression and support vector machines.The recovery follows by inserting the kernel covariance representation into the model equations.Source: §A.6 · p. 25

B Rigorous proof of the main result

• The formulation allows teacher and student matrices U ∈ R^n×p and V ∈ R^n×d to act through potentially non-separable functions.The training and generalization errors are defined for the resulting estimator, with auxiliary variables introduced for the proof and connected to replica variables in Appendix C, eq. (C.24).Source: B Rigorous proof of the main result · p. 26
• The proof analyzes jointly Gaussian teacher-student quantities through their covariance and defines overlaps that organize the fixed-point analysis.The tuple involving the teacher instance and student variables is bivariate Gaussian, and the overlaps are stated to be fundamental to the analysis.Source: B Rigorous proof of the main result · p. 26
• The main result computes the estimator distribution in the weak sense from six scalar fixed-point equations having a unique solution.The proof uses overlap variables rather than the spectral decomposition, which is mathematically equivalent but does not simplify the expressions here.Source: B Rigorous proof of the main result · p. 26

B.1 Necessary assumptions

• (A1) assumes an independent teacher vector θ0 drawn from an arbitrary distribution, including deterministic bounded-norm vectors, with non-vanishing finite squared norm.This assumption ensures that the teacher distribution is non-vanishing.Source: §B.1 · pp. 26–27
• (A2) requires well-behaved covariance matrices, convergent spectral distributions defining the overlaps, and bounded maximum singular values in the high-dimensional limit.The covariance blocks are positive definite, while the cross-correlation matrix Φ may have zero singular values; compact spectral support immediately verifies the summability condition.Source: §B.1 · pp. 26–27
• (A3) requires proper lower-semicontinuous convex functions r and g, coercivity of r + g, and scaling conditions ensuring finite Gaussian expectations.These conditions prevent non-diverging inputs from producing diverging outputs or sub-differentials and support Gaussian concentration of Moreau envelopes; coercivity covers ridge, LASSO, and elastic-net setups.Source: §B.1 · p. 27
• The remaining assumptions impose independence and finite generalization moments, finite dimensional ratios α = n/d and γ = p/d, and stronger regularity for linear or exponential finite-sample rates.The rate conditions include sub-Gaussian teacher coordinates, pseudo-Lipschitz functions, bounded covariance eigenvalues, and, for exponential rates, separability plus ridge or Lipschitz regularization.Source: §B.1 · p. 27

B.2 Main theorem

• The asymptotic characterization is built from scalar potentials involving Gaussian variables, Moreau envelopes, and the optimization problem defined over τ1, τ2, κ, η, ν, and m.The quantities are defined using independent Gaussian vectors and transformed teacher variables before being assembled into the potential E.Source: §B.2 · pp. 27–28
• Under Assumption (B.1), all previously defined quantities admit finite limits as n, p, and d tend to infinity.These limits provide the asymptotic regime required by the main theorem.Source: §B.2 · p. 28
• The potential E is jointly convex in (m, η, τ1) and jointly concave in (ν, κ, τ2), giving a structured optimization problem whose optimality conditions yield self-consistent fixed-point equations.The fixed-point characterization is stated through the optimality conditions of the scalar optimization problem.Source: §B.2 · p. 28
• The theorem’s limiting variables include proximal-operator definitions and jointly Gaussian variables whose covariance specifies the relevant asymptotic distribution.These variables connect the estimator-level statement to the scalar characterization.Source: §B.2 · p. 28
• Theorem 4 establishes finite-dimensional asymptotic formulas for the training loss and generalisation error under Assumption (B.1), with explicit concentration constants and probability control.The result applies to any optimal solution of the estimator and permits regularisation to be removed when evaluating training loss.Source: §B.2 · p. 28
• Theorem 5 extends the result from losses to the estimator itself, controlling observables for Lipschitz and separable pseudo-Lipschitz test functions applied to the optimal solution.For a larger class of test functions, concentration remains available but loses exponential rates.Source: §B.2 · p. 28

B.3 Theoretical toolbox

• The theoretical toolbox reviews known results used throughout the proof and supplies proofs of straightforward consequences that are not explicitly available in the literature.This subsection supports the derivation of the main theorem rather than adding a separate empirical result.Source: §B.3 · p. 29

B.3.1 A Gaussian comparison theorem

• The Convex Gaussian Min-max Theorem [27], a tight form of an inequality from, compares Gaussian primary and auxiliary optimization problems.For compact convex sets and convex-concave objectives, deviations of the primary value are bounded by corresponding auxiliary-problem deviations.Source: §B.3.1 · p. 29
• The framework defines acceptable primary and auxiliary optimization problems, analyzing the asymptotic primary problem through the simpler auxiliary problem.This reformulation is the central operational use of the comparison theorem.Source: §B.3.1 · p. 29

B.3.2 Proximal operators and Moreau envelopes : differentials and useful functions

• Moreau envelopes and proximal operators are introduced for closed, proper functions, with the envelope jointly convex and almost everywhere differentiable.The proximal operator is the unique minimizer of the strongly convex optimization problem defining the Moreau envelope.Source: §B.3.2 · p. 29
• For convex lower-semicontinuous functions, the Moreau envelope preserves pseudo-Lipschitz order, using firm nonexpansiveness of the proximal operator.The appendix also defines pseudo-Lipschitz functions and derives this preservation property through proximal and identity-minus-proximal mappings.Source: §B.3.2 · pp. 29–30
• The useful functions h1, h2, h3, and h4 have monotonicity properties: h1 is nondecreasing, while h2, h3, and h4 are nonincreasing.The proofs rely on firm nonexpansiveness, convexity in the envelope parameter, and Moreau decomposition.Source: §B.3.2 · pp. 30–31
• The appendix states a useful proximal inequality for convex functions, derived from monotonicity of the subdifferential and the resolvent representation of the proximal operator.This inequality extends a related one-dimensional result from.Source: §B.3.2 · p. 31

B.3.3 Useful concentration of measure elements

• Gaussian Poincaré inequality yields concentration results for Moreau envelopes of appropriately scaled convex functions and for finite-order pseudo-Lipschitz functions.Under Assumptions B.1, the Moreau-envelope concentration argument establishes Gaussian integrability before applying Poincaré and Chebyshev inequalities.Source: §B.3.3 · p. 32
• An exponential concentration lemma for separable order-2 pseudo-Lipschitz functions is additionally cited from [70].This complements the Poincaré-based bounds with a result stated without proof.Source: §B.3.3 · pp. 32–33

B.4 Determining a candidate primary problem, auxiliary problem and its solution.

• The original learning problem is reformulated into an acceptable primary problem suitable for applying Theorem 6.The reformulation partitions the Gaussian variables and introduces an equivalent optimization formulation.Source: §B.4 · p. 33
• Lemma 8 proves that optimal w, z, and λ are bounded by dimension-independent constants, allowing the problem to be recast over compact feasibility sets.The argument uses coercivity and lower semicontinuity of g + r, bounded minimizers, covariance assumptions, and random-matrix norm bounds.Source: §B.4 · pp. 34–35
• Lemma 9 reduces acceptable auxiliary optimizations to scalar optimization problems, initially assuming a deterministic teacher vector θ0.The reduction conditions Gaussian matrices, separates components along the teacher-related direction, and repeatedly uses convex duality and Moreau envelopes.Source: §B.4 · pp. 35–37
• For θ0 not in Ker(Φ⊤), the high-dimensional minimization is reduced to a scalar problem over six parameters with dimension-independent bounds on m, κ, and η.The derivation introduces orthogonal decompositions, Lagrange multipliers, convex conjugates, and Moreau envelopes before obtaining the six-parameter form.Source: §B.4 · pp. 36–40
• For θ0 in Ker(Φ⊤), a separate equivalent auxiliary optimization is derived using the positive-definite covariance factor and Moreau-envelope reductions.This case preserves compactness almost surely and concludes the proof of Lemma 9.Source: §B.4 · pp. 40–41

Geometry

• E_n is continuous, jointly convex in (m, η, τ_1), and jointly concave in (κ, ν, τ_2).These properties follow from the convexity of the Moreau-envelope terms, perspective functions, and minimization or maximization over convex variables.Source: §B.5 · pp. 41–42

Asymptotics

• The asymptotic potential E is continuously differentiable, retains the same joint convexity and concavity structure, and is obtained as the consistent infinite-dimensional limit of E_n.The convergence is established through strong-law and concentration arguments, followed by uniform convergence on compact feasibility sets.Source: §B.5 · pp. 42–44

Optimality conditions

• Every feasible solution of the asymptotic minimization problem satisfies an explicit set of fixed-point equations derived from the zero-gradient condition.The equations can also be translated into replica notation using table (C.24).Source: §B.5 · p. 44

Strict curvature

• For fixed (η, m, τ_1), E is jointly strictly concave in (τ_2, κ, ν), while on S_{∂ν,τ_2} it is jointly strictly convex in (η, m, τ_1).These strict properties support uniqueness and characterization arguments near minimizers.Source: §B.5 · p. 45

Connection to learning

• The limiting potential uniformly converges to the stated asymptotic form, linking the scalar optimization problem to the original learning problem and leading to estimators based solely on noise.The link uses the tight inequalities from Theorem 6; the noise-only consequence is stated for the following section.Source: §B.5 · p. 46

B.6 Back to the original problem : proof of Theorem 4 and 5

• Theorem 5 is proved by showing that finite-size optimizers converge to the asymptotic solution, with Theorem 4 following as a consequence.Strong convexity, concentration bounds, and an ε-net argument control convergence from the finite-dimensional auxiliary problem to its asymptotic counterpart.Source: §B.6 · pp. 47–49
• Exponential convergence rates rely on the stated regularity and covariance-eigenvalue assumptions; relaxing them to finite-order pseudo-Lipschitz functions yields linear rates.The proof uses concentration for sub-Gaussian and sub-exponential terms, Moreau envelopes, and bounded covariance eigenvalues.Source: §B.6 · pp. 48–49

B.7 Relaxing the deterministic teacher assumption

• Random teacher vectors can be handled by conditioning on the teacher and then averaging the resulting Theorem 5 expressions, provided independence from A, B, and f0’s randomness.The finite-size rates depend on teacher randomness and covariance-eigenvalue decay, whose conditions for retaining exponential rates are not fully investigated.Source: §B.7 · p. 49

B.8 The ’vanilla’ teacher-student scenario

• When teacher and student share the same covariance distribution, Ω = Φ = Ψ = Σ, the general equations reduce to explicit vanilla teacher-student fixed-point equations and asymptotic estimators.The covariance matrix Σ is positive definite with sub-Gaussian eigenvalue decay.Source: §B.8 · p. 50

C Equivalence replica-Gordon

• For separable loss with ridge regularization and random Gaussian teacher weights, the rigorous Theorem 5 result provides an exact analytical match to the replica prediction.The ridge case also removes the need to assume that Ω is invertible.Source: C Equivalence replica-Gordon · p. 51, §C.1 · p. 51

C.1 Solution for separable loss and ridge regularization

• The ridge-regularized separable-loss solution is derived through Moreau-envelope expectations and corresponding optimality conditions, simplified using Stein’s lemma.The derivation uses standard normal variables s and h with y = f0(√ρs).Source: §C.1 · pp. 51–52

C.2 Matching with Replica equations

• The fixed-point equations from the scalar minimization and replica methods match exactly after an explicit change of variables and Gaussian integration.The appendix identifies the notation mapping between replica quantities and the Gaussian-comparison proof, matching the first three equations directly and the remaining three after transformation.Source: §C.2 · p. 53

D Details on the simulations

• The simulation appendix provides full numerical details and makes implementations of the described pipelines available in the GCMProject repository.The implementation is used to generate the plots in the main manuscript.Source: D Details on the simulations · p. 54

Experimental setup

• Ridge regression is evaluated on centered, globally normalized MNIST and fashion-MNIST images with binary labels, using fixed regularization and subsampled training sets.MNIST and fashion-MNIST each use 6 × 10^4 samples and 28 × 28 inputs; labels separate even from odd digits or clothes above from below the waist.Source: §D.1 · p. 54

Feature maps

• The student representation compares scattering features, erf random features, their limiting kernel, and neural-network features at epochs 0, 5, 50, and 200.The scattering and random-feature maps each use 1953 features, while neural features come from the first two layers of a 2352-unit-per-layer network trained with Adam on MSE loss.Source: §D.1 · p. 54

Self-consistent equations

• Theoretical curves use empirical whole-dataset covariances and avoid explicitly estimating teacher weights by replacing them with the combinations appearing in the self-consistent equations.For binary labels, the construction sets ρ = 1; the kernel covariance is approximated with a 20 × 1953-dimensional random-feature space, whose accuracy is checked in Fig. 5.Source: §D.1 · p. 55, Figure 5 · p. 55

Limitations

• The covariance-based theory is expected to match simulations only when the training sample count n is much smaller than the available universe n_tot.As n approaches n_tot, deviations emerge; for MNIST 8-versus-9 NTK regression, theory predicts perfect generalization near n_tot while simulated error reaches a plateau.Source: §D.1 · pp. 55–56, Figure 6 · p. 56
• The universality construction breaks down for nonlinear predictors, including sign-based classification even when the estimator remains the ridge-regression solution under square loss.This mismatch is demonstrated for logistic regression on grayscale CIFAR10 and square-loss binary classification on MNIST.Source: §D.1 · p. 56, Figure 7 · p. 56

Data generation

• A dcGAN trained on CIFAR10 maps 100-dimensional Gaussian latent vectors to 32 × 32 × 3 images, enabling independent synthetic samples for simulation and covariance estimation.The GAN was trained for 50 epochs without data augmentation and generated 3072-dimensional CIFAR10-looking images.Source: §D.2 · p. 57

Teacher construction

• A teacher trained on real CIFAR10 images assigns binary odd-even labels to generated images through a learned feature map and final weights.The teacher uses the first two network layers as ϕt and the final layer as θ0, achieving 78% classification accuracy on the odd-even task.Source: §D.2 · p. 57

Simulations

• The student is trained on n = 30000 independently generated images with teacher-assigned labels, and logistic regression is evaluated on fresh generated samples across training snapshots.The pipeline averages points and error bars over 10 independent runs and also includes an identity-feature grayscale CIFAR10 variant.Source: §D.2 · pp. 57–58

Pipeline overview

• The complete synthetic-data pipeline first trains a dcGAN to generate images and then applies a teacher trained on real CIFAR10 to assign their labels.This workflow is illustrated in Fig. 8.Source: Figure 8 · p. 58

Self-consistent equations

• For synthetic GAN data, the theoretical curves use self-consistent equations supplied with teacher weights and Monte Carlo estimates of the population covariances.The covariances were estimated from n = 10^6 samples with precision of order 10^-5.Source: §D.2 · p. 58

Limitations

• The same universality underlies successful applications to real data with arbitrary feature maps, but the argument is heuristic, restricted to u = v, and its extension to the generic case remains open.Generalisation-loss terms require concentration results for inverse covariances, and a rigorous treatment of the broader formulas is left for future work.Source: E Ridge regression with linear teachers · pp. 59, 61

Model and random-matrix formulation

• For ridge regression with a linear teacher, Gaussian covariates yield closed-form expected training and test losses that can be analyzed as a random-matrix-theory problem.The setup uses a linear teacher y = Uθ0 and ridge regression on the student data matrix V.Source: E Ridge regression with linear teachers · pp. 59–60

Universality mechanism

• The theory’s training and test losses depend, under concentration, on population and empirical covariance spectra and traces of their products rather than explicitly on the data distribution.Universality requires empirical spectra and covariance-product traces to converge to the Gaussian-predicted values.Source: E Ridge regression with linear teachers · p. 60
• Random-matrix universality supports extending the asymptotic predictions to many non-Gaussian distributions sharing the same population covariances, and potentially to realistic data.Known results cover i.i.d. components beyond Gaussianity and broader classes of correlated random vectors.Source: E Ridge regression with linear teachers · p. 60

Concrete u = v example

• In the restricted u = v case, the training-loss terms depend on the limiting covariance eigenvalue distribution, whose Marcenko–Pastur law remains robust beyond Gaussian data.The law applies when dimension and sample size grow with fixed ratio α = n/d.Source: E Ridge regression with linear teachers · pp. 60–61