Surprises in High-Dimensional Ridgeless Least Squares Interpolation

• Overparameterized deep networks can fit even random labels, yet interpolation and poor generalization are distinct, with high-dimensional interpolation sometimes nearly optimal.These observations motivate analyzing interpolation beyond the conventional association between zero training error and overfitting.Source: §1 · p. 1
• The paper studies linear regression with centered features having covariance Σ and centered noise with variance σ^2.The data consist of i.i.d. feature-response observations in R^p, with responses generated under the stated linear model.Source: §1 · p. 1
• Because p > n makes the least-squares minimizer nonunique, the paper compares minimum-ℓ2-norm interpolation with ℓ2-penalized ridge regression.Min-norm regression selects the least-squares solution with smallest ℓ2 norm, whereas ridge penalizes the coefficient norm squared.Source: §1 · p. 1

2. We denote these

• The study evaluates min-norm interpolation and ridge prediction risk in the proportional regime p ≍n, emphasizing p > n and varying the geometry of (Σ, β).The considered cases include isotropic, well-specified and misspecified, latent-space, and nonlinear features; min-norm interpolation arises as λ →0 when the design has full row rank.Source: §2 · p. 2, §1.1 · p. 2

1.1 Summary of results

• The results establish double descent, overparametrized risk minima under misspecification or latent structure, and nonlinear risks matching Gaussian-feature risks.In isotropic well-specified models, risk diverges near γ = 1 and ultimately minimizes underparameterized; strong approximation bias can move the global minimum beyond γ = 1, while latent-space risk can decrease toward γ →∞. Optimally tuned ridge dominates min-norm except in aligned latent settings, and cross-validation is asymptotically optimal.Source: §1.1 · p. 4, Bias and variance · pp. 5–6

1.2 Connection to neural networks

• Neural-network linearization yields high-dimensional random-feature regression whose gradient-descent limit is the minimum-ℓ2-norm interpolator, but the correspondence is restricted to lazy training and fixed features.The analysis additionally uses a universality hypothesis, while the paper notes that learned representations and fully nonlinear training may change interpolator behavior.Source: §1.2 · pp. 6–7

1.3 Related work

• The work builds on research connecting interpolation, minimum-norm regression, neural networks, and kernel ridge regression, including lazy-training analyses by Jacot et al. (2018), Du et al. (2018b,a), and Chizat and Bach (2018b).Prior work also studies related kernel limits and nonlinear mean-field regimes as hidden units diverge.Source: §2 · p. 2, §1.3 · p. 7
• Prior work studied ridge or min-norm regression mainly for isotropic Gaussian or independent-entry covariates, whereas this paper extends the analysis to general covariance structures and misspecified models, including cases where optimal risk occurs at γ > 1.Dicker (2016), Dobriban and Wager (2018), Advani and Saxe (2017), and Belkin et al. (2019) provide earlier proportional-asymptotic analyses; the paper extends this line beyond those settings.Source: §1.3 · p. 7
• The paper obtains non-asymptotic risk approximations with explicit, nearly optimal error bounds under weaker assumptions on the geometry of (Σ, β) than Wu and Xu (2020) and Richards et al. (2020).Earlier work showed that λ = 0+ can be optimal or nearly optimal when β relates to Σ's eigenvectors; the paper reviews these generalizations in Section 4.Source: §1.3 · p. 7
• For nonlinear random features, the paper handles rectangular matrices whose prediction variance depends on singular values and vectors by applying a leave-one-out resolvent analysis to an extended matrix.This differs from prior kernel random-matrix results focused on symmetric matrices; Mei and Montanari (2019) later obtained a complete characterization of the nonlinear random-features risk.Source: §1.3 · p. 8

1.4 Outline

• The paper’s structure covers background, five linear-model settings, the nonlinear model, and appendices containing nearly all proofs.Sections 3–7 address isotropic features, correlated features, misspecified models, ridge regularization, and cross-validation; Section 8 treats nonlinear features.Source: §1.4 · p. 8

2.1 Data model and risk

• Conditional prediction risk is defined on an independent test point, with a corresponding bias–variance decomposition conditional on the training features X.Source: §2.1 · p. 8

2.2 Ridgeless least squares

• The minimum-ℓ2-norm least-squares estimator is equivalently the pseudoinverse solution and the zero-ridge limit, interpolating training responses when X has rank n.Source: §2.2 · pp. 8–9
• Gradient descent initialized at zero converges to the min-norm least-squares solution when 0 < t ≤ 1/λmax(X^T X).Source: §2.2 · p. 9

2.3 Bias and variance

• The min-norm estimator’s bias and variance are expressed using the sample covariance ˆΣ = X^T X/n and the projection Π = I − ˆΣ+ˆΣ onto X’s null space.Source: §2.3 · p. 9

2.4 Underparametrized asymptotics

• As n,p → ∞ with p/n → γ < 1, least-squares risk converges almost surely to σ^2γ/(1−γ), reflecting variance alone; overparametrization instead yields nonzero bias β^TΠΣΠβ.Source: §2.4 · p. 9

3 Isotropic features

• Isotropic features reduce dependence on β to its squared norm r2 = ∥β∥2, while Section 4 addresses covariance geometry.This simplification follows from rotational invariance in the Gaussian case and extends to general isotropic features.Source: §3 · p. 10, §3.1 · p. 10, §3.2 · p. 10

3.1 Limiting bias

• The limiting bias converges almost surely to r2(1−1/γ) in the overparameterized regime.The bias concentrates around its expectation under the isotropic model.Source: §3.1 · p. 10

3.2 Limiting risk

• Theorem 1 establishes the isotropic limiting risk for independent, unit-variance features with finite 4 + η moment and p/n → γ.For γ < 1, bias is absent and variance increases; for γ > 1, bias increases while variance decreases. Relative to null risk r2, performance depends on SNR: it can improve below the interpolation threshold when SNR ≤ 1 and above a threshold when SNR > 1.Source: §3.2 · p. 10

3.3 Limiting ℓ2 norm

• The limiting squared ℓ2 norm increases on (0, 1), diverges at γ = 1, and decreases on (1, ∞).This profile resembles the limiting risk curve.Source: §3.3 · p. 11

4 Correlated features

• The framework represents correlated-feature geometry through Σ’s eigenvalues and β’s coefficients in the corresponding eigenvector basis, while assuming independent standardized entries in z.The analysis permits arbitrary deterministic positive-definite Σ, subject to the stated spectral regularity conditions.Source: §4 · pp. 11–12

4.1 Prediction risk

• Finite-sample bias and variance converge to deterministic functions of the empirical covariance spectrum, β’s reweighted spectrum, and γ = p/n under a bounded-spectrum, non-interpolation-threshold setting.The approximation holds with probability at least 1 − Cn^-D when λ_min(Σ) > 1/M, uniformly in finite n and p.Source: §4.1 · pp. 12–13, §4 · p. 12
• Under weak convergence of the spectral measures, normalized bias and variance converge almost surely to B(H, G, γ) and V(H, γ), requiring weaker assumptions than prior asymptotic analyses.The non-asymptotic result also implies asymptotic predictions under minimal weak-convergence assumptions.Source: §4.1 · p. 13

4.2 Equidistributed coefficients

• For equidistributed coefficients G = H and γ > 1, the limiting bias and variance simplify; in the isotropic case, they are Bequi(H, γ) = 1 − γ^-1 and Vequi(H, γ) = 1/(γ − 1).Equidistribution means β has no special relation to Σ, with components roughly equally distributed across covariance eigenvectors.Source: §4.2 · pp. 13–14, §4.1 · p. 13

4.3 Limiting ℓ2 norm

• The min-norm estimator’s limiting ℓ2 norm is characterized almost surely by c0(H, γ) and integrals over the covariance and β-weighted spectral measures, with distinct formulas for γ < 1 and γ > 1.This result generalizes the corresponding isotropic analysis.Source: §4.3 · p. 14

5 Misspecified model

• In the misspecified model, adding features beyond interpolation can reduce both bias and variance, and asymptotic risk can attain its global minimum on (1, ∞).This contrasts with the well-specified isotropic model, whose risk is globally minimized at γ = 0.Source: §5 · p. 14

5.1 Data model and risk

• The model observes X but not W, fits the min-norm estimator using only X and y, and evaluates prediction risk while integrating over training noise and unobserved features.Lemma 2 separates the usual regression risk from the misspecification bias; under jointly Gaussian features, that bias is M(β, θ) = θ^T Σw|xθ.Source: §5.1 · pp. 14–15

5.2 Isotropic features

• For isotropic features, the limiting risk contains irreducible misspecification bias, an additional bias when γ > 1, misspecification variance, and ordinary noise variance.The result assumes i.i.d. zero-mean unit-variance entries with a finite moment of order 8 + η, with κ denoting the fraction of signal captured by observed features.Source: §5.2 · pp. 15–16

5.3 Polynomial approximation bias

• With polynomial approximation-bias decay, least-squares risk can beat the null risk for γ < 1 only under sufficient decay, while for SNR > 1 it can also beat it for γ > 1.For SNR > 1 and sufficiently small a, the global minimum occurs after γ = 1; for SNR ≤ 1, the γ > 1 risk decreases toward the null risk from above.Source: §5.3 · p. 16

5.4 A latent space model

• In the latent-space model, observed features are noisy measurements of latent covariates, and the prediction risk decreases for γ > 1, reaching its asymptotic global minimum as γ →∞.As p grows, feature noise averages out, making min-norm regression approach the idealized procedure that observes the latent covariates directly.Source: §5.4 · p. 19

6 Ridge regularization

• Theorem 5 gives non-asymptotic deterministic approximations for ridge bias and variance, uniformly over covariance matrices with nearly optimal sample-size dependence.Under the stated eigenvalue, aspect-ratio, and regularization conditions, the prediction risk decomposes into bias plus variance with high probability.Source: §6 · p. 20

6.1 Isotropic features

• For isotropic ridge regression, the optimal limiting regularization is λ* = σ^2γ/r^2, yielding the simplified risk expression R_X(β̂_λ; β) → σ^2γm(−λ*).The limiting risk is characterized using the explicit isotropic solution for m(z).Source: §6.1 · p. 21
• In isotropic features, optimally tuned ridge has strictly lower asymptotic risk than min-norm least squares across r^2, γ, and κ, with risk minimized near γ = 1 as SNR increases.Figures 7 and 8 compare the well-specified and misspecified settings; the misspecified risk additionally includes the approximation-bias term r^2(1 − κ).Source: §6.1 · p. 21, Figure 7 · p. 22, Figure 8 · p. 22

6.2 Latent space model

• In the latent-space model, risk is minimized at large overparameterization γ ≫ 1 for all probed λ, while regularization smooths the γ = 1 divergence and can make risk monotonically decreasing.The lower envelope over regularization is monotonically decreasing in γ, and Figure 9 displays the corresponding risk curves.Source: §6.2 · p. 21, Figure 9 · p. 23
• At large overparameterization, the optimal regularization is λ → 0 even with noisy observations, because covariate noise can act like an additional ridge term.The kernel perturbation is approximated by K(X,X) + η^2I_n, providing implicit regularization; Figure 10 confirms λ → 0 despite σ_ξ > 0.Source: §6.2 · pp. 21, 24, Figure 10 · p. 23

7.1 Limiting behavior of CV tuning

• A Sherman-Morrison-Woodbury shortcut expresses ridge leave-one-out CV as a weighted average of training errors, avoiding refitting for each observation.The same asymptotic results hold when the normalization factor n is replaced by n − 1.Source: §7.1 · p. 24
• For isotropic features, ridge leave-one-out CV error converges almost surely to prediction error in proportional high-dimensional asymptotics.The result assumes p/n → γ ∈ (0, ∞), with i.i.d. zero-mean, unit-variance features having a finite 4 + η moment.Source: §7.1 · p. 25
• CV- or GCV-tuned ridge achieves the same asymptotic performance as optimally tuned ridge.This identifies the ridge risk curves as asymptotic risks under CV tuning.Source: §7.1 · p. 25

7.2 Shortcut formula for ridgeless CV

• The leave-one-out shortcut extends to min-norm least squares when p > n and λ = 0+ by taking the ridge limit through a kernel representation.The extension assumes rank(X) = n and resolves the zero-over-zero terms in the ordinary shortcut formula.Source: §7.2 · p. 25
• The ridgeless shortcut also applies to positive-definite kernel matrices and to models with an unpenalized intercept after double-centering and using pseudoinverses.In the kernel setting, it gives leave-one-out CV for kernel ridgeless regression.Source: §7.2 · p. 25

8 Nonlinear model

• The universality motivation extends to neural-network feature maps by matching the second-order statistics of gradient features to an equivalent Gaussian model.This is presented as an expectation for lazy-training regimes rather than as the theorem’s nonlinear-feature construction.Source: §8 · p. 26
• The nonlinear model uses x_i = ϕ(Wz_i), with Gaussian inputs z_i, Gaussian weights W scaled by 1/d, and componentwise activation ϕ.Conditionally on W, observations are independent, but their coordinates are dependent because they derive from the lower-dimensional input z_i.Source: §8 · p. 26, §8.1 · p. 26

8.1 Limiting risk

• Purely nonlinear standardized random features have asymptotic bias and variance exactly matching the linear isotropic model, rigorously confirming universality despite dependent coordinates.The result assumes Gaussian inputs and weights, componentwise activation, polynomial growth, E[ϕ(G)] = 0, E[ϕ(G)^2] = 1, and E[Gϕ(G)] = 0.Source: §8.1 · pp. 26, 28
• For ϕ_abs, finite-sample risks agree excellently with the asymptotic curve and depend on p/n rather than d/n.The simulation uses ϕ_abs(t) = a(|t| − b), with a and b chosen to satisfy the standardization conditions.Source: Figure 11 · p. 27, §8.1 · p. 28

8.2 Proof outline for Theorem 8

• The resolvent analysis shows that the empirical covariance spectrum converges to the Marchenko–Pastur law, as in the independent-entries model.The convergence holds almost surely and in L1 when p/n → γ and d/p → ψ.Source: §8.2 · p. 29
• The proof connects asymptotic resolvents to min-norm least-squares variance through a Taylor–Laurent expansion, whose coefficient D0 gives the ridgeless limit.For p/n → γ ∈ (0, ∞) and d/p → ψ ∈ (0, 1), lim λ→0+ lim n,p,d→∞ V_X(β̂_λ; β) = D0 almost surely.Source: §8.2 · p. 29

A.1 Proof of Theorem 5: Preliminaries

• The linear-model proof preliminaries establish a high-order finite-difference bound and analytic, monotone control of a modified Stieltjes-transform solution u(λ, η).The auxiliary lemma assumes bounded spectral support, a nonvanishing fraction of eigenvalues above 1/M, and p/n bounded away from 1; ridge bias and variance then approach interpolation values as λ → 0+.Source: §A.1 · pp. 29, 31, 33

A.2 Proof of Theorem 5, bias term

• The bias proof normalizes ||β||2 = 1 because the bias is homogeneous of degree 2, under bounded population covariance.It works on an analytic domain where Re(η) > −1/λmax(Σ), with ||Σ||op ≤ M and ||R||op ≤ 1/λ.Source: §A.2 · pp. 33–34
• An anisotropic covariance local law controls the resolvent representation uniformly on the complex domain with probability at least 1 − Cn−D.The argument uses Knowles and Yin (2017, Theorem 3.16.(i)) for λ satisfying Re(λ) > n−2/3+ε0.Source: §A.2 · p. 33
• Analyticity and differentiability of the self-consistent solution yield the predicted bias expression after taking Im(λ) → 0 and differentiating at η = 0.The construction uses rn, mn, and Fn together with the eigenvalue transformation si(η) = si/(1 + ηsi).Source: §A.2 · p. 34

A.3 Proof of Theorem 5, variance term

• For the variance term, the proof applies the same local-law framework to analytic resolvent quantities and controls their derivatives uniformly for Re(λ) above the spectral-edge scale.The derivative of Ln is computed using the definition of mn.Source: §A.3 · p. 35

A.4 Generalization to heavier tail covariates: Proof of Theorem 6

• The heavier-tail extension truncates and recenters covariates, then uses Bai-Yin operator-norm bounds to show the truncated remainder becomes negligible.The truncated variables have bounded moments of all orders, allowing Theorem 5 to be applied to XM and ΣM.Source: §A.4 · p. 36
• Taking M → ∞ and using HM ⇒ H and GM ⇒ G completes the extension from truncated models to the original heavier-tailed covariance model.The proof combines the truncation estimates with Borel-Cantelli to obtain the limiting result.Source: §A.4 · p. 36

B.1 Proof of Theorem 2

• The proof reduces minimum-norm regression to ridge regression with λ ≤ 1 and treats the bias and variance terms separately.Variance is independent of β, while bias is homogeneous in ||β||2, allowing the normalization ||β||2 ≤ 1.Source: §B.1 · pp. 36–37
• The bias comparison is controlled spectrally through the eigenvalue decomposition of S_X = X^T X/n and the smallest nonvanishing singular value.Under λ_min(Σ) ≥ 1/M and |p/n − 1| ≥ 1/M, the singular-value lower bound holds with probability at least 1 − Cn^-D, following Bai and Yin (1993).Source: §B.1 · p. 37
• The resulting bias difference satisfies B_X(β̂_λ; β)^1/2 − B_X(β̂; β)^1/2 ≤ C(M)λ.The min-norm bias is also bounded by ||Σ||op ≤ M.Source: §B.1 · p. 37
• A fixed-point analysis shows c*(λ) remains bounded and changes by at most C(M)λ as λ varies from zero.Monotonicity of f and bounds on its derivative imply C(M)^-1 ≤ c*(λ) ≤ C(M), while the implicit function theorem gives |∂λc*(λ)| ≤ C(M).Source: §B.1 · p. 37
• Substituting the fixed-point expansion into the relevant equations yields the claimed bound (105).The argument uses uniformly bounded O*(λ) remainders and repeats the same type of control across the intermediate equations.Source: §B.1 · p. 38
• Combining bounds (104) and (105) with Theorem 5 gives the desired high-probability result for n^-2/3+1/M λ ≤ 1.The proof then sets λ = n^-1/7 and chooses ε sufficiently small; the variance term follows by an analogous argument using |x(x + λ)^-2 − x^-1| ≤ 2λ/x^2.Source: §B.1 · p. 38

C.1 Proof of Proposition 2

• The Bai-Yin theorem ensures X^T X is almost surely invertible for sufficiently large n, yielding zero projection and bias terms.This conclusion follows from λmin(Σ) ≥ c and the lower bound on the smallest eigenvalue of Z^T Z/n.Source: §C.1 · p. 38

C.2 Proof of Theorem 1

• Theorem 1 is proved by expressing bias and variance through sample-covariance spectra, then applying Marchenko-Pastur and Stieltjes-transform limits.The proof also uses pseudoinverse identities, generalized Marchenko-Pastur convergence, Bai-Yin eigenvalue bounds, and an exchange of limits.Source: §C.2 · pp. 39–40

C.3 Proof of Corollary 2

• Corollary 2 follows by showing that B(H, H, γ) converges to B_equi(H, γ) as ∥β∥2 approaches r2.The argument uses homogeneity and compares the resulting expression with the definition of B_equi(H, γ).Source: §C.3 · p. 40

C.4 The case of equicorrelated features

• For ρ-equicorrelated features, Corollary 7 gives the asymptotic risk by reducing the spectral limit to a point mass at 1 − ρ.Although λmax(Σ) diverges, decomposing Σ into an isotropic term and a rank-one term preserves the theorem’s conclusion.Source: §C.4 · p. 41

C.5 Autoregressive features

• For ρ-autoregressive features, the appendix replaces unavailable closed forms with numerical integration and root-finding for v(0) and v′(0).Known eigenvalues approximate integration against dH, after which the asymptotic risk is evaluated; Figure 13 varies ρ from 0 to 0.75.Source: §C.5 · pp. 41–42
• The autoregressive analysis is based on the Silverstein equation, which relates the companion Stieltjes transform v to the limiting spectral distribution H.Differentiating the equation provides v′(0), enabling numerical evaluation of the risk.Source: §C.5 · pp. 41–42

C.6 Proof of Theorem 7

• Theorem 7 begins by analyzing generalized cross-validation, whose rotationally invariant denominator replaces each diagonal smoother entry with the average trace.The proof studies GCV first because it is easier to analyze, then relates it to leave-one-out cross-validation.Source: §C.6 · p. 42

GCV denominator

• The denominator limit follows directly from the Marchenko–Pastur theorem and the Stieltjes transform of its law.The spectral eigenvalues si = λi(XT X/n) determine the denominator, whose almost-sure limit is rewritten using m = mFγ.Source: §C.6.1 · pp. 42, 44

GCV numerator

• The numerator limit uses independent mean-zero quadratic forms, the Marchenko–Pastur theorem, and convergence of Stieltjes-transform derivatives.The quadratic-form argument applies because δ and A are independent, while Vitali’s theorem controls derivative convergence.Source: §C.6.1 · pp. 44–45

GCV convergence

• GCV converges almost surely to the asymptotic ridge-regression prediction error at tuning parameter λ.The identification follows by reparametrizing the limit with the companion Stieltjes transform and matching it to the ridge-risk expression.Source: §C.6.1 · p. 45

Uniform convergence

• The GCV-tuned ridge estimator converges almost surely to the limiting risk σ2γm(−1/α).Uniform convergence of both the GCV criterion and prediction risk over the tuning interval transfers minimization from finite samples to the limiting criterion.Source: §C.6.1 · p. 46

C.6.2 Analysis of CV

• The dependent quadratic forms in CV are handled by deleting observation i and applying the Sherman-Morrison-Woodbury formula before invoking almost sure quadratic-form convergence.This separates the dependent feature vector from the independent leave-one-out matrix, enabling the limiting analysis.Source: §C.6.2 · p. 47
• The CV denominator remainder vanishes almost surely after trace approximations, moment bounds, union bounds, and the Borel-Cantelli lemma control the maximum deviation.The argument also uses Marchenko-Pastur and Bai-Yin bounds to control spectral quantities.Source: §C.6.2 · pp. 47–48
• CV has the same almost sure asymptotic risk limit as GCV, with uniform convergence and CV-tuned risk convergence following by analogous arguments.The proof replaces CV denominators by their limits, controls the remainder almost surely, and matches the limiting denominator through the Silverstein equation.Source: §C.6.2 · p. 49

C.7 Misspecified model results

• Misspecified-model results follow by reducing the misspecified case to a well-specified case through the substitutions in (21).Source: §C.7 · p. 49

C.8 CV and GCV simulations

• CV, GCV, and optimal tuning show excellent agreement and remain close to their common asymptotic risk curves throughout the simulations.Source: §C.8 · p. 49
• Figures 15 and 16 plot finite-sample risks for CV, GCV, and optimal ridge tuning alongside asymptotic risk curves under well-specified and misspecified models, respectively.Source: Figure 15 · p. 50, Figure 16 · p. 50

D Nonlinear model: Proof of Theorem 8

• The nonlinear-model section proves Theorem 8 and sets up a subsequent generalization to activations with a non-vanishing linear component.The broader result assumes E[Gϕ(G)] ≠ 0 for standard normal G.Source: D Nonlinear model: Proof of Theorem 8 · p. 51

D.1 A more general theorem

• Theorem 9 characterizes the asymptotic resolvents m1,n and m2,n under standardized, polynomial-growth activations as n, p, d →∞ with p/n →γ and d/p →ψ.The limits are defined through unique fourth-degree equations and hold almost surely and in L1.Source: §D.1 · p. 51
• The theorem yields a deterministic Stieltjes-transform limit for the sample covariance spectrum, reducing to the Marchenko–Pastur transform when c1 = 0.The corollary identifies Sn(ξ) → s(ξ), where s uniquely solves the stated equations and is analytic in the upper half-plane.Source: §D.1 · p. 52
• The asymptotic resolvents determine the ridgeless prediction-variance limit D0 for ridge regression when γ ≠ 1, with p/n →γ ∈(0, ∞) and d/p →ψ ∈(0, 1).Specifically, lim λ→0+ lim n,p,d→∞ VX(β̂λ; β) = D0 almost surely.Source: §D.1 · p. 52

D.2 Proof of Theorem 9

• The proof follows Cheng and Singer’s resolvent-based approach, constructs an orthogonal activation-function decomposition, and establishes a unique solution with |m1|, |m2| < r0 for ξ > ξ0.It also proves ||E||op ≤ εn ≡ (log n)^M/n1/2 with probability at least 1 − O(n−1), where M is an absolute constant.Source: §D.2 · p. 52, §D.2.1 · p. 56, §D.2.2 · p. 57

D.2.3 Derivation of Eqs. (134)

• D.2.3 Derivation of Eqs. (134): The derivation applies Schur complements and decomposes vectors and matrices into components parallel and orthogonal to w_p.This produces the reduced matrices and fluctuation terms used to obtain the displayed resolvent expressions.Source: §D.2.3 · pp. 59–60
• D.2.3 Derivation of Eqs. (134): The error matrix has operator norm at most ε_n = (log n)^c/√n, after which Woodbury’s formula evaluates the required low-rank correction.The remaining terms are computed analogously to the previous section and substituted into the target equation.Source: §D.2.3 · p. 60

D.2.4 Completing the proof

• For G ∼ N(0, 1), a degree L = L(ε) polynomial ϕL approximates ϕ in mean square, while F has a unique fixed point mL = F(mL).F maps D(r0) × D(r0) into D(r0/2) × D(r0/2), with Lipschitz constant at most 1/2.Source: §D.2.4 · p. 60
• D.2.4 Completing the proof: Almost surely, m1,n(ξ) and m2,n(ξ) converge to m1(ξ) and m2(ξ) for all ξ ∈ C+, yielding Eqs. (135) and (136).The proof first establishes convergence in the upper half-plane above a sufficiently large imaginary threshold, then extends it to all ξ ∈ C+.Source: §D.2.4 · p. 60

D.3 Proof of Lemma D.5

• The spectrum separates into eigenvalues above and below a fixed threshold, with almost-sure limits characterized for every γ ≠ 1 and c1 ∈ [0,1).Lemma 15 introduces S+ and S− using a threshold ρ0 and states the corresponding almost-sure limits.Source: §D.3 · p. 62
• The empirical spectral measure converges almost surely to a limiting probability measure through convergence of the associated transforms.The proof establishes Q_n(ξ) → q(ξ), then uses weak convergence to obtain μ_n ⇒ μ∞ almost surely.Source: §D.3 · p. 62
• The limiting positive-spectrum transform is analytic near zero, proving Eq. (141) with q+(0) = D0 and γc = D−1.This identifies the key constants in the limiting spectral description.Source: §D.3 · p. 63

D.4 Proof of Corollary 8 and Corollary 6

• Setting s = t = 0 yields Sn(z2) → s(z2) almost surely and in L1, and simplifying the governing equations produces Eqs. (138)–(140).The derivation combines the specialized equations and compares them with Eq. (260).Source: §D.4 · p. 63

D.5 Proof of Theorem 8

• For Theorem 8, the variance formula follows by taking c1 → 0 in the earlier theorem, solving the resulting quadratic equations, and applying Lemma 4.The same specialization also recovers Lemmas 3 and 4 by setting t = 0 and c1 = 0 in Theorem 9 and its companion lemma.Source: §D.5 · pp. 63–64
• The resulting asymptotic bias is the same as for the linear model with random isotropic features.The proof uses Σ0 = Ip when c1 = 0 and μMP({0}) = (1 − γ−1)+, together with dominated convergence.Source: §D.5 · p. 64
• When c1 = 0, the empirical spectral distribution converges almost surely to the Marchenko–Pastur law μMP.This spectral convergence is used to analyze the asymptotic bias through the eigenvalues of ˆΣ.Source: §D.5 · p. 64