PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy

Lorenzo Rizzi, Arie Wortsman Zurich, Bruno Loureiro

arXiv:2608.28564v1stat.MLcs.LG

TL;DR

Understanding how power-law anisotropy changes kernel ridge regression is challenging for single-index targets because kernel eigenfunctions are not explicit. The paper characterizes the kernel spectrum and generalization asymptotically, finding distinct variance and bias regimes for weak and strong anisotropy.

  • Problem

    For single-index models, expressing targets in the kernel basis is challenging because the kernel eigenfunctions are not explicit.

    Source: §5.1 · p. 16

  • Method

    The authors use the asymptotic spectrum of inner-product kernels to derive deterministic equivalents for kernel ridge regression bias and variance under power-law covariance anisotropy.

    Source: §6 · p. 20

  • Results

    For α<1, variance peaks at integer polynomial transitions and dampens as α grows; for α>1, variance becomes sample-size independent and bias transitions depend on target decay.

    Source: §1 · p. 2, §6 · p. 20

  • Takeaways & Limitations

    The single-index analysis illustrates how eigenbasis-based risk behavior can emerge for targets defined directly in input space.

    Source: §5 · p. 15

    The assumed multi-dimensional coefficient decay does not map straightforwardly to an exact source condition because sorting products across multi-indices entails nontrivial combinatorial constraints.

    Source: §4.1.2 · p. 11

Abstract

from arXiv · show

We study kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent $α\geq 0$ for polynomial inner-product kernels. We derive asymptotically sharp expressions for the kernel spectrum and the generalization error in the polynomial high-dimensional regime $n=Θ(d^κ)$, revealing how anisotropy reshapes the learning curves. For weak anisotropy ($0<α<1$), the problem remains effectively high-dimensional and retains some features of the isotropic case, while departing from it in others: the variance still peaks at integer sample complexities $κ\in\mathbb{N}$, but these peaks are progressively damped as $α$ grows; meanwhile, for targets strongly aligned with the data's principal directions, the bias drops at fractional sample complexities, decoupling the bias transitions from the interpolation peaks. For strong anisotropy ($α> 1$), the effective dimension of the problem is constant, and the variance stops depending on sample size altogether, plateauing under ridgeless interpolation or vanishing at an explicit rate under fixed ridge penalty. The bias undergoes a sharp transition governed by the target's decay rate: below a threshold, learning is abrupt rather than gradual; above it, the bias decays as a power law that recovers the classical source and capacity rates. We finally specialize these results to single-index targets, showing how the alignment of the index with the data's principal directions determines the effect of anisotropy on learning. Together, our results clarify how the input geometry shapes the kernel features and fundamentally impacts its generalization properties.

1 Introduction & motivation

Kernel scaling laws are established, but how input geometry shapes kernel features and learning remains largely open; this work gives a sharp account of kernel ridge regression under anisotropic inputs.

1 Introduction & motivation

  • For polynomial inner-product kernels under power-law anisotropic data, the authors derive an asymptotically exact high-dimensional spectrum and closed-form bias and variance equivalents.The analysis applies in the polynomial regime n = Θ(d^κ).Source: §1 · p. 2
  • For 0 < α < 1, principal-direction alignment lets degree-m target components become learnable at n = Θ(d^{m(1−α)}), while variance peaks remain at integer κ and are damped as α grows.The fractional bias thresholds separate learning transitions from interpolation peaks.Source: §1 · p. 2
  • For strong anisotropy, variance becomes sample-size independent, while bias transitions depend on target decay: below a threshold learning is abrupt or delayed, and above it bias follows a power law.Without regularization variance saturates at a constant; with constant ridge penalty it vanishes at an explicit rate, and the power-law bias recovers source and capacity rates.Source: §1 · p. 2
  • In single-index models, alignment of the index with principal data directions determines distinct learning behavior in feature space.The paper characterizes anisotropy’s effects from the kernel spectrum through the resulting learning curves.Source: §1 · p. 2

Further related work

2 Setting & Background

This section defines KRR’s generalization problem and motivates studying how anisotropic data geometry shapes the kernel and its excess risk. The analysis focuses on power-law anisotropic Gaussian inputs in a polynomial high-dimensional regime.

Setting

  • KRR minimizes regularized empirical risk in the kernel’s RKHS, and the section studies its predictor’s excess risk relative to the Bayes target.The target is f⋆(x) = E[y|x].Source: §2 · p. 3
  • Because the input distribution determines the kernel operator’s eigensystem, it also determines the RKHS geometry that influences generalization.The operator eigenfunctions and eigenvalues define the kernel features and an explicit representation of the RKHS.Source: §2 · pp. 3–4

Background

  • For isotropic inputs, studied kernel eigenspectra have degree-wise eigenvalue degeneracy and produce staircase excess risk, with degree-m target components requiring at least n = O(d^κ) samples.Such explicit data-geometry-to-risk results are scarce because diagonalizing the kernel operator is generally difficult (Misiakiewicz, 2022).Source: §2 · p. 4

Setting

  • The study examines Gaussian inputs with covariance eigenvalues following a power law, where α controls anisotropy from isotropic data at α = 0 toward effectively low-dimensional data for large α.The covariance is normalized to have trace 1; the motivation is that real data can exhibit direction-dependent variance and power-law structure.Source: §2 · p. 4
  • The analysis seeks sharp excess-risk descriptions as d grows with ψ := n/d^κ = Θ_d(1), primarily for integer κ but also for fractional κ.Source: §2 · p. 4

Assumptions

  • The assumptions use a finite-degree polynomial inner-product kernel with nonnegative coefficients and a degree-D polynomial target with independent, mean-zero label noise.A degree-D kernel cannot fit target components above degree D, so those components can be treated as irreducible noise; D is taken large enough.Source: §2 · pp. 4–5

3 High-dimensional limit of the kernel spectrum

The section sharpens bounds on the anisotropic kernel spectrum in the high-dimensional limit and identifies how power-law covariance decay changes the problem’s effective dimension and spectral structure.

3 High-dimensional limit of the kernel spectrum

  • For polynomial kernels, each multi-index monomial contributes a new independent kernel component, motivating a corresponding eigenvalue.Source: §3 · p. 5
  • In the high-dimensional limit, if R₀(Σ)→∞, the upper and lower bounds of Wortsman and Loureiro (2025) are tight; since Tr(Σ)=1, this is equivalent to Tr(Σ²)=o_d(1).Source: §3 · p. 5
  • The normalization rα(d), the problem’s effective dimension, diverges sub-linearly as d→∞ for 0<α<1, preserving genuinely high-dimensional structure; for α>1, it saturates to a finite constant and is effectively finite-dimensional.Source: §3 · p. 6
  • For 0 < α < 1, the spectrum has gaps and disconnected sectors, so a simple capacity condition cannot accurately characterize generalization.Source: §3 · p. 6

4 Generalization properties

The section establishes high-probability deterministic equivalents for kernel ridge regression excess risk in the anisotropic power-law setting and characterizes how anisotropy affects variance and bias across sample-complexity regimes. It distinguishes integer-scale variance peaks from target-dependent bias transitions under weak anisotropy, and examines variance and target- and ridge-dependent bias behavior under strong anisotropy.

4 Generalization properties

  • With probability tending to one, the excess risk is approximated by deterministic bias and variance equivalents under the stated assumptions, with relative approximation error tending to zero.This establishes the deterministic-equivalent framework for studying anisotropic generalization.Source: §4 · p. 7

4.1.1 Variance

  • For weak anisotropy, variance peaks at integer κ and vanishes at fractional κ; increasing α suppresses the peaks and progressively flattens the multiple-descent curve.As α approaches 1 from below, the variance tends to zero, and the distinction between integer peaks and fractional valleys disappears.Source: §4.1.1 · pp. 8–10

4.1.2 Bias

  • For weak anisotropy, target decay rate separates bias transitions: when ω < 1/2, features are learned at integer κ, while when ω > 1/2, degree m features require n = Θ(d^(m(1−α))).In the slowly decaying case, the fraction of degree-κ features learned depends on sample-complexity scale ψ; for ω > 1/2, the accessible degree reaches ⌊κ/(1−α)⌋.Source: §4.1.2 · pp. 12–13
  • Anisotropy can let targets aligned with principal directions be learned before the matching variance peak: those directions cross the effective-regularization threshold earlier and incur a smaller variance cost.The bulk of the shell, largely orthogonal to the target, crosses later at n = Θ(d^m), producing a variance peak without a matching bias gain.Source: §4.1.2 · p. 13

4.2 Strongly anisotropic regime

  • For strong anisotropy α > 1, the effective dimension converges to the finite constant ζ(α) as d grows, and the section states a deterministic-equivalent variance result.The supplied passage introduces Proposition 4 but does not include its formula or further variance behavior.Source: §4.2 · p. 13, §4.2.1 · p. 13

4.2.1 Variance

  • For fixed positive ridge, variance vanishes asymptotically, whereas without ridge it is constant independent of κ and sample size in the strongly anisotropic regime.For α > 1, the effective dimension is constant; the extracted variance statements do not preserve the explicit asymptotic rate.Source: §4.2.1 · pp. 13–14
  • In the weakly anisotropic regime, variance peaks at integer polynomial sample complexities, while for ω > 1/2 the bias drops at fractional scales n ≍ d^{m(1−α)}.For 0 < ω < 1/2, target decay accelerates learning of the active κ-th shell; for ω > 1/2, the bias transitions decouple from the integer variance peaks.Source: §4.2.1 · p. 14

4.2.2 Bias

  • For ω ∈ [0, 1/2), ridgeless learning leaves components m ≥ κ unresolved until n ∼ d^{κ+ϵ}, while finite ridge resolves only degrees up to ⌊κ/α⌋ with delayed learning.Finite ridge smooths transitions at resonances κ = rα and delays degree-m learning to n ∼ d^{mα}, trading reduced variance for increased bias.Source: §4.2.2 · p. 15
  • When ω > 1/2, bias tends to zero for either λ = 0 or λ = Θ(1), with a convergence-rate crossover at ω = α + 1/2.For n = Θ(d^κ), the resulting rates recover source-and-capacity rates.Source: §4.2.2 · p. 15
  • At α = 1, weakly and strongly anisotropic predictions match continuously: unregularized variance vanishes at the boundary, and the bias predictions remain consistent.As α approaches 1 from either side, the corresponding variance terms smoothly tend to zero; the κ-th shell also becomes entirely unlearned in the limit from below.Source: §4.2.2 · p. 15

5 Case study: single-index models

The section studies single-index targets f⋆(x) = g(v⊤x) and uses a Hermite-basis ansatz to connect their alignment and decay to the learning behavior under anisotropy.

5 Case study: single-index models

  • A single-index model depends on x only through g(v⊤x); normalizing v by E[(v⊤x)²] = 1 permits an explicit Hermite expansion of the target.The scalar link g is assumed continuous and in L2(pX).Source: §5 · pp. 15–16

5.1 Hermite ansatz

  • The Hermite ansatz conjectures that squared target energy is asymptotically distributed alike in the kernel and Hermite bases, allowing deterministic equivalents to use Hermite coefficients.The paper reports numerical evidence for the conjecture and cites extensive numerical support for a more general version by Karkada et al. (2026).Source: §5.1 · p. 17

5.2 Learning a SIM under anisotropy

  • For a localized index aligned with principal direction j, each polynomial degree contributes only its fully diagonal eigentensor, with shell energy E(m) = g_m^2.The analysis distinguishes leading directions (j = 1) from last directions (j = d), across weak and strong anisotropy.Source: §5.2 · pp. 17–18
  • Power-law single-index targets map to an effective decay exponent ω = α + γ, while sparse and delocalized limits correspond to ω → ∞ and ω = α/2, respectively.Averaging over a random delocalized index gives ω = α/2, equivalent to a power-law target with γ = 0; together these cases map SIM configurations into the general power-law framework.Source: §5.2 · pp. 18–19
  • For α > 1 and γ ≥ 0, power-law SIMs have ω > 1/2, so their asymptotic generalization bias vanishes in the thermodynamic limit.The bias depends on the interplay between α and γ in this regime.Source: §5.2 · p. 19

6 Conclusion · A Properties of the Kernel Operator · B Deterministic Equivalents

The paper characterizes kernel ridge regression under power-law anisotropy through sharp kernel-spectrum and generalization-error asymptotics, identifying distinct weak- and strong-anisotropy regimes. It also summarizes the bias transitions and provides appendices on the kernel spectrum and conditions for deterministic equivalents.

6 Conclusion

  • The analysis derives closed-form deterministic equivalents for bias and variance in both ridgeless and macroscopic-ridge regimes from the asymptotic spectrum of inner-product kernels.The conclusions distinguish two qualitative regimes according to the power-law exponent α.Source: §6 · p. 20
  • For α < 1, variance depends on sample size and has interpolation peaks at integer polynomial transitions that weaken as α grows; for α > 1, it is sample-size independent.In the strong-anisotropy regime, variance either saturates at a positive ridgeless limit or tends to zero with order-one ridge; both regimes approach vanishing variance at α = 1.Source: §6 · p. 20
  • For α < 1, bias partially resolves the current polynomial shell when ω < 1/2, but ω > 1/2 unlocks higher-order features beyond that shell.Increasing ω below the threshold reduces the unlearned fraction of the current shell.Source: §6 · p. 20
  • For α > 1, bias plateaus when ω < 1/2, while ω > 1/2 enables decay toward zero with a further rate transition at ω = α + 1/2.The plateau reflects failure to learn features of degree m ≥ κ; the two decay rates apply within the learning phase.Source: §6 · p. 20

A Properties of the Kernel Operator

  • Appendix A proves the kernel-spectrum proposition for multi-indexed eigenvalues under the stated trace and positive-coefficient assumptions.The proposition assumes Tr(Σ²) = o_d(1) and h_k > 0 for all k in [D].Source: A Properties of the Kernel Operator · p. 24

B Deterministic Equivalents

  • Appendix B sets up deterministic equivalents by separating lower- and higher-degree eigenfunctions and stating the concentration, target-tail, and label-noise assumptions needed to apply the theorem.The section says those assumptions must be proved in this setting before using the deterministic equivalents.Source: B Deterministic Equivalents · pp. 25–27

B.1 The Weakly Anisotropic Setting … D Weakly anisotropic regime

The appendices verify conditions for deterministic equivalents in weak and strong anisotropy, reparameterize kernel and target spectra by polynomial shells, and identify when diagonal corrections matter.

B.1 The Weakly Anisotropic Setting

  • For n = Θ(d^κ), the relative approximation ν_λ,m is 1 when κ is not an integer and O(poly log(n)) when κ is an integer.When κ is an integer, the threshold set includes the whole degree-κ shell.Source: §B.1 · pp. 27–28
  • Polynomial eigenfunctions of polynomial inner-product kernels are hypercontractive, and the proof establishes diagonal concentration for the higher-order conditions.The passage identifies hypercontractivity and concentration of the higher-order kernel diagonal as requirements for the higher-order conditions.Source: §B.1 · pp. 29–31
  • In the weakly anisotropic regime, the proof establishes the assumptions needed to apply deterministic equivalents.It combines the effective-dimension, lower- and higher-order, and spectral conditions.Source: §B.1 · p. 33

B.2 The Strongly Anisotropic Setting

  • For strong anisotropy, deterministic equivalents remain valid when the ridge parameter satisfies λ > c for a constant c independent of n and d.The construction has m ≤ n^(1−δ₀), and the passage requires positive ridge regularization.Source: §B.2 · p. 33

C Reparametrization of the kernel eigenvalues

  • The spectral reparameterization groups eigenvalues by polynomial shell and relaxes ordered index sums to free sums, neglecting asymptotically subleading repeated-index corrections.The target expansion uses symmetrically extended weights to make the same free-sum conversion.Source: C Reparametrization of the kernel… · pp. 33–34
  • The free-sum approximation can fail for targets concentrated on diagonal features, where the multinomial correction cannot be ignored.The passage contrasts these targets with dense or delocalized targets, for which bulk terms dominate.Source: C Reparametrization of the kernel… · p. 34

D Weakly anisotropic regime

D.1 Kernel State Equation

The section derives a macroscopic kernel state equation for ξ by separating polynomial-degree contributions into regimes around κ and κ/(1−α). It reduces the relevant multidimensional integrals to one-dimensional Irwin–Hall integrals.

D.1 Kernel State Equation

  • For m < κ, the degree-m terms vanish asymptotically; macroscopic integration boundaries occur at m = κ or when κ/(1−α) is an integer.The boundary behavior is summarized in Figure 10.Source: §D.1 · p. 35, Figure 10 · p. 36
  • The Irwin–Hall identity rewrites the m-dimensional integral as a one-dimensional integral over the distribution of a sum of m independent uniform variables.This reduction enables the analysis of the branches with m ≥ κ.Source: §D.1 · p. 37
  • At m = κ, ψ(κ) converges to a finite macroscopic value, evaluated through a convergent integral.This satisfies the state equation’s requirement that ψ(m) remain Θ(1).Source: §D.1 · p. 39
  • For κ < m ≤ κ/(1−α), the diverging upper boundary dominates, yielding the same asymptotic expansion as for m > κ/(1−α), where fractional dimension scalings cancel.The resulting higher-degree contribution is self-regularizing.Source: §D.1 · p. 39
  • The assembled state equation implicitly determines ξ, with higher-degree tails contributing an effective ridge λ_eff = λ + sum_{m>κ} h_m and the explicit resonant term ψ(κ).The resonant contribution is finite and macroscopic.Source: §D.1 · p. 39

D.2 Variance

In the polynomial high-dimensional limit, the variance sum is determined solely by the shell m = κ; lower-degree, intermediate, and super-critical shells all vanish.

Case a: m < κ

  • For m < κ, τ(m) = Θ(d^(m−κ)) and vanishes in the thermodynamic limit.These lower-degree shell contributions strictly tend to zero.Source: §D.2 · p. 40

Case b: m > κ/(1−α)

  • For m > κ/(1−α), the super-critical tail vanishes, including at α = 1/2, where logarithmic growth is overcome by polynomial decay.When α > 1/2, the exponent κ − 2m(1−α) is strictly negative in this regime.Source: §D.2 · pp. 40–41

Case c: κ < m ≤ κ/(1−α)

  • Throughout κ < m ≤ κ/(1−α), τ(m) vanishes, with decay bounded by O(d^(κ−m)) or e O(d^((κ−m)(1/α−1))).At α = 1/2, a logarithmic factor is suppressed by the polynomial decay.Source: §D.2 · p. 41

Synthesis

  • Only the m = κ shell contributes to the variance in the high-dimensional limit.All other contributions in the shell sum vanish as d → ∞.Source: §D.2 · p. 41

Finite-size scaling

  • As dimension increases, empirical variance curves collapse onto the asymptotic theoretical prediction.Figure 11 shows this finite-size scaling for κ = 1, α = 0.3, and λ = 0.Source: Figure 11 · p. 42

D.2.1 Fractional sample complexity κ

For noninteger κ, there is no resonating shell m = κ, so the threshold integral term ψ(κ) vanishes and the kernel state equation simplifies. The variance also lacks a macroscopic contribution and is governed by the adjacent integer shells.

D.2.1 Fractional sample complexity κ

  • Without the resonating shell m = κ, the threshold integral term ψ(κ) vanishes and the self-consistency equation simplifies.This applies when κ is not an integer.Source: §D.2.1 · p. 42
  • For noninteger κ, variance tends to zero, with its asymptotic decay set by the slower-decaying shell among ⌊κ⌋ and ⌈κ⌉.The effective variance decay exponent is the minimum absolute decay rate of these two bounding shells.Source: §D.2.1 · p. 42
  • For α < 1/2, τ(⌈κ⌉) = Θ(d^(κ−⌈κ⌉)); for α > 1/2, the variance term scales as Θ(d^((⌈κ⌉−κ)(1−1/α))).The α > 1/2 regime is described as having a localized integral.Source: §D.2.1 · p. 42

D.3 Bias · Slow decay target ω < 1 · E Strongly anisotropic regime

The bias separates into target-dependent unlearned energy and a target-independent correction, with shell contributions governed by target decay and sample-complexity scaling. For slow-decay targets, some shells saturate at their full energy, while the strongly anisotropic regime expresses eigenvalues up to a dimension-independent multiplicative constant.

D.3 Bias

  • The bias is written as target-dependent unlearned energy divided by a target-independent correction, with the target contribution expanded over polynomial shells.The unlearned energy depends explicitly on the target coefficients in the kernel eigenbasis; the correction term does not depend on the specific target.Source: §D.3 · pp. 42–43
  • For fast-decay targets with ω > 1/2, the shell-energy series converges in the thermodynamic limit and the shell prefactor scales as Θ(d^0).The source condition uses target anisotropy ω and a shell-dependent scalar coefficient.Source: §D.3 · p. 43
  • Away from transition boundaries, a shell retains its full spectral energy when the relevant exponent is negative, but its contribution vanishes when the exponent is positive.The analysis defines an effective sample-complexity scaling to describe these shell transitions.Source: §D.3 · p. 44

Slow decay target ω < 1

  • For slow-decay targets, unnormalized energy per shell grows without bound, so the shell index must scale with d to keep total energy finite.The analysis approximates the shell sum by a Riemann integral.Source: Slow decay target ω < 1 · p. 44
  • For m above the scaling threshold, the leading shell contribution is identified with the shell’s spectral energy.The passage states U(m) = E(m) in the indicated regime.Source: Slow decay target ω < 1 · p. 45
  • For κ < m ≤ κ/(1−α), the shell contribution saturates to its full energy because the integral’s leading divergence comes from the upper boundary.The lower-boundary contribution is asymptotically sub-leading in this intermediate regime.Source: Slow decay target ω < 1 · p. 46

E Strongly anisotropic regime

  • In the strongly anisotropic regime α > 1, the eigenvalues are rewritten with a dimension-independent multiplicative constant, so subsequent formulas hold up to an overall constant.The section introduces this eigenvalue representation to establish formulas for the α > 1 regime.Source: E Strongly anisotropic regime · p. 47

E.1 Kernel State Equation and Variance … E.2.1 Ridgeless limit

The analysis separates shell-integral asymptotics by the relation between shell order and sample-complexity exponent, then derives variance behavior across regularization regimes and relates bias to the target’s unlearned energy and decay rate.

E.1 Kernel State Equation and Variance

  • For m > κ, the shell integral has an Irwin–Hall-governed poly-logarithmic divergence, while at m = κ it scales as ∆^(κ−1)/(κ−1)!, with ∆ ≃ κ log(f̂(d)).The critical case diverges more slowly when f̂(d) → ∞.Source: §E.1 · pp. 47, 49

E.1.1 Ridgeless variance

  • The ridgeless scaling introduces a super-poly-logarithmic f(d) because, without it, generic ψ(m) terms would diverge logarithmically and the sum over shells would need to balance those logarithmic powers.Source: §E.1.1 · p. 50
  • As α → 1+ in the ridgeless limit, the variance vanishes identically for sufficiently large n.Source: §E.1.1 · p. 51

E.2 Bias

  • The bias is B = U(1−τ): τ = 1−1/α in the ridgeless limit, whereas τ → 0 under regularization λ = Θ_d(1).Source: §E.2 · p. 51

E.2.1 Ridgeless limit

  • The fast decay regime (ω > 1/2) is further partitioned into two distinct subcategories, while the slow decay regime is characterized by ω < 1/2.Source: §E.2.1 · p. 52

Ultra-fast decay ω > α + 1 · Slow decay ω < 1

In ultra-fast decay, every polynomial feature is learned and deep shells govern the vanishing bias; in slow decay, learning depends on whether a shell lies below, above, or at the critical degree m=κ.

Ultra-fast decay ω > α + 1

  • In ultra-fast decay, the kernel learns every polynomial feature, and deep shells dictate the rate at which unlearned energy and macroscopic bias vanish.The resonant shell m=κ shares the deep-shell polynomial decay rate.Source: Ultra-fast decay ω > α + 1 · pp. 52, 55
  • The resonant shell m=κ decays at the same polynomial rate as shells m>κ, avoiding the log(d)^(κ−1) explosion.This behavior is described for the ultra-fast decay regime.Source: Ultra-fast decay ω > α + 1 · p. 55

Slow decay ω < 1

  • For slow decay ω<1/2, subcritical shells m<κ are perfectly resolved and contribute vanishing unlearned energy.The target coefficients scale with d to keep the target’s total energy finite in this regime.Source: Slow decay ω < 1 · p. 55
  • For slow decay, supercritical shells m>κ retain their full spectral energy: U(m)→E(m).Source: Slow decay ω < 1 · p. 56
  • In slow decay, the resonant shell m=κ remains completely unlearned, retaining its full target energy because of the sub-polynomial divergence of f(d).Without f(d), the resonant shell would behave differently.Source: Slow decay ω < 1 · p. 56
  • The slow-decay analysis combines the shellwise results into a total unlearned-energy limit and summarizes the corresponding unregularized bias.Source: Slow decay ω < 1 · pp. 56–57

E.2.2 Regularized regime · F Single-Index Models

The regularized analysis distinguishes learned subcritical shells from resonant and deep-shell behavior, while the single-index analysis uses Hermite expansions and covariance-adjusted weights to characterize targets.

E.2.2 Regularized regime

  • Macroscopic regularization λ = Θd(1) imposes a strict resolvent scaling, and the shell-energy analysis adapts ridgeless computations using updated integral boundaries.Source: §E.2.2 · p. 57
  • Subcritical shells m < κeff are perfectly learned under macroscopic ridge, with unlearned energy U(m) → 0.Source: §E.2.2 · p. 57
  • Super-critical shells m > κeff map to the ridgeless m > κ regime, with strong polynomial decay replacing the ridgeless sub-polynomial regularizer.Source: §E.2.2 · p. 58
  • At resonant m = κeff, the upper integration bound remains finite; for ω > 1/2, the unlearned energy vanishes with the super-critical power-law scaling.Source: §E.2.2 · p. 58
  • For ω < 1/2 and integer κeff, the critical-shell core integral evaluates to 1/(κeff−1)!, yielding an exact form for its unlearned energy.Source: §E.2.2 · pp. 58–59

F Single-Index Models

  • The single-index analysis expands the L2 link function in one-dimensional Hermite polynomials and transforms the index weights to w = Σ^1/2v, with ||w||2 = 1.Source: F Single-Index Models · p. 60
  • For sufficiently dense functions, the resulting single-index analysis asymptotically neglects diagonal terms while extracting the target parameters.Source: F Single-Index Models · p. 61

G Hermite Ansatz Conjecture · H Technical and Auxilliary Lemmas

Numerical experiments support the Hermite Ansatz for cumulative target energies and show that it predicts single-index-model bias across weak and strong anisotropy. The accompanying appendix provides technical and auxiliary lemmas, including a Gaussian Poincaré inequality.

G Hermite Ansatz Conjecture

  • The cumulative-energy comparison uses exact finite-d Hermite coefficients for single-index targets and an asymptotic kernel spectrum that closely approximates the finite-d spectrum.The target coefficients are sorted by kernel eigenvalues, while kernel-eigenbasis coefficients and spectrum are estimated by empirical diagonalization.Source: G Hermite Ansatz Conjecture · pp. 61–62
  • The empirical kernel spectrum and Hermite-basis cumulative target energies closely match their kernel-eigenbasis counterparts, supporting the Hermite Ansatz across anisotropy regimes.Figure 13 compares the empirical and asymptotic spectrum and the cumulative curves for different α values.Source: G Hermite Ansatz Conjecture · p. 62, Figure 13 · p. 63
  • For sparse single-index targets, empirical bias matches the asymptotic predictions: it tends to a nonzero value at α = 0.35 and to zero at α = 1.5.The strongly anisotropic zero limit follows the predicted scaling laws, which depend on regularization strength.Source: G Hermite Ansatz Conjecture · p. 62, Figure 14 · p. 64
  • For weak anisotropy, power-law single-index bias follows the predicted feature-learning regimes, including convergence to zero when the degree-three link is fully learnable.At α = 0.8, the predicted cutoff is the fifth-order feature; the simulations use a link with no fifth-order coefficient.Source: G Hermite Ansatz Conjecture · p. 62, Figure 15 · p. 64
  • For strong anisotropy and γ > 0, the experiments support bias convergence to zero under both predicted scaling cases, distinguished by ω relative to α + 1/2 and regularization.The cases compare ω above versus below α + 1/2, with predictions for λ → 0 and λ = 1.Source: G Hermite Ansatz Conjecture · p. 62, Figure 16 · p. 65

H Technical and Auxilliary Lemmas

  • The technical appendix includes auxiliary lemmas H.1–H.2 and states a Gaussian Poincaré inequality for Gaussian inputs and differentiable functions in H.3.The supplied passages do not include the bounds stated in H.1–H.2 or the inequality’s formula.Source: H Technical and Auxilliary Lemmas · pp. 65–66

Related papers

Loading 2608.28564v1…