PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy

Lorenzo Rizzi, Arie Wortsman Zurich, Bruno Loureiro

arXiv:2608.28564v1stat.MLcs.LG

TL;DR

The paper asks how anisotropic data geometry changes kernel spectra and KRR generalization beyond assumed capacity conditions. It derives sharp spectral and risk asymptotics for polynomial kernels in the polynomial high-dimensional regime, finding distinct weak- and strong-anisotropy learning curves, and studies single-index targets as a case study.

  • Problem

    Existing kernel scaling analyses often assume feature spectra or derive them only in linear models, leaving the effect of data geometry on learning largely unresolved.

  • Method

    The paper analyzes polynomial inner-product KRR under power-law anisotropic Gaussian inputs using asymptotically exact kernel spectra and deterministic equivalents for bias and variance.

  • Results

    Weak anisotropy preserves integer variance peaks but permits fractional bias transitions, whereas strong anisotropy makes variance sample-size independent and gives decay-governed bias transitions.

  • Takeaways & Limitations

    Input anisotropy reshapes both kernel features and learning curves, with principal-direction alignment producing distinct single-index learning behaviors.

  • Takeaways & Limitations

    The target-decay ansatz is not straightforward to map onto an exact univariate source condition because sorting multi-index products imposes nontrivial combinatorial constraints.

Abstract

from arXiv · show

We study kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent $α\geq 0$ for polynomial inner-product kernels. We derive asymptotically sharp expressions for the kernel spectrum and the generalization error in the polynomial high-dimensional regime $n=Θ(d^κ)$, revealing how anisotropy reshapes the learning curves. For weak anisotropy ($0<α<1$), the problem remains effectively high-dimensional and retains some features of the isotropic case, while departing from it in others: the variance still peaks at integer sample complexities $κ\in\mathbb{N}$, but these peaks are progressively damped as $α$ grows; meanwhile, for targets strongly aligned with the data's principal directions, the bias drops at fractional sample complexities, decoupling the bias transitions from the interpolation peaks. For strong anisotropy ($α> 1$), the effective dimension of the problem is constant, and the variance stops depending on sample size altogether, plateauing under ridgeless interpolation or vanishing at an explicit rate under fixed ridge penalty. The bias undergoes a sharp transition governed by the target's decay rate: below a threshold, learning is abrupt rather than gradual; above it, the bias decays as a power law that recovers the classical source and capacity rates. We finally specialize these results to single-index targets, showing how the alignment of the index with the data's principal directions determines the effect of anisotropy on learning. Together, our results clarify how the input geometry shapes the kernel features and fundamentally impacts its generalization properties.

1 Introduction & motivation

This work addresses how data geometry shapes kernel features and generalization, extending kernel scaling analysis beyond assumed spectra and linear models. It characterizes anisotropy-dependent learning behavior, including fractional bias transitions, damped variance peaks, and finite-dimensional strong-anisotropy regimes.

  • Motivation: Kernel scaling analyses typically assume capacity conditions or derive spectra only for linear models, leaving geometry-dependent learning consequences largely open.The paper targets this gap by studying kernel ridge regression under anisotropic inputs.
  • Contributions: Polynomial inner-product kernels under power-law anisotropy admit asymptotically exact spectra and closed-form deterministic equivalents for bias and variance.These results apply in the polynomial high-dimensional regime n = Θ(d^κ).
  • Weak anisotropy: For 0 < α < 1, targets aligned with principal directions can learn degree-m components at n = Θ(d^{m(1−α)}), yielding fractional thresholds κ = m(1−α).These thresholds occur earlier than the isotropic n = Θ(d^m) transitions described in the paper.
  • Weak anisotropy: For 0 < α < 1, variance retains interpolation peaks at integer κ, while increasing anisotropy progressively suppresses their magnitude.Thus bias transitions and variance peaks need not coincide.
  • Strong anisotropy and single-index models: For strong anisotropy, variance becomes sample-size independent, while bias transitions depend on target decay and can recover source and capacity power-law rates.The paper also studies how single-index alignment with principal directions produces different feature-space learning behaviors.

2 Setting & Background

The paper formulates kernel ridge regression for regression data with anisotropic Gaussian inputs and polynomial inner-product kernels, then asks how the input distribution changes the RKHS geometry and excess risk. It studies the polynomial high-dimensional limit under power-law covariance decay and polynomial targets with label noise.

  • KRR setting: Kernel ridge regression minimizes a regularized empirical risk over the RKHS associated with a symmetric positive definite kernel.The estimator is evaluated through excess risk relative to the Bayes predictor.
  • Data geometry and RKHS: The input distribution determines both the risk geometry and the kernel operator’s eigenfunctions and eigenvalues, thereby implicitly determining the RKHS geometry.This motivates studying the effect of pX on KRR generalization.
  • Research question: Explicit geometry-to-risk results are scarce because diagonalizing the kernel operator is generally difficult and prior analyses largely concern isotropic distributions.This paper studies the corresponding anisotropic setting.
  • Data model: The analysis uses anisotropic Gaussian inputs pX = N(0, Σ), with covariance eigenvalues decaying as a power law and normalized by TrΣ = 1.The exponent α interpolates between isotropic and effectively low-dimensional data.
  • Asymptotic regime: The high-dimensional regime takes d → ∞ with fixed ψ := n/d^κ = Θ_d(1), allowing both integer and fractional sample-complexity exponents κ.The primary focus is on integer κ, with fractional cases explored when relevant.
  • Kernel and target assumptions: The kernel is a finite-degree polynomial inner-product kernel, and the target is a polynomial of degree D with independent mean-zero label noise.A degree-D kernel fits at most degree-D target components; orthogonal components act as additional irreducible noise.

3 High-dimensional limit of the kernel spectrum

The paper sharpens prior spectral bounds for polynomial inner-product kernels under anisotropic Gaussian data and relates the resulting spectrum to an effective dimension controlled by covariance normalization. The effective dimension diverges for weak anisotropy but saturates for strong anisotropy, producing qualitatively different spectral structures.

  • Spectral structure: Polynomial-kernel features correspond to multi-index monomials, with eigenvalues determined by products of covariance eigenvalues across the index coordinates.This explains why anisotropic covariance directly reshapes the kernel spectrum.
  • Sharp asymptotics: The paper makes prior upper and lower kernel-spectrum bounds asymptotically tight when the covariance participation ratio diverges, equivalently Tr(Σ^2) = o_d(1) under TrΣ = 1.This yields an asymptotically exact spectrum in the considered high-dimensional setting.
  • Effective dimension: For 0 ≤ α < 1, the effective-dimension normalization grows as Θ_d(d^{1−α}); for α = 1 it grows as Θ_d(log d); for α > 1 it approaches ζ(α).The normalization is the quantity controlling the effective dimensionality of the problem.
  • Anisotropy regimes: When 0 < α < 1, the effective dimension diverges sublinearly, whereas for α > 1 it saturates to a finite constant and the problem becomes effectively finite dimensional.These are the paper’s weak- and strong-anisotropy regimes.
  • Isotropic comparison: In the isotropic case, features within each degree shell are degenerate, with Θ(d^m) eigenvalues of magnitude Θ_d(d^{-m}) at frequency m.Anisotropy removes this shell degeneracy and produces a more structured spectrum.
  • Spectral consequences: In weak anisotropy, the spectrum develops gaps and disconnected sectors, so a simple capacity condition cannot accurately characterize generalization.This differs from the strong-anisotropy capacity behavior inherited from the input exponent.

4 Generalization properties

The paper gives asymptotically sharp deterministic equivalents for KRR generalization under power-law anisotropy, showing that anisotropy separates variance peaks from bias transitions and changes the strong-anisotropy limit.

  • The anisotropic kernel spectrum supports asymptotically exact deterministic equivalents for KRR bias and variance in the polynomial high-dimensional regime.These results establish the conditions needed to analyze excess risk under anisotropic power-law data.
  • 4.1.1 Variance: For weak anisotropy, variance peaks at integer sample complexities and vanishes at fractional complexities, producing a multiple-descent pattern.At fixed integer scaling, the variance peaks at finite sample complexity and vanishes at both very small and very large prefactors.
  • 4.1.1 Variance: As α increases toward 1, variance peaks are progressively suppressed and multiple descent is gradually reduced.The suppression follows the decreasing contribution of the second degree of freedom as α approaches 1 from below.
  • 4.2.1 Variance: For strong anisotropy, the variance becomes independent of sample size without regularization and vanishes at an explicit rate under a fixed positive ridge penalty.The effective dimension is finite in this regime, yielding a constant ridgeless variance and sample-size-decoupled regularized behavior.
  • 4.1.2 Bias: For sufficiently aligned targets, weak anisotropy lets degree-m features become learnable at fractional scaling κ = m(1−α), separating bias drops from integer variance peaks.For rapidly decaying target coefficients, the kernel learns features up to order floor(κ/(1−α)); for slowly decaying targets, learning remains constrained by the active degree.
  • 4.2.2 Bias: In strong anisotropy, slowly decaying targets leave a finite unlearnable portion and require n ∼ d^(κ+ε) to learn the κ-th degree, whereas faster decay yields smooth power-law bias decay.The transition is governed by the target decay rate and recovers classical source and capacity rates above the threshold.

5 Case study: single-index models

The single-index case maps input-space targets onto the paper’s anisotropic kernel framework through Hermite expansions and an effective decay rate determined by index alignment and target decay. This mapping yields distinct learning behaviors for localized, dense, and power-law targets.

  • Single-index setup: A single-index model depends on x only through the scalar projection v⊤x, with v normalized by E[(v⊤x)^2] = 1.The scalar link function is denoted g.
  • Hermite ansatz: The Hermite ansatz approximates cumulative target energy in the kernel eigenbasis using Hermite coefficients, with numerical evidence supporting this correspondence.This lets the deterministic equivalents be evaluated using Hermite coefficients with asymptotically negligible relative error.
  • Localized targets: A localized index aligned with principal direction j retains only the fully diagonal degree-m feature, with shell energy E(m) = g_m^2.Learning behavior must then distinguish leading directions j = O(1) from trailing directions j = O(d).
  • Power-law targets: Power-law single-index targets with decay γ have effective decay rate ω = α + γ, combining intrinsic target decay with data anisotropy.Their tensor coefficients match the framework θ_i1,...,im ∼ (i1...im)^−ω.
  • Learning regimes: Dense and localized single-index configurations map into the unified power-law framework through different effective decay limits, linking index alignment to the resulting learning curve.The sparse leading-direction case corresponds formally to ω → ∞, whereas the delocalized case has ω = α/2.
  • Power-law targets: For strongly anisotropic data α > 1 and γ ≥ 0, the effective rate satisfies ω > 1/2, so the asymptotic generalization bias vanishes.This places such power-law targets in the fast-learning regime.

6 Conclusion

The paper derives asymptotically sharp kernel spectra and deterministic equivalents for KRR under power-law anisotropic Gaussian data. It finds qualitatively different variance and bias regimes below and above α = 1.

  • Main results: The analysis obtains an asymptotically exact high-dimensional spectrum for polynomial inner-product kernels and closed-form deterministic equivalents for KRR bias and variance.These results cover ridgeless and macroscopic-ridge settings.
  • Variance: For weak anisotropy α < 1, variance depends on sample size and exhibits interpolation peaks at integer polynomial transitions n ∼ d^ℓ, while increasing α dampens them.This is the multiple-descent regime.
  • Variance: For strong anisotropy α > 1, variance becomes sample-size independent, saturating positively without regularization or vanishing under an order-one ridge penalty.The two regimes meet continuously at α = 1, where V → 0.
  • Bias: When α < 1 and ω < 1/2, features above the current polynomial shell remain unlearned, whereas ω > 1/2 unlocks higher-order features up to κ_eff = ⌊κ/(1−α)⌋.The learned fraction of the current shell also depends on ω.
  • Bias: The paper summarizes the bias regimes through a phase diagram over anisotropy α and target decay ω, with critical boundaries marking qualitative changes.The diagram is formulated in the ridgeless limit.

B.1 The Weakly Anisotropic Setting

The weakly anisotropic analysis establishes deterministic equivalents in the polynomial regime n = Θ(d^κ) by verifying concentration, effective-rank, and spectral conditions. It also identifies distinct behavior at integer and non-integer κ.

  • Proof strategy: For 0 < α < 1 and n = ψd^κ, the proof chooses a degree threshold m(n) and verifies the assumptions needed for deterministic equivalents.The case α = 0 follows by the same arguments.
  • Concentration: The proof establishes concentration of higher-order diagonal features and combines it with the effective-rank estimates to satisfy the theorem’s spectral assumptions.This completes the weak-anisotropy justification for applying the deterministic equivalents.
  • Effective-rank control: The relative approximation bound satisfies ν_λ,m = 1 for non-integer κ and ν_λ,m = O(poly log(n)) when κ is an integer.Integer transitions therefore require a weaker logarithmic control.
  • Effective-rank control: At non-integer κ, the selected threshold lies below the next shell, giving λ_⌊ηn⌋,m = 0 and ν_λ,m(n) = 1.At integer κ, the threshold reaches the κ-th shell instead.
  • Concentration: Polynomial kernel eigenfunctions satisfy the required hypercontractivity bounds, which supports both lower- and higher-order concentration conditions.The argument uses the fact that these eigenfunctions are Gaussian polynomials.
  • Strong-anisotropy contrast: The strong-anisotropy extension requires ridge regularization λ > c for a constant c independent of n and d.This requirement follows because the higher-order trace vanishes while the regularized tail rank must remain controlled.

D.1 Kernel State Equation

The kernel state equation is derived by decomposing shell contributions under polynomial scaling and evaluating their asymptotic regimes. A resonant shell at m=κ contributes a finite macroscopic term, while higher-order tails generate effective ridge regularization.

  • Regime decomposition: The analysis separates shell contributions according to their degree m relative to κ and κ/(1−α).These regimes determine the asymptotic behavior of the integration boundaries and shell integrals.
  • Integral reduction: The Irwin-Hall trick converts the m-dimensional integral into a one-dimensional integral over the sum of logarithmic coordinates.The substitution y_i=e^{u_i} makes the variables uniformly distributed on an interval of logarithmic width log(d).
  • Resonant shell: ψ(κ) remains finite and macroscopic when m=κ, with the resonant contribution explicitly given by 1/[κ!(κ−1)!].The resulting quantity satisfies the state-equation constraint ψ(m)≤ψ=Θ(d^0).
  • Higher-order modes: For m>κ/(1−α), fractional powers of d cancel, producing a dimension-free self-regularizing contribution from higher-degree modes.The approximation is valid because both integration boundaries diverge and the polynomial term dominates the denominator.
  • State-equation synthesis: The assembled state equation identifies higher-order tails as an effective ridge penalty, λ_eff=λ+P_{m>κ}h_m, together with the resonant ψ(κ) term.This equation implicitly determines the macroscopic parameter ξ.

D.2 Variance

The variance is obtained by summing shell contributions, but asymptotically only the resonant shell m=κ survives when κ is an integer. Away from integer κ, adjacent shells determine a decaying variance.

  • Sub-critical shells: For m<κ, the variance shell scales as Θ(d^{m−κ}) and vanishes in the thermodynamic limit.These sub-critical contributions are strictly negligible because m−κ<0.
  • Super-critical shells: For m>κ/(1−α), the shell contribution vanishes, with the decay mechanism depending on whether α is below, equal to, or above 1/2.The upper boundary dominates for α<1/2, logarithmic factors appear at α=1/2, and the lower boundary dominates for α>1/2.
  • Intermediate shells: In the intermediate band κ<m≤κ/(1−α), τ(m)→0, bounded by O(d^{κ−m}) or O(d^{(κ−m)(1/α−1)}).For α=1/2, the decay includes a logarithmic factor that remains dominated by polynomial suppression.
  • Resonance: Only the shell m=κ yields a macroscopic nonzero variance contribution in the integer-κ case.The variance expression therefore reduces asymptotically to the resonant term.
  • Finite-size scaling: Empirical variance curves progressively collapse onto the asymptotic theoretical prediction as the system dimension increases.The figure varies d while fixing κ=1, α=0.3, λ=0, and ψ=0.8.

D.2.1 Fractional sample complexity κ

When κ is non-integer, the absent resonant shell removes the macroscopic variance peak. The asymptotic variance instead decays at a rate set by the slower-decaying adjacent integer shell.

  • Fractional sample complexity: If κ∉N, the threshold term ψ(κ) vanishes because there is no shell exactly at m=κ.The kernel state equation consequently simplifies, and no shell contributes a macroscopic variance term.
  • Adjacent-shell control: For non-integer κ, the variance decay is controlled by τ(⌊κ⌋) or τ(⌈κ⌉), the slower-decaying neighboring integer shell.The effective decay exponent is the minimum absolute decay rate among these two shells.
  • Anisotropy-dependent rates: For α<1/2, the upper adjacent shell scales as Θ(d^{κ−⌈κ⌉}), whereas for α>1/2 it scales as Θ(d^{(⌈κ⌉−κ)(1−1/α)}).Thus anisotropy changes the fractional-κ variance decay through localization of the integral.

D.3 Bias

The bias is expressed as target-weighted unlearned energy combined with a variance-like correction, and its asymptotics depend on target decay and shell regime. The analysis distinguishes fast-decaying targets from slower-decaying ones.

  • Bias decomposition: The bias decomposes as B=U(1−τ), where U=ν²⟨θ,(Λ+νI)⁻²θ⟩ captures target-dependent unlearned energy.The vector θ contains target coefficients in the kernel eigenbasis, while τ does not depend on the specific target.
  • Target model: The target coefficients are modeled through an anisotropy parameter ω and shell-dependent coefficients θ_m, with shell energy defined by squared weights.The analysis assumes dense, symmetric target coefficients in the thermodynamic limit.
  • Fast-decay targets: For ω>1/2, the target spectral energy converges and the prefactor θ_m remains intensive, Θ(d^0).The convergent source-weight series permits dominated-convergence arguments in the bias asymptotics.
  • Effective threshold: Defining κ_eff=κ/(1−α), the bias asymptotics are organized by whether shell degrees lie below, within, or above this effective threshold.Away from transition boundaries, termwise limits determine whether denominators approach one or diverge.
  • Shell asymptotics: For m>κ/(1−α), the leading-order target contribution can reduce to the shell spectral energy, U(m)=E(m).In the intermediate regime, the lower integration boundary is asymptotically sub-leading because the divergence is driven by the upper boundary.

E.1 Kernel State Equation and Variance

For α > 1, shell-integral asymptotics determine the kernel state equation and variance: unregularized variance remains finite, whereas fixed ridge suppresses kernel contributions and makes variance vanish.

  • For α > 1, super-critical shells m > κ produce poly-logarithmic integral divergence governed by the Irwin-Hall density.
  • At the resonant shell m = κ, the integral grows as Δ^(κ−1)/(κ−1)!, rather than following the generic constant-boundary behavior.
  • The resulting variance is given by a final asymptotic formula, and it vanishes identically for sufficiently large n as α approaches 1 from above.
  • The ridgeless state equation yields τ = 1 − 1/α, while fixed ridge makes τ tend to zero.

Ultra-fast decay ω > α + 1

In the ultra-fast decay regime, sub-critical target shells are learned, while deep shells retain their full spectral energy and determine the asymptotic bias decay.

  • For sufficiently fast target decay, the kernel learns every polynomial feature and the final bias follows the corresponding power-law scaling.
  • The total unlearned energy and macroscopic bias vanish, with the convergence rate dictated by deep shells m > κ.
  • When m < κ, the kernel perfectly resolves the sub-critical shells because their unlearned-energy exponent is strictly negative.
  • When m > κ, the unlearned energy saturates to the full spectral energy E(m), so deep shells remain unlearned.
  • The resonant shell m = κ also remains completely unlearned, retaining its full initial target energy because of the diverging sub-polynomial factor f(d).

E.2.2 Regularized regime

Under fixed ridge regularization, the effective threshold becomes κ_eff, with sub-critical shells learned, deep shells behaving as in the ridgeless case, and the resonant shell changing most substantially.

  • With λ = Θ(1), the resolvent scaling changes the effective threshold from κ to κ_eff, determined by the regularized integration boundaries.
  • Shells with m < κ_eff have vanishing integrals and are perfectly learned under macroscopic ridge regularization.
  • Deep shells with m > κ_eff map directly to the unregularized super-critical regime because strong polynomial decay dominates the asymptotics.
  • The resonant shell m = κ_eff is the main departure from ridgeless behavior because the absence of f(d) stabilizes its upper integration boundary.
  • For fast-decaying targets, the resonant shell joins the power-law scaling of super-critical modes and its unlearned energy vanishes accordingly.

G Hermite Ansatz Conjecture

The numerical tests compare Hermite- and kernel-basis representations and evaluate single-index-model bias across anisotropy regimes, supporting the proposed asymptotic descriptions.

  • The cumulative target energies computed in the Hermite and kernel eigenbases closely agree, while the theoretical spectrum closely matches empirical diagonalization.
  • For sparse single-index models, weak anisotropy yields a non-zero limiting bias, whereas strong anisotropy yields bias tending to zero under the predicted regularization-dependent scaling.
  • In weak anisotropy, power-law single-index targets with effective ω < 0.5 are learned through the κ-th feature, while higher effective ω changes the learning regime.
  • When the link expansion is truncated at degree 3, the bias drops to zero because the kernel learns the entire target function.
  • In strong anisotropy, simulations test both sides of the threshold ω = α + 1/2 and compare ridgeless and fixed-ridge bias scalings, supporting the conjectured predictions.
Loading 2608.28564v1…