PaperlayerView exact revision on arXiv

Source-linked AI summary

The Geometry of Statistical Feature Learning in Mean-Field Langevin Dynamics

Zong Shang, Tomoya Wakayama, Guillaume Lecué, Taiji Suzuki

arXiv:2606.31429v2math.STcs.LGstat.MLstat.TH

TL;DR

Statistical accounts of feature learning must explain both which features training learns and how they improve estimation. The paper gives a base–fiber formulation and analyzes spherical MFLD, finding low-temperature concentration and prediction rates for Gaussian index models.

  • Problem

    Prior work largely examined whether hidden-layer features change during training, while a statistical account asks what is learned and why it improves estimation.

    Source: §1 · p. 2, §2 · p. 7

  • Method

    The paper defines feature learning through a base–fiber organization and analyzes spherical MFLD as a Wasserstein gradient flow of a negative-entropy-regularized empirical risk.

    Source: §1 · p. 3, §2.3 · p. 12

  • Results

    Low-temperature measures concentrate near hidden indices or on parity-selected single-index spaces, while well-specified models attain d/N and Md/N prediction rates, up to logarithmic factors.

    Source: §4 · p. 25

  • Takeaways & Limitations

    The learned feature space aligns with the regression signal, so the latent low-dimensional structure need not be supplied to the statistician in advance.

    Source: §4 · p. 25

    Parameter-recovery guarantees deteriorate when the number of indices is very large or two indices have very small angular separation.

    Source: §3.4.2 · p. 22

Abstract

from arXiv · show

We introduce a geometric formulation of statistical feature learning for supervised regression. Feature learning is defined through a base--fiber decomposition: the base is the feature-side geometry produced by training, and the fiber is the learned feature space where estimation is performed. We prove this property for spherical mean-field Langevin dynamics, viewed as the Wasserstein gradient flow of a negative entropy-regularized empirical risk. In Gaussian multi-index models, the low-temperature stationary distribution concentrates near the hidden indices, forms a multi-spike structure, and yields parameter recovery with high probability, even though negative entropy regularization penalizes concentration. This concentration has a sharp transition at temperature $\lambda\asymp 1$. In Gaussian single-index models, the stationary measure satisfies a concentration property, with parity determining whether it lives on $S_2^{d-1}$ or $\mathbb{RP}^{d-1}$. The induced learned feature space aligns the regression signal and yields rates $d/N$ and $Md/N$, up to logarithmic factors.

1 Introduction

The introduction asks what features training learns and why those features improve estimation, then frames feature learning geometrically and studies it through mean-field Langevin dynamics.

1.1 Mean-field Shallow Neural Networks

  • A mean-field shallow network separates hidden weights from output weights, yielding a base of hidden-layer marginals and an associated fiber of functions.Source: §1.1 · pp. 3–4

1.2 Mean-field Langevin Dynamics

1 Introduction

  • Feature learning is formalized as a base–fiber structure: training selects a feature-side base point, and estimation occurs in its induced learned feature space.Source: §1 · p. 3, §1.3 · p. 6

1.3 Our contributions

  • In Gaussian multi-index models, low-temperature stationary measures form multi-spike structures near hidden indices, yielding parameter recovery despite entropy penalizing concentration.Source: §1.3 · p. 6
  • Parity determines whether single-index concentration occurs on the sphere or real projective space; learned-feature alignment yields d/N and Md/N prediction rates up to logarithmic factors.Source: §1.3 · p. 6
  • MFLD also has a self-regularization property, while LASSO exhibits feature learning under a support recovery condition.Source: §1.3 · p. 6

2 The Base–Fiber Geometry of Feature Learning

The section frames feature learning in MFLD as a base–fiber decomposition: training evolves a data-dependent feature space in which the regression signal aligns with exploitable directions. It describes how prediction is then performed by a latent coefficient estimator within that learned space, connecting feature selection to interpretability, generalization, and ridge-like self-regularization.

2 The Base–Fiber Geometry of Feature Learning

  • Feature engineering prespecifies a feature space but may leave the regression signal poorly aligned with exploitable directions, motivating a definition of feature learning through in-fiber alignment.The section frames its definition as a first attempt to quantify whether training-produced features and low-dimensional target representations reduce estimation error.Source: §2 · pp. 7–8

2.1 The Alignment Property in a Fixed Fiber

  • In a fixed RKHS fiber, alignment means the estimator can exploit the target’s concentration in principal covariance directions for faster estimation, but cannot improve an unfavorable alignment.Ridge regression, gradient flow, gradient descent, and principal components regression are among the estimators identified as satisfying alignment through Feature Space Decomposition.Source: §2.1 · pp. 8–9

2.2 A geometric and statistically oriented formulation of Feature Learning

  • Feature learning combines evolving features, a data-dependent representation with an effectively k-dimensional target component, and a head estimator that exploits their alignment for an O(k/N) prediction rate.The definition requires k=o(N); the resulting rate is described as oracle-like, as in a k-sparse signal regime in the learned feature space.Source: §2.2 · pp. 10–11
  • The paper distinguishes problem-specific adaptivity from minimax adaptivity: MFLD is adaptive for well-specified Gaussian single- and multi-index models, but counterexamples show failure for misspecified single-index links.The stated adaptivity concerns the directions of the low-dimensional structure for a specific regression problem, not uniform minimax performance over a class.Source: §2.2 · p. 11

2.3 The Base–Fiber structure of MFLD

  • MFLD’s hidden-layer marginal evolves the initial random-features space into a learned RKHS, while the final head estimates within the fiber over that learned base point.The nonlinear Fokker–Planck flow induces the path H(0) → H(t) → Hfeat; the latent estimator predicts from the learned feature map rather than directly from the input.Source: §2.3 · pp. 12–13
  • The learned feature space Hfeat is a data-dependent RKHS selected by the trained measure, with predictors represented by coefficients in L2 of that measure.The factorization gives the measure a feature-side role and the coefficient function a role in forming the predictor.Source: §2.3 · p. 13

2.4 Self-Regularization and Alignment in the Learned Feature Space

  • The learned features are the top k eigenvectors of Hfeat, though identifying their physically meaningful significance requires case-specific analysis.The feature space is used to analyze interpretability and generalization.Source: §2.4 · pp. 13–14
  • The estimation error decomposes into latent-estimator error and approximation error to Hfeat, making feature alignment and the estimator’s ability to exploit it central.The paper notes that the created k-dimensional representation may support a final rate, but the supplied passage truncates before specifying it.Source: §2.4 · p. 14
  • For 0 < A < ∞ and λ > 0, the latent estimator admits a data-dependent convex regularizer that is 1/A^2-strongly convex and yields ridge-like self-regularization.MFLD’s latent estimator is a RERM on Hfeat, implicitly favoring signals supported mostly on its top eigenvectors; its alignment property is anticipated but not formally proved.Source: §2.4 · pp. 14–15
  • Proposition 3 justifies ridge-regression modeling of the output layer: MFLD-trained networks produce this regularization implicitly, without requiring it to be imposed.Prior feature-learning studies often imposed output-layer ridge regression through two-stage or two-timescale training [BBSS22] [TS24].Source: §2.4 · p. 15

3 Low-Temperature Geometry and Estimation for Spherical MFLD in Gaussian Index Models

This section studies how low-temperature geometry in spherical MFLD on Gaussian index models concentrates learned features around hidden directions and compresses the learned function class to a low-dimensional space aligned with the regression signal. It connects this dimension reduction to estimation rates and uses signal approximation, alignment, and feature-kernel evolution to establish feature learning and non-laziness.

3.1 Gaussian index models as a lens on Statistical Feature Learning

  • Rather than supplying index structure in advance, the section asks whether generic MFLD discovers it from data; the analyzed dynamics trains only the hidden layer and has a data-random stationary measure.Source: §3.1 · pp. 15, 17

3.2 Low-Temperature Multi-Spike Structure on the Base

  • For λ=o(1), the multi-index stationary measure forms spikes around the true parameters despite entropy penalizing concentration, with a sharp transition at λ ≍ 1.The dynamics achieves parameter recovery without an explicit sparsity-inducing regularizer.Source: §3.2 · pp. 17–18
  • In single-index models, odd information index concentrates near w⋆ on the sphere, while even index concentrates near [w⋆] on projective space, corresponding to antipodal spikes.The probability outside the relevant radius-ρ neighborhood is bounded by ε/ρ².Source: §3.2 · pp. 18–19
  • The low-temperature geometry follows from Frobenius convergence of the first M stationary moment tensors in multi-index models and the IE(σ)-th tensor in single-index models.Source: §3.2 · pp. 19–20
  • The results concern the long-time limit, at t = Omega(exp(N)), rather than finite-time training; how finite-time moment errors affect the geometry remains open.Source: §3.2 · p. 20

3.3 Dimension Reduction, Top-k Signal Approximation - Theoretical Framework Based on the Feature-Learning Property

  • The low-temperature geometry compresses the infinite-dimensional nonlinear model to a first-order space of dimension at most M(d + 1), with negligible L2(PX) approximation error.Source: §3.3 · p. 20
  • The learned feature subspace represents the main regression signal with k = M(d + 1) directions, leaving negligible weighted tail energy outside the top k.Source: §3.3 · p. 20
  • The estimation argument reduces the model around f⋆ to a small number of leading learned directions, then applies localized uniform convergence around those directions.The feature-learning property makes the main component of f⋆ effectively approximable in the learned feature space.Source: §3.3 · p. 21

3.4.1 The d/N convergence rate in the Gaussian single-index model

  • Under the single-index assumptions, the learned low-dimensional structure yields d/N estimation rates, and MFLD attains minimax-optimal rates up to logarithmic factors for both f⋆ and w⋆.The framework localizes stochastic control around leading learned directions rather than the original infinite-dimensional model.Source: §3.4 · p. 21, §3.4.1 · p. 21, §3.3 · p. 21

3.4.2 The dM/N convergence rate in the Gaussian Multi-Index Model

  • For Gaussian multi-index prediction, the low-dimensional representation yields dM/N rates, while the general result requires separated indices and its constants can deteriorate rapidly with M or small separation.The minimax-optimal rate is achieved for λ ≲ 1/N; when λ = 0, only statistical properties are established, not algorithmic convergence.Source: §3.4 · p. 21, §3.4.2 · p. 22
  • With a frame condition making the true directions nearly orthonormal, the multi-index result gives an M-uniform prediction rate with leading term Md/N and no hidden exponential dependence on M.This refinement is presented as complementary to, not a replacement for, the general separated-index theorem.Source: §3.4.2 · p. 23

3.5 Feature-Learning Property and Non-Laziness of Spherical MFLD

  • Recovery of hidden multi-index structure separates the learned feature kernel from the rotationally invariant initialization, while the learned feature space represents the signal and satisfies alignment with k = M(d + 1).The kernel evolution establishes non-laziness for trajectories whose kernels converge to the terminal learned kernel.Source: §3.5 · pp. 24–25

4 Summary and Outlook

The paper frames feature learning as a base–fiber geometry and applies it to spherical mean-field Langevin dynamics. Its low-temperature geometry supports parameter recovery and minimax prediction rates, while the outlook identifies limits of the spherical setting.

4 Summary and Outlook

  • The base–fiber formulation treats training as producing a feature-side base that induces a learned feature space for estimation, where the signal is concentrated in a small set of leading directions.Feature learning is characterized by a data-dependent RKHS representation that a final head estimator can exploit.Source: §4 · p. 25
  • Spherical MFLD is viewed as a Wasserstein gradient flow of a negative entropy-regularized empirical risk, whose long-time limit selects the base point of the learned geometry.The selected base is the stationary hidden-layer marginal, which induces the learned fiber and feature space.Source: §4 · p. 25
  • In Gaussian multi-index models, low-temperature stationary measures form multiple spikes near hidden indices and yield parameter recovery; single-index concentration depends on link parity.Negative entropy regularization penalizes concentration, making the multi-spike structure a property of the low-temperature stationary geometry.Source: §4 · p. 25
  • For well-specified links, alignment in the learned feature space yields minimax prediction rates d/N for single-index and Md/N for multi-index models, up to logarithmic factors.The effective low-dimensional structure is formed by Wasserstein dynamics rather than supplied in advance.Source: §4 · p. 25
  • The analysis trains only the hidden layer; extending it to full MFLD would require studying dynamics over the full parameter space and whether its geometry still supports estimation.The authors highlight misspecified index models as a setting where output-layer evolution may be needed to form the appropriate feature space.Source: §4 · pp. 25–26

A Mathematical and Variational Preliminaries

  • Appendix A collects notation and develops Hermite calculus and variational identities used repeatedly in the proofs.Its notation subsection defines the projection space and other mathematical conventions used in the paper.Source: A Mathematical and Variational… · p. 26, §A.1 · p. 26

A.2 Hermite functions

The appendix develops Hermite tools for Gaussian projections and connects homogeneous Hermite subspaces to symmetric tensors. It also establishes properties of an entropy-constrained variational solution and an entropy chain rule.

A.2 Hermite functions

  • Directional degree-m Hermite projections span the homogeneous subspace C_m, and their inner products give an isometric correspondence with order-m symmetric tensors.The correspondence uses the L2(PX) inner product on C_m and the Frobenius inner product on the tensor space.Source: §A.2 · pp. 27, 29

A.3 The variational solution

  • For each u in (−A, A), the variational problem has a unique minimizing density p_t(u), and its value ψ is the convex conjugate of the log-partition function.The solution parameter t(u) is unique, and ψ has a uniform curvature lower bound on (−A, A).Source: §A.3 · pp. 29–31

A.4 Chain rule of negative Shannon entropy

  • The relative entropy for the joint law of output and hidden weights decomposes by the chain rule into marginal and conditional contributions.The decomposition uses the hidden-weight marginal and the conditional output-weight distribution.Source: §A.4 · pp. 31–32

B The Feature Learning Property of the LASSO

The appendix interprets LASSO feature learning through support selection and proves the feature-learning property under support recovery, alongside consistency and nonzero signal conditions.

B The Feature Learning Property of the LASSO

  • The paper identifies selected coordinates as learned features even though LASSO selects from a pre-existing feature set rather than constructing new features.The learned feature space is the span of the canonical basis vectors on the estimated support.Source: B The Feature Learning Property of the… · p. 32
  • Under support recovery, the LASSO’s selected-coordinate feature space represents the regression function exactly, since the approximation term vanishes.Support recovery is sufficient, though not necessary, for the factorization conditions concerning approximation in the learned feature space.Source: B The Feature Learning Property of the… · pp. 32–33
  • The feature-learning definition also captures utilization: under support recovery, the target has no tail energy in the selected feature space, and LASSO consistency supplies the estimation component.The paper distinguishes this utilization condition from feature learning in the narrower sense.Source: B The Feature Learning Property of the… · pp. 32–33
  • LASSO satisfies the paper’s feature-learning property under support recovery and consistency, with a nonzero signal required for feature evolution from the zero initialization.The learned features are the canonical coordinates indexed by the recovered support.Source: B The Feature Learning Property of the… · p. 33

C Self Regularization (implicit bias) of MFLD

At stationarity, the output layer of MFLD has a self-regularized structure. Reparameterizing by the hidden-layer marginal and conditional output-weight mean yields a strongly convex regularized estimation problem on the learned coefficient space.

C Self Regularization (implicit bias) of MFLD

  • Once the hidden-layer distribution is stationary, the output-layer estimator is a regularized empirical risk minimizer with strongly convex regularization.The regularizer is induced by the conditional output-weight distribution under the entropy-regularized formulation.Source: C Self Regularization (implicit bias)… · pp. 33, 35
  • The empirical risk depends only on the hidden-layer marginal φ and the conditional output-weight mean a, enabling the optimization to be expressed in terms of these two quantities.The resulting risk is the squared prediction loss for the mean-field model parameterized by φ and a.Source: C Self Regularization (implicit bias)… · pp. 34–35
  • Minimizing over conditional output-weight distributions reduces their entropy contribution to an integral of ψ(a(w)) over the hidden-layer marginal.The minimizing conditional distribution is the unique tilted density with prescribed mean a(w).Source: C Self Regularization (implicit bias)… · pp. 34–35
  • After fixing the learned marginal, the conditional mean estimator is a convex regularized M-estimator on L2(φ̂λ), with a 1/(2A^2)-strongly convex penalty.The regularized problem combines empirical squared loss with Ψ(a).Source: C Self Regularization (implicit bias)… · pp. 34–35

D Proof of the existence of feature-learning for spherical MFLD

The section proves that spherical mean-field Langevin dynamics has the feature-learning property in Gaussian index models: its learned measure concentrates near the true directions, and this localization yields sharp estimation rates.

D Proof of the existence of feature-learning for spherical MFLD

  • The proof establishes concentration of the learned measure near the true directions, with the resulting localization and low-dimensional structure leading to sharp estimation rates.This is the stated feature-learning property for spherical MFLD in Gaussian index models.Source: D Proof of the existence of… · p. 36

D.1 Proof of Theorem 1 via feature learning

The proof of Theorem 1 establishes a high-probability guarantee under a small-temperature condition, combining prediction-error localization, parity-dependent concentration and alignment with the hidden direction, and empirical-process controls for estimation. It also shows that the entropy-regularized empirical minimization problem attains a minimizer.

D.1 Proof of Theorem 1 via feature learning

  • For every λ ≥ 0, the entropy-regularized empirical minimization problem admits a minimizer.The objective is lower semicontinuous on a compact space, and the reference measure has finite objective value.Source: §D.1 · p. 36
  • Prediction error localizes φ near w⋆ for odd IE(σ), and near {±w⋆} for even IE(σ).The proof uses a tensor localization bound and the geometry of the odd- and even-degree cases.Source: §D.1 · pp. 37–39
  • A Taylor expansion decomposes fφ−f⋆ into a first-order component in a space of dimension at most 2d+1 and a second-order remainder in a convex hull of Q1.This decomposition supplies the finite-dimensional and remainder controls used in the empirical-process argument.Source: §D.1 · pp. 41–42
  • Combining the first-order and remainder bounds yields localized empirical-process control for prediction differences.The argument bounds the first-order component over a finite-dimensional ball and controls the remainder using Lemma 6.Source: §D.1 · p. 42
  • A localized uniform deviation bound for squared-loss differences follows from contraction and the prediction-process bound.The loss difference is expressed through prediction error and noise, then controlled using the contraction principle.Source: §D.1 · pp. 42–43
  • When λd ≤ 1, Lemma 8 bounds the approximation term Aλ by Cappψ(λd), which the proof uses to derive the stated theorem bounds.Capp depends only on (Lσ, Mσ).Source: §D.1 · pp. 44–45
  • 1 − 4e^-x is the probability guarantee in the rewritten Theorem 1 for x ≥ 1, N ≥ 2, and λd ≤ c0.The constants depend only on the listed activation and noise parameters.Source: §D.1 · pp. 44–45
  • For odd IE(σ), the theorem gives a concentration result tested against any 1-Lipschitz function on the sphere.The proof invokes Lemma 15, Proposition 4, and Theorem 1.Source: §D.1 · p. 45
  • The passage bounds the displayed neighborhoods B(w⋆;ρ) and B(−w⋆;ρ) by r*ρ², and defines BRP([w⋆];ρ) using projective distance at most ρ.The class [w⋆] is taken modulo sign.Source: §D.1 · p. 45
  • The symmetric measure on {w⋆, −w⋆} is not concentrated around w⋆ on the sphere: its test-function deviation is at least 1 with probability 1.For G(w) = ⟨w, w⋆⟩, the measure has mean zero.Source: §D.1 · p. 45

D.2 Proof of Theorem 2

The proof constructs a nonnegative polynomial that vanishes at the hidden directions, then uses tensor and Voronoi-cell arguments to relate localization to regression error.

D.2 Proof of Theorem 2

  • A Hermite–tensor projection identity and orthogonal projection relate projected regression differences to tensor-moment differences.The proof applies the identity to the signed measure difference between the learned and target measures.Source: §D.2 · p. 46
  • The constructed nonnegative polynomial vanishes exactly at the hidden directions.The proof uses this polynomial in its measure-concentration argument.Source: §D.2 · p. 46
  • The Voronoi-cell argument controls local barycenters and mass using the regression error.The displayed bounds use the L2(P_X) regression error to control these concentration quantities.Source: §D.2 · pp. 48–49

For the localized loss class

The localized-loss argument places the loss class inside a sum of a low-dimensional subspace ball and a convex hull; Theorem 2 states a high-probability conclusion under a temperature condition.

For the localized loss class

  • The localized loss class is contained in the sum of a ball in an M(d + 1)-dimensional subspace and a scaled convex hull.The class consists of loss differences for functions within L2(P_X) radius r of the target.Source: For the localized loss class · pp. 49–50
  • Under Assumption 2 and λd ≤ c0, Theorem 2 states a conclusion for every m with probability at least 1−4e^−x.The supplied theorem passage does not include the content of that conclusion.Source: For the localized loss class · p. 50

D.3 Proof of Theorem 3

The proof uses a low-degree polynomial comparison to establish Theorem 3’s bounds under its stated regularity and sample-size conditions.

D.3 Proof of Theorem 3

  • The proof compares against a low-degree polynomial Qrip built from the component orthogonal to the hidden-index subspace.The construction uses the projection onto Range(W⋆) and the residual component of w.Source: §D.3 · pp. 50–51
  • Theorem 3 assumes λd≤c0, with constants depending on the listed model and noise parameters.The theorem statement also requires x≥1 and N≥2.Source: §D.3 · p. 52
  • When N≥d log(C0dN), the theorem’s second statistical term is absorbed.Source: §D.3 · p. 52

D.4 Proof of Corollary 2

The proof verifies the feature-learning properties by using the constant-one feature representation for the spherical MFLD estimator and a finite-dimensional coefficient subspace.

D.4 Proof of Corollary 2

  • The estimator’s alignment condition is immediate because its latent estimator and feature coefficient are both the constant-one function.The proof takes ωN≡0 for the alignment condition.Source: §D.4 · p. 52
  • The self-regularization step projects onto a coefficient subspace U of rank at most M(d+1).The proof invokes Ky Fan’s maximum principle for this subspace.Source: §D.4 · p. 52
  • The proof obtains the remaining feature-space item from Theorems 1 and 2 after identifying the learned feature representation with the estimator.Source: §D.4 · p. 52

D.5 Proof of Corollary 1

The proof transfers concentration of the learned measure near hidden directions into kernel and Hermite-projection comparisons, treating single- and multi-index cases separately.

D.5 Proof of Corollary 1

  • In the single-index case, the proof uses the parity of the activation’s Hermite degree when the measure is supported on {w⋆, −w⋆}.Source: §D.5 · pp. 53–54
  • For the multi-index case, Hermite-chaos projections provide a lower bound on the kernel discrepancy through the maximum projection difference over indices.Source: §D.5 · p. 54
  • A measure concentrated near the hidden directions yields a kernel discrepancy bounded by a quantity of order √r∗.The proof uses the activation’s boundedness and Lipschitz property to control the kernel change.Source: §D.5 · pp. 53–54

E Supplementary Lemmas

The supplementary lemmas establish interpolation and convergence statements, geometric control near hidden indices, and a caveat about the log-Sobolev proof constants.

E Supplementary Lemmas

  • A second-moment bound around a point yields tail control for Lipschitz observables through Markov’s inequality.Source: E Supplementary Lemmas · p. 55
  • If any label lies outside the activation’s range, no feature measure in the stated class can interpolate the full training sample.Source: E Supplementary Lemmas · p. 55
  • For small δ, Lemma 16 identifies a unique dominant hidden-index coordinate and bounds squared distance to its associated direction by Cηδ.Source: E Supplementary Lemmas · pp. 55–56
  • The appendix establishes convergence of the mean-field Langevin flow to its variational limit together with a convergence rate.The supplementary proof also states convergence of the KL divergence.Source: E Supplementary Lemmas · p. 58
  • The Holley–Stroock argument gives log-Sobolev constants with exponential dependence on λ, which the authors conjecture could be improved to polynomial dependence on λ⁻¹.The stated concern applies at least to the multi-index problem.Source: E Supplementary Lemmas · p. 59

E.1 A counter-example

The appendix gives a counterexample showing that the feature-learning result need not extend to misspecified single-index models. It contrasts the paper’s learned feature structure with norm- and compression-based bounds.

E.1 A counter-example

  • For globally Lipschitz σ, a Sobolev single-index target exists with positive L2(PX) approximation error over the bounded-parameter learner, so the result does not automatically extend to misspecification.The model outputs are uniformly Lipschitz, while the constructed target can be separated from all such outputs.Source: §E.1 · pp. 59–60

F.1 Comparison with norm-based bounds and compression bounds

  • Norm-based rates use parameter or margin-normalized norms, but their norm, kernel, or local-region metric is fixed before training [NTS15] [BFT17] [NBMS18].Some analyses localize around the learned predictor, yet the structure used for that analysis remains fixed [BBM05].Source: §F.1 · p. 60
  • Compression rates can depend on the trained network’s representation size, but they do not explain why training finds a feature space where the target has low effective dimension [AGNZ18, ZVA+19, SAN20].Examples include pruning, quantization, and shorter codes that preserve empirical error approximately.Source: §F.1 · p. 60
  • MFLD learns a problem-dependent feature structure through its hidden-layer distribution, inducing a data-dependent RKHS whose aligned principal directions support sharper, problem-dependent rates.The theory contrasts this learned structure with both fixed norms and structure assessed only after compression.Source: §F.1 · p. 60

F.2 Proof of Proposition 1

The appendix establishes that high-temperature MFLD lacks the concentration phenomenon, clarifies activation assumptions, and gives additional problem classes covered by its guarantees. A misspecified single-index target can also be represented as a well-specified two-index model.

F.2 Proof of Proposition 1

  • When λ=ω(1), the stationary measure does not exhibit the concentration phenomenon.The appendix states this as the high-temperature conclusion of Lemma 17.Source: §F.2 · p. 61

F.3 On the activation functions

  • The multi-index assumption requires the first M Hermite coefficients of σ to be nonzero, a finite non-degeneracy condition not guaranteed by smoothness or boundedness.Unshifted symmetric activations can have vanishing coefficients of the opposite parity; generic shifts typically avoid the exceptional cases.Source: §F.3 · p. 61

F.4 Additional Classes of Problems Effectively Solved by MFLD

  • For K-admissible σ and λd sufficiently small, Theorem 2 certifies learning classes A_{d,K,Δ,σ} when the stated risk threshold R_{N,λ,x,σ}(K,Δ)≤ε² holds.The theorem’s guarantee is stated with probability at least 1−4e^-x.Source: §F.4 · p. 62
  • A target misspecified as a single-index model is well-specified as a two-index model with separation Δ=2, and 2-admissible σ yields rate O((d log(dN)+x)/N) in the low-temperature regime.The rate holds up to constants depending on σ and the noise level.Source: §F.4 · p. 62

Related papers

Loading 2606.31429v2…