PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework

Yanyang Xiao

arXiv:2610.21964v1cs.LGcs.AImath.SPstat.ML

TL;DR

Existing accounts leave open an asymptotic explanation of spectral bias, including why preferred components are generally sinusoidal, while finite-domain Fourier analysis can introduce aliasing. The analysis defines pk(x) as an order-k polynomial and states that matrix Eq. (17) has a decaying property, with symmetric matrix B of bounded spectral norm and diagonal matrix D. The graded structure of H supports an explanation of spectral bias through eigenvalue decay and eigenvector shape.

  • Problem

    Existing accounts leave open an asymptotic explanation of spectral bias, including why preferred components are generally sinusoidal, while finite-domain Fourier analysis can introduce aliasing.

    Source: §1 · p. 2

  • Method

    The analysis defines pk(x) as an order-k polynomial and states that matrix Eq. (17) has a decaying property, with symmetric matrix B of bounded spectral norm and diagonal matrix D.

    Source: §2 · p. 3, §4.2 · p. 8

  • Results

    The graded structure of H supports an explanation of spectral bias through eigenvalue decay and eigenvector shape.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The framework relates learning bias to activation-function properties and potentially to network architecture.

    Source: §1 · p. 2

    The analysis assumes finite truncation of H, and treats activations with real-axis singularities as non-smooth for certain translation parameters.

    Source: §3.2 · p. 5, §4 · p. 7

Abstract

from arXiv · show

Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.

1. Introduction

The paper develops an aliasing-free, asymptotic account of spectral bias using the dynamics of frequency components. It aims to explain why components are generally sinusoidal and extend the framework beyond toy networks and specific activations.

1. Introduction

  • Finite-domain Fourier analysis can contaminate lower-amplitude frequency components with aliasing, motivating an aliasing-free representation.The authors also seek a quantitative, asymptotic account of spectral bias and its origin.Source: §1 · p. 2
  • The theory is presented as applicable to sensible activation functions and as requiring no closed-form integral to explain eigenvector shape.The paper contrasts this scope with prior work on eigenvector shape limited to ReLU.Source: §1 · p. 2
  • The paper connects its motivation to natural images, sounds, and EEG, which it describes as low-frequency dominated.It suggests that sinusoidal frequency components coincide with principal components of these signals.Source: §1 · pp. 2–3
  • The framework represents network outputs with an orthogonal-polynomial-based General Fourier Series and studies its coefficient dynamics through matrix H.The paper then analyzes H's eigenvalues and eigenvectors, and discusses links to the loss landscape and effective dimension.Source: §1 · p. 3

2. General Fourier Series Expansion Derived from Loss

The paper defines a loss-induced inner product and uses its orthonormal polynomial basis to expand the network output as a Generalized Fourier Series. The polynomial degree plays a role analogous to frequency in Fourier modes.

2. General Fourier Series Expansion Derived from Loss

  • The fitting setup targets function values at sampled inputs in [-1, 1], with the high-dimensional case left under consideration.As the sample count tends to infinity, the setup defines a theoretical loss under the input distribution.Source: §2 · p. 3
  • The loss induces a norm and inner product, under which the authors use an orthonormal polynomial basis to represent functions.The stated basis is complete and has convergence trends like Chebyshev approximation.Source: §2 · pp. 3–4
  • The network fitting function is expanded in a Generalized Fourier Series using the polynomial basis.In one dimension, the degree-k polynomial has k zeros, giving its index a frequency-like role analogous to Fourier modes.Source: §2 · pp. 3–4

3. Evolution of Generalized Fourier Series in Gradient Descent

The paper derives gradient-descent dynamics for the generalized Fourier-series coefficients and relates their evolution to the eigenvalues of H. The derivation treats activation smoothness cases and specifies a real-axis singularity caveat.

3.1. Gradient of Loss in View of Generalized Fourier Series

  • The gradient of the loss is expanded in the generalized Fourier-series view to derive how its coefficients change under gradient descent.The coefficient dynamics are expressed as a linear system whose eigenvalues govern the evolution.Source: §3.1 · p. 4, §3.2 · p. 6
  • For analytic activations, the coefficient analysis assumes the nearest singularity lies away from the real axis and relates the Bernstein ellipse parameter to that distance.The passage gives the parameter bound in terms of B, the distance to the nearest singularity.Source: §3.1 · p. 5
  • For activations continuous up to a specified derivative order, the smoothness of parameter derivatives varies by parameter, with a corresponding coefficient bound.For ReLU, the derivative's total variation has parameter-dependent asymptotic behavior; the analysis uses Chebyshev expansion, with Legendre expansion stated to have the same asymptotics for uniform weighting.Source: §3.1 · p. 5

3.2. Gradient Descent

  • Gradient descent is written as a continuous-time ODE in the small-step-size limit, and the dynamics of the generalized Fourier-series coefficients are derived from it.The eigenvalues of H govern the coefficient dynamics; the paper notes that H_c(θ) is time-dependent.Source: §3.2 · pp. 5–6
  • An activation with a real-axis singularity can become non-smooth for certain translation parameters.The authors characterize this as probably not a very interesting activation function.Source: §3.2 · p. 5

4. Structure of the matrix H

The paper studies H through a shallow-network example and analyzes its graded structure. Under the stated smoothness conditions, this structure yields decay bounds for H's eigenvalues.

4. Structure of the matrix H

  • Because H is infinite-dimensional, the analysis assumes finite truncation, justified by the fast decay of the coefficients.Direct analysis of the infinite-dimensional problem is left for future work.Source: §4 · p. 7

4.1. 1-Hidden Layer DNN

  • For a one-hidden-layer network, activation smoothness gives decay bounds for the derivatives used to compute H.The bounds distinguish parameter derivatives, with the stated derivative order depending on whether the parameter is a translation, scale, or output weight.Source: §4.1 · pp. 7–8

4.2. Graded Matrix

  • The graded structure writes H using a diagonal matrix D and a symmetric matrix B of bounded spectral norm.The paper relates this structure to eigenvalue decay, with the decay depending on activation smoothness.Source: §4.2 · pp. 8–9

5. Related Problems That Encounter Similar Matrix H

The loss Hessian and the GFS coefficient matrix have identical eigenvalues, and the loss landscape is extremely flat because it has many near-zero eigenvalues. The base-function matrix shares the same eigenvalue decay, though it is numerically not full rank.

5.1. Hessian of Loss Function

  • The loss Hessian matrix HL(θ) and coefficient matrix Hc(θ) have exactly the same eigenvalues.Eigenvalues beyond either matrix’s dimension are defined as zero.Source: §5.1 · p. 10
  • The loss landscape is extremely flat because it has vast numbers of near-zero eigenvalues.The passage points to Eq. (24) for this result.Source: §5.1 · p. 10

5.2. Effective Dimension of Base Function

  • The base-function matrix HΥ has the same eigenvalue-decay property as Hc(θ) and HL(θ).The passage notes an ideal solution for the output parameters, but the supplied text does not give its form.Source: §5.2 · p. 10
  • The sampled base-function matrix is numerically not full rank.The associated equation is described as arising in an extreme learning machine, or when fixing hidden weights and fitting only output weights.Source: §5.2 · p. 10

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The author reports that AI-assisted technologies were not used to construct the theory, which was developed around 2018. AI was used for grammar and spelling correction, and some AI coding was used to generate a figure.

Appendix A. Lemmas

The appendix states bounds for Chebyshev coefficients and approximation errors under specified smoothness conditions. It also introduces an analytic-function setting based on continuation into a Bernstein ellipse.

Appendix A.1.1. Theorems for absolutely continuous functions

  • For k ≥ ν + 1, the Chebyshev coefficients satisfy |a_k| ≤ 2V/[πk(k − 1)···(k − ν)] ≤ 2V/[π(k − ν)^(ν+1)].The stated conditions include absolute continuity through derivative ν−1 and bounded variation V of derivative ν.Source: Appendix A · p. 13
  • For n > ν, the stated Chebyshev projection and interpolant bound is ∥f − p_n∥∞ ≤ 4V/[πν(n − ν)^ν].The passages introduce this result under the same conditions as Lemma 1.Source: Appendix A · p. 13

Theorems for analytical functions

  • The analytic-function results assume continuation into an open Bernstein ellipse Eρ, where the function is bounded by M.The supplied passage gives |a_0| ≤ M but does not include the rest of the coefficient bound.Source: Appendix A · p. 13

Appendix A.2. General Theorems

For admissible weight functions, the GFS converges inside the largest Bernstein ellipse of analyticity and diverges outside it. The ellipse governs asymptotic geometric coefficient decay, while the weight function changes the basis and prefactors.

Lemma 8. GFS approximation

  • For a general weight function ρ, the stated convergence result assumes standard Szegő-class conditions.The passage identifies a standard sufficient condition but does not supply it in the provided text.Source: Appendix A.2 · p. 15

General theorems about polynomial approximations

  • For admissible weight functions, the GFS converges uniformly on compact subsets inside Eρ⋆ and diverges outside; its asymptotic geometric coefficient decay is governed by that ellipse.The weight function changes the polynomial basis and prefactors, while analyticity determines the ellipse of convergence.Source: Appendix A.2 · p. 15

Related papers

Loading 2610.21964v1…