PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Understanding the Spectral Bias of Deep Learning

Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, Quanquan Gu

arXiv:1912.01198v3cs.LGstat.ML

TL;DR

The paper asks how neural networks preferentially learn lower-complexity functions and whether this spectral bias has a rigorous mechanism. It analyzes over-parameterized two-layer networks through neural tangent kernel eigendirections, proving eigenvalue-dependent convergence and showing that lower-degree spherical harmonics are easier to learn on the unit sphere.

  • Problem

    Existing work had not provided a satisfying theoretical explanation for why neural networks learn lower-complexity functions faster.

  • Method

    The paper decomposes neural-network training along neural tangent kernel eigenfunctions and analyzes convergence under general data distributions, with a spherical-harmonic case study.

  • Results

    Training directions converge at rates determined by their neural tangent kernel eigenvalues, and lower-degree spherical harmonics are learned more easily for uniform spherical data.

  • Takeaways & Limitations

    The theory provides a rigorous explanation of spectral bias without specific assumptions on the target function and supports faster learning of lower-frequency components.

  • Takeaways & Limitations

    The results are restricted to the neural tangent or lazy-training regime, and some spherical-data conditions depend exponentially on k or d.

Abstract

from arXiv · show

An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions. The priority of learning functions with low complexity might be at the core of explaining generalization ability of neural network, and certain efforts have been made to provide theoretical explanation for spectral bias. However, there is still no satisfying theoretical result justifying the underlying mechanism of spectral bias. In this paper, we give a comprehensive and rigorous explanation for spectral bias and relate it with the neural tangent kernel function proposed in recent work. We prove that the training process of neural networks can be decomposed along different directions defined by the eigenfunctions of the neural tangent kernel, where each direction has its own convergence rate and the rate is determined by the corresponding eigenvalue. We then provide a case study when the input data is uniformly distributed over the unit sphere, and show that lower degree spherical harmonics are easier to be learned by over-parameterized neural networks. Finally, we provide numerical experiments to demonstrate the correctness of our theory. Our experimental results also show that our theory can tolerate certain model misspecification in terms of the input data distribution.

1 Introduction

The paper explains spectral bias by linking neural-network training dynamics to neural tangent kernel eigendirections, whose eigenvalues determine convergence rates. For uniform spherical data, lower-degree spherical harmonics are learned more readily, with experiments supporting the theory.

  • Motivation: Spectral bias describes neural networks learning lower-complexity components faster during training.This phenomenon has been proposed as a possible explanation for good generalization by over-parameterized networks.
  • Motivation: Existing analyses and experiments provide evidence for frequency-dependent learning but do not fully explain the underlying mechanism.Prior work studied Fourier spectra, low-degree polynomials, and neural tangent kernel behavior.
  • Main results: The paper proves that training error decomposes along neural tangent kernel eigendirections, with each direction converging according to its corresponding eigenvalue.The theorem applies under sample-complexity and over-parameterization conditions and gives precise residual control for arbitrary data distributions.
  • Spherical case: For uniform data on the unit sphere, neural tangent kernel eigenfunctions are spherical harmonics, and lower-degree harmonics are easier to learn.The eigenvalue characterization is stated as μ_k = Ω(max{k^-d-1, d^-k+1}) and improves a prior Ω(k^-d-1) bound when d is much larger than k.
  • Main results: Lower-frequency components can be learned with fewer examples and narrower networks at faster convergence rates.As network width increases, additional higher-frequency components become learnable, but more slowly.

2 Related Work

Related work studies over-parameterized neural networks through neural tangent kernel convergence, spectral analysis, and implicit bias. This paper builds on those lines while targeting frequency-dependent learning rates and width requirements.

  • Neural tangent kernel: Neural tangent kernel analyses establish convergence and generalization results under different width or target-function assumptions.Some results require width depending on the smallest Gram-matrix eigenvalue, while others assume finite NTK-induced RKHS norm.
  • Paper positioning: The paper distinguishes its goal from prior global-convergence and generalization analyses by studying how different target components are learned.Existing techniques do not directly determine how many hidden nodes are needed for simple or complex components.
  • Kernel spectra: The neural tangent kernel can be represented through an integral operator whose spectrum is studied using harmonic decompositions and spherical harmonics.Prior work analyzed eigenvalue decay for kernels on the unit sphere.
  • Implicit bias: Implicit-bias research asks which properties training algorithms impose when multiple optimal solutions exist.This literature covers gradient descent, stochastic gradient descent, and mirror descent across several settings.

3 Preliminaries

The preliminaries define the two-layer ReLU network, its gradient-descent training setup, and the neural tangent kernel and associated integral operator. For two-layer networks, the NTK combines degree-0 and degree-1 arc-cosine kernels and is a Mercer kernel.

  • Problem setup: The model is a two-layer fully connected network with ReLU activation trained by square loss on n examples.The first and second layer weights are represented by W1 and W2, respectively.
  • Problem setup: Inputs are assumed to lie on the unit sphere and follow an unknown distribution, while labels satisfy |y_i| ≤ 1.The setup uses d-dimensional spherical inputs represented in R^(d+1).
  • Optimization: Training initializes both layers randomly and updates the parameters using gradient descent with step size η.Algorithm 1 specifies Gaussian initialization and iterative updates W^(t+1) = W^(t) − η·∇L_S(W^(t)).
  • Neural tangent kernel: In the infinite-width limit, inner products of network gradients define the neural tangent kernel.The paper analyzes training in the neural tangent or lazy-training regime.
  • Neural tangent kernel: For two-layer networks, the NTK is κ(x,x′) = ⟨x,x′⟩κ1(x,x′) + 2κ2(x,x′), combining two arc-cosine kernels.The two components have degrees 0 and 1.
  • Integral operator: The NTK defines an integral operator on square-integrable functions, and its continuity, symmetry, and positive semidefiniteness make it a Mercer kernel.The operator is defined relative to the input distribution τ.

4 Main Results

The paper analyzes spectral bias through NTK eigendirections, showing that larger-eigenvalue components converge first, and specializes the result to spherical harmonics under uniform sphere inputs.

  • 4.1 Convergence Analysis of Gradient Descent: Theorem 4.2 refines training analysis by tracking convergence along eigendirections defined by NTK eigenfunctions rather than only the overall loss.The projected residual along the first r_k eigendirections is controlled under sample-size and width conditions.
  • 4.1 Convergence Analysis of Gradient Descent: Components associated with larger NTK eigenvalues are learned first, while smaller-eigenvalue components require more time, examples, and network width.This provides the paper’s mechanism for spectral bias and applies to arbitrary target functions or labelings.
  • 4.2 Spectral Analysis of NTK for Uniform Distribution: For uniform inputs on the unit sphere, the NTK admits a Mercer decomposition whose eigenfunctions are spherical harmonics.The spectrum is analyzed through the integral operator and its distinct eigenvalues, including multiplicities of harmonic eigenspaces.
  • 4.2 Spectral Analysis of NTK for Uniform Distribution: The spherical spectrum satisfies µ0 = µ1 = Ω(1), odd degrees k = 2j + 1 have µk = 0, and even-degree eigenvalues follow the stated maximum-order bound.The bound is reported as µk = Ω(max{ k^-d-1, d^-k+1 }) in the d ≫ k regime and improves on the earlier Ω(k^-d-1) bound.
  • 4.3 Convergence for Uniformly Distributed Data: The uniform-sphere convergence conditions have exponential dependence on k or d, which the paper characterizes as reasonable and unavoidable in the corresponding high-dimensional settings.With n training points, the paper argues that learning more than n independent components cannot generally be expected.

5 Experiments

Experiments test the predicted component-wise convergence using spherical-harmonic targets, more general functions, and three non-uniform input distributions. Across these settings, lower-frequency or lower-order components are learned faster, with approximately linear convergence in the logarithmic analysis.

  • With 4096 hidden neurons and 1000 training samples, the experiments train two-layer networks using vanilla gradient descent.
  • 5.1 Learning Combinations of Spherical Harmonics: The spherical-harmonic experiment measures residual projection lengths onto approximately orthonormal component vectors.
  • 5.1 Learning Combinations of Spherical Harmonics: The lowest-frequency residual converges to zero first, followed by the second-lowest, while the highest-frequency component converges last.
  • 5.1 Learning Combinations of Spherical Harmonics: Increasing the scale of higher-frequency components does not change the ordering: low frequencies are still learned first.
  • 5.1 Learning Combinations of Spherical Harmonics: Log-scale curves show that different projection lengths exhibit convergence close to linear, matching the theoretical analysis.
  • 5.2 Learning Functions of Simple Forms: For trigonometric and even-degree polynomial targets, residual projections eventually converge at predicted rates despite non-monotonic behavior early in training.
  • 5.3 Non-uniform Input Data Distributions: Under piece-wise uniform, normalized non-isotropic Gaussian, and normalized Gaussian-mixture inputs, lower-order polynomial components remain relatively faster to learn.
  • 5.3 Non-uniform Input Data Distributions: These non-uniform-input results suggest tolerance to some model misspecification, even though spherical harmonics are no longer exact NTK eigenfunctions.

6 Conclusion

The paper explains spectral bias in two-layer ReLU networks by linking gradient-descent convergence to neural tangent kernel eigenvalues. It derives sphere-specific eigenvalue orders and supports the theory with synthetic experiments.

  • The paper theoretically justifies spectral bias through detailed convergence analysis of two-layer ReLU networks.
  • Gradient descent converges at different rates along different directions, with each rate determined by the corresponding neural tangent kernel eigenvalue.
  • For uniform data on the unit sphere, the paper gives an explicit order for neural tangent kernel integral-operator eigenvalues.
  • Synthetic-data experiments support the theoretical result.
  • Spherical harmonics form the paper's sphere-based harmonic framework, including orthonormal bases and addition formulas for their decomposition.

B.1 Proof of Lemma 4.1

The proof of Lemma 4.1 uses concentration inequalities to show that empirical inner products of bounded orthonormal functions are well controlled from independent samples.

  • The proof begins by applying concentration inequalities to the empirical inner products generated by i.i.d. samples.
  • The functions are orthonormal in L2, and their values are uniformly bounded by M on the sphere.
  • Hoeffding's inequality provides high-probability bounds, and a union bound extends them simultaneously across all indexed function pairs.

B.2 Proof of Theorem 4.2

The proof connects finite-width neural-network dynamics to neural tangent kernel eigendirections, controls the iterates and residual projections, and then derives the theorem using concentration and harmonic decompositions.

  • The proof's core idea is to connect neural-network gradients throughout training with the neural tangent kernel.
  • Finite-sample eigenvectors are compared with neural tangent kernel eigenfunctions through concentration results and approximate orthogonality.
  • The key residual lemma provides a more careful characterization of residual dynamics along different directions than earlier NTK-regime analyses.
  • The resulting residual bound decays geometrically with the relevant eigenvalue, up to approximation and finite-sample terms.
  • An induction argument keeps all gradient-descent iterates inside a neighborhood of initialization under the stated width and step-size conditions.
  • The proof combines concentration, finite-dimensional eigenspace control, and explicit kernel calculations to complete the theorem and corollaries.
  • For sphere-supported data, Mercer decomposition and harmonic identities identify the neural tangent kernel eigenvalues using spherical-harmonic components.
  • The sphere-specific spectral calculation accounts for parity effects, including vanishing eigenvalue terms for selected odd or even orders.

C.2 Proof of Lemma B.4

The proof of Lemma B.4 combines recursive error control, uniform network-function bounds, and eigenvalue perturbation estimates to establish finite-width training guarantees. Under neighborhood and width conditions, the error follows a kernel-driven recursion with a controlled remainder.

  • Finite-width comparison: A separate perturbation lemma bounds the difference between infinite-width NTK eigenvalues and their finite-width counterparts with high probability.This connects the kernel-based convergence analysis to the finite-width network analyzed in the lemma.
  • Recursive error control: The proof uses a recursive formula for the prediction error, expressing the next iterate through the infinite-width NTK matrix plus a bounded remainder.The recursion is y − ŷ(t1+1) = [I − (ηmθ^2/n)K∞](y − ŷ(t1)) + e(t1), with the remainder controlled proportionally to the current error.
  • Recursive error control: Under the stated ball and width conditions, the remainder satisfies a high-probability bound proportional to ω^1/3ηmθ^2 times the current prediction error.The iterates must remain inside B(W(0),ω), while m and ω satisfy logarithmic and polynomial constraints.
  • Auxiliary controls: Auxiliary lemmas provide uniform bounds on network values and parameter-dependent quantities throughout the neighborhood around initialization.These bounds support the decomposition of the remainder into multiple terms and the matrix-form conclusion of the recursion proof.
  • Error bound: The resulting error estimate decays geometrically at a rate governed by the minimum relevant NTK eigenvalue, up to initialization- and remainder-dependent terms.The displayed bound contains a factor (1 − ηmθ^2λ_rk/2)^t1 together with an additive term involving (ηmθ^2λ_rk)^−1 and the initial error.

D.2 Proof of Lemma C.3

This proof establishes a uniform bound on network function values within a neighborhood of the initialization by combining gradient and initialization controls.

  • The proof applies Lemmas D.1 and B.3, then uses triangle inequality to control the network-output variation.

E.1 Proof of Lemma D.2

The proof bounds changes in network derivative features around initialization, using ReLU indicator stability, Lipschitz continuity, and concentration results.

  • The proof decomposes derivative changes into contributions from perturbations in the first-layer weights and activation indicators.
  • Lemma E.1 controls the deviation of derivative-feature matrices from their initialization values throughout B(W(0), ω).
  • First-layer gradient changes are bounded using concentration and initialization estimates for the data-dependent derivative terms.
  • The second-layer gradient bound follows from the 1-Lipschitz continuity of σ and triangle inequality.

F Appendix F: Additional Experimental Results

Additional experiments compare training-data and test-data projection lengths for Gegenbauer components, revealing generalization at low orders and overfitting at the highest order.

  • Low-order Gegenbauer polynomials are learned and generalize well, whereas the high-order component overfits the training data.
  • Figure 7 compares projection lengths onto vectors determined by training data with functions estimated using test data.
Loading 1912.01198v3…