Overview frequency principle/spectral bias in deep learning

Zhi-Qin John Xu; Yaoyu Zhang; Tao Luo

Source-linked AI summary

Overview frequency principle/spectral bias in deep learning

Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo

arXiv:2201.07395v4Jan 19, 2022cs.LG

TL;DR

Deep neural networks are highly successful but remain theoretically difficult to understand, especially regarding their generalization and frequency-dependent behavior. This paper overviews the Frequency Principle through phenomenological experiments and Fourier analysis, finding that DNNs generally fit low-frequency components before high-frequency ones. The resulting perspective identifies both low-frequency strengths and high-frequency limitations while motivating practical algorithms and further theory.

Problem
DNNs can generalize well despite over-parameterization, yet the theoretical understanding of their behavior, capabilities, and limitations remains incomplete.
Method
The paper synthesizes Frequency Principle studies using simple synthetic experiments, Fourier analysis, and verification on higher-dimensional data.
Results
DNNs generally fit functions from low to high frequencies during training, exhibiting an implicit low-frequency or spectral bias.
Takeaways & Limitations
The Frequency Principle explains DNN strengths on low-frequency functions, difficulties with high-frequency functions, and motivates algorithms for faster high-frequency learning.
Takeaways & Limitations
The Frequency Principle may not hold in settings such as PDE solving when the loss function contains gradient information.

Abstract

from arXiv · show

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights on this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) -- DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by onedimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.

1 Introduction

The paper overviews the Frequency Principle, which describes DNNs fitting low-frequency components before high-frequency components. This perspective offers a phenomenological route to understanding DNN capabilities, limitations, and practical behavior.

DNN theory remains important because these models are successful in applications yet can fail critically and remain difficult to understand.
The Frequency Principle clarifies DNN strengths on low-frequency data and difficulties with high-frequency data, while inspiring algorithms and frequency-based studies.
The Frequency Principle addresses questions about why over-parameterized DNNs can generalize well despite fitting training data.
A phenomenological approach uses simple synthetic problems to visualize DNN training before applying Fourier analysis to quantify frequency-dependent convergence.
DNNs often learn target functions from low to high frequencies, revealing an implicit low-frequency bias also called spectral bias.

2 Empirical study of F-Principle

Empirical studies first demonstrate the Frequency Principle on one-dimensional synthetic functions and then extend it to images and high-dimensional datasets. Across these settings, DNNs generally capture low frequencies before higher frequencies, although measuring frequency in high dimensions requires specialized methods.

2.1 Frequency principle in low-dimensional problems: One-dimensional experiments use Fourier components and relative frequency errors to show that DNNs fit simpler frequency components before more oscillatory ones.The target includes three designed frequencies, and convergence is tracked with ΔF(k).
2.1 Frequency principle in low-dimensional problems: The first frequency peak converges very quickly, the second more slowly, and the third more slowly still.
2.1 Frequency principle in low-dimensional problems: In two-dimensional image regression, the DNN learns a coarse-grained image before producing increasingly detailed structure.
2.1 Frequency principle in low-dimensional problems: The low-frequency preference can benefit image restoration but also limits DNNs’ ability to learn high-frequency structures.
2.2 High-dimensional data: High-dimensional analysis distinguishes response frequency from image-input frequency and faces a Fourier-transform curse of dimensionality.For a d-dimensional function, evaluating two frequency points per dimension requires 2^d evaluations.
2.2.1 Projection method: The projection method reduces analysis to selected frequency directions and shows low frequencies dominating and converging before higher frequencies in MNIST and CIFAR10.
2.2.2 Filtering method: The filtering method separates low- and high-frequency components with Gaussian smoothing, and low-frequency parts converge faster across MNIST and CIFAR10 network settings.The comparison includes tanh fully connected, shallow ReLU convolutional, and VGG16 models.

3 Theoretical study of F-Principle

Theoretical studies explain the F-Principle through Fourier-domain dynamics, activation-function regularity, and eigenvalue structure, establishing low-frequency components as faster-converging modes under several settings. These analyses span idealized, continuous, NTK, finite-sample, and broader network regimes, while retaining important scope limitations.

Frequency-domain foundation: Fourier analysis makes the F-Principle theoretically tractable by exposing different convergence rates for flat and oscillatory components.The frequency perspective is presented as a way to analyze neural-network training beyond its black-box characterization.
Idealized mechanism: For tanh networks, low-frequency gradients dominate training when weights are small because an exponential frequency factor suppresses higher frequencies.The analysis attributes the dominance to exp(−|πk/2w_j|), with weaker low-frequency dominance for larger w_j.
NTK and eigenvalue analysis: In the NTK regime, components associated with larger kernel eigenvalues converge faster, and these eigenvectors are lower-frequency for two-layer networks.This supplies a rigorous low-frequency-bias explanation through eigen-decomposition of the kernel dynamics.
Dense data distributions: Harmonic analysis of two-layer ReLU networks connects kernel eigenvectors with frequencies and derives successive fitting of higher spectral modes as sample size grows.The strongest explicit results assume uniform or related distributions on a sphere, while empirical studies find real data often align with large-eigenvalue, low-frequency eigenvectors.
Generalizations: Theoretical extensions cover arbitrary data distributions, general activation functions, continuous formulations, and multilayer networks under regularity and convergence assumptions.The multilayer result characterizes the high-frequency loss decay qualitatively rather than distinguishing detailed effects of network structure.
Scope and limitations: High-dimensional predictions can be difficult to interpret because the relevant differentiation order depends on dimension and may be fractal.This is identified as a limitation of the variational interpretation in high-dimensional problems.
Variational interpretation: The linear F-Principle links neural-network training to variational formulations whose solutions can resemble linear or cubic splines, with α = d as a critical point.For α < d the variational problem is trivial away from training points, whereas α > d yields a solution with certain regularity.

4 Understanding and studying DNN based on F-Principle

The F-Principle provides a frequency-based way to study DNN training, generalization, and failure modes. Its behavior depends jointly on activation and loss, favoring low frequencies in common settings but changing when these choices emphasize higher frequencies.

F-Principle analysis uses experiments and Fourier representations to study DNN settings, generalization, and frequency-based approaches beyond the principle itself.The overview also considers algorithms inspired by this perspective.
Activation and loss: Activation functions shape frequency convergence: standard activations support low-frequency convergence, while a Ricker activation decaying from higher frequencies removes the observed faster-convergence ordering.With smaller a, the Ricker function decays from a higher frequency, and no frequency converges faster in the reported experiment.
Activation and loss: Gradient information in the loss makes high frequencies converge much faster by multiplying Fourier components by frequency ξ.The gradient-augmented loss explicitly gives greater priority to higher-frequency components than the common mean squared loss.
Activation and loss: The joint effect of activation and loss means the F-Principle is readily observed for common choices but may not hold in settings such as PDE losses containing gradient information.The paper identifies activation and loss together as determinants of frequency convergence behavior.
Generalization: When the F-Principle is violated, a DNN can fit training data yet produce oscillatory test outputs, whereas frequency-following training yields smoother outputs and better generalization.The comparison changes only the Ricker activation parameter a; the low-frequency-biased network remains smooth at test points, while the other remains oscillatory.
Generalization: Fourier analysis links generalization differences to target spectra: low-frequency-dominated MNIST/CIFAR10 data generalize well, whereas parity-like high-frequency functions generalize badly.For parity, aliasing creates artificial low-frequency components that are captured first, while high-frequency components are compromised, producing poor generalization.

4.4 Early stopping

Early stopping exploits the F-Principle by halting training before high-frequency noise is fitted, improving generalization and robustness. The associated energy analysis explains why low-frequency fitting is preferred and why high-frequency targets generalize less well.

Early stopping: Early stopping avoids fitting noisy high-frequency training components, yielding better generalization on contaminated data.Test loss decreases initially, then rises as training captures higher-frequency noise.
Early stopping: At the best-generalization turning step, the DNN output is smooth and captures the dominant low-frequency peak.The training loss is already very small, while test loss is minimized at the turning step.
Robustness: Because low-frequency functions are more robust to input perturbations, early stopping can also enhance DNN robustness.A cited two-layer DNN study reconstructs true labels from noisy data using input-weight training with early stopping.
Frequency-selective fitting: Minimizing FP-energy favors low-frequency fitting because its frequency weight amplifies high frequencies and diminishes low frequencies.This preference selects a low-frequency fit among possible steady states.
Quantitative understanding: Generalization error decreases with more training data at a 1/√n rate, but worsens as the target contains more high-frequency components.The same frequency dependence is consistent with an NTK-regime error bound in which higher frequencies correspond to smaller eigenvalues.

4.6 Frequency perspective for understanding experimental phenomena

The F-Principle provides a frequency-based account of several training phenomena, including information-plane compression and increasing output complexity. It also motivates methods that reduce high-frequency learning costs or improve high-frequency representation.

Information-plane compression: The F-Principle explains information-plane compression as training shifts from continuous low-frequency components to discretized high-frequency components.The shift corresponds to decreasing output entropy as more high-frequency structure is captured.
Output complexity: DNN output complexity increases during training because progressively higher frequencies enter the output.With common small initialization, the initial output is often close to zero.
Depth: A deep frequency principle interprets each hidden layer through an effective target determined by the preceding layer’s output and the true labels.This perspective is proposed to understand how depth can accelerate training.
Algorithm design: The F-Principle motivates frequency-aware interventions, including noise penalties, frequency filters, multiscale inputs, and specialized activations.These approaches target the difficulty of learning high-frequency components in applications such as reinforcement learning and image generation.
Computational efficiency: Early training can be exploited computationally because low-frequency components are learned quickly and are often dominant and robust in real data.Early-Bird subnetworks and low-precision early training are reported to save computation without sacrificing generalization performance.

5 F-Principle for DNN-based methods for solving PDEs

The F-Principle is presented as an important feature of DNN-based numerical algorithms. The section compares their frequency convergence with conventional iterative and finite-element methods and reviews ways to address high-frequency difficulty.

Frequency behavior: DNN-based algorithms exhibit an F-Principle that is important for understanding their behavior in scientific computing.The section reviews frequency convergence differences with iterative and finite-element methods.
Section scope: The section organizes its analysis around comparing convergence, interpreting the difference from an iterative perspective, and reviewing methods for overcoming high-frequency challenges.These are the stated components of the review.

5.1 Parameterize the solution of a PDE

For PDE solving, neural networks parameterize the solution and are trained through variational, boundary-penalized, or least-squares objectives. Accuracy is assessed by comparing the learned solution with the true solution.

PDE parameterization: Poisson’s equation is used as an intuitive example for parameterizing a PDE solution with a neural network.The network represents u(x; θ), where θ denotes the DNN parameters.
Variational formulation: A deep Ritz approach minimizes a variational problem whose solution can be proved to solve Poisson’s problem.Its energy functional defines the target objective.
Boundary enforcement: The discretized objective combines an interior-domain term with an L2 penalty enforcing the Dirichlet boundary condition.The penalty is weighted by β and uses samples from the domain and boundary.
Least-squares formulation: Physics-informed or least-squares approaches train the network with a residual-based loss, including a least-squared-error objective.The section presents this as an alternative to the variational approach.
Accuracy assessment: Solution accuracy is evaluated by computing the distance between the neural-network solution and the true solution.This comparison provides the stated measure of learning accuracy.

5.2 Difference from conventional algorithms

The section compares DNN-based PDE solvers with conventional numerical methods, showing contrasting frequency-wise convergence and different behavior in over-parameterized R-G methods.

DNNs learn PDE solutions from low to high frequencies, whereas Jacobi iterations learn high frequencies before low frequencies.
For the Jacobi method, lower frequencies converge more slowly because their iteration eigenvalues are larger.
In a Poisson example, low-frequency Fourier components converge faster for the DNN, while high-frequency components converge faster for Jacobi.
A hybrid scheme initializes Jacobi iterations with a partially trained DNN to combine faster low-frequency learning with Jacobi’s high-frequency convergence.
R-G methods use local or global basis functions, including finite-element and spectral bases, to represent numerical solutions.
Over-parameterized R-G methods can produce singular solutions, whereas two-layer DNN solutions remain stable and nonsingular for large model size.

5.3 Understanding F-Principle by comparing the differential operator and the integrator operator

The section explains the contrasting frequency behavior of differential-operator optimization and neural-network integral-kernel optimization. Differential operators emphasize high frequencies, while the neural kernel favors low frequencies.

The analysis compares DNN training with Jacobi-style differential-operator iterations to explain their opposite frequency-ordering behavior.
For the differential operator, lower-frequency modes converge more slowly under mean-square optimization.
The neural-network kernel favors lower-frequency modes, creating competition between the integral-kernel and differential-operator effects.
Input differentiation weights Fourier components by frequency, giving higher-frequency modes larger weights and faster convergence.

5.4 Algorithm design to overcome the challenge of high-frequency

The section reviews methods designed to overcome DNNs’ difficulty with high-frequency components, including frequency conversion, multiscale architectures, adaptive activations, and explicit high-frequency weighting.

The F-Principle identifies high-frequency learning as a limitation of DNN-based algorithms.
PhaseDNN converts high-frequency components to lower-frequency spectra for learning and maps the learned representation back afterward.
MscaleDNN uses differently scaled inputs to perform radial frequency conversion for high-dimensional high-frequency problems.
MscaleDNN-1 applies multiple scales within a network, whereas MscaleDNN-2 sums subnetworks whose inputs use different scales.
Fourier-feature networks encode inputs with sinusoidal features at selected frequencies before neural-network processing.
Other approaches use adaptive activation scales or assign larger loss weights to high-frequency components.

6 Anti-F-Principle

The section describes anti-F-Principle behavior, in which high frequencies receive unusually strong priority instead of the usual low-frequency preference. Such behavior can arise from the loss function or large network weights.

Anti-F-Principle examples occur when high-frequency components receive sufficiently high priority during optimization.
Losses containing input gradients can weight high frequencies more strongly, so the observed frequency preference depends on competition between loss weighting and activation regularity.
Large network weights can reduce low-frequency dominance and increase initialization fluctuations, potentially producing high-frequency priority.

7 Conclusion

The F-Principle is presented as a general implicit bias that helps understand and design DNNs, while leaving important theoretical and mechanistic questions open.

The F-Principle provides a basic framework for understanding DNNs and inspiring their design.
Its generality extends beyond gradient training, where empirical studies also find the F-Principle holds.
A theory for general DNNs with arbitrary sample distributions and their generalization error remains unclear.
Beyond NTK, the F-Principle does not characterize the exact details of DNN training, motivating study of loss landscapes, width, depth, and initialization.
The loss landscape of a DNN has been described as containing the critical points of all narrower DNNs.