Source-linked AI summary
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, Zheng Ma
TL;DR
The paper studies how DNNs train across frequencies, addressing the behavior of gradient-based learning and its implications for generalization and scientific computing. Using Fourier-based response-frequency analysis, experiments and idealized theory show that DNNs often fit low frequencies before high frequencies. This F-Principle contrasts with conventional numerical schemes and is associated with an implicit low-frequency bias.
Problem
Understanding DNN training and its implications for generalization and numerical computation remains a fundamental problem, including how the F-Principle behaves on high-dimensional data.
Method
The paper uses projection and filtering Fourier-analysis methods across datasets and network settings, supplemented by idealized theory about activation-function regularity.
Results
DNNs often fit target functions from low to high frequencies, consistently across MNIST/CIFAR10 settings, while conventional numerical schemes learn higher frequencies faster.
Takeaways & Limitations
The F-Principle implies that DNNs are implicitly biased toward fitting training data with low-frequency functions, helping explain differing generalization behavior across target functions.
Takeaways & Limitations
Using DNNs alone may be limited by slow convergence at high frequencies, motivating combinations of DNNs and conventional methods.
Abstract
from arXiv · showhide
We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies -- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.
1 Introduction
The paper identifies a Frequency Principle in which DNNs commonly fit target functions from low to high frequencies. It examines this behavior across datasets, architectures, losses, numerical schemes, and theoretical settings, connecting it to generalization.
- Frequency Principle: DNNs often fit target functions from low to high frequencies during training.At early stages, low-frequency components are fitted first; higher-frequency components emerge as training continues.
- Empirical evidence: Projection and filtering methods consistently demonstrate the F-Principle on MNIST and CIFAR10 across architectures including VGG16 and various loss functions.The experiments include fully connected and convolutional networks, as well as different activation and loss functions.
- Scientific computing: DNNs learn lower frequencies faster than conventional numerical schemes, while DNN-based schemes can greatly accelerate low-frequency convergence.The comparison is made against schemes such as the Jacobi method, which converges faster for higher frequencies.
- Theory: Idealized theory attributes the F-Principle to the smoothness or regularity of commonly used activation functions.The paper presents an intuitive explanation under idealized assumptions.
- Generalization: The F-Principle implies an implicit low-frequency bias that is consistent with good generalization on low-frequency-dominant real data and poor generalization on high-frequency-dominant functions such as parity.The paper contrasts datasets such as MNIST/CIFAR10 with parity-function spectra concentrated at high frequencies.
2 Frequency Principle
The paper defines response frequency through Fourier analysis of input-output mappings and evaluates frequency-wise convergence during DNN training. Because exhaustive high-dimensional frequency analysis is impractical, it uses projection and filtering approaches to test the low-to-high fitting pattern.
- Frequency definition: In this paper, frequency refers to response frequency of an input-output mapping, not image or input frequency.For classification, it measures how rapidly the output changes as input coordinates such as pixel intensities vary.
- Frequency definition: Response frequency is estimated for general input-output mappings using the standard nonuniform discrete Fourier transform.The transform is applied to mappings between inputs and outputs under the data distribution.
- Convergence analysis: Frequency-wise convergence is measured by the relative error ΔF(k) = |ĥ_k − ŷ_k|/|ŷ_k| between DNN and target Fourier components.The target and DNN outputs are evaluated on the training points, allowing convergence to be examined at selected frequencies.
- Practical evaluation: Evaluating every frequency in dimensions of O(10^2) or O(10^3) is impractical because of the curse of dimensionality.The paper therefore fixes a direction for projection analysis and uses filtering for coarse-grained frequency analysis.
- Synthetic demonstration: For a one-dimensional target with three frequency components, the DNN converges fastest at the first frequency peak, followed by the second and third peaks.The experiment uses f(x) = sin(x) + sin(3x) + sin(5x) and tracks relative Fourier-component errors during training.
3 F-Principle in MNIST/CIFAR10 through projection method
The projection method examines response-frequency convergence along selected directions in high-dimensional MNIST and CIFAR10 data. Across both datasets, DNNs capture low frequencies before higher frequencies during training.
- Projection method: The projection method restricts Fourier analysis to one direction, using k⃗ = k⃗p1 because enumerating all frequencies is computationally infeasible in high dimensions.For each dataset, p1 is chosen as the first principal component of the input space.
- Projection method: The method computes the relative frequency error ∆F(k) = |ĥk − ŷk|/|ŷk| for selected frequencies throughout training.The transforms are computed along the selected projection direction for the target and DNN outputs.
- MNIST/CIFAR10 results: Low frequencies dominate the response spectra of both MNIST and CIFAR10, while higher frequencies are progressively captured during training.The reported training/test accuracies are 0.951/0.963 for MNIST and 0.98/0.72 for CIFAR10.
- MNIST/CIFAR10 results: DNNs capture low frequencies first and gradually capture higher frequencies in the selected directions for both real datasets.Similar phenomena are also observed for other output components and directions.
4 F-Principle in MNIST/CIFAR10 through filtering method
The filtering method complements one-direction projection by separating low- and high-frequency components and comparing their relative errors. Experiments on MNIST, CIFAR10, CNNs, and VGG16 show faster convergence for low-frequency components.
- Filtering method: Filtering divides the frequency domain into low-frequency components with |k⃗| ≤ k0 and high-frequency components with |k⃗| > k0.The convergence of the two parts is compared through their relative errors during training.
- Filtering method: If elow < ehigh across different cutoffs, the filtering method identifies faster convergence of lower frequencies in the mean sense.The low- and high-frequency outputs are decomposed from the evolving DNN output and evaluated at each training epoch.
- Filtering method: High-dimensional Fourier costs are avoided by using the Fourier transform of a Gaussian to approximate the high-frequency indicator.The Gaussian transform decays exponentially with frequency, and the corresponding spatial-domain computation uses convolution.
- Experiments: The filtering experiments cover MNIST with a tanh-DNN, CIFAR10 with a ReLU-CNN, and CIFAR10 with VGG16.The compared networks use the stated dataset-specific losses and training configurations.
- Results: Across the examined datasets and networks, low-frequency relative error decreases much faster than high-frequency relative error.The results also hold for both SGD and GD, and are not limited to MSE loss.
5 F-Principle in solving differential equation
The paper compares DNN-based and conventional numerical solvers for Poisson’s equation, finding opposite frequency-wise convergence behavior. Combining the methods can accelerate convergence by pairing DNN learning of low frequencies with Jacobi convergence of high frequencies.
- DNN-based approaches are studied for scientific computing problems including high-dimensional partial differential equations.
- For the one-dimensional Poisson equation, the paper formulates a central-difference discretization and its resulting linear system.
- Jacobi convergence is controlled by iteration-matrix eigenvalues, with lower frequencies converging more slowly than higher frequencies.
- In the Poisson experiment, DNN training fits low Fourier frequencies faster, whereas Jacobi iteration fits high frequencies faster.
- A hybrid solver uses a DNN-trained initial guess to capture low frequencies before applying Jacobi iteration to converge high frequencies.
- Using DNNs alone can be limited by slow high-frequency convergence, motivating combined DNN and conventional numerical schemes.
6 A preliminary theoretical understanding
The paper gives an idealized theoretical explanation for the F-Principle through activation-function regularity. For tanh networks, low-frequency loss components can dominate higher-frequency components during gradient-based training.
- The paper connects activation-function smoothness and regularity with frequency-dependent gradient and convergence priorities.
- The theoretical setting considers a one-hidden-layer DNN with tanh activation and parameters comprising weights and biases.
- The Fourier transform of the network output is used to define amplitude deviation, frequency-wise loss, and total loss.
- By Parseval’s theorem, mean squared error in the spatial domain equals the corresponding L2 loss in the Fourier domain, supporting the frequency-wise convergence analysis.
- Theorem 1 establishes a measure bound for parameter regions where the higher-frequency component has stronger gradient contribution than the lower-frequency component.
- For non-converged frequencies and sufficiently small weights, the lower-frequency gradient exponentially dominates the higher-frequency gradient.
7 Discussions
The discussion interprets the F-Principle as an implicit low-frequency bias that helps explain differing DNN generalization behavior. Fourier analyses show closer recovery for MNIST/CIFAR10 and substantial mismatch for parity-function targets.
- 7.1 Generalization: DNN generalization is examined by comparing Fourier structure in real datasets with that of the parity function.
- 7.1 Generalization: For MNIST/CIFAR10, the Fourier spectrum concentrates at low frequencies, and training-data spectra overlap the full-dataset spectrum at dominant frequencies.
- 7.1 Generalization: The parity function concentrates power at high frequencies and has vanishing power near zero frequency.
- 7.1 Generalization: Well-trained DNN outputs recover dominant low frequencies for MNIST/CIFAR10 but deviate substantially from the parity spectrum across frequencies.
- 7.1 Generalization: Insufficient samples can create artificial low frequencies through aliasing when the target contains significant high-frequency components.
- 7.2 Related explanations: The paper positions Fourier analysis as an additional perspective alongside complexity, loss geometry, optimization stability, and implicit-bias explanations of generalization.