Source-linked AI summary
Training behavior of deep neural network in frequency domain
Zhi-Qin John Xu, Yaoyu Zhang, Yanyang Xiao
TL;DR
The paper asks why DNNs that can overfit often generalize well and studies implicit biases in their training process. Using real and synthetic datasets across varied architectures, activation functions, optimizers, and settings, it identifies the Frequency Principle: DNNs learn dominant low-frequency components before high-frequency ones. The paper relates this behavior to early stopping and observed generalization, while noting sensitivity to initialization scale.
Problem
It remains unclear what general principle underlies DNN optimization and why over-parameterized DNNs often generalize well despite their ability to overfit.
Method
The paper empirically studies DNN training in the frequency domain on real and synthetic datasets across different architectures, widths, depths, optimizers, and activation functions.
Results
DNNs first capture dominant low-frequency components quickly and then capture high-frequency components more slowly, a pattern called the F-Principle.
Takeaways & Limitations
The F-Principle provides insight into early stopping and the good generalization performance often observed for DNNs.
Takeaways & Limitations
Large initialization values can obscure a clear F-Principle and may produce different generalization performance, including clear overfitting.
Abstract
from arXiv · showhide
Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the study of implicit biases underlying the training process of DNNs. In this work, for both real and synthetic datasets, we empirically find that a DNN with common settings first quickly captures the dominant low-frequency components, and then relatively slowly captures the high-frequency ones. We call this phenomenon Frequency Principle (F-Principle). The F-Principle can be observed over DNNs of various structures, activation functions, and training algorithms in our experiments. We also illustrate how the F-Principle help understand the effect of early-stopping as well as the generalization of DNNs. This F-Principle potentially provides insights into a general principle underlying DNN optimization and generalization.
1 Introduction
The paper investigates why over-parameterized DNNs can generalize well despite their capacity to overfit, identifying a training bias toward learning frequency components from low to high.
- DNN training remains difficult to interpret despite the transparency of individual node values and parameters.
- Over-parameterized DNNs often generalize well after fitting training data, even when their parameter count greatly exceeds the dataset size.
- DNNs first quickly capture dominant low-frequency components and later capture high-frequency components more slowly, a behavior termed the Frequency Principle.
- The F-Principle appears across varied DNN widths, depths, training algorithms, and activation functions on real and synthetic data.
- The paper uses the F-Principle to provide insight into early stopping and the good generalization of DNNs.
2 Related works
Related work connects low-frequency fitting with lower function complexity and describes subsequent theoretical and empirical studies extending the F-Principle.
- Over-parameterized DNNs tend to fit training data with low-frequency functions that are naturally lower in complexity.
- Lower-frequency functions have smaller Lipschitz constants, which can lead to smaller generalization error through stability-based analysis.
- Subsequent work strengthened verification of the F-Principle for high-dimensional datasets and developed theoretical studies of the phenomenon.
- The F-Principle has also been used to study different learning trajectories in meta-learning and nonlinear collaborative schemes for deep networks.
3 Experimental setup
The experiments use mean-squared error, Adam optimization, Gaussian initialization, and several DNN configurations spanning fully connected and convolutional architectures.
- All DNNs use Adam with default parameters, mean-squared-error loss, and Gaussian initialization with mean 0.
- Figure 1 compares a fully connected tanh-DNN with a ReLU-CNN on MNIST and CIFAR10 using 10,000 training samples from each dataset.
- Figures 2–5 use a fully connected tanh-DNN with four hidden layers of widths 200-200-200-100, learning rate 2 × 10^-5, and full-batch training.
- The Fourier transform is experimentally estimated on discrete training or test data points.
4 F-Principle
The F-Principle describes DNN training as capturing low-frequency components before high-frequency ones, across synthetic and real-data settings. Frequency-domain analyses of image datasets and known-frequency functions provide empirical evidence for this ordering.
- 4 F-Principle: DNN training follows the F-Principle: low-frequency components are captured first, followed by high-frequency components.This pattern is reported for a general class of low-frequency-dominated functions.
- 4.1 MNIST/CIFAR10: For MNIST and CIFAR10, frequency analysis projects images onto the first principal component before applying non-uniform Fourier analysis.The analysis uses fully connected and convolutional DNNs and examines their output functions during training.
- 4.1 MNIST/CIFAR10: Both fully connected and convolutional DNNs capture image-training frequency peaks in an order from low to high frequencies.The datasets are dominated by low-frequency components along the first principal direction; non-peak components are treated as artifacts of the implicit periodic boundary condition.
- 4.2 Synthetic data: For f0(x) = sin(x)+2 sin(3x)+3 sin(5x) with α = 2, the synthetic target has three main Fourier components and shows precise low-to-high convergence.The data are evenly sampled on [−10, 10] with sample size 600.
- 4.2 Synthetic data: The F-Principle persists across several low-frequency-dominant functions, ReLU and tanh activations, and gradient descent and Adam optimizers.The authors report that the principle always holds in these additional analyses.
5 Understanding the training behavior of DNNs by the F-Principle
The F-Principle explains why over-parameterized DNNs can generalize well and why early stopping helps on noisy data: training captures low frequencies before high-frequency components.
- The F-Principle biases over-parameterized DNNs toward low power at high frequencies outside the training set’s effective frequency range.For low-frequency-dominated functions, this bias can yield well-generalized solutions despite the ability to fit training data exactly.
- Early stopping helps avoid fitting noisy high-frequency components in contaminated training data.The test loss first decreases and then increases as training begins capturing high-frequency noise.
- At the turning step, the DNN captures the dominant frequency peak and achieves the best generalization performance.After this step, the test loss increases while the model fits higher-frequency noise.
- In the contaminated-data experiment, training used 300 points and testing used 6000 evenly sampled points in [−10, 10].
6 Conclusions and discussion
The paper concludes that DNN training commonly follows the F-Principle across architectures, optimizers, and activations, while initialization can complicate the phenomenon and alter generalization.
- The F-Principle was observed across widths from tens to thousands of neurons, depths from one to tens of hidden layers, GD, SGD, Adam, tanh, and ReLU.
- The F-Principle provides insight into the good generalization performance of DNNs and their training behavior in the information plane.
- Large initialization can obscure the F-Principle and produce different generalization behavior from small initialization.
- With standard deviation 10, the DNN overfits test data, whereas standard deviation 0.1 yields good test-data generalization despite fitting the training data in both cases.
7 Appendix
The appendix examines information compression through mutual-information estimates and shows that changing α can make compression appear or disappear while preserving the early low-frequency training behavior.
- Compression vs. no compression in the information plane: Information compression is analyzed using trajectories of DNN training in the information plane.The analysis examines how the F-Principle relates to the compression phase.
- Computation of information: The DNN output T is deterministic given input X, so I(T|X) = 0 and I(X, T) = I(T) before binning.
- Computation of information: With bin size b = 0.05, I(T) and I(Y, T) are approximated using the binned variables T_b and Y_b, T_b.Binning can make I(T_b|X_b) nonzero, with the difference vanishing as bin size shrinks.
- Compression vs. no compression in the information plane: For α = 0, no compression phase appears in I(T), whereas increasing α produces discretized outputs and observable compression.
- Compression vs. no compression in the information plane: Different α values share the same dominant low-frequency components, so their early training courses are similar.For α > 0, I(T) later decreases from near I(f0(x)) toward the lower entropy I(f(x)) as the output becomes discretized.
- Compression vs. no compression in the information plane: Information compression is often observed in classification because discretization is generally inevitable for discrete labels.