Source-linked AI summary
Theory of the Frequency Principle for General Deep Neural Networks
Tao Luo, Zheng Ma, Zhi-Qin John Xu, Yaoyu Zhang
TL;DR
The paper asks how general DNNs learn functions spectrally and develops a Fourier-analysis framework for this question. It proves Frequency-Principle characterizations at initial, intermediate, and final training stages under broad network, data, and loss settings. The results establish low-frequency dominance across these stages and identify function regularity as governing frequency-domain loss decay.
Problem
A rigorous theory of the Frequency Principle for multilayer DNNs with finite width, general activations, data densities, and losses is missing.
Source: §1 · p. 2Method
Results
Theorems show that low frequencies dominate relevant changes in MSE or DNN output across the initial, intermediate, and final stages.
Source: §1 · p. 2, §3.3 · pp. 10–11Takeaways & Limitations
The results provide a theoretical foundation for understanding the training process of general DNNs through the Frequency Principle.
Source: §1 · pp. 2–3, §3 · p. 7The analysis assumes target-function regularity, including f_target ∈ W^{k,∞}, and uses time averaging because instantaneous loss change need not be monotone or have a nonzero denominator.
Source: §2.3 · p. 5, §3.1 · p. 8
Abstract
from arXiv · showhide
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.
1 Introduction
The paper develops a Fourier-analysis theory explaining how general DNNs learn functions from low to high frequencies. It rigorously characterizes this Frequency Principle across training stages under broad architectural, data, and loss settings.
Motivation
- The paper addresses the missing theory of whether analyses for infinite-width, one-hidden-layer networks extend to multilayer DNNs with limited neuron numbers.Prior PDE approaches describe gradient flow in two-layer ReLU networks in the mean-field limit, but their applicability to general DNNs remains unclear.Source: §1 · p. 2
Background
- The Frequency Principle states that DNNs tend to learn target functions from low to high frequencies during training.Empirical studies observe this behavior across benchmark and synthetic data, and prior work uses it to interpret DNN learning.Source: §1 · p. 2
Contribution
- The framework rigorously characterizes the Frequency Principle at the initial, intermediate, and final training stages using Fourier analysis.At the initial and final stages, low frequencies dominate changes in MSE or DNN output; at the intermediate stage, low frequencies dominate MSE reduction over a half-life interval.Source: §1 · p. 2
Scope
- The results apply to multilayer DNNs with commonly used activations, general population densities, and a broad class of loss functions.The analysis includes activations such as ReLU, sigmoid, and tanh.Source: §1 · p. 3
Interpretation
- The analysis identifies a mechanism in which function regularity determines the decay rate of the loss function in the frequency domain.Source: §1 · p. 3
2 Preliminaries
The preliminaries define general multilayer DNNs, their loss-driven gradient-flow training, the regularity assumptions, and lemmas establishing the Sobolev properties needed for Fourier analysis.
2.1 Deep Neural Networks
- The paper models a DNN with H−1 hidden layers, d-dimensional input, one-dimensional output, general activations, and recursively composed layer functions.The input is the zeroth layer and the output is the H-th layer.Source: §2.1 · p. 3
- Network parameters consist of layer weights and biases collected in θ ∈ R^N, with N equal to the total number of parameters.Source: §2.1 · p. 3
2.2 Loss Function and Training Dynamics
- Training is studied with population-weighted MSE and general loss functions, and the parameters follow the corresponding gradient-flow dynamics.The MSE formulation also enables Fourier-domain expressions through the Plancherel theorem.Source: §2.2 · p. 4
2.3 Assumptions
- The assumptions require ftarget ∈ W^{k,∞} for some positive integer k (possibly ∞), bounded population density, nontrivial bounded parameter trajectories, and a C2 loss satisfying the stated local inequalities.Source: §2.3 · pp. 5–6
2.4 Regularity
- Regularity lemmas establish Sobolev membership for the truncated target and hypothesis functions and for the parameter gradient, enabling the subsequent Fourier estimates.Additional lemmas provide weighted Fourier-integrability bounds under the stated assumptions.Source: §2.4 · pp. 6–7
3 Main Results
The section develops quantitative Fourier-based characterizations and stage-specific theorems for the F-Principle, then illustrates the results numerically across training stages and loss functions.
3 Main Results
- The paper characterizes the F-Principle through frequency-resolved loss changes and proves results for initial, intermediate, and final training stages.The framework includes quantities for MSE and general losses, with separate theorems for each stage.Source: §3 · p. 7, §3.1 · p. 8, §3.2 · p. 9
3.1 Characterization of F-Principle
- The characterization requires time averaging when the loss is not monotone or its instantaneous decrement denominator can vanish.Finite decrements over an appropriate training timescale replace potentially undefined infinitesimal ratios.Source: §3.1 · p. 8, §3.2 · p. 9
3.2 Main Theorems
- The initial-stage theorem bounds frequency-resolved loss changes under L2 and general losses, subject to nonzero gradient conditions and constants depending on initialization and time.The general-loss statement uses a gradient-flow formulation distinct from the L2 case.Source: §3.2 · p. 9
- The intermediate-stage theorem applies to general losses over finite training intervals, while finite-decrement ratios remain meaningful even when instantaneous loss derivatives vanish.A half-life condition supplies a training timescale for the finite decrement.Source: §3.2 · pp. 9–10
- The final-stage theorems characterize the F-Principle when training converges to a non-degenerate global minimizer, for both L2 and general losses.The final-stage result is explicitly conditional on convergence to such a minimizer.Source: §3.2 · p. 10
3.3 Discussion and Illustrations
- In the numerical illustration, low frequencies dominate loss and output changes initially, whereas at the final stage only high-frequency peaks remain unconverged.The experiment uses uniformly sampled data on [−3.14, 3.14] and examines MSE and L4 training losses.Source: §3.3 · pp. 10–11
4 Proof of Theorems
The proofs establish the F-Principle across initial, intermediate, and final training stages by bounding frequency-specific dynamics under stated regularity and non-degeneracy assumptions. Numerical illustrations accompany the initial-stage results for MSE and L4 losses.
4.1 F-Principle: Initial Stage (Theorem 1)
- Initial-stage proofs bound high-frequency loss dynamics relative to total loss dynamics using hypothesis and parameter evolution, Cauchy–Schwarz, and gradient nonvanishing assumptions.The accompanying MSE and L4 figures numerically illustrate the theorem quantities across selected frequencies.Source: §4.1 · pp. 11, 13, Figure 1 · p. 12, Figure 2 · p. 13
4.2 F-Principle: Intermediate Stage (Theorem 2)
- The proof framework relies on bounded parameter trajectories and regularity assumptions that support continuity, boundedness, and the required differential estimates.The intermediate-stage argument separately controls numerator and denominator terms before combining them under its interval assumptions.Source: §4.1 · p. 13, §4.2 · pp. 16–17
- The intermediate-stage proof controls frequency contributions over intervals where the loss decreases sufficiently, with bounds depending on the interval length and regularity quantities.The estimates remain valid under a relaxed condition δL(θ(T1)) ≥ L(θ(T2)) for δ ∈ (0, 1), and convergence to a non-degenerate global minimizer removes the dependence on T2 − T1.Source: §4.2 · pp. 15, 17, 18
4.3 F-Principle: Final Stage (Theorem 3)
- Final-stage proofs use convergence to a non-degenerate minimizer and its Hessian to establish the corresponding frequency bound for L2 and general loss functions.The general-loss proof combines the earlier theorem with the non-degenerate-Hessian assumption.Source: §4.3 · pp. 18–19