PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Spectral Bias of Neural Networks

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, Aaron Courville

arXiv:1806.08734v3stat.MLcs.LG

TL;DR

Neural networks can fit arbitrary mappings, but the paper investigates whether their learning and parameterization favor simpler frequency content. Using Fourier analysis of deep ReLU networks, it studies spectral bias, manifold geometry, and perturbation robustness, finding preferential low-frequency learning alongside geometry-dependent high-frequency expressivity.

  • Problem

    Neural networks can fit random mappings while generalizing natural data, motivating the question of what implicit regularization favors simpler solutions.

  • Method

    The paper uses Fourier analysis of ReLU networks, combining analytic spectral treatment with studies of training dynamics, parameter perturbations, manifold geometry, and real-data effects.

  • Results

    Deep ReLU networks exhibit spectral bias by favoring low frequencies, while complex manifold shapes facilitate higher-frequency learning and lower frequencies are more robust to parameter perturbations.

  • Takeaways & Limitations

    The findings connect smooth-function preference and frequency robustness with the geometry of the data manifold, showing that expressivity depends on both network and data structure.

  • Takeaways & Limitations

    The reported spectral-decay rate is not guaranteed by piecewise smoothness alone, and the training explanation assumes the noisy target’s Lipschitz constant exceeds the network’s at initialization.

Abstract

from arXiv · show

Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with $100\%$ accuracy. In this work, we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets \emph{easier} with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

1. Introduction

The paper reframes neural-network implicit regularization through Fourier analysis, finding that deep ReLU networks favor low frequencies during learning and parameter perturbations. It also studies how ReLU structure and data-manifold geometry shape frequency expressivity.

  • Gradient descent learns low-complexity functions faster, offering a shifted view of implicit regularization in neural networks.
  • Fourier analysis reveals a spectral bias: neural networks prioritize learning lower-frequency modes despite approximating arbitrary functions.
  • Lower-frequency components are more robust to random perturbations of network parameters than higher-frequency components.
  • Complex data-manifold shapes can facilitate learning higher frequencies, with a theoretical analysis of this geometry–expressivity relationship.
  • The analysis focuses on ReLU networks because their continuous piecewise-linear structure permits analytic treatment of their Fourier behavior.

2. Fourier analysis of ReLU networks

The paper analyzes ReLU networks as continuous piecewise-linear functions and derives their Fourier representation from their linear regions. The resulting spectrum has direction-dependent decay, with bounds governed by network geometry and Lipschitz behavior.

  • 2.1. Preliminaries: A ReLU network is a scalar function built from affine layers and elementwise activations σ(u)i = max(0, ui).
  • 2.1. Preliminaries: ReLU networks are continuous piecewise-linear functions whose convex-polytope regions correspond to hidden-neuron activation patterns.
  • 2.2. Fourier Spectrum: The Fourier transform is decomposed through the network’s linear regions and the Fourier transforms of their indicator functions.
  • 2.2. Fourier Spectrum: Theorem 1 expresses ReLU-network Fourier components as a rational function and applies to arbitrary width and depth.
  • 2.2. Fourier Spectrum: Spectral decay is anisotropic: it is k^−d−1 in almost all directions but can be as slow as k^−2 along specific directions.The slower directions are orthogonal to faces bounding the linear regions.
  • 2.2. Fourier Spectrum: The Fourier-spectrum numerator is bounded by NfLf, where Nf is the number of linear regions and Lf is the network’s Lipschitz constant.The Lipschitz bound scales exponentially with depth and polynomially with width.
  • 2.2. Fourier Spectrum: Piecewise smoothness alone does not guarantee the stated spectral decay rate.The paper gives |x| as a continuous function smooth except at zero whose Fourier decay is k^−1.5.

3. Lower Frequencies are Learned First

The experiments show that ReLU networks learn lower-frequency components before higher-frequency ones, even when higher frequencies have larger target amplitudes. This spectral bias appears in synthetic and MNIST settings, is linked to gradually increasing parameter norms, and makes higher-frequency components less robust to perturbations.

  • Synthetic Experiments: Networks fit lower frequencies first, even when higher-frequency target components have larger amplitudes.The experiment compares equal-amplitude components with amplitudes increasing from A1 = 0.1 to A10 = 1.
  • Synthetic Experiments: Higher frequencies are less robust to random parameter perturbations than lower frequencies.The perturbation experiment evaluates isotropic parameter changes after convergence and links high-frequency expression to finely tuned parameter configurations.
  • Synthetic Experiments: The network’s spectral norms increase as training progresses toward fitting higher frequencies.The paper relates this trend to the Lipschitz bound and the gradual increase of parameter norms during gradient descent.
  • Synthetic Experiments: The Fourier-spectrum analysis predicts that the rate of residual change decreases with increasing frequency during training.For fixed direction, the paper derives a frequency-dependent decay of the residual’s training dynamics and reports agreement with Experiment 1.
  • Real-Data Experiments: On MNIST, low-frequency noise harms validation performance more, while high-frequency noise is learned later and causes an early validation-loss dip.The experiments use radial-wave noise added to the MNIST target and evaluate validation against the uncorrupted target.
  • Real-Data Experiments: The MNIST results show that spectral bias persists in high-dimensional, non-synthetic data.As higher-frequency noise is gradually fitted, validation loss increases while training loss continues to decrease.

4. Not all Manifolds are Learned Equal

The paper shows that data-manifold geometry changes the effective difficulty of learning high-frequency signals: more complex manifolds can make them easier to express and learn. The proposed mechanism is that the embedding maps low input-space frequencies into higher latent-space frequencies.

  • Motivation and setup: Complex manifold shapes can facilitate learning higher-frequency functions, showing that spectral bias depends on data geometry.The paper studies lower-dimensional manifolds embedded in the input space and presents experiments and theory for this interaction.
  • Mechanism: Low-frequency functions in input space can acquire high-frequency components when restricted to complex lower-dimensional manifolds.The embedding map mediates the relationship between the network spectrum and the spectrum of the restricted function.
  • Regression on curved manifolds: Increasing the manifold complexity parameter L attenuates spectral bias and makes the regression task easier.For the flower-shaped curves γL, the results show improved learning as L grows.
  • Classification on curved manifolds: Increasing L also improves classification performance for the same target signal across the tested frequency settings.This reproduces the regression behavior under binary cross-entropy loss.
  • Mechanism: The theoretical kernel Pγ encodes correspondences between input-space frequencies k and latent-space frequencies l through the manifold embedding γ.Its nonzero values are characterized using critical points involving the Jacobian of γ.
  • Mechanism: For the flower-shaped embeddings, increasing L allows larger latent frequencies to be expressed using the same input-space frequency.The coordinate functions of the embedding contain a frequency mode at L, providing the mechanism illustrated by the experiments.

5. Related Work

Related work situates the paper among results on neural-network expressivity, implicit regularization, harmonic analysis, and comparisons with kernel machines and nearest-neighbor methods.

  • Expressivity: Earlier work established that neural networks can approximate arbitrarily complex functions, including with width-bounded networks and many linear regions in deep ReLU networks.These results motivate studying properties beyond raw expressivity.
  • Implicit regularization: This paper differs from prior implicit-regularization work by focusing on a learning bias toward simple functions with dominant lower-frequency components.Its Fourier analysis also reflects width and depth effects on expressivity.
  • Harmonic analysis: Prior harmonic-analysis studies used transforms or Fourier-spectrum support to analyze neural-network approximation and complexity.The cited approaches include ridgelet methods and Fourier analysis of two-layer networks.
  • Comparative perspective: The discussion compares deep neural networks with kernel machines and K-nearest neighbors in smoothness, expressivity, and parameter efficiency.It reports that DNN spectra decay faster than those of KNNs.

6. Conclusion

The paper concludes that deep ReLU networks exhibit spectral bias toward smooth, low-frequency functions, while manifold geometry non-trivially affects high-frequency expressivity. It also identifies Fourier analysis as a promising tool for studying model sensitivity and complexity.

  • Conclusion: Neural networks can approximate arbitrary functions but favor low-frequency functions, producing a bias toward smooth functions called spectral bias.This conclusion summarizes the paper’s Fourier analysis of deep ReLU networks.
  • Conclusion: The geometry of the data manifold affects expressivity in a non-trivial way, including the expression of high-frequency functions on complex manifolds.The conclusion connects manifold geometry to the paper’s expressivity results.
  • Future work: Fourier transforms may help quantify how quickly a function changes locally and thereby analyze model sensitivity and complexity.The paper presents this as a direction for future research.

A.1. Experiment 1

The experiment trains a six-layer, 256-unit ReLU network on multi-frequency sine-wave targets and tracks Fourier components during full-batch optimization. The setup compares equal-amplitude targets with targets whose higher frequencies have larger amplitudes.

  • Experimental setup: A 6-layer ReLU network with 256 units per layer is trained to regress a superposition of sine waves with frequencies 5, 10, 15, ..., 50.The target is sampled at 200 uniformly spaced points in [0, 1].
  • Experimental setup: The experiment uses equal amplitudes Ai = 1 for all frequencies in one setting and amplitudes increasing from 0.1 to 1 in the other.The second setting assigns larger amplitudes to higher frequencies.
  • Optimization: The network is trained for 80000 full-batch gradient-descent steps with Adam, using the sampled target values as supervision.The passage specifies Adam rather than stochastic gradient descent.
  • Measurements: The normalized Fourier magnitudes and layerwise weight spectral norms are averaged over 10 runs with different random phases.The spectral norms are estimated with 10 power iterations.
  • Measurements: The experiment monitors the network spectrum during training and visualizes intermediate learned functions and loss curves.The spectrum is evaluated at the target frequencies as training progresses.
  • Generalized frequency analysis: A generalized frequency analysis uses Gaussian RBF kernel eigenvectors, whose increasing indices approximately correspond to increasing sinusoidal frequencies.The kernel matrix is eigendecomposed, and the eigenvector spectrum is evaluated on 50 uniformly spaced samples.

A.5. Qualitative Ablation over Architectures

The ablation tests how depth, width, and parameter norm affect a ReLU network’s ability to fit high frequencies. Depth and weight clip have stronger apparent effects than width, consistent with the spectral-bias theory.

  • A.5. Qualitative Ablation over Architectures: The experiment fits a δ-function whose constant spectrum tests the network’s ability to represent large frequencies.Depth, width, and weight clip are varied as architectural and parameter-norm factors.
  • A.5. Qualitative Ablation over Architectures: Increasing depth significantly improves the network’s ability to fit higher frequencies.The depth increases linearly in the ablation.
  • A.5. Qualitative Ablation over Architectures: Increasing width also helps fit higher frequencies, but its effect is considerably weaker despite exponential width increases.This comparison holds depth fixed.
  • A.5. Qualitative Ablation over Architectures: Increasing weight clip, or maximum parameter norm, helps the network fit higher frequencies.Constraining the Lipschitz constant prevents learning higher frequencies, supporting tightness of the O(Lf) bound.
  • A.5. Qualitative Ablation over Architectures: Across the ablations, lower frequencies are learned first, providing further evidence of spectral bias.The observations are reported as consistent with Theorem 1.

A.6. MNIST: A Proof of Concept

This section compares learning signals on a complex MNIST-derived manifold with learning them on a flat manifold of the same dimension. It also connects spectral bias to connected prediction paths in image space.

  • A.6. MNIST: A Proof of Concept: Learning random labels is harder when input samples lie on the same complex manifold than when they lie on a flat manifold of equal dimension.The comparison assumes the manifold hypothesis and uses white noise, whose expected spectrum is coordinate-system independent.
  • A.6. MNIST: A Proof of Concept: The MNIST manifold is approximated using a 64-dimensional denoising-autoencoder feature space embedded into the 784-dimensional input space.The decoder supplies the embedding used for the complex-manifold condition.
  • A.6. MNIST: A Proof of Concept: The flat comparison samples 784-dimensional vectors while setting all but the first 64 components to zero, with white-noise scalar targets.Two identical networks are trained under identical conditions, and loss curves are averaged over 10 runs.
  • A.6. MNIST: A Proof of Concept: The result complements earlier findings that random labels are easier to fit when random inputs occupy the full-dimensional input space.Here, the comparison instead holds manifold dimension below the ambient input dimension.
  • A.6. MNIST: A Proof of Concept: The paper motivates connected prediction regions by arguing that low-frequency bias makes isolated constant-prediction bubbles rare.The subsequent path experiment constructs connections among same-class and adversarial images.
  • A.6. MNIST: A Proof of Concept: Paths between all image pairs for all CIFAR-10 labels avoid every decision boundary, and their average length is only (3.0 ± 0.3)% above linear interpolation.The paths connect training and adversarial images as well as adversarial pairs.

B. The Continuous Piecewise Linear Structure of Deep ReLU Networks

The paper exploits the continuous piecewise-linear structure of ReLU networks to analyze their Fourier transforms. A recursive polytope-boundary construction explains the resulting frequency decay and its exceptional directions.

  • B. The Continuous Piecewise Linear Structure of Deep ReLU Networks: ReLU networks are continuous piecewise-linear functions whose linear regions are convex polytopes, with affine behavior on each region.The regions correspond to activation patterns, and every continuous piecewise-linear function can be represented by a ReLU network.
  • B. The Continuous Piecewise Linear Structure of Deep ReLU Networks: Because general ReLU networks may grow linearly rather than be square-integrable, their Fourier transforms are interpreted as tempered distributions when needed.This extends the Fourier analysis to networks whose ordinary Fourier transform does not exist as an L2 function.
  • B. The Continuous Piecewise Linear Structure of Deep ReLU Networks: The Fourier transform of a polytope is recursively expressed through Fourier transforms of its lower-dimensional boundary faces, with an O(k^-1) weight at each step.The recursion terminates when the frequency projection onto a face is zero.
  • B. The Continuous Piecewise Linear Structure of Deep ReLU Networks: The face-poset computation represents polytopes of different dimensions as nodes in a weighted directed acyclic graph and sums weighted contributions over face paths.Each path ends at the first face whose projected frequency is zero.
  • B. The Continuous Piecewise Linear Structure of Deep ReLU Networks: For generic frequencies, paths reach zero-dimensional vertices and yield an O(k^-d) spectrum, while exceptional orthogonal directions can produce dominant O(k^-r) terms with r < d.The exceptional directions lie on lower-dimensional subspaces determined by faces.

C.3. On Theorem 1

The section derives Fourier-spectrum behavior for ReLU networks and connects spectral decay to parameter gradients, training dynamics, and network smoothness constraints.

  • Theorem 1: The spectrum decays according to polytope geometry, with contributions supported in subspaces orthogonal to faces of the network’s linear regions.The relevant codimension depends on the direction of the Fourier vector.
  • Theorem 1: Relaxing the weight clip enables the network to fit higher frequencies, indicating that the O(L_f) dependence in the spectral bound can be tight.Here L_f is the maximum norm of the region-specific weight matrices.
  • Parameter-gradient decay: The parameter-gradient spectrum decays one power of k more slowly than the function spectrum, because differentiating phase terms contributes an additional factor proportional to k.If f = O(k^-∆−1), then ∂f/∂θ = O(k^-∆).
  • Training dynamics: The convergence analysis studies full-batch gradient descent on targets containing a single Fourier frequency and relates the residual and parameter-gradient magnitudes to that frequency.The target is represented by a Fourier-domain Kronecker delta.
  • Manifold effects: The manifold analysis assumes periodic, bounded embeddings and examines when oscillatory contributions can remain large for fixed input-domain frequency.Slow decay of the embedding’s Fourier coefficients can make the relevant expression arbitrarily large under stated conditions.

D. Volume of High-Frequency Parameters in Parameter Space

The section characterizes how parameter-space volume shrinks for parameters that express increasingly high frequencies, while contrasting this behavior with expressivity in deep networks and kernel machines.

  • Definition and monotonicity: The parameter set contributing more than ε to frequencies above cutoff k is denoted Ξ_ε(k) within a bounded parameter space.The sets are nested: higher cutoffs produce subsets of the sets for lower cutoffs.
  • Volume bound: The relative volume of Ξ_ε(k) is O(k^-∆−1), where 1 ≤ ∆ ≤ d, so high-frequency-expressing parameters occupy a shrinking fraction of parameter space.The exponent depends on the relevant Fourier direction through ∆.
  • Comparison with other models: Deep networks can approximate arbitrary target functions while using parameter-efficient piecewise-linear representations whose numbers of linear regions grow polynomially with width and exponentially with depth.This contrasts with the high-frequency parameter-volume result rather than denying universal approximation.
  • Comparison with other models: Kernel machines can also approximate arbitrary targets, but Gaussian-kernel counts scale linearly with the number of sign changes in the target function.The passage contrasts this scaling with the parameter efficiency of deep networks.

E.2. K-NN Classifier vs. DNN classifier

The comparison evaluates KNN and DNN frequency spectra on the same manifold-based classification task and finds that the DNN produces smoother predictions than the KNNs considered.

  • Experimental setup: The experiment trains KNN classifiers with various K values on a binarized k = 150 frequency signal defined on an L = 20 manifold.Probability predictions are evaluated over a box interval in R^2.
  • Spectral measurement: The frequency spectrum is obtained by applying a 2D FFT to the prediction map and integrating over angular components to produce ζ(k).The angle is parameterized by ϕ.
  • Results: The DNN matches a K = 1 KNN at lower radial frequencies, but its spectrum decays faster than every KNN spectrum considered.Faster spectral decay indicates smoother predictions in this comparison.
  • Results: Unlike DNNs, KNNs do not necessarily perform better for larger manifold complexity L, suggesting that KNNs do not exploit the manifold geometry in the same way.This conclusion comes from repeating the experiment associated with Figure 9 using KNNs.
Loading 1806.08734v3…