PaperlayerView exact revision on arXiv

Source-linked AI summary

On understanding and overcoming spectral biases of deep neural network learning methods for solving PDEs

Zhi-Qin John Xu, Lulu Zhang, Wei Cai

arXiv:2501.09987v1math.NAcs.NA

TL;DR

DNNs tend to learn low frequencies before high frequencies, complicating PDE solutions with wide frequency content. This review surveys approaches to address that bias and finds that many draw on classical approximation theories and numerical methods. The cited examples report frequency-wide or high-frequency capabilities in particular settings, while systematic analysis of frequency-dependent convergence remains limited.

  • Problem

    DNNs tend to fit functions from low to high frequencies, creating challenges for learning PDE solutions with a wide range of frequencies.

    Source: §1 · p. 3

  • Method

    The review surveys methods for overcoming spectral bias in DNN-based PDE learning and relates many of them to classical approximation theories and numerical methods.

    Source: §1 · p. 3, §8 · p. 32

  • Results

    PhaseDNN is reported to achieve uniform convergence across a broad frequency spectrum for wave propagation, while MscaleDNN captures high-frequency solution content that the compared FC-DNN failed to capture.

    Source: §4 · p. 12, §5.1.1 · p. 19

  • Takeaways & Limitations

    The reviewed methods trace roots to approaches including wavelets, wave-ray MGM, domain decomposition, multigrid, and multi-level methods.

    Source: §8 · p. 32

    The frequency dependence of convergence across PDE-learning formulations has received little study, and rigorous systematic comparisons of their pros and cons across problem types are lacking.

    Source: §2.2.2 · p. 6, §8 · p. 34

Abstract

from arXiv · show

In this review, we survey the latest approaches and techniques developed to overcome the spectral bias towards low frequency of deep neural network learning methods in learning multiple-frequency solutions of partial differential equations. Open problems and future research directions are also discussed.

1. Introduction

The review frames spectral bias as a key challenge for DNNs learning PDE solutions with wide frequency content, despite their potential for high-dimensional and complex-geometry problems. It surveys methods developed to address that challenge.

1. Introduction

  • DNN-based methods offer potential for high-dimensional problems and complex geometries where traditional numerical methods face challenges.The review attributes this potential to DNN compositional structure and meshless random sampling.Source: §1 · p. 2
  • Multigrid methods achieve frequency-uniform convergence for elliptic PDEs, while the review describes DNN-based PDE methods as still in early development.The review contrasts this mature multigrid capability with DNNs’ low-to-high frequency learning preference.Source: §1 · p. 3
  • DNN training typically learns low-frequency content before higher frequencies, challenging its use for PDE solutions with a wide frequency range.The review identifies overcoming this spectral bias as necessary for DNN-based PDE methods to approach the robustness and practicality of multigrid methods, including on high-dimensional problems.Source: §1 · p. 3
  • The review surveys background and methods developed to overcome spectral bias in DNN-based PDE learning.It identifies open research challenges and future directions as part of its scope.Source: §1 · p. 3

2. Preliminaries on DNN learning of PDE solutions

The preliminaries introduce DNN approximation and PDE-learning formulations, then contrast residual-based PINN and variational Deep-Ritz losses. These formulations have different frequency-training trade-offs, and a rigorous systematic comparison across problem types is lacking.

2.1. Deep neural network (DNN)

  • DNNs can approximate broad classes of functions, but finding satisfactory solutions in practice also depends on appropriate optimization methods.The cited approximation results include universal approximation under activation-function conditions and approximation of Barron-space functions without the curse of dimensionality.Source: §2.1 · p. 4

2.2. DNN based methods for solving PDEs

  • DNN-based PDE learning can use unlabeled empirical losses, with domain samples drawn at non-zero probability.For time-dependent problems, the text notes that accounting for causality can significantly improve solution performance and accuracy.Source: §2.2 · p. 5

2.2.1. PDE residual based learning – physics-informed neural network (PINN)

  • PINNs and related residual-based methods seek PDE solutions by minimizing least squares of the PDE residual.The described PINN formulation also uses a boundary loss to enforce boundary conditions, with interior and boundary samples drawn randomly at each iteration [p31][p32].Source: §2.2.1 · pp. 5–6

2.2.2. Variational energy based Deep-Ritz method

  • PINN differentiation may accelerate high-frequency convergence but can destabilize training, whereas Deep-Ritz can train more stably with slower high-frequency convergence.The review says both formulations generally retain preferential low-frequency convergence, and rigorous systematic comparisons across problem types are lacking.Source: §2.2.2 · p. 6

3. Spectral bias of deep neural networks

Deep neural networks tend to fit functions from low to high frequencies, challenging their use on solutions with a wide range of frequencies. The section reviews empirical evidence and qualitative and linear quantitative analyses of this spectral bias.

3. Spectral bias of deep neural networks

  • DNNs tend to fit target functions from low to high frequencies, making solutions with a wide range of frequencies challenging to learn.Source: §3 · p. 7

3.1. Empirical evidence of spectral bias

  • A three-frequency sine target is learned from low to high frequencies, and spectral bias also appears across high-dimensional settings, architectures, losses, and non-gradient optimizers.The cited experiments report spectral bias with L-BFGS and quasi-Newton methods as well as common optimization methods.Source: §3.1 · p. 7

3.2. Qualitative theory of spectral bias

  • For small training weights, activation-spectrum decay makes low frequencies dominate gradients; derivative-informed losses can counter the bias by upweighting high frequencies [42].The review says tanh and ReLU exhibit similar frequency-domain decay, so the bias is also observable with these activations.Source: §3.2 · p. 9

3.3. Linear quantitative theory of spectral bias

  • NTK-based linear-frequency analysis estimates convergence frequency by frequency and confirms the observed spectral bias under network linearization.The review presents linearization as an effective approximation as network width tends to infinity.Source: §3.3 · pp. 9–11

4. Frequency shifting approach

Frequency shifting addresses spectral bias by moving high-frequency components into a lower-frequency range for training, then mapping learned representations back to their original frequencies. PhaseDNN uses smaller subnetworks for frequency ranges and accurately approximates a high-frequency Helmholtz solution that a normal fully connected network fails to capture.

Frequency shifting approach

  • PhaseDNN uses phase shifts to convert high-frequency data components into a lower-frequency spectrum for learning, then reverses the shifts to recover the original frequencies.Its architecture comprises smaller sub-DNNs, each approximating a specific range of high-frequency content.Source: §4 · p. 12
  • For a band-limited function, decomposition into frequency bands lets a shift by each band’s center frequency move its spectrum to a low-frequency interval learnable by a small DNN.Once the shifted component is learned, removing the phase factor recovers its original component; combining the components approximates the function across the full frequency range.Source: §4 · pp. 12–13
  • For λ = 205 and μ = 3, PhaseDNN accurately approximates the high-frequency Helmholtz solution, while a normal fully connected network fails to capture its oscillations.The shift frequencies are selected as 0, λ, and μ; the shifts can be chosen as training hyperparameters.Source: §4 · p. 14, Figure 3 · p. 15
  • The survey also describes a prior-dictionary PINN approach [80] and a multiple-output network for representing PhaseDNN components.These approaches are presented as alternatives to using multiple networks for the PhaseDNN components.Source: §4 · pp. 14–15
  • PhaseDNN is effective for low-dimensional high-frequency functions, but the number of Fourier terms grows exponentially with dimension, creating high-dimensional difficulties.The shift frequencies may be selected as training hyperparameters.Source: §4 · pp. 14–15

5. Frequency scaling approach

Frequency scaling methods convert higher-frequency content to lower frequencies before training. The survey describes radial scaling in MscaleDNN and physical-scaling approaches that produce multiscale effects.

Frequency scaling approach

  • MscaleDNN uses radial scaling in the frequency domain, equivalently in the physical domain, to convert higher-frequency content to lower frequencies before training.Fourier feature networks are another related approach.Source: §5 · p. 15
  • Activation-function optimization and random-feature methods use physical scaling to achieve multiscale effects.Source: §5 · p. 15
  • Fourier feature networks are identified as another approach related to frequency scaling.Source: §5 · p. 15

5.1. Multiscale DNN (MscaleDNN)

MscaleDNN uses radial scaling and frequency-localized activations to help learn high-frequency content through lower-frequency functions. The section describes its architectures, reports success on oscillatory PDE examples, and presents a diffusion-model explanation for reduced spectral bias and guidance on scale selection.

5.1. Multiscale DNN (MscaleDNN)

  • MscaleDNN [85] uses radial scaling to turn high-frequency approximation into lower-frequency learning, while localized-frequency activations help separate frequency content among subnetworks.Source: §5.1 · p. 16

5.1.1. The idea of Multiscale DNN

  • The method partitions B(Kmax) into M concentric annuli, downscales each high-frequency region to a low-frequency one, and trains a DNN on the scaled function; scale factors can be trainable.Source: §5.1.1 · pp. 16–17
  • The first hidden-layer activation affects MscaleDNN performance; reported variants include sReLU, quadratic ReLU, sinusoidal, residual sinusoidal, and ellipse Gaussian RBF activations.An orthogonality penalty between low- and high-frequency output subspaces was also reported to improve PDE-solving error.Source: §5.1.1 · p. 19
  • On a 2-D oscillatory Poisson problem, MscaleDNN captures high-frequency solution content while the vanilla FC-DNN fails; Liu et al. also report success on high-frequency fitting and PDEs with complex geometry.The comparison uses eight subnetworks with scales {1, 2, 4, 8, 16, 32, 64, 128}, each sized 2-400-400-400-1, against an FC-DNN sized 2-3200-3200-3200-1.Source: §5.1.1 · pp. 18–19, Figure 5 · p. 20

5.1.2. Spectral bias reduction property of the MscaleDNN, and selection of scales.

  • Under a vanishing learning rate and an infinite-width, two-layer sine-activation assumption, a derived diffusion model matches practical MscaleDNN error decay on oscillatory-function approximation [95].Source: §5.1.2 · pp. 20–21
  • Diffusion-coefficient support expands to higher frequencies as the scale index s increases, providing a mathematical explanation for MscaleDNN spectral-bias reduction; scale values should depend on the target solution.Dynamically selected scales were reported to improve DNN solutions for PDEs with oscillatory solutions [96].Source: §5.1.2 · pp. 21–22, Figure 6 · p. 21

5.2. Fourier feature networks

Fourier feature networks lift inputs through fixed sinusoidal features, with frequency parameters set by fixed choices or sampled distributions. The survey relates this representation to MscaleDNN and describes variants used in PDEs and other applications.

5.2. Fourier feature networks

  • Fourier feature networks set first-layer weights to fixed values, biases to zero, and use sinusoidal activations to lift the input into a feature representation.Source: §5.2 · p. 22
  • With unit amplitudes and scale-based frequencies, Transformer positional encoding resembles MscaleDNN scales, while Gaussian-sampled frequencies are another option.A multiscale input structure was used for neural radiance fields in view synthesis; the resulting network is MscaleDNN with sine activation.Source: §5.2 · p. 22
  • PDE-oriented Fourier-feature extensions include Gaussian-sampled frequencies and multi Fourier feature networks with a Gaussian-sampled matrix and tunable standard deviation.Another reported representation combines Fourier features with scaled inputs [101].Source: §5.2 · p. 22

5.3. Activation function optimization

This section surveys activation-function strategies for mitigating spectral bias, from adaptive scaling to richer neuron-level and network-level activation choices.

5.3. Activation function optimization

  • Activation-function optimization seeks to improve DNN accuracy on multiple-frequency functions and mitigate spectral bias.The surveyed choices include B-spline and sinusoidal activations, alongside standard activation functions.Source: §5.3 · p. 23

5.3.1. Adaptive activation function

  • Adaptive activation σ(na·) uses a trainable scale a and integer n; numerical results validate learning up to frequencies such as sin(12x).The scaling factor plays a role similar to the scales in MscaleDNN.Source: §5.3.1 · p. 23

5.3.2. Reproducing activation functions

  • Reproducing activation functions combine basic activations through learnable linear combinations, with poly-sine-Gaussian activations reported effective for mixed-frequency solutions [108].The reported PDE example includes sin(2πx1) sin(2πx2), and the poly-sine-Gaussian choice uses {x, x2, sin(βx), e−(βx)2}.Source: §5.3.2 · pp. 23–24

5.3.3. Multiple activation neural network

  • Kronecker neural networks enrich each neuron with a collection of activations, and their Rowdy-Net variant adds scaled sine or cosine functions.Theory shows faster initial loss decay than feed-forward networks under suitable conditions.Source: §5.3.3 · p. 24

5.4. Random Feature Method

The random feature method combines random neural-network basis functions with local partition-of-unity models, optionally adding a global component to capture large-scale solution features.

5.4. Random Feature Method

  • Random feature models build local solutions around partition-of-unity centers and piece them together to represent local PDE variations.Each local model uses random feature functions with fixed randomly chosen parameters; the scaling coefficient helps cover a wider frequency range.Source: §5.4 · p. 25
  • A global component can be added above the PoU-based local basis when that basis alone captures large-scale solution features less efficiently.This yields a multi-scale basis combining local and global components.Source: §5.4 · p. 26
  • A random feature method solves a low-dimensional PDE with wide frequency range at almost machine accuracy, including on complex geometries.The method is meshless and uses QR-based least squares rather than gradient-based training, which the review identifies as the reason for machine accuracy.Source: §5.4 · p. 26

6. Hybrid approach

Hybrid methods combine neural and classical iterative solvers to address their complementary frequency behavior: DNNs favor lower frequencies, while iterative methods reduce high-frequency errors.

6.1. Jacobi-DNN

  • Jacobi-DNN first trains a DNN approximation for Poisson’s equation, then applies Jacobi iterations from that initialization to reduce high-frequency error.Related deep-learning-initialized iterative approaches are reported effective for low-dimensional PDE and eigenvalue problems [111].Source: §6.1 · p. 27

6.2. HINTS

  • HINTS alternates a trained DeepONet iterator with a numerical iterator, balancing convergence across eigenmodes and improving computational efficiency.The DeepONet is trained offline; the numerical iterator is used for a proportion 1−1/nr of iterations, while DeepONet takes proportion 1/nr.Source: §6.2 · pp. 27–28

6.3. MgNet, Multi-level, Multi-grade Nets

  • MgNet replaces multigrid prolongation, restriction, and selected smoothing operations with trainable CNNs, retaining fine- and coarse-grid connections between frequency ranges [114, 115].Because it uses grids, MgNet is limited to low-dimensional PDEs.Source: §6.3 · pp. 28–29
  • A hierarchical approach recursively trains additional networks on residuals, each correcting the sum of the preceding networks [116].A later approach assigns coefficients to the networks to improve accuracy.Source: §6.3 · p. 29
  • Multi-grade learning feeds each network the previous network’s output and has been empirically reported faster than one-grade learning on multi-frequency problems [118].Source: §6.3 · p. 29

7. Multi-scale neural operators and diffusion models

The review describes multiscale neural operators and diffusion-based approaches for addressing spectral bias in operator learning. In a 1-D Helmholtz example, MscaleFNO captures oscillations and reaches lower relative error than Normal FNO after 500 training epochs.

7. Multi-scale neural operators and diffusion models

  • Neural operators learn mappings from physical input functions to solution functions and can predict for unseen inputs after training.The cited discussion notes that learning an operator can avoid retraining a solution-parameterizing network for different materials or boundary conditions.Source: §7 · p. 30
  • Multiscale FNO and multiscale DeepONet use scaled inputs to alleviate spectral bias in neural operators [124] [125].Source: §7 · p. 30
  • A hierarchical attention neural operator leveraging diffusion models was proposed to address spectral bias, and generative modeling techniques have been used to learn wide-frequency PDE operators [129] [130].Source: §7 · pp. 31–32
  • After 500 epochs, MscaleFNO relative error is approximately 4 × 10−2, while Normal FNO remains around 8 × 10−1.The figure also shows Normal FNO failing to capture oscillations, while MscaleFNO captures global behavior and local detail.Source: Figure 7 · p. 31, Figure 8 · p. 32

8. Conclusions and open problems

The review synthesizes methods for mitigating spectral bias in DNN-based PDE learning and identifies open questions about training dynamics, frequency-dependent convergence, and practical solver design.

8. Conclusions and open problems

  • Reviewed methods often build on classical ideas, including wavelets, wave-ray multigrid, domain decomposition, multigrid, and multilevel approaches.Examples include multiscale DNNs, Fourier features, PhaseDNN, Random feature methods, and Mg-Net.Source: §8 · p. 32
  • Understanding how network weights evolve during training and relate to interpolation and expressivity may guide models and analysis tools for mitigating spectral bias.The review identifies information bottleneck and weight condensation results as starting points for this work, particularly for high-dimensional PDEs and complex geometries.Source: §8 · pp. 32–33
  • Training studies associate the fitting phase with low-frequency learning and the diffusion phase with high-frequency learning, motivating tests across the reviewed DNN models [135].The fitting phase decreases loss and increases input-output mutual information; the diffusion phase is dominated by noise and chaotic weight evolution.Source: §8 · p. 33
  • In two-layer ReLU networks, neurons can condense onto a few directions early in training, limiting expressivity to very low-frequency content before more directions emerge [137] [138] [140] [141].The review identifies how new DNN models alter this condensation process as an open question for understanding bias mitigation and training efficiency.Source: §8 · pp. 33–34
  • Quantitative frequency-domain convergence analysis remains unresolved for nonlinear networks, while suitable tools for varied PDE losses, min-max training, SDE-based DNNs, and operator learning remain open problems.Existing rigorous frequency analysis covers linear-regime fully connected networks and two-layer multiscale networks; the review anticipates further PINN analysis.Source: §8 · p. 34
  • The review does not address hyperparameter selection, including loss penalties, activation functions, and learning rates, although progress on spectral bias may have broad application impact.The authors describe further algorithmic and mathematical progress as potentially far-reaching across application areas.Source: §8 · pp. 34–35

Related papers

Loading 2501.09987v1…