Source-linked AI summary
On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
Sifan Wang, Hanwen Wang, Paris Perdikaris
TL;DR
PINNs struggle with high-frequency and multi-scale PDE solutions because their learning is biased toward NTK-dominant eigen-directions. The paper uses NTK analysis to design spatio-temporal and multi-scale Fourier-feature architectures, which improve performance on challenging forward and inverse problems, while requiring frequency-scale selection informed by the target solution.
Problem
PINNs struggle to approximate high-frequency or multi-scale functions because spectral bias favors lower-frequency components, limiting their performance on such PDE problems.
Method
The paper analyzes limiting NTKs and uses spatio-temporal and multi-scale Fourier feature mappings to construct PINN architectures for multi-scale PDEs.
Results
The proposed methods demonstrate effectiveness on challenging multi-scale forward and inverse PDE problems, including benchmarks where conventional PINNs fail.
Takeaways & Limitations
NTK analysis provides a principled basis for designing PINN architectures and training approaches for PDE solutions with complex multi-scale spatio-temporal features.
Takeaways & Limitations
The architectures require careful selection of the number and scale of Fourier feature mappings, using prior knowledge about the target PDE solution’s frequency distribution.
Abstract
from arXiv · showhide
Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction-diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at \url{https://github.com/PredictiveIntelligenceLab/MultiscalePINNs}.
1 Introduction
PINNs can struggle to train on high-frequency or multi-scale PDEs because spectral bias favors lower-frequency components. This work analyzes the issue through NTK theory and proposes Fourier-feature-based architectures to address it.
- PINNs have shown broad promise across computational-science applications, but their effectiveness and failure mechanisms remain mathematically insufficiently understood.
- High-frequency and multi-scale PDEs can make PINN training increasingly stiff, producing unbalanced gradients across loss terms.
- Spectral bias prevents conventional fully connected networks from accurately learning high-frequency functions, and NTK analysis identifies it as a leading cause of PINN failure on multi-scale targets.
- The paper studies fully connected networks and PINNs through limiting NTKs to design models for multi-scale PDEs.
- The authors interpret spectral bias as NTK eigenvector bias, show Fourier mappings modulate NTK-eigenvector frequencies, and use this analysis to engineer multi-scale architectures.
- The proposed work evaluates its architectures on benchmarks where conventional PINNs fail, including representative multi-scale problems.
2 Physics-informed neural networks
PINNs approximate unknown PDE solutions with neural networks trained to fit data and satisfy differential equations and boundary conditions. A 1D multi-scale Poisson example demonstrates that conventional training can fail despite extensive optimization.
- PINNs represent an unknown quantity u(x) governed by a PDE and boundary conditions, with time treated as an additional spatio-temporal coordinate when needed.
- The unknown solution is approximated by a deep neural network whose parameters are optimized using a composite physics-informed loss.
- Boundary and residual collocation points are randomly sampled in the computational domain at each gradient-descent iteration.
- The illustrative Poisson solution combines low-frequency macro-scale structure with high-frequency micro-scale structure.
- After 10^7 gradient-descent iterations, the conventional PINN remains unable to learn the correct solution, although a fully connected network can approximate it with sufficient interior data.
- Figure 1 compares predicted and exact solutions, pointwise error, residual and boundary losses, and relative L2 error during training.
3 Methodology
The paper analyzes spectral bias through NTK eigenspaces and uses Fourier feature embeddings to modulate which frequencies neural networks learn efficiently. It then develops multi-scale and spatio-temporal architectures for PINNs that address high-frequency and multi-scale PDE solutions without adding trainable parameters.
- 3.1 Analyzing spectral bias through the lens of the Neural Tangent Kernel: NTK eigenvalues determine the convergence rates of target-function components, so networks learn first along eigenvectors associated with larger eigenvalues.For conventional fully connected networks, eigenvalues decrease as eigenfunction frequency increases, producing spectral bias against high-frequency components.
- 3.2 Fourier feature embeddings: Random Fourier features embed coordinates using Gaussian frequencies controlled by σ, with larger σ producing higher-frequency NTK eigenfunctions and narrower eigenvalue gaps.This reshapes the NTK eigenspace and can accelerate learning of high-frequency target components.
- 3.2 Fourier feature embeddings: Fourier feature mappings must be scaled appropriately because overly large σ can overfit sparse data with high-frequency oscillations.The network can fit all training pairs while failing to recover the intended low-frequency target function.
- 3.3 Multi-scale Fourier feature embeddings for physics-informed neural networks: For PINNs, NTK eigenvectors help explain why fully connected networks learn PDE solutions and residuals from low to high frequencies, hindering multi-scale problems.The limited number of interior data points makes this frequency preference especially consequential for PDE approximation.
- 3.3 Multi-scale Fourier feature embeddings for physics-informed neural networks: The proposed architectures combine multiple Fourier embeddings with different σ values, including separate spatial and temporal embeddings for spatio-temporal problems.Outputs are combined through concatenation and a linear layer, or through point-wise multiplication followed by a linear layer; neither design adds trainable parameters or significantly increases floating-point operations.
4 Results
The experiments show that conventional PINNs and single-scale Fourier features struggle with multi-scale or high-frequency PDE solutions, whereas appropriately designed multi-scale spatio-temporal features can achieve accurate predictions. Adaptive loss weighting is additionally necessary for the wave example.
- 1D Poisson equation: The multi-scale Fourier feature architecture approximates the 1D Poisson solution with a relative L2 error of 1.36e-03.The model uses two mappings initialized with σ1 = 1 and σ2 = 10.
- 1D Poisson equation: Plain PINNs and single Fourier feature mappings produce relative L2 errors ranging from 10% to above 100% on the 1D Poisson problem.σ = 1 captures low frequencies, whereas σ = 50 captures high-frequency oscillations but misses low-frequency content.
- High frequencies in a heat equation: A spatio-temporal Fourier feature architecture captures the heat-equation oscillations with a relative L2 error of 1.78e-03.The spatial and temporal mappings are initialized with σ = 200 and σ = 1, respectively.
- High frequencies in a heat equation: Single Fourier feature mappings and conventional PINNs yield relative L2 errors around 100% for the high-frequency heat equation.The tested single mappings use σ ∈[1, 1000].
- Wave propagation: For wave propagation, the spatio-temporal Fourier feature network initially fails, while adaptive weighting produces agreement with the exact solution and relative L2 error within 0.1%.Both experiments use the same network, Fourier mappings, and hyper-parameter settings; the difference is adaptive loss weighting.
- 2D Gray-Scott equation: The proposed architecture reconstructs Gray-Scott concentration fields accurately, whereas conventional PINNs fail to learn the fields and infer the unknown diffusion rates.Both approaches are trained for 120,000 gradient-descent iterations under the reported network settings.
5 Discussion
The discussion frames Fourier feature networks through NTK eigenvectors and presents the resulting architectures as a principled approach for multi-scale, time-dependent PDEs. It also acknowledges that selecting feature scales requires prior spectral information and that broader theory remains open.
- Discussion: Fourier feature mappings determine the frequency of the eigenvectors of the resulting limiting NTK.This links the analysis of Fourier feature networks to mechanisms underlying spectral bias.
- Discussion: The proposed architectures target forward and inverse time-dependent PDEs whose solutions contain complex multi-scale spatio-temporal features.The paper evaluates them on benchmarks where conventional PINN approaches fail.
- Limitations: The architectures require choosing the number and scale of Fourier feature mappings so that NTK eigenvector frequencies roughly match the target solution.This requires prior knowledge of the target PDE solution’s frequency distribution, which may be unavailable in complex dynamical systems.
- Future directions: The paper leaves open rigorous theories for general-network NTK eigensystems, PINN-specific eigensystems, and feature embeddings for shocks or boundary layers.It also raises initialization questions related to the location of desirable minima in the loss landscape.
A Definition of fully-connected neural networks
The appendix defines a scalar-valued fully-connected neural network recursively across hidden layers and specifies its output and NTK parameterization.
- Network definition: A scalar-valued fully-connected network is defined recursively using weight matrices, bias vectors, and a coordinate-wise smooth activation function.The recursion applies across hidden layers h = 1, . . . , L.
- Network output: The final network output is obtained by applying the last-layer weights and bias to the final hidden representation.The last-layer parameters have dimensions W^(L) ∈R1×dL and b^(L) ∈R.
- Parameterization: All network parameters are initialized as independent Gaussian variables N(0, 1) under the NTK parameterization.The parameter set θ includes weights and biases from the input through the final layer.
B Proof of Lemma 3.1
The supplied proof passages show a derivative-based argument involving the kernel and coordinate variables, ending with the statement that the proof is complete.
- Derivative calculations: The proof differentiates the kernel with respect to each coordinate x_l.The passages first introduce x = (x1, . . . , xl, . . . , xd), then take first and second derivatives.
- Conclusion: The lemma proof concludes after the stated derivative calculations.The supplied proof text ends with “This concludes the proof.”
C Proof of Proposition 3.2
The proof derives the eigenfunctions of the cosine kernel by converting the eigenvalue equation into an ordinary differential equation and then determining eigenvalues through a two-dimensional linear system. The resulting eigenvalue calculation is expressed through the characteristic polynomial of the associated matrix.
- The eigenfunction satisfies g′′(x) = −b2g(x), so it has the form g(x) = C1 cos(bx) + C2 sin(bx).The constants C1 and C2 are arbitrary coefficients.
- Substituting this trigonometric form into the kernel equation reduces the calculation to integrals involving products of sine and cosine terms.The cosine difference identity separates the expression into cosine and sine components.
- The coefficients satisfy a two-dimensional linear system obtained by matching the cosine and sine components of the kernel equation.The system is written in terms of the constants C1 and C2 and the integral quantities I1, I2, and I3.
- The kernel eigenvalue is determined by an eigenvalue of the associated matrix A through its characteristic polynomial.The proof then evaluates the polynomial using expressions involving b and sin(2b).
- The proof concludes after identifying the resulting eigenvalue relationship for the cosine kernel.
D Proof of Proposition 3.3
The proof uses the block structure and positive definiteness of the kernel matrices at zero to establish orthogonal diagonalizations with strictly positive diagonal eigenvalue matrices. Their invertibility then completes the argument.
- Assumption (i) in Proposition 3.3 directly yields the initial block-kernel relation used in the proof.
- Assumption (ii) states that Kuu(0) and Krr(0) are positive definite and therefore admit orthogonal diagonalizations.
- The diagonal matrices Λu and Λr contain the eigenvalues of Kuu(0) and Krr(0), respectively.Because all eigenvalues are strictly positive, both diagonal matrices are invertible.
- The proof ends after using these properties to establish the proposition.