PaperlayerView exact revision on arXiv

Source-linked AI summary

Operator-informed initialization for Fourier features physics-informed neural networks

Juan Molina, Paris Perdikaris, Mircea Petrache, Matías Courdurier, Francisco Sahli Costabal

arXiv:2610.03378v1cs.LG

TL;DR

PINNs can learn high-frequency structure slowly, motivating an analysis of frequency-dependent residual dynamics. The paper derives those dynamics for Fourier-feature PINNs and uses the PDE symbol to guide initialization. Experiments report improved learning behavior across several PDEs, with benefits depending on the geometry of weakly constrained frequency regions.

  • Problem

    PINNs can converge slowly and lose predictive accuracy on sharp-gradient, localized, or oscillatory solutions, difficulties linked to neural-network frequency bias.

    Source: Introduction · pp. 1–2

  • Method

    The paper derives residual frequency dynamics for Fourier-feature PINNs and adapts the initialization sampling distribution to the geometry induced by the PDE symbol.

    Source: §7 · p. 22

  • Results

    The initialization mitigates frequency bias and improves optimization and residual convergence, with larger gains for transport, wave, and Helmholtz equations than for Laplace, whose strategies show comparable dynamics.

    Source: §7 · pp. 22–23

  • Takeaways & Limitations

    The advantage is more pronounced when weakly constrained spectral regions have extended geometry not efficiently captured by isotropic Gaussian initialization.

    Source: §7 · pp. 22–23

    The method relies on a meaningful PDE symbol and is naturally applicable to linear and semilinear constant-coefficient PDEs; fully nonlinear PDEs generally lack a unique linear symbol for this construction.

    Source: §7 · p. 23

Abstract

from arXiv · show

Physics-Informed Neural Networks (PINNs) typically exhibit spectral bias, where some frequencies of the target function converge more slowly than others. In this work, we analyze the training dynamics of Fourier Feature PINNs in the Neural Tangent Kernel regime to address this limitation. We derive an explicit evolution equation to estimate the residual error in the frequency domain, demonstrating that the convergence rate of specific frequencies is primarily governed by the product of the differential operator's symbol and the spectral density of the initialization weights. Leveraging this theoretical insight, we propose an informative initialization strategy that tailors the initial weight distribution to the specific PDE being solved. With this method, we can diminish the operator-induced spectral bias, balancing the convergence rates across the frequency spectrum and achieving better prediction accuracy. Numerical experiments on linear and nonlinear partial differential equations confirm that this initialization strategy improves learning dynamics and approximation accuracy across frequencies compared to standard initialization methods, with no additional training cost.

Introduction

PINNs combine data-driven learning with PDE-residual minimization, but can converge slowly on solutions with sharp gradients, localized structures, or oscillations. These difficulties are linked to neural networks’ tendency to learn low frequencies before high frequencies.

Introduction

  • PINNs embed governing physical laws in the training objective and combine data-driven learning with PDE-residual minimization.They provide an alternative to classical numerical solvers and have applications including fluid mechanics and bioengineering.Source: Introduction · p. 1
  • Sharp gradients, localized structures, and highly oscillatory targets can bring slow convergence and reduced predictive accuracy in PINNs.These difficulties are closely related to neural networks’ intrinsic frequency bias.Source: Introduction · pp. 1–2
  • Frequency bias is the tendency of gradient-trained neural networks to learn low-frequency components earlier and faster than high-frequency ones.It is also called spectral bias or the frequency principle.Source: Introduction · p. 1
  • Understanding the mathematical mechanisms governing frequency bias in PINNs is important for improving their efficiency and predictive capabilities.Many physically relevant solutions have multiscale or broadband frequency content.Source: Introduction · p. 2

1. Background

The background reviews frequency-bias research, PINN-specific training dynamics, and Fourier-feature networks as a controllable spectral representation. It also situates the paper among initialization and frequency-aware approaches to mitigating bias.

Background

  • Frequency-bias studies span eigenvalue and data-distribution analyses, infinite-width characterizations, and continuous Fourier-domain descriptions of individual modes (Ronen, Jacobs, Kasten and Kritchman, 2019; Cao, Fang, Wu, Zhou and Gu, 2021; Luo, Ma, Xu and Zhang, 2022b; Luo, Ma, Wang, Xu and Zhang, 2022a; Zhang, Luo, Ma and Xu, 2021; Molina, Petrache, Costabal and Courdurier, 2026; Luo et al., 2022b).The cited literature also includes shallow-network analysis (Xu, 2018).Source: §1 · p. 2
  • PINN frequency bias is complicated by a loss combining PDE residuals with boundary, initial, and data constraints, whose components can converge at different rates (Wang, Yu and Perdikaris, 2022).Frequency-dependent effects specific to PINNs have also been reported (Xu, Zhang and Cai, 2025).Source: §1 · p. 2
  • Fourier-feature networks expose frequency content through a prescribed sampling distribution, allowing spectral properties to be changed without changing the subsequent architecture.Their frequency distribution also controls the kernel’s spectral properties and effective bandwidth in the NTK regime (Tancik et al., 2020).Source: §1 · p. 2
  • In the infinite-width NTK regime, gradient-descent training has a deterministic kernel description and a linear function-space evolution equation (Jacot, Gabriel and Hongler, 2018; Lee, Xiao, Schoenholz, Bahri, Novak, Sohl-Dickstein and Pennington, 2019; Arora, Du, Hu, Li, Salakhutdinov and Wang, 2019b; Arora, Du, Hu, Li and Wang, 2019a).The framework has been used to study PINN dynamics and architectural effects on spectral convergence.Source: §1 · pp. 2–3
  • Prior mitigation strategies include PDE-agnostic initialization, pretrained-weight transfer, frequency transformations, domain decomposition, and adaptive Fourier encodings (Liu, Zhang, Peng, Zhou and Yao, 2022; Tarbiyati and Nemati Saray, 2024; Cai, Li and Liu, 2020; Cai, Zhu, Shen, Wang and Cao, 2024; Li et al., 2023; Liu et al., 2025).The paper contrasts these methods’ problem-specific or heuristic frequency manipulations with deriving dynamics from the operator.Source: §1 · p. 3
  • The paper presents a unified framework deriving frequency-dependent dynamics from the differential operator, in contrast to approaches tailored to particular PDE classes, architectures, or training settings.The initialization principle adapts to the PDE symbol without redesigning the architecture or training procedure for each problem.Source: §1 · p. 3

Main Contributions

The paper analyzes residual frequency dynamics for Fourier-feature PINNs, proposes initialization guided by the PDE symbol, and tests its effects across PDEs and training settings.

Main Contributions

  • The theoretical framework describes residual frequency dynamics for two-layer Fourier-feature networks in the NTK regime and links convergence to the PDE symbol and initial parameter density.Frequency regions where the symbol is small or vanishes strongly influence residual convergence, providing an explanation of operator contributions to frequency bias in PINNs.Source: Main Contributions · p. 3
  • The PDE-informed initialization adapts first-layer parameter sampling to the geometry induced by the PDE symbol.Source: Main Contributions · p. 3
  • The proposed initialization consistently mitigates frequency bias, accelerates optimization, improves residual convergence, and reduces variability across random initialization seeds.The theoretical predictions are validated through numerical experiments on several PDEs.Source: Main Contributions · p. 3
  • The method remains effective for nonlinear PDEs, deep networks, and ADAM optimization, and can improve established PINN frameworks without additional training cost.The initialization modifies the sampling distribution during initialization.Source: Main Contributions · p. 3

2. Problem setup

The setup extends a neural-network and gradient-flow framework to PINNs and focuses on the interior differential operator’s contribution to training dynamics. It defines the PDE, network, loss, and NTK assumptions used to derive linearized residual dynamics.

2. Problem setup

  • The analysis extends the neural-network and gradient-flow framework of Molina et al. (2026) to the PINN setting.Source: §2 · p. 3

2.1. Operator formulation of the PDE

  • For time-dependent problems, the solution is defined on a space–time domain; for stationary problems, the domain is spatial, and the PDE is written in operator form with boundary and applicable initial conditions.The analysis focuses on the interior differential operator because it captures the main mechanism underlying the method’s spectral behavior.Source: §2.1 · p. 4

2.2. Neural network model and training dynamics

  • The solution is approximated by a single-hidden-layer neural network, and the practical PINN loss combines the PDE residual with boundary and initial-condition constraints.The setup assumes positive boundary and initial-condition loss weights.Source: §2.2 · p. 4

2.3. Linearized residual dynamics

  • In the infinite-width NTK setting, the residual dynamics generalize classical NTK dynamics by incorporating the adjoint differential operator as a differential preconditioner.The assumptions include i.i.d. parameter initialization from zero-mean, finite-variance distributions; the derivation also assumes sufficient regularity and neglects boundary terms.Source: §2.2 · p. 4, §2.3 · p. 5

3. Frequency-domain formulation

The frequency-domain analysis assumes a linear constant-coefficient differential operator and sufficiently dense, approximately uniform data sampling. It derives residual-frequency dynamics in which the operator symbol modulates spectral behavior.

3. Frequency-domain formulation

  • Fourier transformation converts derivatives of a linear constant-coefficient operator into multiplication by a frequency-variable polynomial.For the symbol of D*_xD_x, the passage specifies a non-negative frequency-dependent response.Source: §3 · p. 5
  • The proof transforms the residual dynamics into frequency space and applies Parseval’s identity.The proof is described as analogous to Luo et al. (2022a), with details in Appendix A.Source: §3 · pp. 5–6
  • The operator symbol modulates residual-frequency dynamics under the stated assumptions, as formalized by the frequency-domain result.The lemma's dynamics ignore boundary contributions; the assumptions include a linear constant-coefficient operator and sufficiently dense, approximately uniform sampling.Source: §3 · pp. 5–6

4. Frequency dynamics for Fourier Feature networks

For Fourier Feature networks, the residual dynamics in frequency space are modulated by both the PDE symbol and the initialization distribution. Extending the analysis to full PINN losses introduces nonlocal boundary and initial-condition effects, illustrated for the heat equation.

4. Frequency dynamics for Fourier Feature networks

  • Under the NTK assumptions, Fourier Feature residual dynamics take the form of a diffusion-reaction equation modulated by the PDE symbol and initialization distribution.In this setting, the dynamics operator is local and has relatively weaker cross-frequency influence during training.Source: §4 · p. 6

4.1. Extension to full PINNs losses

  • Adding boundary and initial-condition losses introduces nonlocal trace operators into the frequency dynamics, alongside boundary contributions from integration by parts.The boundary and initial operators generally have nonlocal Fourier representations.Source: §4.1 · p. 6

4.2. Example: heat equation in one spatial dimension

  • For the heat equation with Dirichlet boundaries, the boundary operator couples spatial frequency modes through convolution with the kernel cos(2π⋅).The initial-condition operator is also a trace at τ = 0 and leads to another nonlocal operator.Source: §4.2 · p. 7

5. Qualitative analysis of the residual frequency dynamics

Residual frequencies evolve according to both initialization density and the PDE symbol, while boundary and initial conditions remain essential where the interior operator provides weak control. The paper uses this analysis to design PDE-informed initialization distributions that balance concentration near relevant symbol level sets with broader frequency coverage.

5. Qualitative analysis of the residual frequency dynamics

  • In the frozen-feature regime, frequencies receiving little initialization probability mass evolve slowly or, in the limit, not at all.The wide-network NTK regime predicts first-layer parameters remain close to initialization, making this simplified setting relevant to full dynamics.Source: §5 · pp. 7–8
  • Frequencies where the PDE symbol is small receive weak interior-residual control, so boundary and initial-condition terms remain essential to determine the physically admissible solution.Where the symbol vanishes, the interior contribution vanishes and evolution depends on the remaining terms.Source: §5 · p. 8
  • Figure 2 shows residual errors persisting where the PDE symbol is small, and training dynamics deteriorate as the initialization polynomial becomes more misaligned with the true operator symbol.The interior symbol specifies spectral geometry; it does not imply that the interior term dominates dynamics in those regions.Source: §5 · p. 8, Figure 2 · p. 9
  • For homogeneous linear constant-coefficient PDEs on the whole space, the solution spectrum is supported within the zero set of the PDE symbol.On bounded domains, spatial localization spreads an ideal spectrum into a neighborhood of that set; a source may amplify components in low-symbol regions.Source: §5 · p. 8

5.1. PDE-informed initialization to mitigate the frequency bias

  • The proposed initialization concentrates smoothly near low-level sets of the PDE symbol while retaining a Gaussian component for broader isotropic frequency coverage.The broader component is useful for nonlinear problems, where interactions can generate frequencies not captured by the principal linear symbol.Source: §5.1 · pp. 9–10
  • For linear PDEs the method uses pure PDE-informed initialization, whereas nonlinear PDEs use λ = 0.5 to balance it with the Gaussian component.For nonlinear PDEs, a pure PDE-informed initialization may become overly specialized during long training because nonlinear interactions generate additional frequencies.Source: §5.1 · p. 10
  • An inverse-symbol weighting is sensitive to ε near symbol zeros, may overconcentrate sampled frequencies, and performed worse than the proposed exponential construction.This alternative was considered as a formal way to compensate directly for the symbol factor in interior convergence rates.Source: §5.1 · p. 10

6. Numerical Experiments

The numerical experiments assess PDE-informed Fourier-feature initialization across PINN benchmarks and settings, focusing on convergence, prediction accuracy, and seed-to-seed variability. They report improvements in frequency-dependent learning disparities, with gains varying across benchmarks and training configurations.

6.1. Dependence of the learning rate on the initialization weights.

  • Increasing high-frequency sampling raises PDE residuals proportionally to w² and the dominant loss-Hessian eigenvalue approximately as maxₖ wₖ⁴; stable gradient descent requires jointly choosing frequency distribution and learning rate.The grid’s Nyquist bound is approximately 50, while the wave example’s solution contains frequencies only up to 5.5, so rejection sampling covers a substantially wider range than necessary.Source: §6.1 · p. 11, §6.1.1 · pp. 12–13

6.2. Training dynamics and solution accuracy

  • On multiscale transport, Gaussian σ_w=8–12 often gives low training loss but large prediction error, whereas σ_w=2–6 yields more reliable approximations.Small empirical PINN loss need not indicate an accurate solution because collocation loss and domain-wide relative L2 error measure different things.Source: §6.2 · p. 13, Figure 4 · p. 14

6.3. PDE-Informed vs. Standard Initialization

  • PDE-informed initialization gives comparable dynamics for Laplace, moderate improvements for heat, and faster convergence with lower prediction errors for Helmholtz, transport, and wave equations.The gains track the symbol’s zero-set geometry: they are largest for nontrivial characteristic or circular zero sets and smaller when the zero set is an isotropic isolated point.Source: §6.3 · p. 15, Figure 5 · p. 16
  • For Burgers, transport, and wave equations, PDE-informed initialization reduces spectral error more uniformly across frequency bands and reaches low error earlier than Gaussian initialization.The two strategies show similar spectral-error behavior for the nonlinear cubic equation.Source: §6.3 · p. 15, Figure 6 · p. 17

6.4.1. Nonlinear dynamics

  • For the nonlinear Laplace and Burgers equations, PDE-informed initialization more clearly improves prediction accuracy than training-loss dynamics; for Sine–Gordon and Klein–Gordon, it improves both.The nonlinear initialization uses the principal linear operator’s symbol to construct its frequency geometry.Source: Figure 5 · p. 16, §6.4.1 · p. 17

6.4.2. ADAM dynamics

  • Under ADAM, PDE-informed initialization generally yields faster convergence, lower prediction errors, and more stable behavior across seeds, though Gaussian initialization sometimes has lower individual-seed training loss.The experiments use exponential learning-rate decay with factor 0.90 every 50 iterations.Source: §6.4.2 · pp. 17–18, Figure 7 · p. 18

6.4.3. Multilayer case

  • In multilayer networks, PDE-informed initialization improves convergence and prediction accuracy across the tested equations under both GD and ADAM, with larger gains for transport than heat.The tested architecture is [2, 2000, 200, 1].Source: §6.4.3 · pp. 18–19, Figure 8 · p. 19

6.5. Validation on State-of-the-Art PINN Benchmarks

  • The JAX-PI validation tests whether changing only the initialization benefits advection, Allen–Cahn, and Kuramoto–Sivashinsky benchmarks within the repository’s existing training framework.The reported setup uses multilayer width-256 networks, frozen Fourier-feature embeddings, and established optimization pipelines; the supplied passages do not state the benchmark outcomes.Source: §6.5 · pp. 19–20, Figure 9 · p. 20
  • Across all three benchmarks, PDE-informed initialization yields faster or comparable optimization and less variability across seeds than Gaussian initialization.The improvement is especially pronounced for advection, where training loss and relative L2 error decrease substantially faster; nonlinear problems show smaller but consistent differences and improved or comparable final prediction errors.Source: §6.5 · pp. 20–21
  • Under the simplified baseline configuration, PDE-informed initialization gives dynamics comparable to SOTA, with faster convergence and moderately better predictions for Allen–Cahn and Kuramoto–Sivashinsky.The baseline configuration uses NTK/gradient-based loss weighting, learning-rate decay, and non-dimensionalization; Allen–Cahn also shows noticeably faster early convergence.Source: §6.5 · pp. 20–21, Figure 10 · p. 21
  • PDE-informed initialization and JAX-PI stabilization methods appear to address similar frequency-bias difficulties, but initialization adds virtually no computational cost.The proposed method only changes the Fourier-frequency sampling distribution before training; the passage suggests some JAX-PI heuristics alleviate frequency bias through optimization rather than explicit distribution changes.Source: §6.5 · p. 21
  • Both initializations accurately reconstruct benchmark solutions, but PDE-informed initialization generally lowers reconstruction errors, especially across the advection domain.Allen–Cahn and Kuramoto–Sivashinsky show smaller visual differences, with slightly lower reconstruction errors for PDE-informed initialization.Source: §6.5 · p. 21, Figure 11 · p. 22

7. Discussion

The paper derives how PDE symbols and Fourier-frequency initialization jointly shape residual convergence, then uses that analysis to construct an operator-informed initialization. Experiments report gains that depend on the geometry of weakly constrained frequencies, while identifying limits for fully nonlinear PDEs.

7. Discussion

  • In the NTK regime with frozen Fourier frequencies, residual dynamics depend on the initialization density and PDE symbol, especially where the symbol is small or zero.The dimension and geometry of the PDE symbol's zero set identify spectral directions where the interior residual provides the weakest control.Source: §7 · p. 22
  • The proposed initialization adapts Fourier-frequency sampling to the PDE-symbol geometry and concentrates frequencies near regions where the symbol is small.Unlike isotropic standard Gaussian sampling, it explicitly incorporates information about the differential operator.Source: §7 · p. 22
  • Across the tested PDEs, PDE-informed initialization mitigates frequency bias, accelerates optimization, improves residual convergence, and reduces seed variability, with gains varying by PDE-symbol geometry.Gains are comparable for Laplace, moderate for heat, and substantially larger for transport, wave, and Helmholtz equations with extended frequency-space structures; effectiveness also extends to nonlinear PDEs, deep networks, and ADAM without extra training cost.Source: §7 · pp. 22–23
  • The advantage is more pronounced when weakly constrained frequency regions have nontrivial geometry that isotropic Gaussian sampling does not efficiently capture.The reported gains are small for Laplace, moderate for anisotropic heat, and larger for transport, wave, and Helmholtz equations.Source: §7 · pp. 22–23
  • Unlike approaches that alter architectures or training procedures, this method changes only initialization and is designed to adapt across PDEs through their Fourier symbols.The discussion contrasts it with PDE-agnostic initialization, transfer learning, architectural changes, and frequency transformations.Source: §7 · p. 23
  • The method relies on a meaningful PDE symbol and is naturally applicable to linear and semilinear PDEs with constant coefficients.Fully nonlinear PDEs generally lack a unique linear symbol capturing their dominant dynamics, making construction of the frequency distribution more challenging.Source: §7 · p. 23
  • The authors suggest aligning initialization distributions with an operator's spectral structure as a broader principle for improving optimization.They identify inverse problems, integral and nonlocal equations, and operator-learning architectures as possible settings for future extension.Source: §7 · p. 23

A. Derivation of the FB equation for̂ 𝑟

The proof section places its derivation in the framework of tempered distributions and relates the PINN residual dynamics to a regression result, with an additional factor from the differential operator. The supplied passages identify this proof strategy but do not state the full evolution equation.

A. Derivation of the FB equation for̂ 𝑟

  • The proof uses the theory of tempered distributions, with Schwartz functions on R^d and tempered distributions in their dual space.The stated convention is that distributional relations hold in duality with Schwartz functions.Source: A. Derivation of the FB equation for̂ 𝑟 · p. 25
  • The proof follows Molina et al. (2026), modifying their regression-task dynamics to account for the differential operator.The passage identifies the operator contribution as a multiplicative symbol factor.Source: A. Derivation of the FB equation for̂ 𝑟 · p. 26
  • For the PINN setting, the cited derivation says the dynamics differ from the regression case by the PDE symbol factor.The proof then computes the Fourier-feature network expression using steps analogous to Molina et al. (2026).Source: A. Derivation of the FB equation for̂ 𝑟 · p. 26

B. Effect of PDE-symbol mismatch

The transport experiment tests whether initialization quality depends on matching the PDE's frequency geometry. Keeping the training equation fixed, the authors find the best dynamics and prediction accuracy when the initialization uses the true transport velocity, with performance worsening as the assumed velocity diverges.

B. Effect of PDE-symbol mismatch

  • The experiment compares standard Gaussian sampling with PDE-informed initializations built using the true transport velocity 𝑐=1 or incorrect velocities 𝑐=2, 𝑐=5, and 𝑐=10.The training loss always uses the true velocity 𝑐=1; only the initialization polynomial changes, rotating its preferred frequency region when incorrect.Source: B. Effect of PDE-symbol mismatch · p. 27
  • The mismatch experiment supports a link between improvement and PDE-dependent frequency geometry, rather than merely using a non-Gaussian distribution.Among tested initializations, performance is best when the sampling polynomial matches the true operator, and deteriorates as the characteristic sets mismatch more.Source: B. Effect of PDE-symbol mismatch · p. 27
  • Correct-velocity initialization yields the best training dynamics and relative L2 prediction error; both worsen progressively as the assumed velocity moves from 2 to 5 to 10.The 𝑐=2 initialization performs much closer to standard Gaussian initialization than the correct 𝑐=1 initialization.Source: Figure 12 · p. 27, B. Effect of PDE-symbol mismatch · p. 27

C. Other JAXPI benchmarking examples

For the steady lid-driven cavity equations, the initialization is based on the principal linear operator rather than the nonlinear convective terms. At sufficiently high Reynolds numbers, the resulting initialization approaches an isotropic Gaussian density.

C. Other JAXPI benchmarking examples

  • The steady incompressible Navier–Stokes equations are the starting point for the cavity example.The subsection derives the principal symbol for this system.Source: §C.1 · p. 27
  • Because initialization depends only on the principal linear operator, the nonlinear convective terms are discarded to obtain the steady Stokes operator.The reduced system is then Fourier-transformed to obtain its principal-operator symbol.Source: §C.1 · p. 28
  • For sufficiently large Reynolds numbers, the PDE-informed initialization is exactly a Gaussian density up to its normalization constant.The passage relates this limit to the viscous contribution vanishing and the principal symbol being dominated by the incompressibility constraint.Source: §C.1 · p. 28

C.2. Ablation study using PDE-informed initialization

The ablation tests PDE-informed initialization with JAX-PI optimization strategies on Advection and Allen–Cahn, finding that combinations vary in usefulness and can interfere.

C.2. Ablation study using PDE-informed initialization

  • Across Advection and Allen–Cahn, added JAX-PI strategies did not necessarily improve PDE-informed initialization, which already provided a favorable optimization starting point.The comparisons are reported in Tables 4 and 5.Source: §C.2 · p. 28, Table 4 · p. 29, Table 5 · p. 29
  • Gradient normalization consistently yielded faster convergence while maintaining competitive prediction errors.Source: §C.2 · p. 28
  • For Advection, enabling the time-period strategy substantially degraded predictive accuracy, indicating that this combination was incompatible with the proposed initialization.Source: §C.2 · p. 28
Loading 2610.03378v1…