PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning in Sinusoidal Spaces with Physics-Informed Neural Networks

Jian Cheng Wong, Chinchun Ooi, Abhishek Gupta, Yew-Soon Ong

arXiv:2109.09338v2cs.LGcs.AIcs.CEphysics.comp-ph

TL;DR

Accurate PINN training is difficult, and initialization can bias typical models toward flat outputs that satisfy PDE residuals while missing initial or boundary conditions. The paper analyzes this issue and introduces sinusoidal input mappings that increase gradient variability. Across a range of forward and inverse problems, sf-PINNs improve accuracy over standard PINNs.

  • Problem

    Accurate PINN training remains difficult, in part because typical models can initialize near flat outputs that satisfy PDE residuals while missing initial or boundary conditions.

    Source: §I · p. 2

  • Method

    The paper analyzes PINN behavior at initialization and introduces sinusoidal input mappings in sf-PINNs to increase input-gradient variability.

    Source: §I · p. 2, §IV · p. 6

  • Results

    sf-PINNs achieve better accuracy than standard PINNs, with a few orders of magnitude improvement across forward and inverse problems spanning multiple physics domains.

    Source: §I · p. 2

  • Takeaways & Limitations

    The paper concludes that sinusoidal input mappings can improve PINN training across different baseline settings, while the exact feature-map composition is less impactful than using a sinusoidal input layer.

    Source: §VI · p. 14

Abstract

from arXiv · show

A physics-informed neural network (PINN) uses physics-augmented loss functions, e.g., incorporating the residual term from governing partial differential equations (PDEs), to ensure its output is consistent with fundamental physics laws. However, it turns out to be difficult to train an accurate PINN model for many problems in practice. In this paper, we present a novel perspective of the merits of learning in sinusoidal spaces with PINNs. By analyzing behavior at model initialization, we first show that a PINN of increasing expressiveness induces an initial bias around flat output functions. Notably, this initial solution can be very close to satisfying many physics PDEs, i.e., falling into a local minimum of the PINN loss that only minimizes PDE residuals, while still being far from the true solution that jointly minimizes PDE residuals and the initial and/or boundary conditions. It is difficult for gradient descent optimization to escape from such a local minimum trap, often causing the training to stall. We then prove that the sinusoidal mapping of inputs, in an architecture we label as sf-PINN, is effective to increase input gradient variability, thus avoiding being trapped in such deceptive local minimum. The level of variability can be effectively modulated to match high-frequency patterns in the problem at hand. A key facet of this paper is the comprehensive empirical study that demonstrates the efficacy of learning in sinusoidal spaces with PINNs for a wide range of forward and inverse modelling problems spanning multiple physics domains.

I. INTRODUCTION

PINNs combine neural networks with physics-based loss terms, supporting forward, inverse, and meta-PDE problems where labeled data may be costly or scarce. This paper investigates why typical PINNs can be difficult to train and how sinusoidal input mappings affect their behavior.

A. Overview of a typical PINN

  • PINNs represent physical processes with neural networks and use PDE residuals, alongside applicable data, initial-condition, and boundary-condition losses, to constrain their outputs.Source: §I · p. 1, A. Overview of a typical PINN · p. 3

I. INTRODUCTION

  • PINNs have been used for mesh-free forward PDE solutions, inferring unknown physics from limited data, and meta-modeling across varying conditions or parameters.Source: §I · p. 1
  • Accurate PINN training remains difficult, and balancing loss components is nontrivial because their magnitudes and convergence rates differ.Source: §I · p. 2
  • The paper links typical PINNs’ initial bias toward flat outputs to deceptive PDE-residual minima and introduces sinusoidal mappings as a way to increase input-gradient variability.Source: §I · p. 2
  • Experiments report that sf-PINNs improve accuracy by a few orders of magnitude over standard PINNs across forward and inverse problems in multiple physics domains.Source: §I · p. 2

B. Empirical observation of training issue

In a 1D convection-diffusion example, changing the PDE-to-boundary-condition loss weighting shifted typical PINN outputs between flat and sloped forms, but neither matched the true solution.

B. Empirical observation of training issue

A. Behavior at initialization of a typical PINN

The section analyzes a tanh PINN with Xavier initialization and derives how its input-gradient behavior changes with network width. It connects vanishing input gradients at initialization to outputs that trivially satisfy classes of differential equations.

A. Behavior at initialization of a typical PINN

B. Implication to PINN training

The initialization bias can place a PINN near a loss minimum that satisfies PDE residuals but not initial or boundary conditions. Gradient descent may then struggle to reach the true solution, particularly for high-frequency or steep-gradient problems.

B. Implication to PINN training

  • A near-zero initial input gradient can trap a PINN at a deceptive minimum that reduces PDE loss while remaining far from the solution satisfying PDE and boundary conditions.Source: B. Implication to PINN training · p. 4
  • Escaping the trap is especially difficult when the true solution has high frequencies or steep gradients, because sensible solutions may be inaccessible near initialization.Source: B. Implication to PINN training · p. 4
  • Tanh saturation as weights grow can also drive gradients toward zero and impede further gradient-based training.Source: B. Implication to PINN training · pp. 4–5
  • The near-zero-gradient bias differs from the low-frequency learning bias often treated as a useful regularizer in data-fit DNNs, because PINNs prioritize minimizing all loss terms.Source: B. Implication to PINN training · p. 5

C. Other activation and initialization methods for PINNs

Typical PINNs can lose input gradients as input-layer weight variance grows, while sinusoidal activations retain more backward-pass gradient signal. This section examines alternative activation and initialization choices for controlling that behavior.

C. Other activation and initialization methods for PINNs

  • Although He initialization can provide larger initial input gradients, it may still lack the gradient variability needed for accurate PINN training.For sigmoid, the initial input-gradient distribution converges to zero faster than for tanh or sin.Source: C. Other activation and initialization… · p. 5
  • As input-layer weight variance approaches infinity, a typical PINN almost surely has zero input gradient away from the origin.The analysis also reports that changing the weight distribution affects input gradients disproportionately across input locations.Source: C. Other activation and initialization… · p. 5
  • At large activation-input variance, sin retains the highest backward-pass gradient proportion, about 0.5.For small variance below 5e-2, the plotted quantity is approximately 1 across the compared activations.Source: Figure 3 · p. 5, C. Other activation and initialization… · p. 5
  • For tanh, the input-gradient relationship is approximately linear at small weight variance or input magnitude, then approaches zero as either grows.The reported spatial sensitivity matters in PINNs because training samples are uniformly drawn across the input domain.Source: C. Other activation and initialization… · p. 6

IV. PINN TRAINING IN SINUSOIDAL SPACES

The paper replaces a typical PINN’s input layer with a trainable sinusoidal feature mapping. Its initialization can raise input-gradient variability across the domain, while sinusoidal features also offer a useful starting representation for problems with sinusoidal patterns.

IV. PINN TRAINING IN SINUSOIDAL SPACES

  • sf-PINN replaces the typical PINN input layer with a sinusoidal mapping whose frequency and phase parameters are optimized during training.The mapping uses W1 for feature frequency and b1 for phase lag.Source: §IV · p. 6
  • The mapping’s initial weight variance is used to scale the input-gradient distribution, and sf-PINNs are less sensitive to later hidden-layer activation and initialization choices.The input-layer weights are sampled from N(0, σ^2), with zero bias.Source: §IV · pp. 6–7
  • Increasing input-layer weight variance can elevate sf-PINN input gradients throughout the domain and provide variability to escape an early local-minimum trap.The stated upper bound is (n+1)2π^2σ^2 when |x| or σ is large; equality holds for a one-hidden-layer sf-PINN.Source: §IV · p. 6
  • For problems with naturally occurring sinusoidal patterns, sinusoidal features provide an informed guess that may make the loss landscape easier to optimize.The output is not necessarily constrained to be periodic within the spatial-temporal domain.Source: §IV · p. 7

A. sf-PINN with different activations and initialization schemes

The section compares sf-PINNs with standard PINNs across activations and initialization schemes on convection-diffusion and lid-driven-cavity problems. sf-PINNs improve training and accuracy across the reported settings.

A. sf-PINN with different activations and initialization schemes

  • For convection-diffusion, all Xavier-initialized standard-PINNs fail to obtain a good solution; He initialization improves results except with sigmoid, while tanh sf-PINN is most accurate.The problem has a steep gradient near x_1=1.Source: A. sf-PINN with different activations… · pp. 7–8
  • For the lid-driven cavity, only He-initialized standard-PINNs with sin activation converge to a plausible solution; sf-PINNs improve results across schemes, with sin slightly more accurate than tanh.Figure 5 compares velocity-magnitude contours against ground truth for Xavier-initialized models at Re=400.Source: Figure 5 · p. 7, A. sf-PINN with different activations… · p. 8
  • Across both problems, sf-PINNs improve training and accuracy relative to corresponding standard-PINNs, with less weight adaptation from initialization.Figure 6 summarizes solution MSE and weight displacement across 20 independent runs.Source: A. sf-PINN with different activations… · p. 8, Figure 6 · p. 8

B. sf-PINN with different sinusoidal features mapping methods

The experiments compare sinusoidal-feature mappings with standard PINNs across forward PDE problems, varying bandwidth and loss weights. sf-PINNs and related variants generally improve accuracy, with outcomes depending on the problem and bandwidth.

Experimental settings

  • The experiments vary σ from 1e-1 to 1e1 and λ from 1 to 1e6, reflecting the bandwidth and loss-weight choices tested.The authors describe σ as controlling the initial gradient distribution and associate it with the sinusoidal features’ frequency range.Source: B. sf-PINN with different sinusoidal… · p. 9, Figure 12 · p. 13

1D transient wave equation

  • Optimized sf-PINN reaches MSE <1e-10 on the 1D transient wave equation, while standard-PINN remains worse after a similar λ search.At default settings, MSE is 2.6e-4 for sf-PINN; standard-PINN has MSE=2e-2. Figure 7 visualizes the compared solutions and errors.Source: B. sf-PINN with different sinusoidal… · p. 10, Figure 7 · p. 10

2D transient N-S equations (Taylor–Green vortex)

  • On the 2D transient N-S equations, appropriately tuned sf-PINN reaches MSE below 1e-9 and preserves the vortices, unlike the distorted standard-PINN solution.With default settings, standard-PINN reaches MSE~1e-5 but distorts the central vortices; Figure 8 compares the solutions and errors.Source: Figure 8 · p. 10, B. sf-PINN with different sinusoidal… · p. 11

1D transient KdV equation

2D Helmholtz equation

Discussion of results across different σ and λ

Bandwidth selection

  • For a problem with frequency sin(6πy), the probability of initializing at least one of 64 features near that frequency rises sharply when σ is between 1 and 6.sf-PINNs in this σ range are more easily trained and yield more accurate solutions; too-large σ reduces the probability.Source: B. sf-PINN with different sinusoidal… · p. 12, Figure 11 · p. 12

Discussion of results across different σ and λ

  • Within appropriate σ ranges, sf-PINNs improve solution MSE by 3–4 orders of magnitude, remain superior across a wide λ range, and standard-PINN never outperforms them.The authors report σ=1 as a practical default and suggest searching between 1e-1 and 1e1 for most encountered problems.Source: B. sf-PINN with different sinusoidal… · pp. 12–13

C. sf-PINN for inverse problem

The inverse experiments infer wave velocity or Reynolds number from dense or sparse observations while treating initial and boundary conditions as unknown. sf-PINNs outperform standard-PINNs on the wave problem and achieve lower solution error on both problems, including sparse-data cases.

Experimental setup

  • The inverse tasks estimate wave velocity c or Reynolds number Re from observations, with exact initial and boundary conditions unknown beyond the observed data.The experiments include dense and sparse observation settings, and use a loss combining data fit with PDE residual.Source: C. sf-PINN for inverse problem · p. 13

1D transient wave equation

  • With 200 sparse wave observations, sf-PINN reaches MSE=3.2e-8 and estimates c=2 with sd=7.3e-6, while standard-PINN has MSE=6e-4 and estimates c=1.999 with sd=1.7e-3.The authors report that sf-PINNs outperform standard-PINNs in both dense and sparse observation scenarios; Figure 13c shows the sparse-data solutions and errors.Source: C. sf-PINN for inverse problem · p. 13, Figure 13 · p. 14

2D transient N-S equations (Taylor–Green vortex)

  • With 600 sparse N-S observations, sf-PINN reaches MSE=6.9e-9 versus 1.4e-5 for standard-PINN, while both estimate ν=Re^-1 near 9.998e-3.The respective estimates are 9.998e-3 (sd=2.4e-6) and 9.97e-3 (sd=3.1e-5); Figure 13d visualizes sparse-data solutions and errors.Source: C. sf-PINN for inverse problem · p. 13, Figure 13 · p. 14

VI. CONCLUSION

The conclusion attributes PINN training difficulty to limited input-gradient variability and an initial bias toward flat outputs. It presents sinusoidal input mappings as a theoretically motivated approach, supported by improvements across forward and inverse problems.

Conclusion

  • Typical PINNs can start near a local optimum that minimizes PDE residuals while remaining far from the solution satisfying both PDE residuals and initial or boundary conditions.The authors connect this trap to limited initial input-gradient variability and flat output functions.Source: §VI · p. 14
  • Sinusoidal mappings initialize PINNs with an appropriate input-gradient distribution, helping sf-PINNs overcome typical trainability issues.The paper reports consistent improvements across baseline settings and forward and inverse problems spanning multiple physics domains.Source: §VI · p. 14
  • Sine-only input mapping can provide many benefits of full Fourier features, while subsequent hidden-layer activation and initialization choices can yield similar performance.The authors emphasize input-layer mapping over the exact sinusoidal feature composition.Source: §VI · p. 14
  • Tuning σ can improve sf-PINN training and reduce the need for sophisticated loss-composition tuning, although it requires additional effort.The paper also notes that PINNs remain computationally expensive and challenging for more complex problems.Source: §VI · pp. 14–15
Loading 2109.09338v2…