PaperlayerView exact revision on arXiv

Source-linked AI summary

An Expert's Guide to Training Physics-informed Neural Networks

Sifan Wang, Shyam Sankaran, Hanwen Wang, Paris Perdikaris

arXiv:2308.08468v1cs.LGmath.NAphysics.comp-ph

TL;DR

PINNs lack common benchmarks, strong baselines, and practical guidance despite their ability to combine data with PDE constraints. This paper integrates training and architecture techniques, evaluates them through reproducible ablations and challenging benchmarks, and releases a JAX library. The resulting pipeline achieves state-of-the-art results and supplies baselines for future comparisons.

  • Problem

    Common benchmarks and strong baselines are missing, making PINN progress difficult to assess systematically and PINN use difficult to guide in practice.

  • Method

    The paper develops an integrated pipeline combining nondimensionalization, architecture choices, loss weighting, causal and curriculum training, reproducible ablations, and a JAX library.

  • Results

    The full algorithm achieves a relative L2 error of 5.84 × 10−4 on the Allen–Cahn example, while disabling Fourier Feature embedding yields 4.35 × 10−1.

  • Takeaways & Limitations

    The benchmarks and reported state-of-the-art results provide strong baselines for fairer, more systematic comparisons of PINN methods.

  • Takeaways & Limitations

    Nondimensionalization does not resolve all training issues, because PDE residuals can remain differently scaled from latent solution functions.

Abstract

from arXiv · show

Physics-informed neural networks (PINNs) have been popularized as a deep learning framework that can seamlessly synthesize observational data and partial differential equation (PDE) constraints. Their practical effectiveness however can be hampered by training pathologies, but also oftentimes by poor choices made by users who lack deep learning expertise. In this paper we present a series of best practices that can significantly improve the training efficiency and overall accuracy of PINNs. We also put forth a series of challenging benchmark problems that highlight some of the most prominent difficulties in training PINNs, and present comprehensive and fully reproducible ablation studies that demonstrate how different architecture choices and training strategies affect the test accuracy of the resulting models. We show that the methods and guiding principles put forth in this study lead to state-of-the-art results and provide strong baselines that future studies should use for comparison purposes. To this end, we also release a highly optimized library in JAX that can be used to reproduce all results reported in this paper, enable future research studies, as well as facilitate easy adaptation to new use-case scenarios.

1 Introduction

PINNs combine observational data with PDE constraints, but systematic evaluation and practitioner guidance remain limited. This work addresses those gaps with an integrated training pipeline, reproducible ablations, benchmarks, and a JAX library.

  • PINNs incorporate noisy experimental data and physical laws by parameterizing unknown functions with deep neural networks.
  • Existing PINN studies commonly use separate benchmarks and compare against the weak original formulation, hindering systematic progress assessment.
  • The proposed pipeline integrates recent methods to address spectral bias, unbalanced back-propagated gradients, and causality violation.
  • Fully reproducible ablations across diverse benchmarks identify setups that consistently achieve state-of-the-art results and are proposed as future baselines.
  • The released JAX library supports reproducing the findings, future research, and adaptation to new use-case scenarios.

2 Physics-informed Neural Networks

PINNs represent unknown PDE solutions with neural networks and train them by minimizing losses that encode PDE, initial, and boundary constraints. The proposed pipeline combines nondimensionalization, suitable architectures, and training algorithms to address major training pathologies.

  • The PDE formulation includes an evolution equation together with initial and boundary conditions governed by differential and boundary operators.
  • The unknown solution u(t, x) is represented by a neural network uθ(t, x), enabling PDE residuals to be defined through the model.
  • PINN training minimizes a composite loss function that combines the constraints arising from the PDE system.
  • Automatic differentiation efficiently computes gradients with respect to input variables and network parameters during optimization.
  • The training pipeline consists of PDE nondimensionalization, suitable network architecture selection, and appropriate training algorithms to address spectral bias, causality violation, and unbalanced gradients.

3 Non-dimensionalization

Nondimensionalization scales PDE variables to dimensionless quantities of order one, supporting more consistent optimization and stable network propagation. It does not eliminate disparities among PDE residual and solution loss terms, so adaptive loss weighting remains necessary.

  • Nondimensionalization transforms a physical system into an equivalent dimensionless system by scaling variables with characteristic values.
  • Nondimensionalization is especially important for physics-informed models involving experimental data or real-world problems.
  • Scaling inputs and outputs to order one supports initialization assumptions and can yield more consistent optimizer steps, faster convergence, and better performance.
  • Even after variables are scaled, PDE residuals and latent solution functions can have substantially different scales in the objective.
  • The pipeline therefore introduces self-adaptive loss weighting based on back-propagated gradients and NTK theory to balance loss-term convergence rates.

4 Network Architecture

The paper uses MLPs to represent latent spatio-temporal functions and recommends architectures balancing expressivity with optimization difficulty. Fourier features address spectral bias, while random weight factorization is a drop-in parameterization intended to accelerate convergence.

  • 4.1 Multi-layer Perceptrons (MLP): MLPs take spatio-temporal coordinates as inputs and predict the corresponding target solution functions.
  • 4.1 Multi-layer Perceptrons (MLP): Networks that are too narrow or shallow lack capacity, whereas overly wide or deep networks can be difficult to optimize; recommended widths are 128–512 and depths 3–6.
  • 4.2 Random Fourier features: MLPs exhibit spectral bias toward low-frequency functions, limiting their ability to learn high frequencies and fine solution structures.
  • 4.2 Random Fourier features: Random Fourier feature embeddings map coordinates into high-frequency signals before the MLP, improving approximation of sharp gradients and complex solutions.
  • 4.2 Random Fourier features: The Fourier-feature scale σ controls encoding frequencies and the NTK eigenspace, with low values causing blurry predictions and high values potentially introducing artifacts.
  • 4.3 Random weight factorization: Random weight factorization expresses each weight matrix as W(l) = diag(s(l)) · V(l), assigning trainable scale factors to individual neurons.

5 Training

The training pipeline addresses temporal-causality violations and imbalanced loss optimization through sequential weighting, adaptive balancing, and curriculum strategies for difficult PDE problems.

  • Respecting Temporal Causality: Temporal weights progressively prioritize PDE residuals along the time axis, preventing later-time residuals from being minimized before earlier residuals become sufficiently small.The weights depend inversely exponentially on cumulative residuals from previous time steps.
  • Respecting Temporal Causality: The temporal-weighting method is computationally inexpensive because it reuses PINN loss values already stored in the training computational graph.The implementation uses lax.stop_gradient to avoid back-propagating through temporal weights.
  • Respecting Temporal Causality: The weighted residual loss is sensitive to the causality parameter ϵ: values that are too small weaken causality, whereas values that are too large can make optimization difficult.The authors recommend a moderately large ϵ and reducing it slightly if temporal weights do not converge to 1.
  • Loss Balancing: Multi-scale PDE losses make fixed manually chosen weights impractical because optimal loss-term weights vary substantially across problems.The pipeline therefore incorporates self-adaptive learning-rate annealing to balance losses during training.
  • Loss Balancing: Gradient-based and NTK-based weighting achieve similar performance, but gradient-based weights are less stable while NTK-based weighting incurs higher computational cost.The pipeline generally recommends gradient-based weighting as the first choice.
  • Curriculum Training: Curriculum training decomposes difficult optimization tasks into manageable sub-tasks for time-dependent PDEs and singular perturbation problems.For temporal problems, the method trains sequential time windows; for singular perturbations, it progressively increases problem singularity.

6 Miscellaneous

The paper recommends practical choices for optimizers, sampling, boundary enforcement, Fourier embeddings, and modified MLPs to improve PINN training and expressiveness.

  • Optimizer and learning rate: Adam consistently performs well without heavy tuning, while weight decay is discouraged for forward problems because it tends to reduce predictive accuracy.An initial learning rate of 0.001 with exponential decay typically yields good results.
  • Sampling strategy: Random collocation-point sampling reduces memory and per-iteration computational cost relative to full-batch sampling and contributes regularization effects that improve generalization.
  • Imposing boundary conditions: Periodic boundary conditions can be imposed exactly by constructing Fourier feature embeddings whose network representations satisfy the periodic constraints.The construction extends from one-dimensional periodic cells to higher-dimensional domains.
  • Fourier feature embeddings: Time coordinates can be concatenated with Fourier features, and a trainable period parameter can enforce periodicity along the temporal axis when appropriate.The period parameter is initialized to the temporal-domain length and can also be set beyond that length for nonperiodic dynamics.
  • Modified MLP: A modified MLP adds two input encoders and merges their features into each hidden layer through point-wise multiplication.The architecture generally outperforms a standard MLP for minimizing PDE residuals but requires greater computational resources.

7 Results

The benchmark suite evaluates Algorithm 1 and its training strategies across challenging PDE problems using ablations, hyper-parameter sweeps, and reproducible baselines. Results show strong accuracy, component-specific effects, scalable implementation, and important limits for difficult long-time or high-Reynolds-number flows.

  • Benchmark design: The study evaluates Algorithm 1 across challenging benchmarks and uses ablations to isolate each methodological component’s effects on relative L2 error and run-time.Components are disabled individually while other settings remain fixed; optimal-model hyper-parameters are selected separately through broader sweeps.
  • Benchmark results: Across the benchmark suite, the proposed model reports state-of-the-art relative L2 errors, supported by comprehensive ablations and hyper-parameter sweeps.Table 1 summarizes the benchmark-level relative L2 errors, while optimal configurations are detailed in the Appendix.
  • Computational efficiency: The implementation supports efficient data-parallel training with weak scaling up to 256 GPUs and utilities for monitoring gradient norms and NTK eigenvalues.The Navier-Stokes scaling benchmark measures 10,000 iterations at a fixed batch size of 40960 per GPU.
  • Training pathologies: Plain PINNs exhibit unbalanced gradients, causality violation, and spectral bias, while the same challenges recur in the advection example.The Allen-Cahn analysis compares gradient distributions, temporal PDE residuals, and NTK eigenvalues during training.
  • Allen-Cahn equation: The full Allen-Cahn pipeline achieves a relative L2 error of 5.84 × 10^-4, while disabling Fourier features increases it to 4.35 × 10^-1.Conventional PINNs run slightly faster at 12.93 minutes, whereas other configurations have relatively similar run-times.
  • Advection equation: The advection model achieves an excellent agreement with the exact solution, with a relative L2 error of 6.88 · 10^-4 when all techniques are integrated.Removing time periodicity, Fourier features, or grad norm weighting substantially increases test error.
  • Difficult flow regimes: Long-term integration without time-marching is highly challenging, and direct training struggles to produce accurate solutions for Reynolds numbers greater than 1,000.For vortex shedding, time-marching partitions [0, 10] into 10 windows; no test error is reported because numerical reference timing varies with discretization.

8 Conclusions

The paper introduces a comprehensive PINN training pipeline targeting spectral bias, imbalanced losses, and causality violation. It evaluates the pipeline through ablations on challenging benchmarks and reports state-of-the-art results intended as strong future baselines.

  • The pipeline addresses spectral bias, imbalanced losses, and causality violation in PINN training.
  • It integrates equation non-dimensionalization, Fourier feature embeddings, loss weighting, causal training, modified MLPs, random weight factorization, and curriculum training.
  • Thorough ablation studies across challenging benchmarks support the pipeline’s effectiveness and produce state-of-the-art results.
  • The benchmark suite is presented as a strong baseline for fairer and more systematic comparisons in future PINN research.

A Spectral Bias through the lens of the Neural Tangent Kernel

The section explains spectral bias using Neural Tangent Kernel theory: networks learn components associated with larger kernel eigenvalues first. In conventional fully connected networks, high-frequency components converge more slowly because their corresponding eigenvalues decrease with frequency.

  • Neural Tangent Kernel theory analyzes the training behavior of deep fully connected networks and the convergence of their target-function components.
  • The training error can be decomposed into the NTK eigen-space, with each component’s convergence rate determined by its corresponding eigenvalue.
  • Networks learn target-function components along larger-eigenvalue directions before components associated with smaller eigenvalues.
  • In conventional fully connected networks, NTK eigenvalues shrink as the frequency of corresponding eigenfunctions increases, slowing high-frequency learning.
  • The section connects this eigenvalue-dependent convergence behavior to the spectral bias observed during PINN training.

B Random Weight Factorization

Random weight factorization reparameterizes each neuron’s weights with trainable scale and direction factors. The analysis shows that this changes the optimization geometry, can reduce distances to minima, and induces neuron-wise adaptive learning rates.

  • Factorized parameterizations form equivalent families of representations, illustrated as hyperbolas in the scale-direction plane for a one-parameter weight.
  • The factorized space can shorten the distance between initialization and global minima compared with the original parameter space.
  • Random weight factorization represents each neuron’s weight vector as a trainable scale factor multiplied by a trainable direction vector.
  • An appropriate factorization can make the initialization arbitrarily close to a proper local minimum in factorized parameter space.
  • The section derives the corresponding gradient updates in the original parameter space and provides theoretical intuition for the reparameterization.
  • Weight factorization rescales the learning rate of each weight by (s^2 + ∥v∥^2), yielding a self-adaptive learning rate for each neuron.

C PINNs can violate causality

The section presents causality violation as a PINN training difficulty and examines temporal residual and loss-weight behavior across several PDE benchmarks. It also reports ablations and configurations for evaluating the proposed training components.

  • C PINNs can violate causality: Causality violation is examined by discretizing the temporal domain and studying PDE residual minimization over successive time intervals.
  • C PINNs can violate causality: For the Allen-Cahn and advection equations, figures track initial-condition loss, PDE residual loss, loss weights, and the minimum temporal residual during training.
  • C PINNs can violate causality: The benchmark sections provide hyper-parameter configurations for the Allen-Cahn, advection, Stokes, Kuramoto–Sivashinsky, cavity, torus, and cylinder problems.
  • C PINNs can violate causality: For the Stokes equation, the training diagnostics track loss convergence and changes in loss weights.
  • C PINNs can violate causality: For Kuramoto–Sivashinsky and Navier–Stokes flows, diagnostics additionally track changes in causal weights during training.
  • C PINNs can violate causality: The lid-driven-cavity ablation study reports relative L2-error convergence under configurations with different components disabled.
Loading 2308.08468v1…