Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next

Salvatore Cuomo; Vincenzo Schiano di Cola; Fabio Giampaolo; Gianluigi Rozza; Maziar Raissi; Francesco Piccialli

Source-linked AI summary

Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next

Salvatore Cuomo, Vincenzo Schiano di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, Francesco Piccialli

arXiv:2201.05624v4Jan 14, 2022cs.LGcs.AImath.NAphysics.data-an

TL;DR

PINNs address the challenge of incorporating physical equations into neural-network methods for scientific computing. This paper reviews PINNs and related collocation-based variants, their components, applications, and customization strategies. The review finds broad methodological and application growth, while identifying unresolved theoretical and implementation limitations.

Problem
Scientific computing needs neural-network methods that incorporate physical knowledge while addressing PDEs and related problems that can be difficult for standard numerical approaches.
Method
The paper conducts an extensive literature review of PINNs, related physics-informed frameworks, their building blocks, applications, learning theory, toolsets, and customization strategies.
Results
The reviewed literature mainly customizes PINNs through activation functions, optimization procedures, neural-network architectures, and loss-function structures across diverse scientific problems.
Takeaways & Limitations
PINNs provide a mesh-free framework that can use physical equations and optional data to address forward and inverse problems within related optimization formulations.
Takeaways & Limitations
The review identifies unresolved theoretical issues and limited investigation of non-feed-forward architectures and network-size selection in PINNs.

Abstract

from arXiv · show

Physics-Informed Neural Networks (PINN) are neural networks (NNs) that encode model equations, like Partial Differential Equations (PDE), as a component of the neural network itself. PINNs are nowadays used to solve PDEs, fractional equations, integral-differential equations, and stochastic PDEs. This novel methodology has arisen as a multi-task learning framework in which a NN must fit observed data while reducing a PDE residual. This article provides a comprehensive review of the literature on PINNs: while the primary goal of the study was to characterize these networks and their related advantages and disadvantages. The review also attempts to incorporate publications on a broader range of collocation-based physics informed neural networks, which stars form the vanilla PINN, as well as many other variants, such as physics-constrained neural networks (PCNN), variational hp-VPINN, and conservative PINN (CPINN). The study indicates that most research has focused on customizing the PINN through different activation functions, gradient optimization techniques, neural network structures, and loss function structures. Despite the wide range of applications for which PINNs have been used, by demonstrating their ability to be more feasible in some contexts than classical numerical techniques like Finite Element Method (FEM), advancements are still possible, most notably theoretical issues that remain unresolved.

1 Introduction

This review situates PINNs within the development of neural-network approaches to PDEs and examines their growing use across scientific computing. It surveys PINN foundations, variants, applications, advantages, and unresolved issues.

Foundations: PINNs integrate physical prior knowledge, typically through governing equations, into neural-network approaches for solving PDEs.Earlier work used neural networks to approximate PDE solutions, while PINNs formalized physics-informed learning as a data-driven solver framework.
Field development: PINN research has expanded from early neural-network PDE approximations toward more sophisticated architectures, optimization practices, automatic differentiation, and discrete time-stepping.These developments were supported by improved hardware, training practices, open-source packages, and automatic differentiation.
Capabilities: PINNs can solve forward and inverse problems using mesh-free optimization, with the same framework supporting PDE solutions and parameter characterization.The approach can use physical equations and boundary conditions without labeled data, while inverse settings may incorporate observations.
Review scope: The survey analyzes PINN building blocks, learning theory, toolsets, applications, future directions, and accuracy and convergence issues.It also reports customization through network depth, layer size, activation functions, and transfer learning.
Review scope: The review examines PINNs as collocation-based methods alongside broader physics-informed and physics-guided neural-network frameworks.Its literature scope includes physics-informed machine learning, physics-guided neural networks, and variants such as Deep Ritz, DGM, and hp-VPINN.

2 The building blocks of a PINN

A PINN represents the unknown field with a neural network and trains its parameters by combining differential-equation, boundary-condition, initial-condition, and optional data losses. Automatic differentiation supplies physical residuals, and optimization updates the network parameters.

Physical problem: The governing problem combines a differential operator F with boundary or initial conditions represented by B on the domain boundary.The formulation supports Dirichlet, Neumann, Robin, and periodic conditions.
Loss construction: PINN training minimizes a weighted loss formed from the differential-equation residual, boundary conditions, and optional labeled data.Forward problems may use only physical equations and boundary conditions, whereas inverse or noisy-data problems can include observations.
Neural-network block: The neural network maps domain variables z to an approximation of the unknown solution u(z).The network is parameterized by θ, which determines the surrogate solution.
Physics-informed block: Automatic differentiation connects the solution network to a physics-informed component that computes derivatives and residual terms.Initial and boundary conditions and available observations are evaluated before the feedback mechanism updates θ.
Feedback mechanism: The feedback mechanism uses an optimizer and learning rate to reduce the combined residual loss and fix the network parameter vector θ.This closes the training loop between residual evaluation and parameter updates.

2.1 Neural Network Architecture

PINN studies have primarily used feed-forward networks while exploring CNN, RNN, GAN, and composite architectures. Architecture choice remains problem-dependent because network size, differentiability, geometry, and training stability affect performance.

Architectures: Most PINN implementations use feed-forward neural networks, while the literature also experiments with CNN, RNN, and GAN architectures.The review synthesizes these mainstream and alternative network types in its architecture survey.
Architecture selection: Selecting network size remains difficult because smaller networks may underrepresent unknown functions, whereas larger networks can be difficult to train on small datasets.The ideal architecture is still an open research question.
Architecture customization: PINN research has investigated customized depth, width, activation functions, and multiple-network designs for different scientific problems.Examples include multiple fully connected networks and architectures tailored to specific equations or interfaces.
Activation function: Activation functions must support the derivative evaluations required by PINNs, making infinitely differentiable choices such as tanh common and motivating alternatives such as Swish.The review reports Swish as improving convergence rate and accuracy in one cited study, while cautioning about second-order derivatives.
CNN architectures: CNNs face difficulties on irregular scientific geometries because Euclidean convolution filters lose invariance on non-uniform meshes.PhyGeoNet addresses this boundary through coordinate transformation to a rectangular reference domain and strict boundary enforcement.
Alternative architectures: Physics-informed recurrent and generative architectures extend PINNs to sequential or stochastic settings by combining learned representations with numerical differentiation or governing SDEs.PhyLSTM connects LSTM networks with a tensor differentiator, while PI-GANs embed stochastic differential equations through automatic differentiation.
Composite architectures: Conservative PINNs combine separate subnetworks over subdomains and reconstruct the complete solution through interface conditions, enabling network parallelization.This design applies domain segmentation to discrete domains.

2.2 Injection of Physical laws

PINNs inject physical laws by differentiating neural-network outputs and incorporating differential-equation residuals into training. Automatic differentiation is the dominant implementation approach, while alternative derivative and network formulations remain limited.

Network formulations: PINN implementations have mostly used feedforward networks, while CNN and RNN applications remain represented among the reported network families.The reviewed table identifies FFNNs, CNNs, and RNNs as the three major neural-network families and reports predominantly FFNN use.
Alternative implementations: Automatic differentiation is used by essentially all PINN implementations, with local fitting reported as an alternative for approximating differential operators.The local-fitting implementation was reported to verify convergence without relying on automatic differentiation.
Residual construction: The differential-equation residual measures mismatch with the governing equation, while boundary and initial residuals measure violations of prescribed conditions.For the exact solution, both the governing-equation and boundary residuals are zero.
Physics injection: PINN physics constraints are formed by differentiating the network with respect to spatiotemporal inputs and embedding the resulting terms in the loss.Repeated differentiation and network composition create the terms used in the partial differential equations.
Derivative computation: Automatic differentiation computes neural-network derivatives through hierarchical chain-rule evaluations across layers.It evaluates derivatives analytically using floating-point operations rather than symbolic strings or numerical approximations.

2.3 Model estimation by learning approaches

PINN model estimation minimizes a combined loss that can incorporate PDE, data, and boundary or initial-condition errors. Training commonly uses gradient-based optimization, but network size, loss weighting, and computational cost create practical challenges.

Loss formulation: PINN parameters are estimated by minimizing a loss function whose terms represent PDE, data, and boundary or initial-condition mismatches.The PDE term enforces the governing equation at collocation points, while data terms fit synthetic or measured observations.
Training data: Training combines relatively few scattered observations with a larger set of collocation points enforcing zero residuals.The resulting optimization can use ordinary stochastic gradient descent without constrained-optimization methods.
Residual enforcement: PDE residual losses are commonly evaluated at randomly selected domain locations, although integral loss definitions are also used.Automatic differentiation computes the derivatives required for the PDE loss without numerically discretizing and solving the PDE.
Weighting and regularization: Loss weights can encode PDE-model fidelity, while regularization is generally needed because network parameters substantially outnumber measurements.Setting the PDE weight to zero removes governing-equation knowledge from optimization.
Boundary conditions: Soft boundary-condition enforcement does not guarantee accurate satisfaction, and no theory currently guides its loss weights.Hard enforcement instead encodes boundary conditions directly into network design.
Loss alternatives: Energy-based losses are reported as simpler to minimize and more computationally efficient, but more sensitive to training-sample size and resolution than PDE-based losses.The comparison concerns alternative loss-function strategies rather than a universal replacement for residual-based losses.
Optimization: PINN training commonly uses Adam and L-BFGS, while noisy-data experiments found optimal learning with increased sample size and L-BFGS alone.The reviewed optimization problem is nonconvex and may contain several local minima.
Optimization: A moderately sized network with four hidden layers and twenty neurons per layer has over 1000 parameters to optimize.Gradient-based optimization may become trapped in local minima, while finding global minima is described as NP-hard.

2.4 Learning theory of PINN

PINN learning theory studies convergence, approximation, generalization, and optimization errors. The review reports theoretical convergence and dimension-related results for selected architectures and PDE classes, while emphasizing that the theory remains incomplete.

Open theory: Theoretical investigations of PINNs remain at an early stage, and substantial work is still needed to characterize their behavior and limits.The review identifies convergence, generalization, approximation, and optimization as active research concerns.
Theoretical foundations: PINN theory aims to establish convergence of the trained neural-network approximation to the physical problem’s solution.The framework adapts numerical-analysis concerns about stability, consistency, and convergence to learning from physical laws and data.
Convergence results: For tanh networks, one result proves that the approximation difference converges to zero as the width of a predefined network tends to infinity.Another result reports strong convergence of minimizers for linear second-order elliptic and parabolic PDEs.
Error decomposition: Theoretical error analysis separates optimization, generalization, and approximation errors, with approximation error depending on architecture.Optimization error is difficult to characterize because training minimizes a generally nonconvex cost function.
Generalization results: Formal generalization-error estimates have been established for selected PDE classes, including linear elliptic, parabolic, hyperbolic, Navier–Stokes, and Kolmogorov-type problems.These results relate generalization error to training error and the number of training points under specified problem settings.
Dimensional dependence: Other estimates suffer from the curse of dimensionality because reducing error requires exponentially increasing training-point counts and network size.A radiative-transfer result instead reports dimensional dependence through a logarithmic factor.
Dimensional dependence: For one reported problem, training error does not depend on dimension but only on the number of training points, indicating no dimensionality curse in that setting.This is a problem-specific result rather than a general conclusion for all PINNs.

3 Differential problems dealt with PINNs

PINNs were introduced for complex nonlinear PDEs and have since been applied across a broad range of differential-equation types. The reviewed applications include ODEs, PDEs, fractional and integro-differential equations, and stochastic differential equations.

Vanilla PINN: The vanilla PINN was introduced for nonlinear PDEs of the form u_t + F_xu = 0, with F_x a nonlinear spatial differential operator.The initial formulation was shown to handle both forward and inverse problems.
Application scope: Subsequent PINN research has addressed ordinary, partial, fractional, integro-differential, and stochastic differential equations.The review organizes these works according to the form of the differential equation.

3.1 Ordinary Differential Equations

PINN-related approaches extend neural differential-equation models to nonlinear dynamical systems by combining learned dynamics, numerical integration, and physics-informed terms. Applications include structural identification and nonlinear seismic-response modeling.

Neural ODEs: Neural ODEs parameterize dynamical systems with neural networks and can serve as continuous representations of residual networks.The vector θ parameterizes the neural-network function f in the ODE formulation.
Physics-informed dynamics: Physics-informed Neural ODEs separate governing physics from an unknown discrepancy term when learning system dynamics.The framework was tested on a cubic-nonlinearity spring-mass system with noisy measurements and on a structure with a negative stiffness device.
Structural dynamics: PINN approaches using deep LSTM networks address nonlinear structural systems under seismic excitation, including steel frames and Bouc–Wen models.The Bouc–Wen model represents a nonlinear system with rate-dependent hysteresis.
Neural integration: Directed graph models and recurrent neural networks can directly implement ODEs while using Euler-based numerical integration.Related work also provides Python tutorials for integrating ODEs with recurrent neural networks.
Structure-preserving models: Symplectic Taylor neural networks use two subnetworks and symplectic integrators instead of the original ODE-net’s Runge–Kutta approach.The method was tested on Hamiltonian systems including Lotka–Volterra, Kepler, and Hénon–Heiles systems.

3.2 Partial Differential Equations

PDEs provide foundational mathematical models for physical phenomena, and their numerical solutions rely on established strategies whose stability and convergence have been extensively studied.

Role of PDEs: PDEs form the building blocks of many mathematical models used to describe physical phenomena.The literature has developed numerical strategies for solving these equations approximately.
Numerical foundations: Classical PDE-solution strategies have established theoretical frameworks for stability and convergence.These properties support approximate solutions of differential problems.

3.2.1 Steady State PDEs

Steady-state PINN studies span elliptic, wave-propagation, mechanics, and inverse problems, with variants adapting architectures, operators, and loss formulations to each setting.

Inverse problems: PINN methods can estimate unknown porous-medium conductivity from finite pressure measurements, and one study reports outperforming a state-of-the-art maximum a posteriori probability method.The inverse problem arises because conductivity is difficult to measure directly.
Variational methods: VPINN reduces the differential-operator order through integration by parts and has been tested on steady Burgers, Poisson, and Schrödinger-related problems.The cited applications include the steady Burgers equation, two-dimensional Poisson’s equation, and Schrödinger Hamiltonians.
Mechanics: Nonlocal PINN formulations using the Peridynamic Differential Operator capture stress and strain concentrations with global functions.The framework was applied to two-dimensional quasi-static linear-elastic and elastoplastic deformation.
Elliptic equations: Steady-state PINNs address elliptic equations, including Poisson problems on regular and irregular domains and Laplace–Beltrami equations on complex surfaces.These studies examine nonsmooth solutions, sample size, network structure, and accuracy.
Elliptic equations: SPINN architectures reinterpret meshless PDE representations to produce partially interpretable sparse neural networks.The approach was proposed for linear elliptic PDEs and Poisson equations.
Wave propagation: EikoNet solves a three-dimensional Eikonal equation for travel-time fields but requires retraining whenever the fixed velocity model changes.PINNs also outperform first-order fast sweeping in accuracy tests, especially for the anisotropic model.

3.2.2 Unsteady PDEs

Unsteady PDE research applies PINNs and variants to diffusion, advection, fluid, plasma, phase-transition, and hyperbolic problems. Reported performance depends strongly on loss weighting, sampling, architecture, and domain decomposition.

Diffusion and phase-change problems: A two-subnetwork PINN with sequentially adapted loss weights improves prediction accuracy for composite-material heat-transfer equations.The model uses separate subnetworks for temperature and degree of cure.
Diffusion and phase-change problems: Dynamic loss weighting mitigates PINN training problems caused by stiffness in gradient-flow dynamics and significantly reduces relative prediction error.The results indicate that loss-function weights are crucial for identifying unknown thermal diffusivity values.
Advection–dispersion problems: PINNs produce accurate advection–dispersion solutions and outperform typical discretization-based methods in the cited comparison.For the same two-dimensional equation, PINNs also perform better than PIELM, with fewer border errors attributed to stronger boundary and initial-condition weighting.
Advection–dispersion problems: PIELM and PINN methods can fail on linear advection equations with sharp-gradient solutions, while MPINN studies examine network size for subsurface parameter and state estimates.The cited limitation concerns sharp gradients, and the MPINN application covers Darcy and advection–dispersion equations.
Hyperbolic and shock problems: Domain decomposition and localized networks help address steep gradients and discontinuities in Burgers and Euler equations.cPINN partitions domains into subdomains, while clustered training points improve accuracy near high-gradient regions.
Fluid and plasma problems: PINNs can infer three-dimensional turbulent plasma fields from limited two-dimensional electron-pressure observations and evolution equations.The cited study uses synthetic plasma generated with a finite-difference algorithm.
Fluid and plasma problems: Res-PINN uses ResNet blocks to improve neural-network stability for Burgers and Navier–Stokes fluid-flow problems.Predicted pressure distributions can match observations even when pressure magnitudes differ.
Hyperbolic and shock problems: Characteristic-form Euler equations combined with data produce better results than conservative-form equations, and training-point placement is essential.These methods target discontinuous flows such as shock and contact waves.

3.3 Other Problems

PINNs extend beyond classical PDEs to fractional, stochastic, and uncertainty-aware problems, including settings with sparse or noisy observations. The reviewed approaches modify differentiation, Bayesian inference, and network structure to handle these broader problem classes.

PINNs have been applied to fractional PDEs and uncertainty estimation beyond classical differential problems.
Fractional PINNs combine automatic differentiation for integer-order operators with numerical discretization for fractional operators.This hybrid construction supports identification of unknown coefficients and operators in fractional PDEs.
Time-dependent fractional orders can be represented using separate neural networks for each order and a larger network for the states.
Bayesian PINNs combine physical principles with scattered noisy observations to predict solutions and quantify aleatoric uncertainty.Bayesian neural networks provide the prior, while Hamiltonian Monte Carlo or variational inference estimates the posterior.
Stochastic PINN studies examine transitions from forward to mixed and inverse problems as measurements shift among stochastic processes.The setting involves approximating solution, forcing, and diffusion-coefficient processes from limited scattered sensor measurements.

3.4 Solving a differential problem with PINN

A PINN is evaluated on a periodic, complex-valued 1D nonlinear Schrödinger problem using initial and boundary losses plus the PDE residual, without extra solution data. The best tested configuration achieves an average MSE of 5,17·10^-04, but errors remain localized and increase over longer time intervals.

The NLS case tests PINNs on periodic boundary conditions and complex-valued solutions over a domain ending at T = π/2.The initial state is ψ(x, 0) = 2 sech(x), and the reference solution is generated with Chebfun.
The network takes time and location as inputs and outputs the real and imaginary parts of the solution.
Training minimizes initial and boundary losses together with the Schrödinger residual at collocation points, using no extra data beyond t = 0.The surrogate-PDE setup sets Ldata = 0.
All tested configurations show similar solution patterns, differing mainly in the order of magnitude of their errors.The experiments vary training data and architectures while comparing MAE and MSE.
5,17·10^-04 is the average MSE of the best tested configuration across the reported NLS solutions.The PINN particularly struggles near (x, t) = (0, π/4) and with symmetric values in t ∈ (π/4, π/2).
Increasing training points decreases error rates, while vanilla PINNs have difficulty maintaining performance over longer time intervals.Doubling initial-condition points has a greater influence than doubling spatial-domain points in this setup.

4 PINNs: Data, Applications and Software

PINN performance depends strongly on the physical formulation and training-point distribution, while applications span hidden-state inference, fluid dynamics, turbulence, materials, and design optimization. The review also identifies temporal extrapolation as a persistent weakness of vanilla PINNs.

4.1 Data: Training-point distribution affects PINN flexibility and approximation quality, especially when point locations are important.The review discusses equispaced, Sobol, and Latin hypercube sampling strategies.
4.1 Data: PINNs can predict latent nonlinear variables when measurements for those variables are missing.
Flows problems: Hidden fluid mechanics infers velocity and pressure fields directly from images using physics-informed constraints.The framework is demonstrated on a three-dimensional intracranial aneurysm.
Flows problems: 3D turbulent fields can be inferred from only 2D data while preserving the governing nonlinear PDEs.The approach is proposed for diagnosing turbulent fields in magnetized collisional plasmas.
Flows problems: Vanilla PINNs perform poorly when projecting solutions beyond the temporal region used for training.This limitation is reported across several Burgers’ equation benchmark problems.
Molecular dynamics and materials related applications: Multi-fidelity PINNs estimate nanofluid viscosity over a wide sample space using very few molecular-dynamics simulations.
Applications: Parameterized PINN geometries can accelerate heat-sink design optimization relative to traditional single-geometry solvers.

4.3 Software

PINN software has expanded into libraries supporting automatic differentiation, complex geometries, hard or soft constraints, adaptive refinement, distributed computing, and diverse PDE formulations. These tools increasingly target both research flexibility and practical problem solving.

PINN libraries generally use feed-forward neural networks and automatic differentiation to compute derivatives for loss functions.
The software ecosystem includes DeepXDE, Modulus, PyDEns, NeuroDiffEq, SciANN, NeuralPDE.jl, ADCME, Nangs, TensorDiffEq, IDRLnet, and Elvet.The packages differ in supported equations, constraints, architectures, geometries, and optimization features.
DeepXDE: DeepXDE supports diverse boundary conditions, complex geometries, scattered points, and residual-based adaptive refinement.Its refinement strategy adds points where the PDE residual is larger until the mean residual falls below a threshold.
NeuroDiffEq and PyDEns impose initial and boundary conditions through network construction, making them physics-constrained neural networks.NeuroDiffEq targets traditional PDEs in two dimensions, while PyDEns configures heat and wave equations.
Modulus supports parameterized geometries, multiplicative filter networks, gradient aggregation, and a broad range of PDE formulations.It can handle strong or weak forms and uses integral loss formulations with global learning-rate annealing.
TensorDiffEq targets scalable PINN inference and inverse problems through multi-worker distributed computing and self-adaptive loss weights.
IDRLnet combines geometric objects, data sources, neural networks, loss metrics, and optimizers for noisy inverse and integral-differential problems.

5 PINN Future Challenges and directions

The review identifies unresolved theoretical, implementation, and application challenges for PINNs, while outlining opportunities for broader architectures and scientific uses. Key open questions concern convergence, optimization, scalability, and whether PINNs can reliably handle difficult physical regimes.

Theoretical difficulties: PINN research still lacks clear evidence that training converges to the correct differential-equation solution.The review links this question to approximation error, stability, network topology, and limited existing theoretical results.
Implementation aspects: PINN implementation remains open across loss weighting, collocation-point selection, initialization, network size, and architectures beyond feed-forward neural networks.Suggested directions include CNNs, RNNs, Fourier neural operators, N-BEATS, Transformers, and sinusoidal representation networks.
Theoretical difficulties: Optimization requires additional theoretical foundations because physical integration introduces interactions among PDE stiffness, gradient descent, and soft constraints.The review specifically identifies the relationship between PDE stiffness and PINN optimization algorithms as a key issue.
Theoretical difficulties: PINNs may scale to high-dimensional PDEs without exponentially increasing computational cost, but the reason for this behavior lacks a formal explanation.The review presents this as an unresolved connection between PINN behavior and learning theory.
SciML framework: PINNs can struggle with high-coefficient convection or viscosity problems because their loss landscapes become more complex under soft PDE constraints.Lower errors were reported when the problem was posed as sequence-to-sequence learning rather than solving the full space-time domain at once.
SciML framework: PINNs offer potential advantages for nonlinear function approximation and diverse engineering applications, but training can take significantly longer than established numerical methods.The review contrasts their nonlinear approximation of functions and derivatives with FEM’s linear approximation.

6 Conclusion

The review traces PINNs from physical-constraint optimization toward a broad family of collocation-based variants and analyzes their architectures, losses, applications, and software. It concludes that substantial development remains, especially for unresolved theoretical issues and optimal training.

Most analyzed studies customize PINNs through activation functions, gradient optimization, neural-network structures, or loss-function structures.
PINNs revive physical-constraint optimization with neural-network approximations while combining data-driven and equation-driven approaches.
PINNs extend physical information through collocation points, where the neural network is forced to obey the physical model equation.
The review covers vanilla PINNs and variants including variational PINNs, with soft and hard treatments of boundary conditions.
The survey analyzes the PINN pipeline from neural networks and physics-based loss construction through equations, applications, and available packages.
Further development is needed for unsolved theoretical issues, optimal PINN training, and solving multiple equations.