Source-linked AI summary
From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning
Juan Diego Toscano, Vivek Oommen, Alan John Varghese, Zongren Zou, Nazanin Ahmadi Daryakenari, Chenxi Wu, George Em Karniadakis
TL;DR
PINNs address scientific problems with incomplete physical knowledge and sparse measurements. This review synthesizes algorithmic developments, applications, uncertainty quantification, theory, and software, highlighting advances in PINNs and PIKANs while noting remaining challenges for robust forward simulations.
Problem
PINNs target ODE/PDE problems with incomplete governing information and sparse measurements that conventional numerical methods may assimilate poorly.
Method
The review compiles major PINN and PIKAN developments across representation models, optimization, differentiation, applications, uncertainty quantification, theory, and software.
Results
Recent developments include stable KAN representations, high-dimensional decompositions, alternative differentiation methods, domain decomposition, and formulations reported to reach 100,000 dimensions on one GPU.
Takeaways & Limitations
PINNs and PIKANs have advanced rapidly and can outperform conventional methods for inverse problems such as parameter estimation and missing-physics discovery.
Takeaways & Limitations
Vanilla PINNs have limited accuracy for chaotic and turbulent long-time integration when no data are available, although modified training and architectures can reach machine precision.
Abstract
from arXiv · showhide
Physics-Informed Neural Networks (PINNs) have emerged as a key tool in Scientific Machine Learning since their introduction in 2017, enabling the efficient solution of ordinary and partial differential equations using sparse measurements. Over the past few years, significant advancements have been made in the training and optimization of PINNs, covering aspects such as network architectures, adaptive refinement, domain decomposition, and the use of adaptive weights and activation functions. A notable recent development is the Physics-Informed Kolmogorov-Arnold Networks (PIKANS), which leverage a representation model originally proposed by Kolmogorov in 1957, offering a promising alternative to traditional PINNs. In this review, we provide a comprehensive overview of the latest advancements in PINNs, focusing on improvements in network design, feature expansion, optimization techniques, uncertainty quantification, and theoretical insights. We also survey key applications across a range of fields, including biomedicine, fluid and solid mechanics, geophysics, dynamical systems, heat transfer, chemical engineering, and beyond. Finally, we review computational frameworks and software tools developed by both academia and industry to support PINN research and applications.
2 Physics-Informed Machine Learning (PIML)
This section is identified only as part of the paper’s Physics-Informed Machine Learning discussion.
- The paper’s Physics-Informed Machine Learning section is dated October 23, 2024.
1. Introduction
The review motivates PINNs by contrasting sparse, incomplete physical information with conventional methods that struggle to assimilate measurements, then surveys major developments and applications.
- Conventional numerical methods can require complete boundary, initial, and material information, making arbitrary assumptions when practical settings contain gaps.
- Sparse thermocouple measurements may be available where quantities such as maximum temperature or heat flux are located, but FEM cannot use them effectively.
- PINNs encode governing physical laws while handling incomplete knowledge, sparse measurements, and missing state variables through an additional residual loss term.
- Since 2017, PINN research has produced substantial advances across scientific domains, including algorithmic improvements addressing limitations of the original formulation.
- The review emphasizes broader algorithmic coverage and applications than earlier reviews, while organizing material on methods, applications, uncertainty, theory, software, and outlook.
2. Physics-Informed Machine Learning (PIML)
PIML combines representation models, governing equations, and optimization to approximate ODE/PDE solutions from physical and observational information without relying on predefined meshes.
- PIML uses neural networks and automatic differentiation to solve forward and inverse PDE problems without predefined grids or meshes.
- A representation model approximates the ODE/PDE solution, while automatic differentiation computes differential and boundary operators from the differentiable model.
- Training minimizes cumulative residuals from governing equations, boundary conditions, and available data, and can extend to systems by stacking constraints.
- Forward problems assume known equation parameters and prescribed boundary conditions, whereas inverse problems infer unknown information and the solution from observations.
3. Algorithmic Developments of PIML
Algorithmic developments improve PIML by modifying representation models, governing-equation treatment, and optimization, including transformations, architectural alternatives, and training workflows.
- PIML training computes solution residuals against physical laws, boundary conditions, and data, then optimizes a multiobjective loss over model parameters.
- PINNs use multilayer perceptrons to approximate ODE/PDE solutions, with trainable weights, biases, and activation functions defining the representation.
- Representation enhancements include normalization, feature expansion, hard constraints, model decomposition, and architectural modifications to improve physical approximation.
- Representation Model Modifications: The review identifies input transformations, residual connections, PINN/KAN layers, adaptive activations, and output transformations as representation-model enhancements.
- Input/Output Transformations: Input and output mappings can improve stability and accuracy by enhancing expressivity, normalizing domains, scaling outputs, or constraining predicted ranges.
Feature Expansions.
Feature expansions transform the input before it enters the representation model, helping address spectral bias and other complex relationships in PIML.
- Feature Expansions.: Input expansions map x ∈ R^n to an expanded input I(x) ∈ R^m before it enters the representation model.The choice of expansion typically depends on the specific problem.
- Feature Expansions.: Random Fourier features have been demonstrated to help mitigate spectral bias in PIML.
- Feature Expansions.: Researchers have also explored polynomial, exponential, Chebyshev polynomial, and gated recurrent unit expansions.
Hard Constraints.
PIML training combines multiple constraints, making optimization difficult; accurately enforcing boundary conditions is therefore important for model reliability and efficiency.
- Hard Constraints.: PIML objectives often combine multiple constraints, significantly complicating optimization.
- Hard Constraints.: Improper enforcement of boundary conditions can degrade neural-network training performance and stability.
- Hard Constraints.: Accurately imposing boundary conditions is crucial for improving model reliability and efficiency.
- Hard Constraints.: Nitsche’s method and the Theory of Functional Connections provide systematic approaches for exactly enforcing Dirichlet boundary conditions.
Dirichlet Boundary Conditions.
Dirichlet boundary conditions can be imposed exactly through penalty-based, functional-connection, or output-transformation methods, including Approximate Distance Functions.
- Dirichlet Boundary Conditions.: hPINNs use penalty methods and augmented Lagrangian approaches to impose hard constraints for various boundary conditions.
- Dirichlet Boundary Conditions.: Approximate Distance Functions impose Dirichlet boundary conditions through output transformations.
- Dirichlet Boundary Conditions.: The composite distance function equals zero on boundary partitions, ensuring the transformed neural-network approximation satisfies u(x) = g(x) there.For M boundary partitions, the composite distance function is formed as the product of the individual distance functions.
- Dirichlet Boundary Conditions.: A suitable input transformation can strictly enforce hard constraints, and Fourier feature embeddings can encode periodicity in smooth univariate functions.
Periodic Boundary Conditions.
Periodic boundary conditions can be imposed using penalty-based, augmented-Lagrangian, hybrid, or specialized architecture-based approaches.
- Periodic Boundary Conditions.: hPINNs employ penalty and augmented Lagrangian methods to enforce periodic boundary constraints.
- Periodic Boundary Conditions.: Hybrid approaches combine multiple techniques for exact imposition of periodic boundary conditions.
- Periodic Boundary Conditions.: Divergence-Free Networks use output transformations to ensure learned vector fields satisfy divergence-free conditions in fluid-dynamics applications.Theoretical results indicate that enforcing divergence-free constraints improves error estimates in linear problems.
- Periodic Boundary Conditions.: SympNets preserve Hamiltonian systems’ symplectic structure through Jacobian matrix factorization techniques.
Complex Constraints.
The original PIML formulation uses a multilayer perceptron as its representation model.
- Complex Constraints.: The original PIML formulation uses an MLP as its representation model.MLPs are built from perceptrons that define network layers.
- Complex Constraints.: Perceptrons form the building blocks of the layers in an MLP.
- Complex Constraints.: The section introduces perceptrons as the layer-level units underlying the original PIML representation.
Perceptron Modifications.
Several modifications target MLP activations and weights, with reported theoretical and empirical improvements in model performance and convergence.
- Perceptron Modifications.: Activation-function modifications were proposed to improve perceptron capabilities.The cited studies reported significant theoretical and empirical performance improvements.
- Perceptron Modifications.: Weight normalization decomposes each weight matrix into magnitude and direction, speeding convergence with minimal computational overhead.
- Perceptron Modifications.: Weight factorization represents the weight matrix as diag(s^(l)) · V^(l), with s^(l) as trainable parameters.The authors reported significant performance improvements both experimentally and theoretically.
- Perceptron Modifications.: PIKANs extend MLP-like representation modeling through Kolmogorov-Arnold Networks.
Other Representation Models.
Alternative representation models include cPIKANs and several other neural architectures, while residual connections are used to improve high-order derivative calculations.
- Other Representation Models.: cPIKANs use Chebyshev polynomials as inner and outer univariate functions in the KAN framework.Their trainable parameters include polynomial coefficients, and the polynomial degree controls the representation.
- Other Representation Models.: The cPIKAN representation was reported as more robust to noise and capable of improving performance with fewer parameters than MLPs.
- Other Representation Models.: Other explored representation models include CNNs, Hermite spline CNNs, GANs, spiking networks, transformers, LSTMs, information-bottleneck architectures, and reinforcement learning models.
- Other Representation Models.: Residual connections are added to PIML architectures to enable accurate calculation of high-order derivatives.
Residual Connections .
Residual connections add layer inputs to transformed outputs, and related approaches use additive or multiplicative connections and separable subnetworks.
- Residual Connections .: A single additive skip connection adds a layer’s input to the output of its representation-model layer.The representation layer may be an MLP layer or a KAN layer.
- Residual Connections .: Studies have explored skip connections implemented through addition or multiplication.
- Residual Connections .: Modified MLPs project inputs into a high-dimensional feature space and update hidden layers through element-wise multiplication and addition.
- Residual Connections .: Separable PINNs use separate subnetworks to approximate the desired solution.
Model Decomposition.
Model decomposition in PIML spans separable representations, uncertainty-aware field decompositions, alternative differentiation strategies, domain decomposition, and staged training. These approaches target computational cost, high-dimensionality, sparse-data inference, scalability, and difficult dynamical or multiphysics problems.
- Separable and tensor decompositions: Separable representations encode each input dimension independently, addressing the curse of dimensionality and achieving up to 60-times speedups on high-dimensional PDEs.Reported examples include the 3D Helmholtz and 4D Navier-Stokes equations.
- Differentiation strategies: Automatic differentiation computes exact derivatives but becomes increasingly challenging as the number of problem dimensions grows.Derivative requirements increase with dimensionality, motivating alternatives such as Gaussian smoothing and stochastic dimension gradient descent.
- Differentiation strategies: Stochastic Dimension Gradient Descent randomly samples dimensional gradient components during training and enables PDE solutions with up to 100,000 dimensions.The method decomposes PDE and residual gradients into components associated with different dimensions.
- Domain decomposition: Domain decomposition partitions complex domains into subdomains whose models communicate through interface conditions, including prediction, residual, or flux continuity.Conservative PINNs enforce flux continuity across subdomain boundaries, while related methods add interface losses.
- Staged and uncertainty-aware decompositions: A staged hidden-field strategy first fits data and boundary conditions, then learns a theoretical representation, and finally incorporates full physics.This approach inferred hidden temperature fields from sparse experimental turbulent-velocity data.
- Broader model enhancements: The review identifies algorithmic advances including self-adaptive weights, feature expansion, fast derivative computation, domain decomposition, adaptive activations, and separable networks as responses to early PINN limitations.It also notes that PINNs can outperform FEM for inverse problems, while forward-problem competitiveness remains debated.