PaperlayerView exact revision on arXiv

Source-linked AI summary

Lecture notes on Physics Informed Neural Networks, Neural Operators, and their applications

Alessandro Bombini

arXiv:2609.17638v1cs.LGcs.AIcs.CEphysics.comp-ph

TL;DR

These notes introduce PINNs and Neural Operators and discuss implementing them for real-world applications. They frame PINN error through three contributions and emphasize the role of sampling strategy in obtaining accurate predictions.

  • Problem

    The course aims to introduce PINNs and Neural Operators and their implementation for applications in engineering, physics, and the oil industry.

    Source: §1.1 · p. 23

  • Method

    The course pairs theory with hands-on JuPyTer notebook practice and discusses implementation from scratch in PyTorch and with PhysicsNeMo.

    Source: §1.1 · p. 23

  • Results

    Three contributions combine to determine the total error between a trained PINN and the true solution.

    Source: §3.3 · p. 95

  • Takeaways & Limitations

    Quasi-random sequences yield lower median error and fewer outliers, highlighting sampling strategy as important for accurate PINN predictions.

    Source: §4.2.2 · p. 126

    The classical representation guarantees existence but does not provide a finite-dimensional parametrisation, an optimisation procedure, or statistical learning guarantees for recovering target-dependent functions from data.

    Source: §5.2.1.1 · p. 196

Abstract

from arXiv · show

This is the set of lecture notes for the PhD course \href{https://www.unibz.it/en/faculties/engineering/phd-computer-science/study-course-offering/2025/36967}{\textit{Physics Informed Neural Network}, held at the University of Bozen/Bolzano} in the academic year 2025/2026. The goal of the course was to introduce the concept of Physics Informed Deep Neural Networks (PINN) and Neural Operators (NOs), discuss their implementation from scratch in PyTorch and using advanced ad-hoc developed open-source libraries such as NVIDia PhysicsNeMo to address real-world problems in various fields (engineering, physics, petroleum reservoir). We discuss recent topics such as Mixture-of-Models, Fourier Neural Operators, Physics-Informed Kolmogorov-Arnold Networks (PIKANs) and Fourier Neural Operators.

the course

The course introduces PINNs and Neural Operators, their implementation, and applications in engineering, physics, and the oil industry. Its opening material combines theory with hands-on practice and begins by introducing functions and differentiability.

1.1 A brief introduction: why should I care?

  • The course aims to introduce PINNs and Neural Operators, their implementation, and applications in engineering, physics, and the oil industry.Source: §1.1 · p. 23
  • Each lecture is planned to combine theory with hands-on practice using JuPyTer notebooks.Source: §1.1 · p. 23

1.2 A brief introduction: what is a Partial Differential Equation?

  • The notes introduce differentiability by starting with a continuous real-valued function on a compact interval and stating that differentiability is defined by a limit.Source: §1.2 · p. 23

1. Lecture 1: General Introduction to the course

Lecture 1 lays mathematical foundations for PINNs and neural operators, reviewing differentiation, PDEs and boundary conditions through illustrative problems before introducing functional analysis and neural-network approximation. It also connects approximation and learning to Monte Carlo integration, sampling, and empirical-risk minimization.

1. Lecture 1: General Introduction to the course

  • The notes review single- and multivariable differentiation, including higher derivatives and the Jacobian as the matrix of partial derivatives.The gradient is described as a vector of derivatives that is a matrix-valued object when the function has multiple components.Source: §1 · pp. 24–25
  • Differential equations relate functions to their derivatives; ODEs involve one variable, PDEs involve fields, and differential problems require boundary or initial conditions.The notes also classify constant-coefficient second-order linear operators as elliptic, parabolic, hyperbolic, or ultra-hyperbolic according to the signs and zeros of their coefficient-matrix eigenvalues.Source: §1 · p. 25

1.2.1.1 A heuristic derivation of 1+1D heat equation as thermal diffusion

  • The heat equation is heuristically derived using continuity with Fick’s law, a random walk, and heat balance, with Taylor expansion used to obtain the limiting diffusion description.The heat-balance approach relates flux to temperature differences between neighboring cells and generalizes to higher dimensions by integrating differences over a ball.Source: §1.2.1.1 · pp. 26–27

1.2.1.2 Heated bar thermalisation via diffusion

  • A heated bar with a cosine initial temperature profile and zero-temperature endpoints is modeled by a diffusion Cauchy problem, solved by separation of variables.The fixed-temperature thermal bath produces outward heat flux, corresponding to heat loss through the boundary.Source: §1.2.1.2 · pp. 27–28

1.2.2 The effect of Robin conditions: the loose end violin string

  • For a violin string, a loose peg is modeled as an elastic restraint that permits endpoint motion and yields a Robin boundary condition involving tension and stiffness.The notes solve the string problem by separation of variables and use orthogonality of eigenfunctions satisfying Dirichlet/Robin conditions to relate modal coefficients to the pluck.Source: §1.2.2 · pp. 28–29
  • The loose-end Robin condition has its greatest effect on lower notes, while its eigenvalue equation requires numerical rather than closed-form solutions.The equation is tan(sqrt(λ)) = α sqrt(λ); numerical solutions are shown for various α.Source: §1.2.2 · pp. 30–31, Figure 1.2 · p. 31

1.3 Recap of Functional Analysis

  • Functional analysis provides the Banach, Hilbert, and Sobolev-space foundations for neural function approximation and weak-form PDE training in Lecture 3.Weak derivatives define Sobolev spaces; L2(Ω) and Wk,2(Ω) are Hilbert spaces, and separable Hilbert spaces allow generalized Fourier-series representations.Source: §1.3 · pp. 31, 35, 36

1.3.1 From functions to functionals

  • Operators map between function spaces, whereas functionals map a function space to scalars; linear differential operators are examples of operators.The Riesz representation theorem identifies each bounded linear functional on a Hilbert space with a unique inner-product representative.Source: §1.3.1 · pp. 36–37
  • Hilbert or Banach spaces support weak PDE formulations and functional-analysis results, whereas C^m(Ω) spaces are not complete under the relevant norms.The notes connect weak formulations to functionals associated with PDEs and state that completeness helps establish existence and uniqueness and supports numerical methods.Source: §1.3.1 · p. 37

1.3.2 The Cybenko’s Theorem

  • Finite sums of continuous, bounded, discriminatory activation functions are dense in C([0, 1]^n) under the uniform norm, establishing the universal approximation property.Thus, for any continuous function and ε > 0, a network with sufficiently many neurons can approximate it uniformly within ε.Source: §1.3.2 · p. 40
  • Deep networks often approximate better with fewer total neurons than shallow networks, and some functions require exponentially more neurons in shallow networks [ES16; Tel16].The notes describe this depth efficiency as more computationally efficient and effective than widening a single layer.Source: §1.3.2 · p. 42

1.4 Recap of Monte Carlo Integration methods

  • Monte Carlo integration gives an unbiased estimator for integrals of functions over bounded domains in any dimension, with convergence rate dependent on N.The notes also describe sampling from a target density by drawing uniform values on [0, 1] and applying the inverse cumulative distribution function.Source: §1.4 · pp. 42, 44
  • O(N^-1/2) is the Monte Carlo integration convergence rate in root-mean-square.Source: §1.4 · p. 45

1.4.1 Relevance for Machine Learning

  • Because the target and input distribution are unknown, learning uses labeled i.i.d. samples to approximate population risk with empirical risk, whose minimization evaluates DNN-target distance by Monte Carlo [Sca24].Source: §1.4.1 · pp. 45–46

Lecture 2: An introduction to numerical resolution of differential

Numerical methods recast differential problems as algebraic ones that standard methods can solve; the lecture introduces grid- and mesh-based approaches before closing with a meshless method.

2.1 Lecture 2: An introduction to numerical resolution of differential

  • FDMs approximate the function on a regular grid and replace derivatives with incremental ratios between neighboring nodes.Source: equations · p. 47
  • FEMs use linear approximations on an irregular mesh and stitching conditions at edges and nodes to solve the PDE's weak formulation.Source: equations · p. 47
  • FVMs use linear approximations on an irregular mesh and impose flux conditions at edges to satisfy continuity.Source: equations · p. 47
  • The meshless Kansa Method [Kan90] uses collocation with an approximator to satisfy the differential equations, recasting the problem as optimization.Source: equations · p. 47

2. Lecture 2: An introduction to numerical resolution of differential equations

This lecture introduces finite differences for approximating derivatives and solving numerical PDE problems, including Poisson, diffusion, inpainting, and Burgers examples. It also connects spatial stencils to CNN kernels and graph diffusion, developing graph-Laplacian spectral properties and their use in dimensionality reduction and graph convolution.

2.1 Finite Difference Methods (FDM)

  • Central differences approximate derivatives with O((∆x)^2) error, while enlarging the stencil can provide higher derivative order or precision.The notes also describe derivatives as discrete convolutions with fixed linear kernels.Source: §2.1 · pp. 48–49

2.1.1 A first application of finite differences: Sobel Edge detection algorithm

  • The Sobel filter combines a first-order difference kernel [−1, 0, +1] with the discrete Gaussian smoothing kernel [1, 2, 1] to detect image edges.Source: §2.1.1 · p. 50

2.1.2 Using FDM to solve Poisson Equation: the Jacobi method

  • For a three-dimensional Poisson problem with mixed Dirichlet/Neumann boundaries, FDM discretizes the Laplacian and Jacobi iteration updates the field until its update error meets a threshold.The iteration imposes boundary conditions after each update.Source: §2.1.2 · pp. 51, 53

2.1.2.1 Jacobi Method as a Relaxation of a Diffusion Equation

  • Whenever it converges, Jacobi iteration can be interpreted as explicit pseudo-time evolution toward the steady state of a corresponding diffusion equation.Source: §2.1.2.1 · p. 53

2.1.3 An application in Computer Vision: image inpainting with Laplace Equation

  • Less than 10% of an image’s pixels sufficed for optimal compression followed by reconstruction with the 2D Laplace PDE.The notes present this as a high-compression inpainting result.Source: §2.1.3 · p. 55

2.1.4 Solving Burgers equation with FDM

  • The Burgers example uses explicit time stepping, whose time-step size is constrained by stability conditions involving advective and diffusive terms.The notes describe the chosen parameters as providing a stable illustrative computation and specify forward Euler in time with central differences in space.Source: §2.1.4 · pp. 55–56
  • For small viscosity, nonlinear advection steepens the Burgers solution into a sharp, shock-like internal layer, while positive diffusion smooths it.A genuine finite-time shock discontinuity occurs in the inviscid limit ν = 0.Source: §2.1.4 · p. 56, Figure 2.5 · p. 57

2.1.5 A heuristic connection to CNN and Graphs: Stencil representation & Heat equation on Graphs

  • A stencil approximates the Laplacian with central differences, linking discretized field operations to CNN kernels convolved with their inputs.The notes describe this analogy for fields in 1D, 2D, and 3D.Source: §2.1.5 · pp. 56–57
  • On an undirected graph, temperature diffusion is written ∂_t u(t) = -α(D - A)u(t), where D - A is the graph Laplacian.A is the symmetric adjacency matrix, and D is the degree matrix.Source: §2.1.5 · p. 58

2.1.5.1 The spectral theory on the graph and Laplacian Eigenmaps as an unsupervised dimensional reduction method

  • For undirected graphs, the graph Laplacian is real symmetric and positive semidefinite, with nonnegative eigenvalues and a simple zero eigenvalue when the graph is connected.The zero eigenvalue corresponds to the constant eigenvector; the notes state that the number of zero eigenvalues equals the number of connected components.Source: §2.1.5.1 · p. 59
  • Laplacian Eigenmaps [BN03] builds a neighborhood graph and represents data using nontrivial low-eigenvalue eigenvectors, retaining slowly varying structure while reducing dimensionality.The method selects low-eigenvalue modes because high-frequency modes may capture fine-scale features or noise.Source: §2.1.5.1 · pp. 60–61
  • The notes relate heat kernels to Laplacian spectral expansions, while cautioning that convergence and regularity of the series are delicate.The heat kernel is also described through its associated integral transform and operatorial representation via exponentiation of the Laplacian.Source: §2.1.5.1 · p. 61
  • Spectral graph convolutions [Bru13; DBV16] filter signals in the graph-Laplacian eigenbasis and transform them back, but explicit eigendecomposition is costly.This cost motivated approximate spectral methods such as ChebNet and GCN that avoid explicit diagonalization.Source: §2.1.5.1 · p. 62

2.2 Finite Element Methods (FEM)

This section surveys approaches for solving PDEs, introducing FEM through its weak-form reduction and Ritz and Galerkin formulations, alongside FVM as a conservation-based alternative. It also covers finite-volume flux updates and meshless approximation, including Kansa collocation, its numerical limitations, and a least-squares extension.

2.2 Finite Element Methods (FEM)

  • FEM rewrites a PDE in weak form, discretizes it with finite-dimensional local-support basis functions, and assembles stiffness and mass matrices to obtain an algebraic problem.For the Laplace problem with pure Dirichlet conditions, the notes give the system -Lα = MJ.Source: §2.2 · pp. 62, 63, 65

2.2.1 Ritz vs Galerkin method

  • Ritz minimizes an energy functional and typically yields symmetric positive definite systems, while Galerkin projects the weak residual onto a test space and also applies without an energy functional.For Poisson’s problem the Ritz and Bubnov–Galerkin formulations coincide; Galerkin also accommodates non-self-adjoint operators, though stability depends on the trial–test pairing.Source: §2.2.1 · pp. 66–68

2.3 Finite Volume Methods (FVM)

  • FVM integrates balance laws over control volumes, representing unknowns as cell averages and enforcing local conservation through a single shared flux on each interior face.Opposite-sign shared fluxes cancel in summed cell balances, giving global conservation up to boundary fluxes and source terms; this structure also suits unstructured meshes.Source: §2.3 · pp. 68–69
  • FVM updates each cell from net boundary flux and internal sources, using numerical face fluxes and a Forward Euler time step.Boundary-face fluxes are chosen consistently with the prescribed boundary condition.Source: §2.3 · pp. 69–70

2.4 Meshless methods; Kansa’s approach

  • Splines support local refinement through knot spacing, while RBF interpolation represents functions as shifted radial profiles, optionally with a low-degree polynomial [Buh00].Spline accuracy also depends on polynomial degree and knot distribution.Source: §2.4 · pp. 70, 72

2.4.1 The Kansa approach: RBF for numerical PDE resolution

  • Kansa approximates the solution with RBF centers, enforces PDE and boundary conditions at collocation points, then optimizes the resulting algebraized error because solvability is not guaranteed.The notes identify stochastic gradient descent and Newton methods as possible optimizers.Source: §2.4.1 · pp. 73–74

An historical note:

  • The classical square Kansa method [Kan90] solves Ma = w with Neq = Nc, but may be singular or ill-conditioned; overtesting Neq > Nc gives a least-squares extension.Conditioning can depend strongly on the kernel and its shape parameter.Source: An historical note · pp. 74–75

Lecture 3: Introduction to Physics

The notes identify geometry and physics as ingredients of a differential problem, then distinguish forward, inverse, and parametric problems by what is known and what solution is sought. They introduce DNNs as function approximators for unknown functions.

Informed Neural Networks

  • A differential problem is described through its domain and boundary behaviour, together with its source and dynamics, which may depend on parameters.Source: Informed Neural Networks · p. 79
  • Forward problems fix geometry and physics; inverse problems add observations while leaving some physical parameters unknown; parametric problems seek solutions across parameters or a solution map.Source: Informed Neural Networks · p. 79

3.1 Forward Problems: Vanilla PINNs

  • For vanilla PINNs, a DNN represents a function from a domain in R^d to R^n, using a function class satisfying a Universal Approximation Theorem's hypotheses.The notes assume familiarity with DNNs and, in particular, MLPs.Source: §3.1 · p. 79

3. Lecture 3: Introduction to Physics Informed Neural Networks

This section presents vanilla and parametric PINNs as meshless PDE solvers and solution learners, covering their limitations and uses in data assimilation, inverse problems, and parameterized design exploration. It also examines sources of error and training behavior through learning-phase, information-bottleneck, and batch-wise SNR analyses.

3. Lecture 3: Introduction to Physics Informed Neural Networks

  • A vanilla PINN uses network derivatives to form a sampled PDE and boundary residual loss, turning the differential problem into optimization over network parameters.The trained network is only a candidate PDE-solution approximation; the relation between discrete residual minimization and actual solution error is treated later.Source: §3 · pp. 81–82

3.1.1 An historical note

  • PINNs did not invent neural-network PDE solvers; their novelty was synthesizing earlier collocation ideas with deep architectures, automatic differentiation, modern optimization, and forward, inverse, and data-assisted problems.The notes attribute their impact to this unified formulation and compelling demonstrations, which helped catalyze wider adoption.Source: §3.1.1 · p. 84

3.1.2 A brief comment on vanilla PINNs

  • Without observational data, vanilla PINNs are meshless collocation solvers akin to Kansa, differing in MLP bases, stochastic sampling, automatic differentiation, and nonlinear residual-loss optimization.Kansa classically yields a dense linear system from analytic RBF derivatives, whereas PINNs use automatic differentiation and nonlinear optimization.Source: §3.1.2 · pp. 84–86

3.1.3 A key comment on vanilla PINNs: strengths... and limits

  • Vanilla PINNs fail on some classes of problems; proposed remedies target specific contexts, and their effectiveness is not guaranteed across contexts.The notes frame PINNs as an additional tool for solving differential problems.Source: §3.1.3 · pp. 86–87
  • Vanilla PINNs rely solely on governing equations as meshless solvers and face difficulties, whereas data-informed PINNs can combine sparse observations and physics for assimilation and inverse problems.Neural-network expressive power supports complex, high-dimensional solution representations.Source: §3.1.3 · p. 87

3.2.1 PINNs for inverse problems

  • PINNs infer unknown physical quantities by fitting observations while enforcing governing equations, but these constraints do not generally eliminate inverse-problem non-identifiability or instability [YMK21a].Additional priors, regularization, uncertainty modeling, or suitable observation design may still be needed, particularly with noisy data.Source: §3.2.1 · pp. 88–89

3.2.2 PINNs for parametric problems

  • Parametric PINNs embed γ in the network input to learn a solution generator across parameterized geometries, PDEs, or boundary and initial conditions.This surrogate can support exploration or optimization without repeatedly solving the PDE, including when γ is discrete, non-differentiable, or the downstream objective is non-smooth.Source: §3.2.2 · p. 92
  • A single parameterized neural solver substantially reduces total design-space exploration cost versus the corresponding collection of OpenFOAM runs.For the heat-sink benchmark, the trained network can be evaluated on new geometries without solving the forward problem again.Source: §3.2.2 · p. 93
  • Parametric PINNs include boundary parameters or representations of initial profiles in the network input to predict solutions across those variations without resolving the PDE each time.They approximate finite-dimensional parameter-to-solution maps, whereas Neural Operators target maps between function spaces.Source: §3.2.2 · pp. 94–95

3.3.1 The PINN Learning Problem as Risk Minimization

  • PINN training minimizes a discrete physics-based risk at practitioner-chosen collocation points, which approximates a continuum risk through numerical quadrature rather than statistical generalization.The discrete objective combines PDE residual and boundary or initial-condition terms, with quadrature weights.Source: §3.3.1 · pp. 96–97

3.3.2 Approximation Error

  • Two-hidden-layer tanh networks can approximate smooth functions and derivatives at algebraic rates, provided the target has the required Sobolev regularity.For a PDE of order m, the notes state that the network should be at least W^{m,∞}; approximation also depends on smoothness, activation, and architecture.Source: §3.3.2 · p. 97

3.3.3 Estimation (Quadrature) Error

  • With N collocation points, the O(N^-1/2) term reflects Monte Carlo quadrature error; C depends on PDE coefficients and domain geometry, and the bound is dimension-independent at least for Kolmogorov-type equations.If collocation points do not adequately resolve regions where the PDE residual is large, adaptive sampling and residual-based refinement can significantly improve PINN accuracy.Source: §3.3.3 · p. 98

3.3.4 Optimization Error

  • PINN optimization can stall at local minima, saddle points, or flat regions, while Adam often reduces residuals early and L-BFGS refines solutions later.Finite-width non-convex training lacks general architecture-independent global convergence guarantees, and optimization may dominate for nonlinear or multiscale problems [Kri+21; WTP21].Source: §3.3.4 · p. 99

3.3.5 Total Error Decomposition

  • The continuous risk of a trained PINN is bounded through optimization, estimation, and approximation errors, which can interact as sampling or network size changes.Insufficient sampling can leave PDE violations between collocation points; larger networks can reduce approximation error while making optimization harder.Source: §3.3.5 · pp. 99–100

3.3.6 Training Dynamics of PINNs: connection to Optimization Dynamics and Information Flow

  • PINN training can exhibit distinct learning phases, with parameter evolution determining which error component dominates over time [Ana+25; Tos+25].The notes connect these dynamics to optimization and information flow.Source: §3.3.6 · p. 100

Hints on DNN training dynamics from Information Bottleneck Theory

  • Information Bottleneck theory describes representations that preserve information relevant to outputs while discarding irrelevant input information; reported dynamics include fitting followed by compression [ST17; TZ15].In the illustrated setup, both input and output mutual information rise during fitting, then input mutual information falls while output mutual information stabilizes.Source: Hints on DNN training dynamics from… · p. 100, Figure 3.7 · p. 101
  • The reported information-bottleneck compression pattern is not universal: it depends on activation functions and estimators, is not required for generalization, and binning estimates can track geometric clustering [ST17; Sax+19; Gol+18].The initial fitting followed by slower compression or diffusion was reported for the networks and estimator studied in [ST17].Source: Hints on DNN training dynamics from… · p. 103
  • Because forward PINNs lack supervised labels, the classical supervised information-bottleneck picture is an analogy, not an identification of the PINN objective with the IB functional.For deterministic networks, I(X;T) can be infinite for continuous inputs, while finite information-plane values depend on choices such as binning, noise, or estimators [AG19; Gol+18].Source: Hints on DNN training dynamics from… · pp. 103–104
  • Batch-wise SNR measures agreement among gradients across batches or partitions at fixed parameters and can be evaluated during full-batch training without treating its noise as SGD noise [Ana+25; Shu+24].High SNR indicates gradient agreement; low SNR indicates heterogeneity and cancellation.Source: Hints on DNN training dynamics from… · pp. 104–105
  • Fitting begins with high SNR and largely aligned, deterministic descent; diffusion follows as subdomain gradients disagree and cancel, SNR stays low or oscillates, and residuals become disordered [Shu+24; Ana+25].The transition is associated with stochastic exploration and improved generalization.Source: Hints on DNN training dynamics from… · pp. 105–107
  • In total diffusion, SNR rises sharply and stabilizes as coherent descent emerges, residuals remain disordered, and generalization error falls rapidly; PINNs entering this phase earliest perform best [Ana+25; Shu+24].The Helmholtz example shows the model closely matching the analytical solution in this phase.Source: Figure 3.9 · p. 105, Hints on DNN training dynamics from… · p. 106
  • PINN losses combining PDE residuals, boundary conditions, and data terms can create gradient conflicts, contributing to misalignment and potentially multiple transitions between deterministic and stochastic regimes.Source: Hints on DNN training dynamics from… · p. 107

Lecture 4: Advanced PINNs methods - Learning strategies, Architectures, Losses, and other

This section surveys five categories of techniques for improving PINN training and connects them to broader machine-learning topics.

approaches

  • The section groups PINN training improvements into adaptive loss weighting, sampling and geometry handling, architectural innovations, optimisation strategies, and best practices.Source: approaches · p. 109
  • It aims to show how ideas from other machine-learning topics may benefit PINN optimisation.Source: approaches · p. 109
  • The techniques are considered across a range of PINN tasks.Source: approaches · p. 109

4.1 Dynamic Hyperparameter Optimisation

This section addresses optimization imbalance, stiffness, and slow convergence in PINNs by reviewing dynamic loss weighting and gradient-direction methods, including NTK-based balancing and M-ConFIG.

4.1.1 GradNorm

  • GradNorm [Che+18b] adapts task weights to align gradient norms with a common baseline and assigns larger magnitudes to tasks with slower relative training rates.The strength of this restoring force is controlled by the additional hyperparameter α.Source: §4.1.1 · p. 111

4.1.2 Homoscedastic Task Uncertainty

  • Homoscedastic task uncertainty [KGC18] derives loss weights from learned task noise, allowing PINNs to balance PDE, boundary, and data terms without manual weight tuning.The log-noise term prevents the trivial solution of increasing uncertainty indefinitely to down-weight a loss.Source: §4.1.2 · pp. 112–113

4.1.3 Learning Rate Annealing

  • Learning Rate Annealing [WTP21] adjusts PDE-to-boundary loss weights using gradient-magnitude ratios smoothed by an exponential moving average.It aims to balance gradient contributions and mitigate stiffness when the losses evolve at different rates.Source: §4.1.3 · p. 114

4.1.4 SoftAdapt

  • SoftAdapt [HTM19] assigns larger softmax-based weights to loss terms that decrease slowly or increase, using changes in loss values rather than gradient magnitudes.The described formula is a subsequent variation; the original paper uses the difference between consecutive loss values.Source: §4.1.4 · p. 115

4.1.5 ReLoBRaLo

  • ReLoBRaLo [BK25] combines moving-average weight smoothing with random lookback and can outperform SoftAdapt in some stiff PINN problems.The lookback reduces sensitivity to transient loss behavior, while smoothing stabilizes weights across iterations.Source: §4.1.5 · pp. 115–116

4.1.6 ConFIG

  • ConFIG [LCT25] constructs a consensus direction from individual loss gradients to reduce destructive interference, modifying gradient directions rather than reweighting loss values.Its gradient-level formulation is compatible with standard optimizers such as Adam, as demonstrated by M-ConFIG.Source: §4.1.6 · pp. 116–117
  • M-ConFIG preserves Adam’s update structure but constructs a conflict-free gradient from loss-specific gradients, resolving gradient conflicts in multi-objective PINN training.It adds consensus-direction computation and projection while retaining Adam’s moment estimation and bias correction.Source: §4.1.6 · p. 118
  • M-ConFIG’s extra cost comes chiefly from loss-specific gradients and consensus construction, scales linearly with loss terms, and is typically less than a few percent of training time [LCT25].The added operations use vector normalisation and inner products, which are negligible compared with backpropagation.Source: §4.1.6 · p. 118

4.1.7 Neural Tangent Kernel-based Adaptive Weighting

  • NTK analysis links training dynamics to kernel eigenvalues; [JGH18] shows that, in the infinite-width limit, the NTK is constant and training follows a linear differential equation.For PINNs, spectral imbalance between PDE residuals and boundary conditions can cause stiffness and slow convergence [WYP22].Source: §4.1.7 · p. 119
  • The [WYP22] strategy periodically scales each loss term using the ratio of the total NTK trace to the trace of its corresponding block.The eigenvalues of the PDE-residual and boundary-condition NTK blocks govern the convergence rates of their respective losses.Source: §4.1.7 · p. 120
  • NTK-based weighting makes loss terms evolve at comparable rates and can improve convergence in stiff PINN problems with unbalanced sensitivities [WYP22].It compensates for NTK anisotropy and helps prevent overfitting to components associated with larger eigenvalues.Source: §4.1.7 · p. 120

4.2 Advanced Schemes

This section surveys advanced techniques for improving PINN optimization, estimation, and training on difficult problems. It covers Sobolev training, sampling strategies, training schedules and initialization, exact Dirichlet boundary enforcement using distance functions and R-functions, and curriculum learning.

4.2.1 Sobolev training

  • Sobolev training minimizes PDE residuals and their derivatives, potentially accelerating convergence and benefiting high-dimensional PDEs [Son+21].Poorly chosen weights can harm convergence and accuracy, while higher-order differentiation adds backpropagation and training cost.Source: §4.2.1 · p. 122

4.2.2 Quasi-random sampling

  • Low-discrepancy sequences improve sampling uniformity, while the finite Hammersley design combines a stratified first coordinate with van der Corput sequences in the remaining coordinates.Unlike Halton, Hammersley requires an upper bound on sample count and is suited to finite design-of-experiments settings [Ham60].Source: §4.2.2 · pp. 123–124
  • Quasi-random designs such as Sobol, Halton, Hammersley, and Faure outperform random or deterministic designs in prediction error, with lower median error and fewer outliers.More uniform sampling improves quadrature of the PDE residual, reducing estimation error and improving training stability.Source: §4.2.2 · p. 126, Figure 4.4 · p. 126

4.2.3 Importance sampling

  • Importance sampling draws more collocation points where PDE residuals are large and reweights them to preserve an unbiased loss estimate.An adaptive strategy periodically updates the sampling density from residual magnitudes, but estimating and updating the distribution adds computational overhead [NGM21].Source: §4.2.3 · p. 127

4.2.4 Approximate distance functions and R-functions for exact boundary conditions

  • A geometry-aware trial space can satisfy Dirichlet data by design, addressing errors and instability caused by soft boundary penalties [SS22].The construction uses an extension of the prescribed boundary data and a scalar field that vanishes on the boundary.Source: §4.2.4 · pp. 128–129
  • R-functions compose primitive implicit geometries into boundary representations, and an approximate distance function gives u=g+ϕuθ, enforcing Dirichlet data exactly with interior PDE-residual training [Rva75; Sha91].The construction can reduce the PINN search space and eliminate Dirichlet penalty terms; [SS22] reports empirical demonstrations across two-dimensional geometries.Source: §4.2.4 · pp. 129–130
  • Smooth regularization must balance signed-distance fidelity near the boundary against gradient underflow, while corners, cusps, and nondifferentiable operations may require smoothing or refinement.A unit normal derivative can be imposed by local rescaling or normalization; differentiability is needed for stable PDE residuals and automatic differentiation.Source: §4.2.4 · pp. 129–130

4.2.5 Curriculum learning

  • Curriculum learning progressively shifts PINN training from easier parameter regimes or constraints to the target problem, aiming to reduce estimation and optimization errors [Ben+09].Approaches include parameter continuation, progressively introduced collocation or boundary/initial data, and dynamic loss weighting; these can be combined.Source: §4.2.5 · pp. 130–131
  • For advection, increasing β makes solutions more oscillatory and can cause vanilla PINNs to fail; starting at small β and advancing toward the target can improve training time and final accuracy [MA23; Kri+21].Curriculum can mitigate gradient pathologies by steering parameters toward favorable regions, especially when easier-task solutions connect continuously to the target; simple schedules or a few intermediate steps may suffice [Ben+09; Kri+21].Source: §4.2.5 · p. 131

4.3 Architectures

This section surveys PINN architectures and training approaches, including network design choices, strong- and weak-form methods, and Fourier- and sine-based representations. It also addresses spectral-bias challenges and describes Transformer and mixture-of-experts architectures, including the roles of coordinate changes and expert specialization.

4.3 Architectures

  • Although many function-approximating architectures can implement PINNs given sufficient regularity, practice may require architectural intervention [Cuo+22].The notes list fully connected, convolutional, recurrent, Bayesian, and generative adversarial networks as architectures used for PINNs.Source: §4.3 · p. 131, Table 4.1 · p. 132

AE CNN

  • PINN designs often benefit from small networks, while spectral bias is also relevant to physics-informed problems and implicit neural representations [Cuo+22] [Row26] [Liu+24a] [Sit+20].The notes connect implicit neural representations to learning continuous functions from coordinates to visual signals.Source: AE CNN · pp. 132–133

4.3.1.1 Adaptive Activations

  • Global adaptive activations add a trainable slope parameter per network, and [JKK20a] reports faster convergence, especially early in training, and greater solution accuracy than fixed activations.The parameter is optimized alongside weights and biases; the notes interpret the change as modifying loss geometry rather than simply rescaling the learning rate.Source: §4.3.1.1 · pp. 133–134

4.3.1.2 Random Weight Factorisation

  • Random Weight Factorisation [Wan+22] writes each neuron’s weight vector as a trainable scale times a trainable direction, yielding a self-adaptive learning rate for each neuron.The notes describe this as a weight-space reparameterisation and relate it to weight normalisation.Source: §4.3.1.2 · pp. 134–135

4.3.2 Deep Galerkin Method, Deep Ritz Method, and Variational PINNs

  • The notes distinguish Ritz energy minimisation, which requires a variational PDE formulation, from Galerkin projection, which can apply without an energy functional.They classify PINN and DGM as pointwise strong-residual methods, Deep Ritz as energy minimisation, and vPINN as using a finite Petrov–Galerkin test space.Source: §4.3.2 · pp. 136–137

4.3.2.1 The Deep Ritz Method

  • The Deep Ritz Method [Yu+18] approximates a trial function with an MLP, evaluates the functional by numerical quadrature, and optimizes the resulting problem.The notes describe stochastic gradient descent over independently sampled domain points, redrawn each epoch.Source: §4.3.2.1 · p. 137
  • For the stated problem, using β = 500, a simple MLP with 811 trainable parameters produced an L2 error of 0.0072.Source: §4.3.2.1 · p. 138

4.3.2.2 The Deep Galerkin Method

  • The Deep Galerkin Method [SS18] is introduced for a Cauchy problem involving a time derivative and a spatial operator.The supplied passage gives the problem setup but no further method details or results.Source: §4.3.2.2 · p. 138
  • DGM redraws samples each epoch and uses an LSTM-inspired gated architecture, introduced as effective for high-dimensional PDEs [SS18].This differs from PINNs, which draw a large point set before training and subsample it each epoch.Source: §4.3.2.2 · p. 139
  • Despite its name, DGM minimizes pointwise strong-residual errors rather than a weak-form Galerkin objective [SS18].The notes characterize it as closer to strong-form residual least squares or meshfree collocation.Source: §4.3.2.2 · p. 140

4.3.2.3 Variational PINNs

  • vPINNs [KZK19] enforce weak-form residuals against a finite set of admissible test functions and minimize a variational loss.The method is described as a deep-learning approach to (Petrov-)Galerkin methods, unlike DGM’s strong-form approach.Source: §4.3.2.3 · p. 141
  • Integration by parts can lower the derivative order required by automatic differentiation; vPINNs trade pointwise collocation for numerically integrated residual moments, making quadrature accuracy part of discretization.For a Poisson residual with zero test functions on the boundary, only first derivatives of the trial solution are needed instead of second derivatives.Source: §4.3.2.3 · p. 141

4.3.3 Spectral Bias and Fourier Embeddings

  • ReLU networks exhibit spectral bias: gradient-based fitting learns low frequencies earlier, while high-frequency components occupy less parameter-space volume, not because they are inexpressible [Rah+19].A ReLU network’s Fourier spectrum decays, and the parameter volume capable of representing high frequencies shrinks.Source: §4.3.3 · p. 142
  • ReLU spectral decay is direction-dependent, not a single isotropic O(k^-n-1) law; along exceptional face-orthogonal directions, the slowest nontrivial decay can be O(k^-2) [Rah+19].The notes caution against interpreting the decay as one fixed exponent across directions.Source: §4.3.3 · p. 144
  • MLPs favor low frequencies, which can aid smooth targets but slow learning of essential oscillatory components, especially in coordinate-based PINNs [Tan+20; Tos+25].The low-frequency preference may also reduce sensitivity to high-frequency noise.Source: §4.3.3.1 · p. 147

4.3.3.1 Fourier Feature Embedding and Fourier Network

  • Positional encoding [RR07], popularized by NeRF [Mil+21], maps coordinates to sinusoidal Fourier features with fixed or learnable modes, expanding the input space for MLPs [Tan+20; WWP21b].The encoding uses sine and cosine components for each selected mode.Source: §4.3.3.1 · pp. 147–148
  • Fourier feature frequency distributions change the effective stationary-kernel bandwidth, potentially accelerating high-frequency convergence, while excessive bandwidth risks overfitting or aliasing [Tan+20].Thus, Fourier features alter both optimization dynamics and the coordinate MLP’s implicit spectral bias.Source: §4.3.3.1 · p. 148

4.3.3.2 Modified Fourier Network family

  • Modified Fourier nets [WTP21] project Fourier features into learned spaces for element-wise hidden-layer updates, while Highway Fourier nets [Nvi26] gate information flow in deep architectures.The modified network uses learnable transformation parameters; highway gates adaptively control how much encoded information passes onward.Source: §4.3.3.2 · p. 149
  • Multi-scale Fourier networks [WWP21b] combine embeddings across frequency ranges; spatiotemporal variants use separate spatial and temporal scales, shared layers, and branch-wise products before output combination.These designs target PDEs with multiscale behavior and allow simultaneous capture of low- and high-frequency features.Source: §4.3.3.2 · pp. 149–150

4.3.4 Sinusoidal Representation Networks

  • SiRen applies sine activations throughout, so its derivatives retain the architecture’s periodic properties and can be supervised directly.Fan-in-scaled initialization supports deep training, and the first layer is scaled by ω, with ω = 30 suggested for many circumstances.Source: §4.3.4 · p. 151
  • A five-layer fully connected SiRen represents a full room, while sinusoidal activations improve neural-SDF object detail and scene complexity [Sit+20].SiRen is fitted to signed distance functions on point clouds.Source: §4.3.4 · p. 152, Figure 4.11 · p. 152

4.3.4.1 A simple example of emergence of Spectral Bias effects in PINN training: continuity equation

  • A simple continuity equation can nevertheless pose substantial difficulty for PINNs, illustrating spectral-bias effects in training [MA23].The example is ∂_t u(t, x) + β∂_x u(t, x) = 0 on [0, T] × [-1, +1].Source: §4.3.4.1 · p. 153
  • For β > 30, PINNs struggle despite Fourier features, curriculum learning, and causal loss, while curriculum variants had comparatively high convergence rates across tested collocation sizes.Curriculum learning over β [MA23] is interpreted as a moving time window, which helps at least over the initial part of the time domain.Source: §4.3.4.1 · pp. 153–154
  • Changing t to τ = βt removes high-frequency modes from the solution but preserves its oscillation count over a longer interval, shifting the challenge to long-time causality and error accumulation.For these convection problems, increasing the PDE coefficient also makes residual optimization progressively harder and creates a more complex loss landscape [Kri+21].Source: §4.3.4.1 · p. 154

4.3.5 Transformers in PINNs

  • PINNsFormer [ZDP23] adapts Transformers to time-evolution PINNs using pseudo-sequences, a spatio-temporal mixer, and physics-informed loss evaluated over generated sequences.Its pseudo-sequence generator maps pointwise inputs to sequential solution approximations, while the mixer projects spatio-temporal features into a higher-dimensional space.Source: §4.3.5 · pp. 154–156
  • A two-hidden-layer, infinite-width network with Wavelet activation is a universal approximator for any real-valued target function.The notes motivate Wavelet activation through the Real Fourier Transform and use this result to explain its expressive power in PINNsFormer [ZDP23].Source: §4.3.5 · pp. 154–155

4.3.6 Mixture-of-Experts

  • MoE-PINNs [BK22] gate weighted predictions from heterogeneous experts; on Burgers, the gate assigned simpler experts to smooth regions and complex ones to discontinuities.Expert 3 was dedicated solely to capturing the discontinuity, while the gate's continuous weights allow smooth transitions between domains.Source: §4.3.6 · pp. 157–159, Figure 4.16 · p. 159

4.4 Optimisation Schemes

This section surveys optimization schemes for PINNs, covering second-order and quasi-Newton methods and the trade-offs motivating L-BFGS and self-scaled Broyden methods. It also describes weighting strategies for balancing loss contributions, including lambda weighting, spatial residual weighting, residual-based attention, and temporal schemes that encourage causal learning.

4.4.1 Optimisers: second order over first order

  • Gradient descent uses the loss gradient, but a learning rate that is too small slows convergence while one that is too large can cause oscillation or divergence.The notes contrast this with Newton’s method, which uses second-order curvature information but requires a Hessian inverse that may need regularization.Source: §4.4.1 · pp. 159–161

4.4.1.1 BFGS and its limited memory version

  • Newton’s method can converge to a maximum or saddle point for non-convex objectives, and its inverse-Hessian computation scales as O(n^3), motivating quasi-Newton methods such as BFGS.BFGS approximates the Hessian with an iteratively updated positive-definite matrix; L-BFGS stores recent curvature pairs, reducing storage and recursion cost to O(mn) rather than O(n^2), with m ≪ n.Source: §4.4.1.1 · pp. 162, 164
  • In some PINN applications, L-BFGS can achieve better performance in fewer iterations than first-order methods, but is more prone to saddle-point traps; Adam warm-up followed by L-BFGS fine-tuning is recommended.The notes describe this as an empirical observation and a mitigation recommended by some studies.Source: §4.4.1.1 · p. 164

4.4.1.2 Self-Scaled Broyden (SSBroyden)

  • Self-Scaled Broyden generalizes the Broyden/BFGS family with scaling parameters for the inverse-Hessian approximation, aiming to improve robustness and global convergence in large-scale, ill-conditioned problems.When its scaling parameters are both 1, the update reduces to standard BFGS.Source: §4.4.1.2 · pp. 164–165

4.4.2.1 Lambda Weighting

  • Lambda weighting changes the local weight assigned to loss terms and can be interpreted as changing the measure, or heuristically adding intrinsic curvature, on the domain.The weighting may depend on training time and PDE time as well as spatial position.Source: §4.4.2.1 · p. 167

4.4.2.2 Signed Distance Function as Lambda Weighting

  • SDF weighting speeds convergence for pressure and all three velocity fields in training on a 17-fin heat sink [Nvi26].The comparison is shown for the initial 100,000 iterations; the closely spaced fins create sharp residual gradients near the heat sink.Source: §4.4.2.2 · p. 169, Figure 4.18 · p. 169

4.4.2.3 Residual Based Attention as Lambda Weighting

  • Residual-based attention (RBA) uses cumulative residuals to focus on difficult spatial and temporal regions without gradient calculations for its weights [Ana+23].The scheme extends the optimizer’s attention span and adds negligible computational cost.Source: §4.4.2.3 · p. 170

4.4.2.4 Temporal Loss Weighting: causal loss

  • Temporal loss weighting with a time-marching schedule [Kri+21] penalizes earlier-time errors more and expands the simulation interval during training.Linear decay is given as an example of temporal loss weighting, while the time interval can move linearly.Source: §4.4.2.4 · p. 171
  • Causal loss [WSP22] weights time chunks so later residual losses have little effect until residuals in earlier chunks decrease.Continuous-time PINNs can violate temporal causality and converge to erroneous transient solutions; causal weights depend exponentially on cumulative residual loss from previous chunks and should remain outside the computational graph.Source: §4.4.2.4 · pp. 172–173

4.5 Recap-ish: An Expert’s Guide to Training Physics-informed Neural Networks

The section recaps PINN training as neural approximation of a differential-equation solution, with sampled points used to impose PDE and initial or boundary constraints. It also summarizes a recommended training recipe that combines embeddings, causal loss, and adaptive loss balancing.

4.5 Recap-ish: An Expert’s Guide to Training Physics-informed Neural Networks

  • PINN training samples the domain for PDE constraints and initial or boundary constraints, uses a neural network to approximate the solution, and forms a multi-objective loss from the resulting residuals.The notes describe the network output uθ(t, x) as an approximation to u(t, x).Source: §4.5 · pp. 173–174
  • The recommended PINN recipe combines equation and variable nondimensionalisation, Fourier embeddings with random weight factorisation, causal loss for time-dependent PDEs, and periodic loss rebalancing by exponential moving average.The cited training pipeline also includes a hyperparameter update based on NTK or gradient norms.Source: §4.5 · p. 174
  • For the recommended NTK update, the default interval is 1,000 epochs with exponential-moving-average parameter α = 0.9; the causal-loss parameter is ϵ = 1.0.The notes caution that back-propagation of the weights wi and λi with respect to network parameters θ should be frozen.Source: §4.5 · p. 174

4.6 [EXTRA]: Neural ODEs

Neural ODEs model a learned vector field through continuous-time evolution, and an adjoint approach computes gradients by solving backward in time without storing the forward computational graph. The notes connect this formulation to residual networks, applications for irregularly sampled sequences, and contrast it with PINNs, which solve a known equation.

4.6 [EXTRA]: Neural ODEs

  • Motivated by residual connections [He+16], a Neural ODE maps x0 to an initial hidden state and solves a learned vector field to a terminal horizon; the solver adapts its steps to the function’s complexity.The output is the final hidden state produced by integrating the ODE.Source: §4.6 · p. 175
  • The adjoint method computes gradients through an additional backward ODE and avoids storing the forward computational graph, enabling constant memory cost with depth.The notes present it as an alternative when the ODE solver is not differentiable or standard backpropagation is inefficient.Source: §4.6 · pp. 175, 177, 178

4.6.1 Neural ODEs Applications and Connections

  • The ResNet-to-Neural-ODE connection is rigorous primarily in the deep limit with shared weights; layer-varying parameters require a more general continuous-time formulation [AN19] [YMW22].The notes describe the ResNet-layer analogy as a forward Euler discretisation [Lu+18].Source: §4.6 · p. 175, §4.6.1 · p. 178
  • Neural ODEs support irregularly sampled time-series modelling, while ODE-RNNs combine continuous hidden-state evolution between observations with recurrent updates at observation times [RCD19].The notes also cite applications in physical sciences, system identification, forecasting, generation, image reconstruction, and graph representation learning [Che+18a] [Zha+25a].Source: §4.6.1 · p. 178

4.6.2 Comparison with Physics-Informed Neural Networks

  • A PINN represents the solution of a known differential equation and penalizes its residual, whereas a Neural ODE learns its vector field by fitting integrated states to observations [Che+18a].For PINNs, the loss combines the known-equation residual at collocation points with data-fitting terms such as initial conditions, boundary conditions, or sparse measurements [MS26].Source: §4.6.2 · p. 179

Neural Operators

This section examines neural operators as approximators of maps between function spaces, including their theoretical guarantees, discretization-invariance properties, and architectures such as DeepONet, KANs, and Fourier-based methods. It also covers physics-informed operator learning for PDEs, including PINO training with data and physical constraints.

5.1.1 Why learning operators?

  • Neural operators learn maps between function spaces, enabling a family of Poisson problems to be addressed by changing the source function.This motivates learning an operator rather than approximating the solution to only one fixed realization.Source: §5.1.1 · p. 182

5.1.2 The Universal Approximation Theorem for functionals (and operators)

  • Under Tauber-Wiener activation and compactness assumptions, universal approximation theorems cover continuous functionals and nonlinear operators [CC93; CC95].The operator theorem guarantees uniform approximation over compact input-function and output-domain sets, but is existential and does not construct the number of terms or sensor locations.Source: §5.1.2 · pp. 184–185

5.1.3 Deep Operator Networks

  • DeepONets combine input-function sensor values and output coordinates through branch and trunk networks to approximate an operator [Lu+21].Training uses triples of input sensor evaluations, output coordinates, and target output values; the trained model supplies a continuous approximated output function for new inputs.Source: §5.1.2 · p. 185, §5.1.3 · pp. 186–187
  • For uniformly refined grids, piecewise-linear interpolation errors κ(m, V) tend to zero, and the theorem requires c(b − a)κ(m, V)e^{c(b − a)} < ε for its parameter-existence result [Lu+21].Total learned-model error also includes neural approximation error, so sensor discretization and branch/trunk network sizes must be balanced.Source: §5.1.3 · pp. 187–188
  • A DeepONet approximation is stated to achieve L2 error below ε for every u in V.The supplied theorem passage gives the approximation bound for the operator output.Source: §5.1.3 · p. 188

5.1.3.1 A prototypical application: learning ∂−1

  • The KAN anti-derivative example learns from sampled functions and evaluated outputs, using Gaussian random fields or Chebyshev polynomials and numerical ODE/PDE reference solutions.In the cited experiments, inputs were integrated to generate solutions; ODEs used Runge-Kutta (4, 5) and PDEs a second-order finite-difference method.Source: §5.1.3.1 · pp. 188–189

5.1.4 Physics-Informed DeepONets

  • The physics-informed DeepONet uses automatic differentiation to impose PDE constraints as soft penalties alongside solution measurements and optional boundary losses.Measurements are taken at field-evaluation locations, while PDE residuals and boundary conditions can be enforced at collocation points.Source: §5.1.4 · pp. 189–190

5.1.4.1 learning ∂−1, the physics-informed way

  • For physics-informed learning of the anti-derivative operator, DeepONet is trained without paired input-output data, using sampled input functions, the condition s(0) = 0, and residual collocation points.In the cited experiment, inputs came from a Gaussian random-field prior and residual collocation points coincided with branch sensors.Source: §5.1.4.1 · p. 191

5.2 Using a different representation theorem for functions: Kolmogorov-Arnold Network

  • The Kolmogorov-Arnold theorem represents multivariate functions through finite compositions of univariate functions, with an extended form covering bounded domains and functions that need not be continuous.The extended result specifies that the outer function’s regularity depends on the target function’s regularity.Source: §5.2 · pp. 191–192

5.2.1 Kolmogorov-Arnold Network

  • KAN makes activation functions learnable with B-splines and adaptive grids, while sparsification, node pruning, and symbolification support compact symbolic representations.The notes describe adding entropy regularization because L1 regularization alone was empirically insufficient for sparsifying KANs.Source: §5.2.1 · pp. 194–195

5.2.1.1 Critiques to KAN

  • KAN’s representation theorem does not guarantee smooth universal inner functions or provide a finite-dimensional learning procedure, and practical evaluations report efficiency, stability, and scalability challenges.The cited critiques distinguish exact representability from learnability and efficient approximation.Source: §5.2.1.1 · p. 196
  • MLPs generally outperform KANs except in symbolic-formula representation, where B-spline activations help MLPs match or exceed KANs; KANs also show stronger catastrophic forgetting [YYW24].Further critiques report slower convergence and higher computational costs for B-spline KANs [PD24], and slower training and hyperparameter sensitivity in scientific machine learning [Shu+24].Source: §5.2.1.1 · pp. 196–197

5.2.2.1 EfficientKAN

  • Efficient-KAN reduces memory costs by applying B-spline basis functions to inputs and combining them with matrix multiplication, avoiding the expanded intermediate tensor [Ble24].Its reformulation works in forward and backward passes; it also replaces activation-based L1 regularization with weight-based L1 regularization to remain compatible.Source: §5.2.2.1 · pp. 197–198

5.2.2.2 Using Radial Basis: FastKAN

  • FastKAN replaces KAN’s cubic B-splines with Gaussian radial basis functions, yielding a faster radial-basis-function network [Li24].The resulting expression can be parallelized.Source: §5.2.2.2 · p. 198

5.2.2.3 Chebyshev-KAN

  • Chebyshev-KAN replaces B-splines with Chebyshev polynomial basis functions and combines them with learned coefficients to produce the layer output [SAK+24].Its inputs are normalized to [−1, 1] in the described implementation.Source: §5.2.2.3 · pp. 198–200

5.2.2.4 Jacobi-KAN

  • Jacobi-KAN replaces Chebyshev polynomials with the more general Jacobi polynomial family, which includes Chebyshev and Legendre polynomials as special cases.The implementation normalizes inputs to [−1, 1].Source: §5.2.2.4 · pp. 200–201

5.2.3 KAN for PDEs: PIKAN and DeepOKan

  • DeepOKAN performs competitively with DeepONet and is more robust to noisy test-time input functions, while cPIKAN uses fewer parameters but more GPU hours [Shu+24].Across the reported benchmarks, MLPs and KANs achieve comparable accuracy, but KANs generally converge more slowly and use data less efficiently [Shu+24].Source: §5.2.3 · p. 203

5.3 Learning Operators, Part II: Neural Operators - formal theory

  • Neural Operators are discretization-invariant methods that output functions and can approximate any operator, according to the comparison in Table 5.2 [Kov+23].The table contrasts model properties including discretization invariance, function-valued inputs and outputs, and operator universal approximation.Source: Table 5.2 · p. 203
  • DeepONet’s branch net constrains inputs to fixed locations despite its point-querying trunk net, breaking discretization invariance; a PCA-based branch modification is one proposed remedy [Hoo+22].Source: §5.3 · p. 203
  • For fixed-parameter discretization invariance, a model must accept arbitrary input discretizations, allow output evaluation at any point, and converge to a continuum operator as refinement increases.Source: §5.3 · p. 204

5.3.1 Neural Operators

  • Neural Operators map functions through pointwise lifting and projection and iterative local-plus-integral updates, with operations defined in function space rather than on a data discretization.Source: §5.3.1 · pp. 206–207
  • Kernel families range from spatial-only F1 to input-dependent F2, which preserves input influence across layers and is usually numerically more performant, to nonlinear, hidden-state-dependent F3.Source: §5.3.1 · pp. 207–208
  • A neural-operator layer with a hidden-state-dependent kernel yields standard Single-Head Attention (SHA) after discretization, making transformer attention a special case of a Neural Operator.The passages describe the attention form as a continuum limit of single-head self-attention with a skip connection.Source: §5.3.1 · pp. 208–209

5.3.2 Universal Approximation Theorems for Neural Operators

  • Following [Kov+23], the notes introduce a standardized framework for Neural Operator universal approximation results and define activation-function classes used in that framework.Source: §5.3.2 · pp. 209–210

5.3.2.1 Discretisation Invariance

  • Neural operators are discretisation-invariant under the theorem’s embedding and continuity assumptions, enabling zero-shot evaluation on finer grids than those used for training.The illustrated Navier–Stokes model is trained on 64 × 64 spatial and 20-point temporal grids and evaluated on 256 × 256 spatial and 80-point temporal grids.Source: §5.3.2.1 · pp. 211–212, Figure 5.11 · p. 212

5.3.2.2 Universal Approximation Theorem

  • Under the stated space and activation hypotheses, continuous operators can be approximated on compact input sets by neural operators; a Hilbert-space bounded-output condition also permits a bounded-activation choice.The theorem states existence for any 0 < ϵ ≤ 1; its additional condition is that ∥G†(a)∥U ≤ M for all a ∈ A.Source: §5.3.2.2 · p. 213

5.4.1 Fourier Neural Operator

  • The Fourier Neural Operator [Li+20] parameterizes a translation-equivariant integral kernel directly in Fourier space and uses FFTs to compute its action.Its Fourier-space parameters support evaluation at arbitrary discretizations and thus zero-shot super-resolution; truncating modes gives O(nkmax) transform complexity, with FFT complexity O(n log n).Source: §5.4.1 · pp. 213–215

5.4.2 Adaptive Fourier Neural Operator

  • Adaptive Fourier Neural Operator [Gui+21a; Gui+21b] models token mixing as continuous global convolution, with block-diagonal channel mixing, soft-thresholded frequencies, and token-shared MLP weights.Soft-thresholding sparsifies Fourier-domain tokens, whose energy is concentrated around low frequencies, focusing model capacity on important tokens and potentially regularizing the network.Source: §5.4.2 · pp. 215–216

5.4.3 Physics Informed Neural Operator

  • PINO [Li+24] combines the Fourier Neural Operator’s supervised data loss with a PDE loss, reducing data needs and enforcing physical constraints on surrogate predictions.The notes describe this as supporting verifiable, accurate, and interpretable surrogate modelling.Source: §5.4.3 · p. 219
  • PINO can evaluate supervised data on a coarse grid and the PDE residual on a finer grid using the same neural operator.Source: §5.4.3 · p. 221
  • PINO also admits a physics-only training regime when labelled input/output solution pairs are unavailable.Source: §5.4.3 · p. 221
  • PINO [Li+24] computes operator derivatives using numerical differentiation, pointwise autograd, or function-wise Fourier-space differentiation, trading approximation, memory, and efficiency.The notes characterize numerical differentiation as fast but approximate, pointwise autograd as exact but memory-intensive, and function-wise Fourier-space differentiation as exact and efficient.Source: §5.4.3 · p. 221
  • Numerical differentiation is fast but can require fine uniform grids or smooth solutions, while pointwise autograd is exact but slower and memory-consuming.Finite differences scale as O(n) and Fourier differentiation as O(n log n); derivative errors can be amplified in the output solution.Source: §5.4.3 · pp. 221–222

D K · (FFT[vL])

The passages cover neural-operator methods and applications in weather forecasting, reservoir simulation, and surface reconstruction from point-cloud measurements, as well as MORPH’s processing of heterogeneous PDE data. They describe approaches including axial attention, large-scale pretraining, Fourier Neural Operators, and Fourier differentiation, with reported results spanning forecast skill and speed, surrogate accuracy, and efficient computation.

D K · (FFT[vL])

  • Fourier differentiation computes FNO derivatives, efficiently via FFT for uniform queries; structured kernels also permit explicit gradient fields.The derivative computation follows numerical Fourier differentiation.Source: D K · (FFT[vL]) · p. 223

5.5.1 Neural operators for Weather forecasting: FourCastNet

  • FourCastNet matches IFS accuracy at short lead times for large-scale variables and outperforms it on fine-scale precipitation and near-surface winds.It generates a week-long forecast in under 2 seconds, approximately 45 000 times faster than IFS on a node-hour basis; training has been scaled to 67 minutes on 3 072 NVIDIA A100 GPUs, achieving 140.8 petaFLOPS in mixed precision.Source: §5.5.1 · pp. 223–224

5.5.1.1 FourCastNet3: Spherical Neural Operator Blocks

  • FCN3 surpasses the IFS ensemble and nearly matches GenCast, while running 8 to 60 times faster in inference.Its geometric SNO foundation supports stable spectra and realistic multiscale dynamics; a 15-day forecast at 0.25° resolution with 6-hourly outputs takes under 60 seconds on one NVIDIA H100 GPU.Source: §5.5.1.1 · p. 227

5.5.2.1 Fourier Neural Operators for ECHELON software

  • ECHELON simulations generate training data for FNO proxies in PhysicsNeMo, which map reservoir parameters to pressure and saturation fields.The workflow reports 10x to 100x inference speedups and strong agreement with ECHELON in well-placement optimization and Norne-field uncertainty cases.Source: §5.5.2.1 · pp. 228–229

5.5.2.2 3D Reservoir Simulation with Physics-Informed Neural Operators

  • The PINO surrogate predicts pressure, saturation, and flux from permeability and porosity fields using a mixed-residual formulation.Introducing flux as an additional vector variable converts the pressure equation into a first-order system and supports direct flux prediction for well-based production analysis.Source: §5.5.2.2 · pp. 231–232
  • Both FNO and PINO achieve R2 scores (>0.95) for pressure and saturation predictions throughout the reservoir simulation period.Their production profiles closely match the reference finite-volume solver, capturing two-phase-flow dynamics and well responses relevant to reservoir management.Source: §5.5.2.2 · pp. 232–233, Figure 5.22 · p. 232, Figure 5.23 · p. 233

5.5.3.1 MORPH

  • MORPH uses UPTF-7 and component-wise convolutions to process heterogeneous PDE data across spatial dimensions without architectural reconfiguration.Its learned cross-attention fuses physical fields, while factorized attention along time, depth, height, and width addresses the quadratic cost O((TDHW)2) of full spatiotemporal attention.Source: §5.5.3.1 · pp. 234–235
  • MORPH’s axial attention factorizes interactions across time, depth, height, and width, reducing complexity to O(T^2 + D^2 + H^2 + W^2) and enabling processing of large 3D grids [Rau+25].For a 64 × 64 × 64 grid, the notes contrast O(12,288) axial-attention operations with O(6.9×10^10) for full attention.Source: §5.5.3.1 · p. 235
  • MORPH is pretrained on six heterogeneous datasets spanning 1D–3D PDEs, with over 20 000 trajectories and released model variants ranging from 7 million to 480 million parameters [Rau+25].The corpus includes compressible Navier–Stokes, MHD, turbulent self-gravitating flows, and reaction–diffusion systems.Source: §5.5.3.1 · p. 236
  • MORPH achieves positive zero-shot GCR on all six out-of-distribution targets after pretraining only on 2D incompressible Navier–Stokes data, without fine-tuning [Rau+25].The targets include compressible flows in 1D, 2D, and 3D, as well as MHD and TGC.Source: §5.5.3.1 · p. 236, Figure 5.25 · p. 236
  • On FNS-KF, pretrained MORPH-FM-S surpasses the scratch-trained model with fewer than 1% of trajectories (128 vs. 20 000); on 1D Diffusion–Reaction, MORPH-FM-Ti outperforms it with 25% [Rau+25].The notes attribute this data efficiency to the diverse pretraining corpus.Source: §5.5.3.1 · p. 237, Figure 5.26 · p. 237

5.5.4 Neural operators for 3D Scene Reconstruction: nPSR

  • nPSR learns the operator mapping the divergence of an oriented-point vector field to an implicit surface with an FNO, then extracts the surface using marching cubes [And+23].Its pipeline converts point clouds to regular-grid fields, so resolution flexibility does not remove the regular-grid constraint.Source: §5.5.4 · pp. 238–239
  • Evaluations retain similar reconstruction quality at 128^3 without retraining, demonstrating cross-resolution capability; the complete nPSR pipeline remains constrained by its regular-grid representation and finite computational resources.Source: §5.5.4 · p. 241

Related papers

Loading 2609.17638v1…