Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEs

Siddhartha Mishra; Roberto Molinaro

Source-linked AI summary

Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEs

Siddhartha Mishra, Roberto Molinaro

arXiv:2006.16144v3Jun 29, 2020math.NAcs.LGmath.AP

TL;DR

PINNs are widely used for PDE approximation, but rigorous explanations of how pointwise residual minimization controls overall error remain limited. This paper develops an abstract stability- and quadrature-based framework for bounding PINN generalization error, illustrates it across three PDE classes, and tests the estimates numerically. The reported experiments are consistent with the theory, including accurate heat-equation approximations up to 100 dimensions and degraded accuracy for shock-forming inviscid problems.

Problem
Rigorous justification for why minimizing PDE residuals at collocation points controls PINN approximation or generalization error remains limited.
Method
The paper derives an abstract generalization-error bound by combining PDE stability estimates with training-error and quadrature-error bounds for residuals evaluated at collocation points.
Results
Numerical results were consistent with the estimates: heat-equation PINNs remained accurate for 100-dimensional problems, while inviscid conservation-law accuracy deteriorated when shocks formed.
Takeaways & Limitations
The framework identifies low training error, sufficiently many quadrature points, PDE stability, and controlled constants as conditions associated with low PINN generalization error.
Takeaways & Limitations
The bounds require control of constants depending on the PDE, quadrature rule, and underlying PINN; estimating these constants is left outside the paper’s abstract analysis.

Abstract

from arXiv · show

Physics informed neural networks (PINNs) have recently been widely used for robust and accurate approximation of PDEs. We provide rigorous upper bounds on the generalization error of PINNs approximating solutions of the forward problem for PDEs. An abstract formalism is introduced and stability properties of the underlying PDE are leveraged to derive an estimate for the generalization error in terms of the training error and number of training samples. This abstract framework is illustrated with several examples of nonlinear PDEs. Numerical experiments, validating the proposed theory, are also presented.

1 Introduction

The paper addresses the limited rigorous justification for PINNs by developing generalization-error bounds for forward PDE approximation. Its framework links training error, collocation-sample count, quadrature, and PDE stability, then illustrates the theory with concrete PDEs and numerical experiments.

Motivation: Rigorous bounds explaining why PINNs approximate PDE solutions well remain scarce despite their broad practical success.The paper identifies limited rigorous justification and few prior approximation-error bounds.
Contribution: The framework connects minimizing PDE residuals at collocation points to bounds on overall approximation error under verifiable PDE hypotheses.The goal is to understand how residual minimization controls unseen-point error.
Contribution: The generalization estimate depends on training error, the number of collocation points, quadrature error, and stability properties of the underlying PDE.The required conditions include low monitored training error, sufficiently many points, PDE stability, and controlled implicit constants.
Examples: The abstract framework covers nonlinear PDE applications and is developed for parabolic equations, scalar conservation laws, and incompressible Euler equations.The paper works out the estimate in three representative examples.
Validation: Numerical experiments are presented to support the proposed theory and the successful approximation of forward PDE problems by correctly formulated PINNs.The experiments are intended to substantiate the abstract error estimates.

2 An abstract framework for Physics informed Neural Networks

The paper formulates PINNs for an abstract nonlinear PDE and derives a generalization-error estimate by combining PDE stability with quadrature-based residual evaluation. The estimate applies broadly, but depends on stability, regularity, quadrature accuracy, and constants tied to the PINN and sampling rule.

Abstract PDE: The abstract PDE maps an unknown solution through a differential operator from X* to Y*, with existence, uniqueness, and stability assumed.The stability constant depends explicitly on solution norms in Z.
Stability: For linear PDEs, the stability assumption reduces to boundedness of the inverse differential operator.The paper notes that many linear PDEs possess such bounded inverses.
Random sampling: Random training points support a root mean square quadrature estimate in very high dimensions, where Monte Carlo quadrature uses rate α = 1.The points are sampled independently and identically with respect to a scaled Lebesgue measure.
Training PINNs: PINNs minimize PDE residuals at quadrature points rather than requiring interior solution data, using the differential operator and PDE inputs.The training strategy avoids potentially expensive simulations needed to generate supervised solution data.
Conditions and limits: The bound is useful when training error is small and quadrature error is controlled, but the relevant constants may depend on the PINN and number of training points.The paper states that these constants require PDE- and quadrature-specific analysis.
Generalization estimate: The abstract estimate bounds generalization error through training error and quadrature error, with constants arising from PDE stability and quadrature estimates.The estimate does not use neural-network structure in its proof and applies to any function in the stated network form.

3.1 The underlying PDEs

The first concrete example is a semilinear parabolic PDE on a bounded domain with smooth initial data and a globally Lipschitz reaction term. These assumptions provide a classical solution with the regularity required by the framework.

Model problem: The example considers a semilinear parabolic equation on a bounded connected domain with a C^k boundary.The space-time solution is formulated in Sobolev spaces over the domain.
Assumptions: The reaction term is assumed globally Lipschitz, with a constant independent of its two arguments.The homogeneous heat equation and linear source term are included as examples.
Regularity: Choosing sufficiently regular initial data yields a classical solution u in C^k([0,T] × D) for k ≥ 2.The stated condition is s̄ > k + d/2 + 1.

3.2 PINNs

For the semilinear parabolic example, the PINN training set combines interior, spatial-boundary, and temporal-boundary points selected from quadrature or sampling schemes. Separate residuals are evaluated and combined in a weighted loss.

Training set: The training set is divided into interior, spatial-boundary, and temporal-boundary points, whose union forms the full set S.Each point type corresponds to a different part of the PDE problem.
Point selection: Interior points may use composite Gauss quadrature for d ≤ 3, low-discrepancy sequences in moderately high dimensions, or random sampling in very high dimensions.Boundary points can likewise be selected by grids, low-discrepancy sequences, or random sampling.
Residuals: The PINN uses a smooth network u_θ and defines interior, spatial-boundary, and temporal-boundary residuals.Smooth activations such as tanh ensure the required derivatives and residuals are well defined.
Loss function: The loss function combines residual contributions using quadrature weights and a hyperparameter λ that balances PDE, initial-data, and boundary-data terms.The weights correspond to the three classes of training points.

3.3 Estimate on the generalization error.

The paper derives a generalization-error estimate for PINNs by combining training-error components, quadrature errors, and stability properties of the underlying PDE. For the semilinear parabolic equation, the estimate is obtained by analyzing the PINN error equation and applying energy estimates and Grönwall’s inequality.

The training error is computed from the loss function and decomposed into interior, temporal-boundary, and spatial-boundary residual components.
The estimate also depends on quadrature errors for interior, temporal-boundary, and spatial-boundary integrations, with positive convergence orders specified for each rule.
Theorem 3.1 estimates PINN generalization error for a semilinear parabolic PDE under a unique classical solution and source-term assumptions.
The proof treats the PINN error as a forced parabolic equation, then uses integration by parts, Cauchy–Schwarz, and Grönwall’s inequality.
The bound becomes small when training-error components are small and the number of training points is sufficiently large, but it weights the boundary residual more heavily.
The estimate depends on the underlying solution through a boundary constant and on dimension through the quadrature error; random sampling requires a cumulative-error modification.

3.4 Numerical experiments.

Numerical experiments test the estimates on one-dimensional and multidimensional heat equations using random training points and ensemble hyperparameter searches. The results show decreasing errors with more boundary points, strong training–generalization correlation, hyperparameter sensitivity, and low errors up to 100 dimensions.

3.4.1 One-dimensional Heat Equation.: The experiments use the linear heat equation, whose explicit solution permits direct computation of generalization error for validating the theory.
3.4.1 One-dimensional Heat Equation.: Randomly sampled interior, spatial-boundary, and temporal-boundary points form the training sets for the one-dimensional experiment.
3.4.1 One-dimensional Heat Equation.: Increasing boundary training points reduces both cumulative generalization error and the computable upper bound, while interior-point count has minimal effect in this case.
3.4.1 One-dimensional Heat Equation.: The theoretical upper bound is almost three orders of magnitude larger than the actual generalization error, consistent with non-sharp inequalities used in its derivation.
3.4.2 Ensemble training: The ensemble search evaluates 360 hyperparameter configurations, retraining each five times and selecting the run with the smallest training loss.
3.4.2 Ensemble training: Generalization-error spreads often vary by two to three orders of magnitude across hyperparameters, while architecture is less sensitive than the loss-balancing parameter λ.
3.4.2 Ensemble training: Training and generalization errors are highly correlated, with generalization error scaling as a square root of total training error.
3.4.3 Heat equation in several space dimensions: Relative cumulative generalization error stays below 0.1% through 10 dimensions and reaches 2.6% at 100 dimensions for the multidimensional heat equation.

4 Viscous scalar conservation laws

This section applies the abstract PINN framework to viscous scalar conservation laws, deriving a generalization-error estimate and testing it on Burgers-type problems. The estimate connects prediction error to training residuals, quadrature errors, PDE-dependent constants, and solution regularity.

Problem setup: The model problem is a one-dimensional viscous scalar conservation law with viscosity coefficient 𝜈>0, smooth flux, initial data, and zero Dirichlet boundary conditions.
PINN construction: Training uses interior, spatial-boundary, and temporal-boundary collocation points, with neural-network residuals defined for each part of the PDE and its conditions.
Error estimate: Theorem 4.1 bounds the PINN generalization error for 𝜈>0 using training errors and constants determined by the PDE and quadrature estimates.
Error estimate: The bound is a concrete realization of the abstract estimate, decomposing training error into four parts and explicitly separating PDE and quadrature constants.
Limitations: As 𝜈→0, PDE-dependent constants can blow up exponentially in time, so the bound may lose control for discontinuous shock solutions.
Numerical experiments: For viscous Burgers’ solutions, PINNs achieve approximately 1% relative generalization error at 𝜈=0.01 and 𝜈=0.005, but about 11% at 𝜈=0.001 and 23% in the inviscid case.
Numerical experiments: For a rarefaction-wave solution with uniformly bounded gradients, the generalization error remains below 2% across viscosity values and decreases slightly as 𝜈→0.

5 Incompressible Euler equations

This section formulates PINNs for incompressible Euler equations in a smooth bounded domain with no-penetration boundaries. It specifies the collocation sets, neural-network outputs, residuals, and weighted training loss used by the abstract algorithm.

Underlying PDE: The incompressible Euler formulation models velocity, pressure, and forcing in dimensions d=2 or 3 on a smooth bounded domain with no-penetration boundary conditions.
Underlying PDE: The analysis is presented for one boundary-condition choice but can be extended to periodic boundaries and to Navier–Stokes equations with viscous terms.
PINN construction: The training set combines interior, spatial-boundary, and temporal-boundary points selected using composite Gauss quadrature or low-discrepancy sequences.
PINN construction: The neural network predicts both velocity and pressure, while the residual system includes velocity, divergence, spatial-boundary, and temporal-boundary residuals.
PINN construction: The loss function uses quadrature weights and a hyperparameter λ to balance PDE residuals against initial and boundary-data residuals.

5.3 Estimate on the generalization error.

For incompressible Euler PINNs, the paper derives a generalization-error bound from training errors, quadrature errors, and stability properties of the underlying classical solution. The proof compares the PINN with the exact solution through an error equation and energy estimates.

Generalization-error estimate: Theorem 5.1 bounds the incompressible Euler PINN generalization error using training errors and constants associated with the PDE and quadrature estimates.The result applies when the exact velocity is a classical solution and the PINN is generated by the stated algorithm.
Proof strategy: The proof defines velocity and pressure differences between the PINN and exact solution, then derives residual-driven equations for those errors.The error velocity satisfies an equation involving the velocity residual, divergence residual, and slip-boundary residual.
Proof strategy: Taking an inner product with the velocity error, integrating by parts, and applying inequalities produces an energy estimate for the approximation error.The estimate is integrated over time and combined with Grönwall’s inequality.
Proof strategy: Quadrature-error bounds and the definitions of the training errors yield the desired generalization estimate.The training errors can be computed a posteriori from the PINN loss function.
Scope and limitation: The Euler bound requires a classical solution with ∇u ∈ L∞(D × (0,T)); large spatial derivatives can make the bound and generalization error large.Smooth solutions are available in two dimensions under sufficiently smooth initial data, while three-dimensional smooth-solution existence is described as an open question.

5.4 Numerical Experiments

The numerical experiments test two-dimensional incompressible Euler PINNs on the Taylor Vortex and Double Shear Layer problems. The PINN is highly accurate for the Taylor Vortex, while the more complicated shear-layer flow exhibits smeared sharp vortices and a higher error.

Experimental setup: The experiments use two-dimensional incompressible Euler equations, with CELU selected as the activation function because it gives better approximation than tanh for Euler equations.The Taylor Vortex uses a periodic domain and an exact solution for comparison.
5.4.1 Taylor Vortex: 8192 interior points and 256 temporal-boundary and spatial-boundary points are used for the Taylor Vortex with low-discrepancy Sobol training points.An ensemble training procedure selects the reported hyperparameter configuration.
5.4.1 Taylor Vortex: 0.012% generalization error is obtained for the best Taylor Vortex hyperparameter configuration, indicating highly accurate approximation.The PINN flow field remains close to the exact solution initially and at later times, with small numerical errors.
5.4.2 Double Shear Layer: 65536 interior points and 16384 temporal-boundary and spatial-boundary points are used for the Double Shear Layer problem.The larger training set reflects the complicated solution and high vorticity of this case.
5.4.2 Double Shear Layer: 3.8% generalization error is reported for the Double Shear Layer, while sharp vortices are smeared out, especially at later times.The PINN approximates the velocity field, and vorticity is computed from it for comparison at three times.

Code

The PINN implementations and ensemble hyperparameter selection use Python scripts supported by PyTorch, which the authors make available for download.

Code: Python scripts supported by PyTorch implement PINN construction, training, and ensemble-based hyperparameter selection.The scripts are available through the cited project repository.

6 Discussion

The discussion presents an abstract stability-based framework for PINN generalization bounds and reports numerical consistency across several PDE examples. It also identifies limitations involving training-error estimation, regularity, constant dependence, and forward-problem scope.

Motivation: The paper’s central question is why minimizing PDE residuals at collocation points can control overall PINN approximation or generalization error.The discussion frames this as a need for rigorous explanation of PINN performance.
Framework: The abstract estimate combines training error, quadrature-sample size, quadrature accuracy, and stability constants for the underlying PDE.Small generalization error follows within the stated framework when training error is small, sampling is sufficient, and the relevant constants are controlled.
Applications: The framework yields bounds for semilinear parabolic PDEs, viscous scalar conservation laws, and incompressible Euler equations.Each bound depends on training and quadrature errors together with stability and regularity estimates for classical solutions.
Numerical validation: The numerical results are consistent with the estimates, including accurate heat-equation solutions in 100 dimensions and deterioration for inviscid conservation laws with shocks.Euler experiments are also reported as consistent with the derived estimate.
Relation to prior work: The approach differs from prior work by requiring no additional regularization terms and by covering nonlinear PDEs admitting the stated stability estimate.The authors present stability estimates and quadrature rules as a unified explanation for performance across many PDEs.
Limitations: A major limitation is that the generalization bound contains a training error that the authors cannot estimate rigorously.The training error comes from a high-dimensional non-convex optimization problem, so the estimate is interpreted conditionally on successful training.
Limitations: The bounds implicitly require regularity of underlying solutions, and constants may depend on the PINN and the number of training points.Estimating PDE- and quadrature-specific constants in terms of neural-network weights is left outside the paper’s scope.
Limitations: The paper addresses forward PDE problems; generalization estimates for inverse problems are deferred to a companion paper.The authors identify inverse problems as an extension rather than part of the present analysis.