PaperlayerView exact revision on arXiv

Source-linked AI summary

Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs

Tim De Ryck, Siddhartha Mishra

arXiv:2106.14473v2math.NAcs.LGcs.NA

TL;DR

High-dimensional PDEs challenge conventional numerical methods, and the paper studies whether PINNs can be given rigorous error guarantees for Kolmogorov equations. It constructs PINNs and bounds solution error through generalization and training error; the results establish polynomial dimension growth under suitable conditions, while a general construction can still grow exponentially with dimension.

  • Problem

    Conventional numerical methods are not suitable for efficiently approximating PDEs with high-dimensional state or parameter spaces.

    Source: §1 · p. 1

  • Method

    The paper constructs PINNs using Dynkin’s formula and combines Kolmogorov PDE stability with an error decomposition and concentration bounds for randomly sampled training points.

    Source: §1 · p. 3, §5 · p. 17

  • Results

    The total L2 error is bounded by generalization error, which is bounded by training error given sufficiently many random samples; the constructed networks and sample counts grow polynomially in dimension under suitable conditions.

    Source: §1 · p. 3, §5 · p. 17

  • Takeaways & Limitations

    For the studied Kolmogorov PDEs, the results provide affirmative answers to the three questions on residual, total error, and training-to-generalization error.

    Source: §1 · p. 3

    The initial construction grows exponentially with spatial dimension, while polynomial network-size growth is proved for a subclass where the dimension-dependent constant grows polynomially, as for constant coefficients.

    Source: §3 · pp. 6, 11

Abstract

from arXiv · show

Physics informed neural networks approximate solutions of PDEs by minimizing pointwise residuals. We derive rigorous bounds on the error, incurred by PINNs in approximating the solutions of a large class of linear parabolic PDEs, namely Kolmogorov equations that include the heat equation and Black-Scholes equation of option pricing, as examples. We construct neural networks, whose PINN residual (generalization error) can be made as small as desired. We also prove that the total $L^2$-error can be bounded by the generalization error, which in turn is bounded in terms of the training error, provided that a sufficient number of randomly chosen training (collocation) points is used. Moreover, we prove that the size of the PINNs and the number of training samples only grow polynomially with the underlying dimension, enabling PINNs to overcome the curse of dimensionality in this context. These results enable us to provide a comprehensive error analysis for PINNs in approximating Kolmogorov PDEs.

1. Introduction

High-dimensional PDEs strain conventional numerical methods, while PINNs offer a way to approximate PDE solutions without large amounts of labeled data. This paper addresses whether PINNs can achieve small residual and solution errors for Kolmogorov equations, and whether training error and network size can be controlled.

Background and context

  • Conventional numerical methods are inadequate for efficiently approximating PDEs with high-dimensional state or parameter spaces, including many-query problems.Such settings arise in scientific models and in uncertainty quantification, optimal design, and inverse problems.Source: §1 · p. 1
  • PINNs address the need for PDE approximation without explicit labeled data or with small amounts of data, which matters when measurements or simulations are costly or infeasible.The approach minimizes pointwise PDE, boundary, and initial-condition residuals.Source: §1 · pp. 1–2

Theory for PINNs

  • The paper asks whether small generalization error is attainable, whether it controls total solution error, and whether small training error controls generalization error with enough training points.These questions form the paper’s proposed theoretical investigation of PINNs.Source: §1 · pp. 2–3

Aims and scope of this paper

  • Earlier analyses addressed error stability and quadrature-based bounds but did not establish small generalization error and imposed assumptions that excluded ordinary random sampling without a validation set.The paper focuses on linear parabolic Kolmogorov equations, including heat and Black-Scholes equations, a high-dimensional class with existing empirical PINN results.Source: §1 · p. 3
  • The paper establishes that PINNs for Kolmogorov equations can achieve arbitrarily small generalization and total error, with network size and required random training samples growing polynomially in dimension under suitable conditions.The construction uses Dynkin’s formula; stability bounds total L2 error by generalization error, and sufficiently many random collocation points bound generalization error by training error.Source: §1 · p. 3

2. PINNs for Linear Kolmogorov Equations

This section specifies the linear Kolmogorov PDE setting and the neural-network representation used to approximate its solution. PINNs are trained by minimizing residuals at points in the domain and on its boundaries, using randomly sampled points to form the loss.

2.1. Linear Kolmogorov PDEs. 

  • Kolmogorov PDEs are linear parabolic equations with affine coefficient functions, and include heat and Black-Scholes equations as examples.The stated domain is D=(0,1)^d; Black-Scholes models option prices based on underlying stocks.Source: §2.1 · p. 4

2.2. Neural Networks.

  • A neural network is built from affine layer maps and activation functions, with its realization mapping the input to the output.For Kolmogorov PDE approximation, the input consists of spatial coordinates and time.Source: §2.2 · pp. 4–5

2.3. PINNs.

  • PINNs minimize pointwise PDE, boundary, and initial-condition residuals, using randomly sampled points to define the training loss.A trained network is a local minimizer of the loss, found with a gradient-descent algorithm such as ADAM or L-BFGS.Source: §2.3 · p. 5

3. Bounds on the approximation error for PINNs

This section constructs PINNs with arbitrarily small residuals for linear Kolmogorov equations and derives bounds connecting generalization error to L2 approximation error. It also establishes approximation rates with dimension dependence that varies by construction and coefficient setting.

Direct approximation

  • Directly approximating a sufficiently regular solution in W2,∞ can make the residual arbitrarily small, but its width O(ε^-d/(s-2-γ)) grows exponentially with dimension.This motivates using the specific structure of Kolmogorov equations and Dynkin’s formula instead.Source: §3 · pp. 5–6

Dynkin-formula construction

  • The dimension-aware construction emulates Dynkin’s formula by approximating the initial data and generator with tanh networks and replacing the expectation with Monte Carlo sampling.The proof proceeds through a temporal Riemann-sum approximation, Monte Carlo approximation, and neural-network approximation.Source: §3 · pp. 6–9

Theorem 3.3

  • Theorem 3.3 constructs tanh networks with arbitrarily small PINN residual whose accuracy-dependent size is independent of d, but whose overall size may still inherit curse-of-dimensionality growth through ρ_d.The neuron count is at most C(dρ_d)^λ ε^-max{5p+3,2+p+β}; the weights grow at most as C(dρ_d)^λ ε^-max{ζ,8p+6}.Source: §3 · pp. 6, 11

Theorem 3.5

  • For constant μ and σ, Theorem 3.5 gives Ψε,d with O(d^λ ε^-max{5p+3,2+p+β}) neurons and weights O(d^λ ε^-max{ζ,8p+6}) for small ε and large d.The result specializes Theorem 3.3 to a setting where ρ_d grows polynomially in d.Source: §3 · pp. 11–12

Total error bound

  • The constructed network also has arbitrarily small total error in L2 and H1, while Kolmogorov-equation stability yields a broader bound making total error proportionate to generalization error.The broader result is introduced for classical solutions with μ ∈ C1(D; Rd) and σ ∈ C2(D; Rd×d).Source: §3 · p. 12

3. Bounds on the approximation error for PINNs

  • The constants C1 and C3 are O(1) and depend only on the Kolmogorov PDE coefficients, whereas C2 depends on the individual PINN approximation.For the PINN Ψ_ε,d constructed in Theorem 3.3, C2 is O(ε).Source: §3 · p. 13

4. Generalization error of PINNs

The section reduces PINN error analysis to bounding generalization error from sampled training error, then connects that bound to the total L2-error for Kolmogorov PDEs. Under suitable sample-size conditions, small training error yields small generalization and total error, with sample counts growing polynomially in dimension.

General error framework

  • The generalization-error framework treats initial- and boundary-data mismatches and the interior PDE residual as function-approximation errors evaluated on independently sampled training points.The setup relates a ground-truth function and its network approximation to the corresponding PINN terms.Source: §4 · p. 14
  • For bounded, Lipschitz training and generalization errors over a compact parameter space, a finite parameter-space cover supports a probabilistic bound relating generalization error to training error.The cover size is bounded in terms of the parameter dimension and the covering resolution.Source: §4 · p. 14

Applications to PINNs

  • The framework also gives cumulative, average-error bounds and is applied to tanh networks for spatial and temporal boundary errors and to PINN errors for linear Kolmogorov equations.The Kolmogorov application uses Lipschitz continuity of networks and their derivatives in the parameters.Source: §4 · pp. 15–16

Bounds and assumptions

  • The PINN corollary requires bounds on training and generalization errors, and the authors note that the bound for the interior error may be a large overestimate.The boundary-error bounds are described as reasonable in size.Source: §4 · p. 16

Training-error condition

  • The training-error argument for a trained PINN additionally assumes global optimization, which makes its training loss no larger than that of the constructed network.The constructed network’s errors are stated to be of order O(ε).Source: §4 · p. 17

Kolmogorov PINN bounds

  • Small training error and sufficiently many samples, chosen as in (4.17), suffice for small PINN generalization and total error, with sample counts polynomial in dimension.The authors state that this overcomes the curse of dimensionality in this setting.Source: §4 · pp. 16–17

5. Discussion

The paper analyzes PINN error for Kolmogorov equations, including heat and Black–Scholes equations, and asks whether small residuals and training errors yield accurate solutions without exponential growth in dimension. It reports bounds linking residual, total, and training errors, while identifying limits to the dimensionality result.

5. Discussion

  • The analysis targets linear parabolic Kolmogorov PDEs, including heat and Black–Scholes equations, and investigates rigorous error bounds and dimensionality.These PDEs can be posed on very high-dimensional spatial domains.Source: §5 · p. 17
  • The authors construct a PINN with arbitrarily small residual, with network size growing polynomially in spatial dimension for a subclass under suitable assumptions.The construction emulates Dynkin’s formula; polynomial growth is established under assumptions on initial data and the stochastic process.Source: §5 · p. 17
  • Stability of Kolmogorov PDEs bounds the PINN’s total L2 error in terms of its generalization error.This addresses the paper’s Q2 error question.Source: §5 · p. 17
  • With sufficiently many randomly chosen training samples, an error decomposition, Lipschitz continuity, and concentration inequalities bound generalization error by training error; sample counts grow polynomially in dimension.This addresses Q3 and alleviates the curse of dimensionality for the sample requirement.Source: §5 · p. 17
  • The paper presents no numerical experiments, and contrasts its analysis with prior work using bespoke regularization or addressing only some error questions.It points to and forthcoming work for numerical experiments, and notes that PINNs do not require interior-domain data unlike the supervised-learning studies it discusses.Source: §5 · pp. 17–18
  • Polynomial growth of the quantity ρ_d is proved only for a subclass of Kolmogorov PDEs; extending that result to general Kolmogorov PDEs remains open.The authors also caution that constants in their estimates may be significant overestimates.Source: §5 · p. 18
  • The authors state that their error decomposition and generalization-to-training-error bounds may apply in other PDE settings, while broader PINN analysis is future work.They mention both linear and nonlinear PDEs and forward and inverse problems.Source: §5 · p. 18

A.1. Auxiliary results.

Appendix A collects auxiliary probability results and a neural-network approximation result used in the paper’s analysis.

A.1. Auxiliary results.

  • The appendix includes auxiliary lemmas involving independent random variables and probability estimates.The supplied statements include i.i.d. random variables and proofs invoking Hölder’s inequality, Fubini’s theorem, and Lemma A.1.Source: §A.1 · pp. 19–20
  • An auxiliary stochastic-process result states existence and uniqueness for a process driven by Brownian motion with affine drift and diffusion functions.The process is specified for an initial random variable in Lp; the supplied statements also refer to its properties.Source: §A.1 · p. 20
  • For the clipped function h(x)=min{1,max{0,x}}, the appendix gives a tanh-network approximation result with two hidden layers and O(N) neurons.The statement is for N ≥ 2 and positive ε and γ.Source: §A.1 · pp. 20–21

Appendix B. Lipschitz continuity in the parameter vector of a neural network and its derivatives

Appendix B develops parameter-wise Lipschitz bounds for neural-network outputs and their derivatives, tools the paper uses to control PINN generalization error.

Appendix B. Lipschitz continuity in the parameter vector of a neural network and its derivatives

  • The appendix establishes that network outputs, Jacobians, and Hessians are Lipschitz continuous in the parameter vector for bounded inputs.It identifies this property as important for bounding PINN generalization error.Source: Appendix B · pp. 21–23
  • The derivative bounds use a twice continuously differentiable activation such as tanh or sigmoid and apply to networks with bounded weights and biases.The appendix states that the differentiable-activation lemma can be adapted for ReLU networks.Source: Appendix B · pp. 21, 22, 24
  • A generalized chain rule for Hessians supplies the derivative representations used in the appendix’s parameter-continuity arguments.The appendix defines Jacobian and Hessian notation for network outputs and applies the chain rule to compositions.Source: Appendix B · p. 22

Appendix C. Additional material for Section 4

Appendix C supplies auxiliary concentration and residual estimates used in the Section 4 analysis.

Appendix C. Additional material for Section 4

  • The appendix includes Hoeffding’s inequality for independent bounded random variables as an auxiliary concentration result.It also uses network-derivative bounds in the displayed estimates for residual quantities.Source: Appendix C · p. 25
  • The residual estimates rely on Lipschitz continuity of neural networks and their derivatives in the parameter vector.The appendix points to Appendix B for that property.Source: Appendix C · p. 25
  • Lemma C.3 estimates PINN generalization and training residual errors for tanh networks with bounded weights and biases on a bounded input domain.Its statement is framed for linear Kolmogorov PDEs and networks with bounded depth and width.Source: Appendix C · p. 25
Loading 2106.14473v2…