PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating a class of inverse problems for PDEs

Siddhartha Mishra, Roberto Molinaro

arXiv:2007.01138v3math.NAcs.LGmath-phmath.AP

TL;DR

The paper studies rigorous generalization-error estimates for PINNs solving PDE data-assimilation and unique-continuation inverse problems. It builds an abstract framework using conditional stability estimates and applies it to four linear PDEs. Experiments report errors below 1% in the considered examples, subject to training and scope limitations.

  • Problem

    Rigorous approximation-error estimates were lacking for PINNs applied to PDE inverse problems, including cases where unknown inputs must be inferred from observations.

  • Method

    The paper derives generalization-error bounds from conditional stability estimates in an abstract framework and applies them to Poisson, Heat, Wave, and Stokes data-assimilation problems.

  • Results

    Errors were less than 1% in the numerical experiments for the considered linear PDE inverse problems, validating the proposed theory.

  • Takeaways & Limitations

    The results provide a unified and, to the authors’ knowledge, first rigorous justification for using PINNs in PDE data-assimilation inverse problems.

  • Takeaways & Limitations

    The generalization bound is conditional on a well-trained PINN, and the concrete examples in the paper are limited to linear PDEs.

Abstract

from arXiv · show

Physics informed neural networks (PINNs) have recently been very successfully applied for efficiently approximating inverse problems for PDEs. We focus on a particular class of inverse problems, the so-called data assimilation or unique continuation problems, and prove rigorous estimates on the generalization error of PINNs approximating them. An abstract framework is presented and conditional stability estimates for the underlying inverse problem are employed to derive the estimate on the PINN generalization error, providing rigorous justification for the use of PINNs in this context. The abstract framework is illustrated with examples of four prototypical linear PDEs. Numerical experiments, validating the proposed theory, are also presented.

1 Introduction

The paper addresses the lack of rigorous PINN approximation-error estimates for PDE inverse problems, focusing on unique continuation and data assimilation. It develops a stability-based framework, illustrates it on four linear PDEs, and validates the theory numerically.

  • Motivation: PDE numerical methods can be computationally costly for large-scale uncertainty quantification, high-dimensional, inverse, multiscale, and multiphysics problems.The cost remains high even on state-of-the-art high-performance computing platforms.
  • Motivation: PINNs integrate PDE information with noisy observational data to infer unknown inputs and the corresponding solution in inverse problems.The unknown inputs may include initial data, boundary conditions, and coefficients.
  • Research gap: Prior rigorous analyses addressed PINN approximation errors for forward problems, but no rigorous estimate was available for PDE inverse problems.Earlier work related forward-problem generalization error to training error and the number of training samples.
  • Approach: The proposed bounds use conditional stability estimates for the underlying inverse problem and require low training error, sufficiently many collocation points, and adequate regularity.The training error is available a posteriori, while the number of collocation points is linked to quadrature error.
  • Scope and validation: The framework covers data assimilation problems for the Poisson, Heat, Wave, and Stokes equations, with numerical experiments validating the resulting generalization-error bounds.These examples span linear elliptic, parabolic, hyperbolic, and indefinite PDEs.

2.1 The underlying abstract PDE

The paper formulates an abstract PDE and its associated unique continuation inverse problem in Banach spaces. Unknown boundary or initial conditions are inferred from measurements on an observation subdomain under conditional stability assumptions.

  • The forward problem: The abstract PDE maps an unknown function through a differential operator, with boundary conditions represented using trace and boundary operators.The formulation also includes initial conditions when the domain is a space-time domain.
  • The forward problem: The forward problem is assumed well-posed when the source and boundary data are known, yielding a unique solution in the solution space.This assumption contrasts with the inverse setting, where boundary or initial data are unavailable.
  • The inverse problem: The inverse problem assumes unknown generic boundary or initial conditions and measurements of the solution on a subdomain D′, called the observation domain.The forward problem may then be ill-posed because uniqueness is not guaranteed.
  • The inverse problem: The solution and missing boundary conditions are determined from the observation data, defining the problem as unique continuation or data assimilation.The data are given only on the observation domain.
  • Stability and extensions: Conditional stability is assumed for sufficiently regular solutions over intermediate domains D′ ⊂ E ⊂ D.The framework also notes possible extensions allowing the measurement set to intersect the boundary.

2.2 Quadrature rules

Quadrature rules approximate the integral loss terms used by PINNs, with points and weights chosen according to the domain dimension and integrand regularity. Different regimes motivate grid, low-discrepancy, sparse-grid, or Monte Carlo methods.

  • Abstract quadrature: Quadrature approximates integrals of functions over D using selected points y_i and positive weights w_i, with an assumed error bound.The error rate is parameterized by α > 0.
  • Dimension-dependent rules: For dimensions d̄ ≤ 4, composite Gauss quadrature on an underlying grid is suitable, with convergence depending on quadrature order and integrand regularity.The rate α depends on the regularity of the integrand.
  • Dimension-dependent rules: For moderately high dimensions, low-discrepancy sequences such as Sobol and Halton points can achieve a rate (log(N))^d̄ N^-1 for bounded Hardy-Krause variation.Sparse grids and Smolyak quadrature rules are also suggested in this regime.
  • Observation-domain quadrature: A separate set of points z_j and weights is used for quadrature over the observation domain D′.These points support approximation of the data-residual integral.

2.3 PINNs

PINNs represent the PDE solution with a feedforward neural network and train its parameters by jointly minimizing PDE and data residuals. The continuous loss is approximated by quadrature and optimized numerically to obtain the PINN approximation.

  • Neural networks: PINNs approximate the inverse-problem solution using feedforward networks composed of affine transformations and scalar nonlinear activations.The network has input, output, and hidden layers, with hidden layers applying affine maps followed by component-wise activation.
  • Training: The network parameters are weights and biases tuned so that the network approximates the target solution of the abstract inverse problem.Training seeks parameters minimizing the mismatch between the neural network and the target.
  • Loss construction: The PINN loss jointly penalizes the PDE residual and data residual, with λ balancing their contributions.The PDE residual measures equation mismatch, while the data residual measures mismatch with observations.
  • Loss construction: The loss integrals are approximated with quadrature points from the interior domain and observation domain.The resulting discrete loss uses the residuals evaluated at the corresponding training sets.
  • Optimization: Optimization approximately minimizes the generally non-convex loss using methods such as ADAM or LBFGS, producing u* = u_θ* as the PINN approximation.The algorithm initializes network evaluations and gradients, runs optimization to an approximate local minimum, and returns the resulting network.

2.4 An abstract estimate on the generalization error

The section derives an abstract generalization-error estimate for PINNs solving PDE inverse problems, expressing the error through training, quadrature, and conditional-stability components. It also identifies training quality, PINN regularity, stability assumptions, sample size, and measurement noise as key conditions and boundaries.

  • Generalization-error estimate: The generalization error is estimated in terms of the PINN’s computable training error after training.The training error depends on the trained parameters and training sets and can be evaluated from the loss functions a posteriori.
  • Generalization-error estimate: Theorem 2.4 bounds the error for a trained PINN under conditional stability and residual-integrability assumptions, using interior and data quadrature training sets.The proof combines the inverse-problem stability estimate with quadrature estimates for the PDE and data residuals.
  • Error mechanisms: The estimate separates error sources into training error, quadrature error, and the conditional stability of the underlying inverse problem.This decomposition is presented as explaining mechanisms behind potentially efficient PINN approximation for unique continuation and data assimilation problems.
  • Conditions: The PINN must be sufficiently well-trained, although training error is controlled only a posteriori rather than beforehand.The authors characterize a well-trained PINN through sufficiently small training errors and identify these errors as part of the generalization gap.
  • Conditions: Accurate quadrature requires a sufficiently regular approximating PINN, with smooth activations such as sigmoid or hyperbolic tangent suggested for enforcing regularity.The required regularity concerns approximation of the PDE and data residuals by the quadrature rules.
  • Scope and practical boundaries: The constants in the bound may grow with the numbers of interior and data training samples, but decay can persist if growth is slower than the bound’s decay terms.For finite samples, the constants can be checked a posteriori from the trained network and computed residuals.
  • Scope and practical boundaries: For small measurement noise, the PINN can still approximate the inverse solution efficiently; with sufficiently few training samples, noise may dominate and Bayesian methods are needed.The noisy-data extension modifies the bound by adding a noise contribution.

3 Poisson’s equation

The Poisson unique-continuation problem recovers a solution from interior observations despite ill-posedness. The paper places PINN approximation within conditional-stability bounds and validates the resulting estimates numerically.

  • Problem formulation: The Poisson equation is posed on D with source term f, while the inverse problem uses observations g on a subdomain D′.The PDE is treated weakly, with u ∈ H^1(D), and D′ is open, simply connected, and smoothly bounded.
  • Problem formulation: The underlying Poisson problem is not necessarily well-posed because u need not belong to H_0^1(D), motivating unique continuation from data on D′.The equivalent elliptic Cauchy problem is identified as an ill-posed PDE inverse problem.
  • Stability and generalization bounds: Conditional stability estimates provide interior and global control, with moduli ω(R)=R^τ and ω(R)=|log(R)|^-τ, respectively.Here τ ∈ (0,1), and the interior estimate applies to open E ⊂ D separated from ∂D.
  • PINN construction: The PINN uses PDE and observation-data residuals evaluated at quadrature points, with a weighted loss and sufficiently regular neural networks.Interior points may be Sobol or grid-based quadrature points, while observation points lie in D′.
  • Stability and generalization bounds: The Poisson PINN generalization error is bounded in terms of training errors and quadrature errors under regularity and interior-domain assumptions.The bound applies on open simply connected E with positive distance from the domain boundary, for τ ∈ (0,1) and constants depending on E.
  • Numerical experiments: With 20^2 training points, the H^1-error is around 1% and the L^2-error around 0.3%, while training takes approximately 21 s.The error decays slowly as training samples increase, consistent with the predicted logarithmic decay; 1% noisy data still yields low errors.

4 Heat Equation

The heat-equation data-assimilation problem infers initial conditions and the full later solution from observations on a space-time subdomain. A Carleman-based conditional stability estimate yields PINN generalization bounds, with boundary residuals included in training.

  • Problem formulation: The heat-equation inverse problem infers unspecified initial conditions and the later solution field from measurements over time.The observations are taken on a space-time subdomain D′_T, while the target solution is defined on D_T.
  • Stability framework: The analysis relies on a classical conditional stability result based on Carleman estimates for the heat equation.The stability theorem applies to weak solutions with source term f ∈ L^2(D_T) and times 0 ≤ T̄ < T.
  • Abstract framework: The heat-equation formulation fits the abstract PINN framework with space-time solution and data-observation spaces and a weak differential operator.The paper explicitly identifies the corresponding abstract spaces and observable.
  • PINN construction: The heat PINN augments interior PDE and observation-data residuals with a spatial-boundary residual to enforce the boundary conditions.Training sets use interior, spatial-boundary, and observation quadrature points, with corresponding quadrature weights in the loss.
  • Generalization bound: Under regularity assumptions, the heat-equation PINN generalization error is bounded using training errors and the three relevant quadrature-error estimates.The estimate is stated for every 0 ≤ T̄ < T and a PINN generated by the specified algorithm and loss functions.

4.4 Numerical experiments

The heat-equation experiments show that PINNs achieve highly accurate data assimilation in one dimension and retain low relative error in very high dimensions, despite shrinking observation domains.

  • 4.4.1 Heat equation in one space dimension: Sobol points are used for interior training, while Cartesian grid points cover data and space-boundary training sets in the one-dimensional experiment.Figure 4 distinguishes the training-point distributions by color and marker.
  • 4.4.1 Heat equation in one space dimension: With 800 training points, the PINN approximates the one-dimensional heat-equation solution to very high accuracy across space and time.The relative-error plot shows that most error is concentrated near the initial time because heat-equation solutions are rapidly damped.
  • 4.4.1 Heat equation in one space dimension: Relative errors in both L2 and H1 norms are significantly less than 1%, even with very few training points.The reported quantities are percentage relative errors in L2(D_T) and L2((0,T);H1(D)).
  • 4.4.1 Heat equation in one space dimension: Increasing the number of one-dimensional training samples produces a saturation effect because the training error is already very small.The authors associate the limited further reduction with difficulty optimizing the loss using the LBFGS optimizer.
  • 4.4.1 Heat equation in one space dimension: Randomly selected training points yield errors as small as Sobol and Cartesian points, with a slight consistent decay as training size increases.The random-point results are averaged over 30 independently chosen training sets and indicate robustness to the training-point choice.
  • 4.4.2 Heat equation in several space dimensions: In multiple dimensions, the relative L2 error remains below 0.1% through n=10 and reaches 2% at n=100.The experiment uses 16,384 randomly chosen training points and evaluates error on 10^5 random test samples.
  • 4.4.2 Heat equation in several space dimensions: The PINN reconstructs the full high-dimensional solution accurately even as the observation domain shrinks exponentially relative to the whole domain.The authors report that error growth appears linear with dimension rather than exponential and compare favorably with a prior forward-problem experiment.

5 The Wave equation

The wave-equation section formulates data assimilation as recovery of unknown initial data and the full solution from observations, then derives PINN generalization bounds using conditional stability under the GCC. Experiments show accurate reconstruction when the GCC holds, while violating it increases error and produces ray-transported errors.

  • 5.1 The underlying inverse problem: Data assimilation recovers unknown initial data and the entire wave-equation solution from source and observation data.The forward problem requires initial conditions, but the inverse formulation infers them from measurements on an observation domain.
  • 5.1 The underlying inverse problem: The geometric control condition requires every generalized bicharacteristic in the space-time domain to intersect the observation region.Under this condition, observability estimates provide well-posedness and a conditional stability estimate for data assimilation.
  • 5.2 PINN approximation: The PINN loss combines PDE, boundary, and data residuals evaluated at quadrature-based training points.Interior, spatial-boundary, and data points are used with corresponding quadrature weights and a hyperparameter.
  • 5.3 Estimates on the generalization error: The generalization error is bounded by training errors and quadrature-error terms, scaled through the wave equation’s observability constant.The bound assumes the observation domain satisfies the GCC and depends on the domain geometry and final time.
  • 5.4 Numerical experiments: Without the GCC, the error is about 1.4% for the largest training set, roughly an order of magnitude above the GCC-satisfying experiment.The error decays as training points increase, but is transported along rays outside the observable boundary region.

6 The Stokes equation

The Stokes section treats velocity and pressure recovery from interior data without boundary conditions, using conditional stability from a three-balls inequality to derive PINN error estimates. Numerical experiments show decaying velocity and pressure errors, although errors are larger when observations cover only about 20% of the domain.

  • 6.1 The underlying inverse problem: The Stokes data assimilation problem infers velocity and pressure from source terms and interior observations because the forward equation lacks boundary conditions.The observation domain is an open simply connected subset of the full domain.
  • 6.2 Conditional stability estimate: A three-balls inequality supplies the conditional stability estimate that places the Stokes inverse problem within the abstract PINN framework.The resulting estimate makes the problem amenable to PINN approximation.
  • 6.3 PINN approximation: The Stokes PINN uses quadrature points in the full domain and observation domain, with residuals for the PDE, divergence, and observed data.The loss combines these residuals through the algorithm’s quadrature-based formulation.
  • 6.3 Estimates on the generalization error: The Stokes generalization bound contains training-error terms and terms decreasing with the numbers of interior and data training samples.The training errors can be computed a posteriori, while increasing sample counts reduces the quadrature-related terms.
  • 6.3 Estimates on the generalization error: The stated generalization estimate directly covers velocity errors, while pressure-error control would require the corresponding pressure Poisson equation.Arbitrary target sets can be handled by covering them with a union of balls.
  • 6.4 Numerical experiments: Velocity-field errors decay slowly but consistently with more training samples, while pressure errors are also reasonably small and decrease despite pressure not being observed.The pressure behavior is related to controlling pressure through the pressure Poisson equation, although no pressure-error estimate is provided.
  • 6.4 Numerical experiments: Observation domains covering approximately 20% of the full domain produce larger velocity and pressure errors than the other three PDE experiments.The authors nonetheless describe reconstructing both fields from this small observation region as significant.

Code

The paper’s PINN experiments and ensemble hyperparameter selection were implemented with Python scripts supported by PyTorch.

  • Code: Python scripts supported by PyTorch implement PINN construction, training, and ensemble hyperparameter selection.The scripts are available from the project repository identified by the authors.

7 Discussion

The paper provides a unified framework and, to the authors’ knowledge, the first rigorous justification for PINNs in data assimilation inverse problems for PDEs. Conditional stability estimates yield generalization-error bounds, while experiments across four linear PDEs report small errors and practical efficiency, alongside clear scope limitations.

  • Problem: The framework targets data assimilation or unique continuation problems where unknown PDE inputs are inferred from possibly noisy observations.Such problems arise particularly in geophysics and meteorology.
  • Method: The PINN algorithm uses quadrature points as training points and the collocated strong-form PDE residual as its loss function.The framework leverages conditional stability estimates for the inverse problem to bound the PINN’s generalization error.
  • Validation: The theory is instantiated for the Poisson, heat, wave, and Stokes equations, with numerical experiments validating the proposed estimates.Each equation has conditional stability estimates established through three-balls inequalities or Carleman estimates.
  • Results: Errors were less than 1% even with a few training samples, while most examples required less than 1–2 minutes of training.PINNs had error amplitudes comparable to finite element methods on fine meshes.
  • Limitations: The generalization bound is stated in terms of training error, which the paper does not estimate because training entails a high-dimensional non-convex optimization problem.The bound should therefore be interpreted as applying when the PINN is trained well.
  • Limitations: Concrete examples cover only linear PDEs, and larger-amplitude observation noise would require a Bayesian framework not developed here.Extending the approach to nonlinear equations such as Navier–Stokes is identified as challenging and future work.
Loading 2007.01138v3…