PaperlayerView exact revision on arXiv

Source-linked AI summary

Failure-informed adaptive sampling for PINNs

Zhiwei Gao, Liang Yan, Tao Zhou

arXiv:2210.00279v3math.NAstat.ML

TL;DR

PINNs can perform poorly when fixed or inefficient sampling misses important solution regions, particularly for singular or localized behavior. The paper introduces FI-PINNs, which use a residual-based failure probability to adaptively add collocation points, and reports rigorous error bounds and effective solution-structure capture across varied PDE problems.

  • Problem

    Fixed or inefficient sampling can fail to capture effective solution regions in PINNs, while existing adaptive methods can be ineffective or costly for high-dimensional or localized problems.

  • Method

    FI-PINNs use a residual-based failure probability and failure-informed enrichment to adaptively add collocation points in failure regions.

  • Results

    FI-PINNs provide rigorous error bounds and effectively capture solution structures across singular, unbounded-domain, and time-dependent PDE tests.

  • Takeaways & Limitations

    Failure probability can serve as a posterior error indicator for adaptively improving PINN training-point placement.

  • Takeaways & Limitations

    The theoretical results have been established only for the linear model, although the reported convergence rates also hold for a nonlinear Burgers’ example.

Abstract

from arXiv · show

Physics-informed neural networks (PINNs) have emerged as an effective technique for solving PDEs in a wide range of domains. It is noticed, however, the performance of PINNs can vary dramatically with different sampling procedures. For instance, a fixed set of (prior chosen) training points may fail to capture the effective solution region (especially for problems with singularities). To overcome this issue, we present in this work an adaptive strategy, termed the failure-informed PINNs (FI-PINNs), which is inspired by the viewpoint of reliability analysis. The key idea is to define an effective failure probability based on the residual, and then, with the aim of placing more samples in the failure region, the FI-PINNs employs a failure-informed enrichment technique to adaptively add new collocation points to the training set, such that the numerical accuracy is dramatically improved. In short, similar as adaptive finite element methods, the proposed FI-PINNs adopts the failure probability as the posterior error indicator to generate new training points. We prove rigorous error bounds of FI-PINNs and illustrate its performance through several problems.

1. Introduction

The paper introduces FI-PINNs to address training-point selection failures in PINNs, especially for complex or unbounded solution regions. It uses residual-informed failure probabilities to adaptively enrich collocation points, with theoretical guarantees and broad numerical testing.

  • Motivation: Fixed training points may miss effective solution regions for complex PDE solutions, while unbounded domains make effective sampling difficult.These challenges motivate adaptive sampling for PINNs.
  • Method: FI-PINNs define a residual-based failure probability as a posterior error indicator for adaptive training-point selection.The strategy parallels posterior error indicators used in adaptive finite element methods.
  • Method: A truncated Gaussian model estimates failure probabilities and generates new training points efficiently compared with RAR and DAS.The approach targets the failure region rather than relying on a large candidate set or training a generative model.
  • Theory: The paper proves rigorous FI-PINNs error bounds in terms of error tolerance and failure probability tolerance.This supplies a theoretical basis for the adaptive procedure.
  • Experiments: Across singular, unbounded-domain, and time-dependent PDE tests, FI-PINNs effectively capture the solution structure.The reported test coverage spans several challenging PDE settings.
  • Scope: The adaptive sampling strategy may extend to formulations such as the deep Ritz method and weak adversarial networks.The paper specifically identifies important sampling as the potentially transferable idea.

2. Preliminaries of PINNs

PINNs approximate unknown PDE solutions with neural networks and train them by penalizing PDE and boundary residuals. Their meshless formulation depends on collocation-point selection, which can contribute to convergence difficulties and motivates adaptive sampling.

  • PINN formulation: PINNs use automatic differentiation and neural networks to solve PDEs by penalizing PDE residuals at sampled domain points.The method is presented as a meshless approach for forward and inverse PDE problems.
  • PINN formulation: A neural network u(x; θ) approximates the unknown solution u(x), with parameters learned through a soft-constrained optimization problem.The optimization balances PDE and boundary losses using a penalty factor λ.
  • Loss construction: The residual r(x; θ) and boundary term b(x; θ) measure how well the network satisfies the PDE and boundary operators.These terms form the components of the loss used to train the network.
  • Sampling: The loss is discretized using a prior-distributed set of collocation points, making training-point selection a key issue in PINNs.The general PINNs framework is presented as relying on such a training dataset.
  • Challenges: PINNs can have difficulty converging to the true solution because their complicated loss landscapes make optimization difficult.The paper positions adaptive sampling as one response among specialized approaches to PINN failure modes.

3. Failure-informed PINNs (FI-PINNs)

FI-PINNs frame large residual regions as failures and adaptively enrich the collocation set there. Their self-adaptive importance sampler updates truncated-Gaussian proposals to estimate failure probability and generate new points efficiently.

  • Failure-informed framework: FI-PINNs define failures through a residual-based limit-state function and use failure probability to assess PINN reliability.The safe set has g(x) < 0, while the failure set has g(x) > 0; sampling can target the failure set when its probability exceeds ϵp.
  • Failure-informed framework: When estimated failure probability exceeds ϵp, FI-PINNs add collocation points from the failure region and retrain the network.The workflow trains the network, estimates failure probability, stops below tolerance, or enriches Dc with adaptive points before repeating.
  • Monte Carlo and importance sampling: Monte Carlo sampling can require O(10^4 ∼ 10^6) samples because localized failure regions may be small and difficult to hit.This makes direct Monte Carlo sampling extremely expensive for PDEs with sharp or highly localized features.
  • Monte Carlo and importance sampling: Importance sampling reduces variance by drawing from a proposal distribution and reweighting samples to approximate the failure probability under the prior.An ideal proposal concentrates on the failure set, but its normalizing constant makes it unavailable in practice.
  • Self-adaptive importance sampling: SAIS iteratively refines truncated-Gaussian proposals using high-LSF samples until enough points fall in the failure region.The final proposal is used to generate samples for estimating failure probability and constructing the adaptive dataset.
  • Self-adaptive importance sampling: The resulting approximated optimal distribution is a truncated Gaussian whose samples support failure-probability estimation and adaptive enrichment.The paper reports that SAIS can self-terminate rapidly with high numerical accuracy and can extend to time-dependent or unbounded-domain problems.

4. Convergence analysis

The convergence analysis establishes an error estimate for an FI-PINNs solution under bounded-domain, operator, boundary-approximation, and residual-boundedness assumptions. The proof links reliability, failure-region size, and residual control.

  • Algorithmic convergence setting: In the algorithm, FI-PINNs estimates failure probability, stops when it is below ϵp, and otherwise generates adaptive points from the failure region.The adaptive dataset is added to the collocation set before the next training iteration.
  • Assumptions and theorem: The analysis assumes a bounded problem domain together with operator, boundary, and residual conditions for the FI-PINNs solution.These assumptions include sufficiently trained boundary data and a bounded residual induced by the network.
  • Assumptions and theorem: Theorem 4.4 provides an error estimate for an FI-PINNs solution when Assumptions 4.1, 4.2, and 4.3 hold.The theorem is stated for a bounded domain and depends on the specified operator, boundary, and residual assumptions.
  • Error-control argument: The proof uses the safe-region bound |r(x; θ∗)| < ϵr and the failure probability to control the contribution of the failure region.For bounded domains, the analysis relates failure probability to the relative area of the failure set.

5. Numerical experiments

The experiments compare Uniform, RAR, and SAIS sampling across several PDE problems. SAIS concentrates collocation points in high-residual regions, generally reducing errors and supporting failure-probability-based stopping.

  • Experiment setup: The experiments compare Uniform PINNs, residual-based adaptive refinement (RAR), and FI-PINNs with SAIS using equal-sized datasets descended from a common initial model.The network uses 2000 initial collocation points and 200 boundary points unless otherwise specified.
  • Two-dimensional Poisson equation: 9.04 × 10^-2 mean L2 error is achieved by SAIS, versus 7.36 × 10^-1 for Uniform and 2.15 × 10^-1 for RAR in the peak problem.The estimated failure probability follows the relative L2 error and falls below tolerance after four iterations.
  • Two-dimensional Poisson equation: For the two-dimensional peak problem, SAIS concentrates updated points near (0.5, 0.5), where the exact solution peaks and decreases rapidly away.The associated SAIS prediction has much smaller absolute error than the other strategies.
  • Convergence properties: The convergence rates with respect to ϵp and ϵr are 1/2 and 1, respectively, matching Theorem 4.4.The experiment fixes one tolerance while varying the other to estimate prediction-error behavior.
  • Burgers’ equation: For the Burgers’ problem, SAIS concentrates samples in high-residual regions and reaches prediction errors of 5.59 × 10^-3 after four updates.The estimated failure probability falls below its tolerance after four updates.
  • High-dimensional Poisson equation: In the high-dimensional Poisson experiment, SAIS errors decrease smoothly while Uniform and RAR do not converge for the example.The failure probability follows the mean L2 error, supporting the FI-PINNs stopping criterion.
  • Unbounded time-dependent problems: Across unbounded and time-dependent tests, SAIS moves collocation points toward high-residual regions and achieves absolute error below 2 × 10^-3 over the test domain.After three updates, the relative prediction error is smaller than 3 × 10^-3.

6. Concluding remarks

The paper presents FI-PINNs, which combine PINNs with adaptive sampling based on a residual-derived failure probability. It also reports rigorous error bounds and successful capture of solution structures across several PDE problem classes.

  • FI-PINNs combines PINNs with adaptive sampling procedures driven by a failure probability defined from the residual.
  • A truncated Gaussian model estimates the failure probability and generates new training points efficiently.
  • The method includes rigorous error bounds for FI-PINNs.
  • Experiments cover PDEs with singular solutions, unbounded domains, and time dependence, with FI-PINNs capturing the solution structure in all tests.

Appendix A. Further discusses

The appendix outlines two extensions of the FI-PINNs framework: more general density models and causality-based weighted residuals.

  • The framework can use more general density models, including Gaussian mixture models.
  • The appendix also discusses using a causality-based weighted residual as the performance function.

A.1. SAIS using Gaussian mixture model.

This appendix extends SAIS with a Gaussian mixture model as an intermediate proposal distribution for a two-peak problem. The resulting samples concentrate in high-residual regions and the predictions closely match the exact solution.

  • SAIS uses a mixed Gaussian model as its intermediate proposal distribution.
  • The Gaussian mixture combines component densities with weights π_m that sum to one.
  • The intermediate proposal is updated adaptively, with Gaussian-mixture parameters estimated from candidate samples using expectation-maximization.
  • In the two-peak test, SAIS samples concentrate where residual error is high and predicted values closely match the exact solution.

A.2. Causal extension of FI-PINNs.

The causal FI-PINNs extension adapts FI-PINNs to time-dependent PDEs by combining time-subdomain decomposition, causal performance weighting, and a weighted residual loss. In the Allen–Cahn test, it achieves low prediction error with fewer than 4000 training points and outperforms causal PINNs at the same dataset size.

  • A.2. Causal extension of FI-PINNs.: Causal FI-PINNs targets time-dependent PDEs through a causal extension of the original framework.
  • A.2. Causal extension of FI-PINNs.: The method divides the domain into time subdomains and trains the network with a weighted residual loss.
  • A.2. Causal extension of FI-PINNs.: Its performance function is modified to Q(x) = w_i|r(x, t_i)|, combining causal weights with the residual in each time subdomain.
  • A.2. Causal extension of FI-PINNs.: In the Allen–Cahn example, causal FI-PINNs achieves a prediction error of 6.5×10^-3 after 6 updates using fewer than 4000 training points.
  • A.2. Causal extension of FI-PINNs.: At the same training-dataset size, causal FI-PINNs produces lower predicted error than conventional causal PINNs, whose relative error is 2.42 × 10^-2.
Loading 2210.00279v3…