PaperlayerView exact revision on arXiv

Source-linked AI summary

B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data

Liu Yang, Xuhui Meng, George Em Karniadakis

arXiv:2003.06097v1stat.MLcs.LG

TL;DR

The paper addresses forward and inverse nonlinear PDE problems with noisy data, where PINNs lack built-in uncertainty quantification. It proposes Bayesian physics-informed neural networks using Bayesian surrogates and posterior inference, and reports that HMC is generally more accurate and robust than VI for posterior estimation.

  • Problem

    PINNs lack built-in uncertainty quantification, restricting their use for PDE problems with noisy data.

  • Method

    B-PINNs combine a Bayesian neural-network surrogate constrained by PDE physics with HMC or VI posterior estimation for noisy-data forward and inverse problems.

  • Results

    B-PINN-HMC demonstrates good accuracy and robustness, whereas B-PINN-VI usually gives unreasonable uncertainties.

  • Takeaways & Limitations

    The experiments identify HMC as more suitable than VI for B-PINN posterior estimation across the investigated PDE problems.

  • Takeaways & Limitations

    Prior selection remains an open problem, and the study tests datasets containing only up to several hundred observations.

Abstract

from arXiv · show

We propose a Bayesian physics-informed neural network (B-PINN) to solve both forward and inverse nonlinear problems described by partial differential equations (PDEs) and noisy data. In this Bayesian framework, the Bayesian neural network (BNN) combined with a PINN for PDEs serves as the prior while the Hamiltonian Monte Carlo (HMC) or the variational inference (VI) could serve as an estimator of the posterior. B-PINNs make use of both physical laws and scattered noisy measurements to provide predictions and quantify the aleatoric uncertainty arising from the noisy data in the Bayesian framework. Compared with PINNs, in addition to uncertainty quantification, B-PINNs obtain more accurate predictions in scenarios with large noise due to their capability of avoiding overfitting. We conduct a systematic comparison between the two different approaches for the B-PINN posterior estimation (i.e., HMC or VI), along with dropout used for quantifying uncertainty in deep neural networks. Our experiments show that HMC is more suitable than VI for the B-PINNs posterior estimation, while dropout employed in PINNs can hardly provide accurate predictions with reasonable uncertainty. Finally, we replace the BNN in the prior with a truncated Karhunen-Loève (KL) expansion combined with HMC or a deep normalizing flow (DNF) model as posterior estimators. The KL is as accurate as BNN and much faster but this framework cannot be easily extended to high-dimensional problems unlike the BNN based framework.

Glossary

The paper uses abbreviations for Bayesian neural-network, PDE, uncertainty-estimation, and stochastic-process methods.

  • B-PINNs means Bayesian physics-informed neural networks.
  • BNNs means Bayesian neural networks.
  • DNF means deep normalizing flow.
  • GPR means Gaussian process regression.
  • HMC means Hamiltonian Monte Carlo, while VI means variational inference.
  • KL means Karhunen-Loève expansion, and PDEs means partial differential equations.

1. Introduction

The introduction motivates physics-informed Bayesian modeling for PDEs with limited and noisy data, combining physical constraints with uncertainty-aware inference.

  • Purely data-driven methods generally require large datasets for accurate results, whereas PDE-based solvers can learn from limited data by encoding physical laws.
  • GPR provides Bayesian uncertainty quantification but has difficulty handling nonlinearities in PDE applications.
  • PINNs are effective for forward and inverse PDE problems but lack built-in uncertainty quantification, limiting use with noisy data.
  • B-PINNs combine a Bayesian neural-network surrogate with posterior estimation by HMC or VI to solve PDEs with noisy data.The framework uses physical laws and scattered noisy measurements to quantify uncertainty.
  • The study compares B-PINNs using HMC or VI with dropout for uncertainty quantification across function-approximation and forward/inverse PDE tasks.
  • A truncated KL expansion is also tested as the surrogate model, paired with HMC or DNF posterior estimation.

2. B-PINNs: Bayesian Physics-informed Neural Networks

B-PINNs formulate PDE solution and parameter inference in a Bayesian framework, using neural-network surrogates, noisy-data likelihoods, and posterior sampling or approximation.

  • The PDE framework distinguishes forward problems, where λ is prescribed, from inverse problems, where λ is inferred from data.
  • The likelihood models scattered measurements of u, f, and b as independent Gaussian observations with known sensor fidelities and potentially different noise sizes.
  • B-PINNs represent the solution with a neural-network surrogate and impose PDE and boundary operators to construct predicted forcing and boundary terms.
  • For inverse problems, B-PINNs assign priors to both surrogate parameters θ and PDE parameters λ and calculate their joint posterior.
  • The framework can represent PDE parameters that are fields depending on x by using another surrogate model.
  • Finite-width neural networks can produce solution priors close to Gaussian while their first and second derivative priors are not close to Gaussian distributions.
  • HMC samples posterior parameters through Hamiltonian dynamics with a Metropolis-Hastings correction, while VI approximates the posterior with a restricted, often factorizable Gaussian family.

3. Results and Discussion

The study systematically compares B-PINN posterior estimators and dropout across one- and two-dimensional PDE-related tasks, using common neural-network settings and repeated sampling.

  • The experiments compare B-PINN-HMC, B-PINN-VI, and PINN dropout on function approximation and 1D/2D forward and inverse PDE problems.
  • B-PINN experiments use neural networks with two hidden layers of width 50 and independent standard Gaussian priors for each parameter.
  • Means and standard deviations are estimated from M = 10,000 samples for all methods in each test case.

3.1. Function regression

The function-regression experiment compares BNN posterior estimators and dropout under noisy observations. HMC closely matches the reference Gaussian-process regression, while VI is less accurate and finite-width BNN prior selection remains open.

  • Experimental setup: The B-PINN is reduced to a BNN for function regression, with likelihood P(D|θ) = P(Du|θ) and 32 noisy training points.The observations use noise ϵu ∼ N(0, 0.12) and are placed in [−0.8, −0.2]∪[0.2, 0.8].
  • Posterior comparisons: BNN-GPR and BNN-HMC produce predictive means similar to the exact function, with larger standard deviations where training data are sparse.BNN-GPR is an approximate Gaussian-process reference based on empirical neural-network covariance estimates.
  • Posterior comparisons: HMC provides posterior estimates similar to BNN-GPR despite a 2701-dimensional parameter vector, whereas VI misestimates uncertainty across different regions.VI uncertainty is larger on x ∈[−0.2, 0.2] and underestimated near the outer intervals.
  • Dropout comparison: The study tests dropout with rates 0.01, 0.05, and 0.2 across two- and four-hidden-layer architectures.The same dropout rate is used during prediction as during training to quantify uncertainty.
  • Limitation: The choice of finite-width BNN priors, including network architecture and parameter distributions, remains an open problem.The appendix discusses how architecture and prior choices influence results.

3.2. Forward PDE problems

Forward-PDE experiments evaluate Bayesian PINNs and dropout on linear and nonlinear equations under two Gaussian noise scales. HMC gives reliable predictions and uncertainty across the tested problems, whereas VI and dropout often provide inaccurate uncertainty or means.

  • 1D linear Poisson equation: The linear Poisson experiment uses noisy sensors for the source term and boundary conditions at noise scales 0.012 and 0.12.Sixteen sensors measure f, while two boundary sensors provide left/right Dirichlet conditions for u.
  • 1D linear Poisson equation: B-PINN-HMC mostly bounds the error between predictive means and the exact linear-Poisson solution within two standard deviations.Its standard deviation increases as the measurement-noise scale increases.
  • 1D linear Poisson equation: For the linear Poisson problem, B-PINN-VI fails to predict u accurately, while dropout matches data but yields inaccurate means and unreasonable uncertainties.Dropout uncertainty changes little between noise scales, and much of the exact solution lies outside its two-standard-deviation intervals.
  • 1D nonlinear Poisson equation: The nonlinear Poisson experiment uses 32 source sensors and two boundary sensors under the same two Gaussian noise scales.The nonlinear equation uses k = 0.7 and λ = 0.01.
  • 1D nonlinear Poisson equation: B-PINN-HMC provides good predictions for both u and f, while VI and dropout struggle to quantify uncertainty accurately for u(x).VI shows large boundary standard deviations at noise scale 0.1 despite boundary observations, and noise-scale effects are weak.
  • 2D nonlinear Allen-Cahn equation: The Allen-Cahn experiment uses 500 randomly distributed source sensors and 25 boundary sensors per boundary under two noise scales.The equation models multiphase flows, with λ = 0.01 and u as the order parameter.
  • 2D nonlinear Allen-Cahn equation: B-PINN-HMC produces means close to the exact Allen-Cahn solution, with errors mostly within two standard deviations that increase with noise.VI and dropout can have errors above 50% in parts of the domain and fail to keep errors within two standard deviations.

3.3. Inverse PDE problems

The inverse PDE experiments estimate unknown reaction parameters from scattered noisy measurements of u and f. B-PINN-HMC consistently provides accurate predictions and uncertainty quantification, outperforming VI and dropout in the reported cases.

  • Problem: The inverse problems estimate unknown reaction parameters from partial measurements of u and f under Gaussian noise.The 1D case uses measurements of f, boundary values, and interior values of u; the 2D case uses randomly sampled measurements and boundary sensors.
  • 1D inverse problem: B-PINN-HMC predicts u and f accurately for both noise cases, while B-PINN-VI and dropout have larger errors at noise scale 0.1.The HMC predictive means fit the exact functions well, whereas VI and dropout errors are larger in the higher-noise case.
  • 2D inverse problem: B-PINN-HMC identifies k with error less than 5% in both 2D noise cases, while B-PINN-VI errors exceed 10%.The HMC standard deviations are of the same order as predictive-mean errors and decrease as the data noise decreases.
  • Limitation: Dropout lacks a theory for choosing an optimal rate, restricting its usefulness for uncertainty quantification from noisy data.The reported standard deviations are not significantly influenced by the noise scale.

4. Comparison with PINNs

The paper compares B-PINN-HMC with standard PINNs on a noisy 1D inverse PDE problem. PINNs fit low-noise data comparably but overfit at higher noise, where B-PINN-HMC is more accurate and also quantifies uncertainty.

  • PINN behavior: Standard PINNs can fit all training points but do not quantify predictive uncertainty.The comparison trains PINNs with Adam for 200,000 steps.
  • Comparison: B-PINN-HMC has similar accuracy to PINNs with small data noise but a significant accuracy advantage with large noise.The comparison is based on the 1D inverse problem and the results in Table 1.
  • Computational cost: PINN computation takes about 10 minutes, compared with about 20 minutes for B-PINN-HMC on the reported small problem.The authors expect the cost difference to increase with data and neural-network size.

5. Comparison with the truncated Karhunen-Lo`eve expansion

The paper replaces the neural-network surrogate with a truncated Karhunen-Loève expansion and compares HMC with deep normalizing flows for posterior estimation. KL-based results are similar to B-PINNs, while KL-HMC is faster in the reported 1D tests but has high-dimensional limitations.

  • KL surrogate: A truncated KL expansion represents the surrogate with orthogonal eigenfunctions, eigenvalues, and mutually uncorrelated random variables.The expansion is truncated to n terms, with θ=(θ1,θ2,...,θn) as the surrogate parameters and prior variables.
  • KL surrogate: The KL surrogate used in the 1D tests has 20 parameters and retains about 92% of the energy, compared with 2701 neural-network parameters.The smaller parameter count enables posterior sampling with a deep normalizing flow.
  • Results: KL-HMC and KL-DNF produce nearly identical results, with predicted means for u and f close to the exact solutions.The predicted reaction-rate results are reported in Table 3.
  • Uncertainty: The KL predictive means fit the exact solution well, with standard deviations increasing as noise increases and errors bounded by two standard deviations.The authors report results similar to those obtained with B-PINNs.
  • Computational cost: DNF training takes about one day versus about 4 minutes for HMC, although DNF makes independent posterior sampling more convenient after training.The authors say this advantage has no significant benefit in the current work but could help other tasks.
  • Computational cost: KL-HMC takes about 4 minutes versus about 20 minutes for B-PINN-HMC in the 1D diffusion-reaction test.The reduced cost is attributed to the truncated KL expansion's smaller number of parameters.
  • Limitation: Truncated KL expansions suffer from the curse of dimensionality for high-dimensional functions, unlike deep neural networks.This limits straightforward extension of the KL-based framework to high-dimensional problems.

6. Summary

The paper develops a Bayesian framework for PINNs that addresses aleatoric uncertainty in forward and inverse PDE problems with noisy data. Comparisons favor HMC for posterior estimation, while the KL-based alternative is accurate and cheaper but limited to low-dimensional problems.

  • B-PINNs combine a Bayesian neural network constrained by the PDE with HMC or VI posterior estimation to quantify uncertainty from noisy data.The framework targets both forward and inverse PDE problems and uses the PDE constraint as a prior.
  • HMC provides good accuracy and robustness, whereas VI usually gives unreasonable uncertainties because it approximates the posterior with a factorizable Gaussian.
  • Dropout-based PINNs can hardly provide satisfactory uncertainty quantification, while PINNs can overfit noisy data and become less accurate than B-PINNs.
  • KL-HMC and KL-DNF are as accurate as B-PINN-HMC, with KL-HMC at reduced cost, but the KL-based framework is limited to low-dimensional problems.
  • The choice of B-PINN priors can significantly influence posterior predictions with small data, and selecting neural-network structures and parameter distributions remains open.

Appendix A. BNNs with different priors

The appendix examines how BNN architecture and parameter priors affect covariance functions and function-approximation predictions. Similar prior covariances correspond to similar posterior predictions across several architectures.

  • Covariance functions are compared for five representative BNN priors with input dimension Nx = 1 using 100,000 independent prior samples.
  • The posterior depends on both the prior and observed data, so surrogate models with similar priors should provide similar posterior distributions for the same observation.
  • When √N_lσw,l is fixed, covariance functions are similar across three tested architecture and weight-scale cases.
  • Predicted means and standard deviations are quite similar for three architectures whose prior covariance functions are similar.

Appendix B. Deep Normalizing Flow Models

The DNF framework learns a bijective transformation from an initial distribution to a target posterior distribution. It trains the transformation by minimizing a KL-divergence objective evaluated through the transformed samples.

  • A potential-flow DNF defines a bijective map G from an initial state at time t = 0 to time T by integrating an ODE.
  • The neural network ϕ(u, t; ζ) takes the concatenation of u and t as input and outputs a real-valued function used in the probability-density ODE.
  • The initial and final densities are P_µI(u) and P_µO(u), respectively, representing the distributions before and after the learned transformation.
  • Forward Euler discretizes the ODE with δt = T/n, yielding G(z) = u_n(z) and a corresponding discretized log-density.
  • For posterior sampling, the target distribution is P(θ|D), and the DNF parameters are trained by minimizing an objective approximating KL divergence to that posterior.
  • The implementation uses T = 1, 50 forward-problem steps or 10 inverse-problem steps, three hidden layers of width 128, 100,000 training steps, and batch size 16.
Loading 2003.06097v1…