PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-fidelity Bayesian Neural Networks: Algorithms and Applications

Xuhui Meng, Hessam Babaee, George Em Karniadakis

arXiv:2012.13294v1cs.LGphysics.comp-ph

TL;DR

The paper addresses multi-fidelity learning when high-fidelity observations are scarce and noisy, seeking accurate predictions, uncertainty quantification, and PDE parameter inference. It combines MAP, Bayesian, and physics-informed neural networks with variational-inference priors and HMC posterior sampling. Across function approximation, inverse PDEs, and real measurements, the method adaptively captures linear and nonlinear correlations, estimates unknown parameters, and supports active-learning uncertainty reduction.

  • Problem

    The paper targets multi-fidelity prediction and inverse problems where high-fidelity observations are scarce or costly, while existing approaches face scalability and correlation-modeling limitations.

  • Method

    The method combines a MAP-trained DNN for low-fidelity data, a BNN for uncertain cross-fidelity correlation, and a PINN for PDE physics, using VI priors and HMC posterior sampling.

  • Results

    Across function approximations, inverse diffusion-reaction systems, and SST data, the method captures linear and nonlinear correlations, quantifies prediction uncertainty, and estimates unknown PDE parameters.

  • Takeaways & Limitations

    Active learning reduces predictive uncertainty and improves accuracy by adding high-fidelity observations where posterior variance is largest.

  • Takeaways & Limitations

    The demonstrations assume noisy measurements with specified Gaussian noise levels and, in the reported study, represent λ as unknown constants rather than general fields.

Abstract

from arXiv · show

We propose a new class of Bayesian neural networks (BNNs) that can be trained using noisy data of variable fidelity, and we apply them to learn function approximations as well as to solve inverse problems based on partial differential equations (PDEs). These multi-fidelity BNNs consist of three neural networks: The first is a fully connected neural network, which is trained following the maximum a posteriori probability (MAP) method to fit the low-fidelity data; the second is a Bayesian neural network employed to capture the cross-correlation with uncertainty quantification between the low- and high-fidelity data; and the last one is the physics-informed neural network, which encodes the physical laws described by PDEs. For the training of the last two neural networks, we use the Hamiltonian Monte Carlo method to estimate accurately the posterior distributions for the corresponding hyperparameters. We demonstrate the accuracy of the present method using synthetic data as well as real measurements. Specifically, we first approximate a one- and four-dimensional function, and then infer the reaction rates in one- and two-dimensional diffusion-reaction systems. Moreover, we infer the sea surface temperature (SST) in the Massachusetts and Cape Cod Bays using satellite images and in-situ measurements. Taken together, our results demonstrate that the present method can capture both linear and nonlinear correlation between the low- and high-fideilty data adaptively, identify unknown parameters in PDEs, and quantify uncertainties in predictions, given a few scattered noisy high-fidelity data. Finally, we demonstrate that we can effectively and efficiently reduce the uncertainties and hence enhance the prediction accuracy with an active learning approach, using as examples a specific one-dimensional function approximation and an inverse PDE problem.

1. Introduction

The paper motivates multi-fidelity modeling for settings where higher-fidelity data improve accuracy but cost more, and identifies scalability and nonlinear-correlation limitations in Gaussian-process approaches. It proposes a Bayesian-neural-network scheme designed to provide uncertainty estimates, adaptively exploit linear and nonlinear correlations, and scale with dataset size.

  • Higher-fidelity data can improve predictions but usually require greater acquisition or computational cost, especially for uncertainty quantification and inverse problems.
  • Multi-fidelity techniques combine variable-fidelity data sources to enhance predictions and quantify uncertainty for cost-effective data acquisition.
  • GPR multi-fidelity training scales as O(N^3), limiting its utility for high-dimensional problems with many training points.
  • GPR models primarily exploit linear correlations, while nonlinear extensions require an added input dimension and larger datasets.
  • The proposed Bayesian-neural-network scheme targets uncertainty estimation, adaptive linear and nonlinear correlation modeling, and scalability with training-dataset size.

2. Methodology

The method combines a MAP-trained network for low-fidelity data, a Bayesian network for uncertain cross-fidelity correlation, and a physics-informed network for PDE constraints. Variational inference supplies priors, while Hamiltonian Monte Carlo produces posterior samples for predictions and unknown PDE parameters.

  • The multi-fidelity model uses a DNN for low-fidelity approximation, a BNN for uncertain low/high-fidelity correlation, and a PINN for PDE physics.
  • MAP trains the first network efficiently because low-fidelity data are generally plentiful and a Bayesian treatment would be computationally prohibitive.
  • The BNN takes high-fidelity locations and low-fidelity predictions as inputs to quantify uncertainty in cross-fidelity correlations.
  • The PDE formulation represents solution, forcing, and boundary observations through differential and boundary operators, with unknown PDE parameters included in the Bayesian parameter vector.
  • HMC samples the posterior over BNN hyperparameters and PDE unknowns, enabling predictive means and standard deviations for solutions and parameters.
  • Mean-field variational inference estimates the prior scale by maximizing the marginal likelihood before posterior sampling.

3. Results

The experiments evaluate the method on function approximation and inverse-PDE settings using noisy, variable-fidelity data and posterior sampling. The supplied methodology passages specify optimization, HMC, and posterior-sample settings used across these experiments.

  • The experiments use Adam optimization for DNN and initial BNN training, followed by adaptive-timestep HMC for posterior estimation.The DNN uses an initial learning rate of 10^-3 and 50,000 steps; the BNN uses the same initial rate and 200,000 steps.
  • Posterior predictions are computed from 1,000 HMC samples to obtain predicted means and standard deviations.

3.1. Function approximation

Multi-fidelity BNNs approximate 1D and 4D functions from noisy low- and high-fidelity data, with predictions and uncertainty estimates that improve over single-fidelity modeling. Increasing low-fidelity data improves predictions when low-fidelity measurements are highly noisy, while multi-fidelity intervals achieve over 90% coverage in the 4D case.

  • 1D function approximation: Multi-fidelity predictions agree well with exact 1D solutions, with most errors bounded by two standard deviations.This holds for both noise-free and moderately noisy low-fidelity data.
  • 1D function approximation: Multi-fidelity modeling significantly improves 1D prediction accuracy compared with single-fidelity modeling using the same BNN.The single-fidelity case uses only high-fidelity data, whereas the multi-fidelity cases incorporate low-fidelity predictions.
  • 1D function approximation: With low-fidelity noise N(0, 0.32), increasing low-fidelity training data from 100 to 1,000 and 1,500 improves low-fidelity and high-fidelity predictions.The high-fidelity predictions for 1,000 and 1,500 points become similar to the noise-free low-fidelity case.
  • 1D function approximation: For sensor-derived low- and high-fidelity data with different accuracy and bias, predicted high-fidelity means agree well with exact solutions under both tested low-fidelity noise levels.The tested low-fidelity noise levels are N(0, 0.05^2) and N(0, 0.3^2).
  • 4D function approximation: In 4D function approximation, multi-fidelity predicted means are much better than single-fidelity predictions at 100,000 random testing locations for both low-fidelity noise cases.The tested cases use noise-free and N(0, 0.05^2) low-fidelity data.
  • 4D function approximation: PICP exceeds 90% for both multi-fidelity 4D cases, indicating that most errors lie within two predicted standard deviations and outperform single-fidelity coverage.PICP measures whether exact values fall within the predicted interval defined by the mean ± two standard deviations.

3.2. Data-driven solutions for inverse PDE problems

The multi-fidelity framework combines neural approximations with physics-informed constraints to reconstruct PDE fields and infer unknown reaction rates from sparse noisy measurements. In the 2D problem, multi-fidelity modeling improves field predictions and parameter accuracy over single-fidelity modeling.

  • Framework: The framework attaches a physics-informed neural network to a Bayesian neural network to test multi-fidelity modeling on PDE-based inverse problems.
  • 1D inverse problem: The 1D inverse problem infers the unknown constant k from limited measurements of u(x) and f(x).
  • 1D inverse problem: The 1D setup uses 10 randomly distributed sensors each for u and f, two boundary sensors for u, and Gaussian measurement noise with standard deviation 0.01.
  • 2D inverse problem: The 2D problem predicts concentration u and source term f throughout the domain while identifying the unknown reaction rate k from limited high-fidelity observations.
  • 2D inverse problem: The 2D multi-fidelity predictions keep computational errors for u and f mostly within two standard deviations, whereas single-fidelity errors are about one order larger.
  • 2D inverse problem: For k, multi-fidelity errors remain within one standard deviation, increase with low-fidelity noise, and are more accurate than single-fidelity results.

3.3. Active learning for reducing uncertainty

Active learning adds noisy high-fidelity observations where posterior uncertainty is greatest, improving function approximation and inverse-PDE predictions. Increasing training samples reduces computational error and improves reaction-rate estimation.

  • Function approximation: The function-approximation experiments begin with 10 randomly sampled high-fidelity measurements and compare noise-free and noisy low-fidelity data.
  • Function approximation: Active learning selects the next observation where posterior variance is maximized and stops when predicted variance falls below 0.05^2.
  • Function approximation: Adding high-fidelity observations significantly improves predicted profiles for both noise-free and noisy low-fidelity cases.
  • Inverse PDE problem: The inverse-PDE active-learning experiment starts with 3 measurements for u and 10 for f, all randomly distributed over x ∈ [0, 1].
  • Inverse PDE problem: Adding more noisy high-fidelity samples for u and f makes the predicted profiles closer to the exact solutions.
  • Inverse PDE problem: Computational error decreases with iteration count, while reaction-rate accuracy improves as the predicted u and f become more accurate.

4. Summary

The MBNN assimilates abundant low-fidelity and scarce high-fidelity data for approximation and inverse PDE problems while quantifying uncertainty. It adaptively captures linear and nonlinear correlations, accurately estimates unknown parameters, and uses active learning to reduce uncertainty.

  • MBNN assimilates large low-fidelity and scarce high-fidelity datasets for function approximation and inverse PDE problems.The low-fidelity data may be simulation data or noisy low-accuracy measurements, while high-fidelity data are noisier measurements with higher accuracy.
  • Its three-network architecture combines low-fidelity fitting, correlation learning with uncertainty quantification, and physical-law encoding.The DNN fits low-fidelity data, the BNN captures low–high-fidelity correlation, and the third network encodes physical laws.
  • The approach adaptively captures both linear and nonlinear low–high-fidelity correlations and quantifies aleatoric and epistemic uncertainty.Aleatoric uncertainty is associated with noisy data, whereas epistemic uncertainty is associated with unknown parameters.
  • Across function approximations, sea-surface-temperature modeling, and diffusion-reaction inverse problems, MBNN provides reasonable predictive uncertainties and accurately estimates unknown parameters.The reported applications include one- and four-dimensional functions, a two-dimensional SST case, and one- and two-dimensional diffusion-reaction systems.
  • MBNN outperforms single-fidelity modeling in all test cases, while active learning efficiently reduces uncertainty by adding high-fidelity data where uncertainty is maximum.Active learning was demonstrated for a function approximation case and an inverse PDE problem.
Loading 2012.13294v1…