Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

1.1. Motivation and scope of the paper

• The framework addresses SciML challenges including ill-posed problems, noisy or gappy data, and high-dimensional computational costs by combining observational data with physical models.SciML methods include data-driven emulation, physics-encoding architectures, and soft physical constraints such as PINNs.Source: §1.1 · p. 2
• The paper unifies uncertainty modeling, UQ methods, evaluation metrics, post-training improvement, and comparative testing, emphasizing PINN and DeepONet frameworks while allowing extensions such as Fourier neural operators [10].Its goal is to establish groundwork for routine UQ use in SciML.Source: §1.1 · p. 3

1.2. Problem formulation

• Four scenarios cover known-operator forward PDEs, mixed deterministic PDEs, mixed stochastic PDEs, and stochastic unknown-operator learning with separate pre-training and inference phases.Problems 1–3 concern neural (S)PDEs, whereas problem 4 concerns neural operators; mixed formulations seek partially unknown solution and parameter fields.Source: §1.2 · pp. 3–4, Table 1 · p. 4

1.3. Novel contributions of this work

• The original contributions integrate posterior inference, priors, noise modeling, calibration, heteroscedastic-data solvers, SPDE polynomial-chaos architectures, noisy operator inference, and out-of-distribution detection into a unified UQ formulation.The work also combines Gaussian process regression with generative adversarial networks for deterministic forward PDEs and addresses heteroscedastic noise in mixed PDE data.Source: §1.3 · p. 5
• The framework further proposes neural-operator out-of-distribution detection for risk-related applications and combines physics with new and historical data containing diverse noise types.These contributions target a unified treatment across scientific machine-learning problem classes.Source: §1.3 · p. 5

1.4. Organization of the paper

• The paper proceeds from PINN and DeepONet foundations through uncertainty modeling, inference and priors, unified problem formulations, evaluation and post-hoc improvement, comparative studies, and discussion of future research.The comparative study includes function approximation and multiple scenarios from Table 1 and Fig. 1.Source: §1.4 · p. 5

2. Neural PDEs and neural operators

• The section presents vanilla PINNs and DeepONets as building blocks for uncertainty-aware PDE solutions and operator mappings, including later extensions with functional priors and SPDE methods.The framework combines available data with physical knowledge for the problems summarized in Table 1 and Fig. 1.Source: §2 · p. 7

2.1. Solving forward and mixed PDE problems: Overview of PINN method

• PINNs approximate PDE solutions with neural networks trained by mean squared error on data and physics-derived residual and boundary terms.For mixed deterministic PDEs, an additional network models the unknown parameter field λ(x), with shared notation for both networks’ parameters.Source: §2.1 · p. 7, Figure 3 · p. 7

2.2. Learning operator mappings: Overview of DeepONet method

• DeepONet learns mappings from input functions λ to solutions u using branch and trunk subnetworks whose w-dimensional outputs are merged by an inner product.For new inputs, interpolation may be required at fixed λ locations, and hybrid physics-and-data training is reported as most effective [7] [8].Source: §2.2 · pp. 7–8, Figure 4 · p. 8

3. Modeling total uncertainty

• Total uncertainty is represented by predictive distributions plus first- and second-order statistics, with terminology summarized in a glossary.The framework integrates these uncertainty concepts into SciML and supports confidence-interval construction.Source: §3 · p. 8

3.1. Uncertainty in function approximation

• Bayesian model averaging combines aleatoric data noise with epistemic parameter uncertainty, while standard point-estimate training omits epistemic uncertainty.Approximate posterior inference produces parameter samples whose predictive distributions are aggregated to estimate total uncertainty, means, and variances.Source: §3.1 · pp. 9–10

3.2. Uncertainty in PINNs

• In PINNs, a single optimized parameter set yields only aleatoric uncertainty, whereas posterior samples produce predictive PDFs containing both epistemic and aleatoric uncertainty.Separate likelihood functions model noisy observations for the f, b, u, and λ data categories.Source: §3.2 · p. 11

3.3. Uncertainty in DeepONets

• In DeepONets, noisy output data are modeled through a likelihood, but epistemic uncertainty requires posterior inference over multiple parameter samples.The pre-training procedure assumes clean λ inputs, limiting direct treatment of noisy input data; concurrent work used Markov chain Monte Carlo with DeepONets [92].Source: §3.3 · pp. 11–12

4. Methods for uncertainty quantification

• Parameter posterior samples are combined through approximate Bayesian model averaging to produce total-uncertainty distributions and first- and second-order statistics.The framework integrates methods for obtaining parameter samples and extends them to historical data, stochastic processes, stochastic differential equations, and noisy physics-informed settings.Source: §4 · p. 12

4.1. Bayesian methods

• Bayesian methods obtain parameter samples through MCMC, variational inference, Monte Carlo dropout, or Laplace approximation, then combine them with approximate Bayesian model averaging for U-PINN and U-DeepONet.HMC and Langevin Dynamics are used in the comparative study; heteroscedastic variants incorporate predicted noise through the likelihood.Source: §4.1 · pp. 12, 14

4.2. Ensembles

• Ensembles approximate posterior uncertainty by collecting multiple loss-landscape modes: deep ensembles retrain independently, snapshot ensembles use one trajectory, and SWAG fits a Gaussian to its optima.These samples are used in the same approximate Bayesian model averaging framework as Bayesian posterior samples.Source: §4.2 · p. 14

4.3. Functional priors (FPs)

• Functional priors use historical data to construct GAN-based priors whose posterior inference can be faster, require less data, and use substantially lower-dimensional parameters than standard Bayesian neural networks.The framework defines GAN-FP, physics-informed GAN-FP, and the novel physics-agnostic BNN-FP and GAN-FP models for inference with noisy data.Source: §4.3 · pp. 15–16

4.4. Solving stochastic PDEs (SPDEs)

• Stochastic-PDE methods represent stochastic processes with GAN generators or polynomial-chaos neural networks, while posterior sampling captures aleatoric, epistemic, and stochasticity uncertainties.The proposed U-NNPC+ adds a decoder neural network for the source term, and GP+PI-GAN1 propagates noisy source-data uncertainty into PDE solution statistics.Source: §4.4 · pp. 16–17

4.5. Towards a unified view of the presented methods

• The unified view separates neural PDE methods, which use governing equations, from neural operators, which use DeepONets as physics surrogates, while sharing posterior-inference algorithms.The conceptual framework maps U-PINN, PI-GAN-FP, GP+PI-GAN, U-PI-GAN, U-NNPC(+), U-DeepONet, PA-BNN-FP, and PA-GAN-FP to the paper’s problem classes.Source: §4.5 · p. 18, Figure 10 · p. 18

4.5.1. Neural PDEs: U-PINN, PI-GAN-FP, GP+PI-GAN, U-PI-GAN, and U-NNPC(+)

• The neural-PDE formulations combine PINNs with Bayesian or GAN-based inference: U-PINN handles mixed PDE data, PI-GAN-FP uses historical priors, U-PI-GAN handles mixed SPDEs, and GP+PI-GAN propagates noisy source data.U-PI-GAN infers generator parameters from stochastic realizations, whereas GP+PI-GAN first fits a Gaussian process to noisy source observations before PI-GAN modeling.Source: §4.5.1 · pp. 18–19
• U-NNPC(+) combines PINNs, polynomial chaos, and posterior inference to estimate solution and unknown-parameter statistics for mixed SPDEs.It receives stochastic realizations of f and potentially u, then performs posterior inference over BNN parameters β and reconstructs stochastic realizations of u.Source: §4.5.1 · p. 19

4.5.2. Neural operators: U-DeepONet, PA-BNN-FP and PA-GAN-FP

• U-DeepONet combines DeepONet with posterior inference over its parameters θ to learn neural operators from paired realizations of u and f.Unlike U-NNPC(+), it discards the additional model M and uses the DeepONet itself as the physics component.Source: §4.5.2 · p. 19
• PA-BNN-FP augments a pre-trained DeepONet with a BNN and posterior inference to reconstruct f and u from new limited, noisy data.The BNN parameters θ are inferred after pre-training the DeepONet on noiseless data.Source: §4.5.2 · p. 19
• PA-GAN-FP replaces the additional BNN with a GAN-FP generator and infers its random input θ to reconstruct f and u from limited, noisy data.The DeepONet and GAN-FP are separately pre-trained on noiseless data before posterior inference.Source: §4.5.2 · p. 19

5.1. Evaluation: Accuracy and uncertainty quality evaluation

• The framework uses metrics spanning predictive statistics or full distributions to support model selection, UQ-method comparison, and overall UQ-design evaluation.Fig. 11 organizes metrics by whether they require test data, a gold standard, or neither; they assess capability, accuracy, consistency, sharpness, dispersion, and distributional agreement.Source: §5.1 · p. 19, Figure 11 · p. 20

5.1.1. Accuracy and predictive capacity

• Accuracy is evaluated with relative ℓ2 error, while mean predictive likelihood measures how probable test data are under the predictive distribution.Bayesian model averaging can improve mean predictions while providing p(u|x, D); high MPL indicates highly probable validation or test data.Source: §5.1.1 · pp. 19–20, Figure 11 · p. 20

5.1.2. Statistical consistency

• Whole-distribution quality is assessed through calibration, where an ideal predictive CDF matches the data-generating process and RMSCE measures deviations between expected and observed interval coverage.Calibration is estimated on average across input locations because at most one sample may be available for each x; over- and under-confidence correspond to overly narrow or broad confidence intervals.Source: §5.1.2 · pp. 20–21, Figure 12 · p. 21

5.2. Post-training improvement: Calibration

• Post-training calibration uses a left-out calibration dataset to improve uncertainty estimates through variance scaling, CDF recalibration, or empirical scaled-residual fitting.Variance scaling changes only uncertainty magnitude, whereas the approaches of Kuleshov et al. [117] and Zelikman et al. [120] can update both predictive means and variances; these transformations act across the domain on average and may worsen in-distribution or out-of-distribution performance.Source: §5.2 · pp. 21–22
• The calibration procedures extend separately to available quantities and realizations in mixed PDEs, stochastic problems, and neural operator learning.Required calibration data vary from spatial observations to stochastic realizations and inputs λ, while separate predictive distributions for f, b, u, and λ can be calibrated when available.Source: §5.2 · p. 22, Figure 13 · p. 22

6. Comparative study

• Across the comparative study, methods are evaluated on in-distribution and out-of-distribution data using epistemic, aleatoric, and total uncertainties, 95% confidence intervals, and calibration plots.Unless otherwise stated, evaluation data are in-distribution; over- and under-confidence are defined relative to the data-generating confidence intervals.Source: §6 · p. 23

6.1.1. Known homoscedastic noise

• For known homoscedastic noise, GP, HMC, MCD, DEns, and SEns are most competitive, but MCD and SEns do not consistently increase uncertainty on out-of-distribution inputs.This limits their ability to distinguish interpolation from extrapolation in risk-sensitive applications.Source: §6.1.1 · p. 24, Table 4 · p. 24, Figure 14 · p. 25
• HMC epistemic uncertainty increases with noise magnitude and decreasing dataset size, whereas MCD does not, despite MCD achieving higher predictive likelihood than HMC across tested cases.The results motivate using multiple metrics because HMC’s larger uncertainty can reduce predictive likelihood while remaining desirable under noisier, smaller-data conditions.Source: §6.1.1 · pp. 26–27, Table 5 · p. 26
• Increasing NN expressivity increases epistemic uncertainty, while learned priors partially reverse this effect and yield similar posteriors across architectures.Post-training calibration with only 2-14 datapoints can substantially reduce RMSCE, but calibration on training data typically overfits and may worsen OOD performance.Source: §6.1.1 · pp. 27–28, Figure 16 · p. 27, Figure 17 · p. 28

6.1.2. Unknown homoscedastic noise

• With unknown homoscedastic noise, cheaper MFVI and LA perform comparably to HMC when priors are learned, although HMC retains the smallest in-distribution calibration error.HMC with Gibbs sampling learns concentrated parameter-prior and noise-scale distributions, with the learned prior approximating the stacked posterior.Source: §6.1.2 · pp. 28, 30, Table 6 · p. 29, Figure 18 · p. 29

6.1.3. Unknown Gaussian heteroscedastic noise

• For unknown Gaussian heteroscedastic noise, h-HMC+FP outperforms BNN-prior methods in mean-prediction accuracy, while h-DEns has highest MPL and h-HMC has best calibration.The GAN-FP learns heteroscedastic noise from historical data; h-HMC, h-MFVI, and h-DEns produce satisfactory aleatoric predictions despite limited data.Source: Table 7 · p. 30, §6.1.3 · pp. 31–32, Figure 21 · p. 31

6.1.4. Summary

• Function-approximation results indicate that learned-prior low-cost methods can match expensive methods, small held-out calibration sets reduce error, prior learning eases architecture selection, and heteroscedastic modeling is satisfactory.Comparing UQ methods requires multiple evaluation metrics, and the study also includes Student-t heteroscedastic noise with posterior tempering.Source: §6.1.4 · p. 32

6.2. Mixed deterministic PDE: Nonlinear time-dependent diffusion-reaction equation

• In the mixed diffusion-reaction PDE, five methods and standard, heteroscedastic, large-noise, extrapolation, and steep-boundary-layer cases are evaluated using noisy mixed measurements of u, f, and λ.The methods treat boundary and interior u measurements identically, allowing noisy, limited, or unavailable boundary and initial data.Source: §6.2 · pp. 32–33, Table 8 · p. 33, Table 9 · p. 33

6.2.1. Standard case: homoscedastic noise and uniform measurements in space and time

• PI-GAN-FP outperforms the BNN-based methods in most standard mixed-PDE comparisons, while HMC and DEns outperform MFVI and MCD; its RMSCE is at least six times smaller.HMC+FP also provides more accurate f(x) and λ(x) estimates than HMC and DEns and is almost perfectly calibrated.Source: §6.2.1 · pp. 34, 36, Figure 25 · p. 36, Table 10 · p. 39

6.2.2. Heteroscedastic noise case: larger noise scales close to the boundaries of the space domain

• Under heteroscedastic noise increasing toward domain boundaries, h-HMC learns relatively accurate noise and uncertainties that cover point-wise errors within 95% CIs in most cases.Epistemic uncertainty increases with noise and decreases with available data, especially for λ(x); mean-prediction accuracy deteriorates in challenging cases.Source: §6.2.2 · p. 37, Figure 26 · p. 37

6.2.3. Summary

• Overall, PI-GAN-FP is more accurate than BNN-based UQ methods, while HMC and DEns perform better than MFVI and MCD and uncertainties cover errors in most challenging cases.The additional cases include heteroscedastic and large noise, extrapolation, and steep boundary layers; HMC, MFVI, and DEns estimate unknown noise satisfactorily.Source: §6.2.3 · p. 37

6.3. Mixed stochastic elliptic equation

• For the mixed stochastic elliptic equation, U-NNPC+, U-NNPC, and U-PI-GAN use three independently trained models, with U-PI-GAN uniquely generating new stochastic samples directly.U-NNPC and U-NNPC+ require new samples of f and λ to produce new u samples, whereas U-PI-GAN samples from its learned distribution.Source: §6.3 · p. 38

6.3.1. Summary

• U-NNPC+ performs better than U-PI-GAN and U-NNPC for noisy stochastic realizations, but ensemble epistemic uncertainty often fails to cover errors and becomes over-confident.All three methods approximate the stochastic processes relatively accurately, including the most significant covariance-spectrum components.Source: §6.3 · p. 39, Table 10 · p. 39, §6.3.1 · p. 40

6.4. Forward deterministic PDE: Nonlinear static diffusion-reaction equation

• GP+PI-GAN outperforms U-PINN in mean accuracy with 15 source-term measurements, but underestimates u uncertainty with six because source and solution are not PDE-coupled.U-PINN instead couples u and f through the PDE, causing their uncertainties to increase together in the sparse-data case.Source: §6.4 · p. 41, Figure 29 · p. 41

6.5. Operator learning and out-of-distribution data detection: 2D flow in heterogeneous porous media

• DeepONet learns the operator mapping λ to u from 1,000 realizations, supporting general limited/noisy inference and special complete-λ inference scenarios.The operator-learning setup uses transformed and standardized conductivity and solution data, with complete or limited λ defined by availability on the pre-specified grid.Source: §6.5 · pp. 42–43, Table 11 · p. 42

6.5.1. Physics-agnostic functional prior: combining a pre-trained DeepONet with BNN and GAN priors for treating noisy inference data

• Combining a pre-trained DeepONet with a historical-data GAN performs better than combining it with a BNN for limited noisy inference data, while BNN results are more calibrated for u.The BNN incurs no additional computational cost, and its calibration advantage persists for in-distribution data.Source: Table 12 · p. 43
• PA-GAN-FP outperforms PA-BNN-FP in accuracy and predictive capacity for ID and OOD data, while PA-BNN-FP is more calibrated and adds no computational cost.Both methods deteriorate substantially on OOD data, with RL2E near 40–50%; PA-BNN-FP is also more calibrated for OOD data.Source: Figure 30 · p. 44, Figure 31 · p. 44, §6.5.1 · p. 45

6.5.2. U-DeepONet: combining DeepONet with deep ensemble for incorporating epistemic uncertainty

• U-DeepONet achieves RL2E = 19% on OOD data versus approximately 40% for individual DeepONets, while its 95% epistemic-uncertainty CI covers errors for both ID and OOD inputs.For ID data, RL2E is 4%; average output standard deviation increases from 0.04 for ID to 0.16 for OOD, reflecting greater ensemble disagreement.Source: §6.5.2 · p. 46, Figure 32 · p. 46, Figure 33 · p. 47

6.5.3. Out-of-distribution (OOD) data detection

• GAN MSE, GAN NLL, and DEns std follow the RL2E trend for OOD data detection; for correlation lengths l = 0.15 and l = 0.25, their values are higher by 60%, 37%, and 57%, respectively.Source: Figure 34 · p. 48, §6.5.3 · p. 49

6.5.4. Summary

• For complete clean inputs, U-DeepONet is more accurate than standard DeepONet on OOD data, its epistemic uncertainty covers errors, and the proposed metrics provide a practical warning for unreliable outputs.For limited noisy inputs, PA-GAN-FP performs better than PA-BNN-FP, whereas PA-BNN-FP is more calibrated and incurs no additional computational cost.Source: §6.5.4 · pp. 49–50

6.6. Performance versus cost

• HMC and deep ensembles are the most computationally expensive but generally achieve the strongest performance, whereas LA, MFVI, and SEns provide good performance–cost trade-offs.Indicatively, HMC costs 4–5 times standard training for 50 leapfrog steps; deep ensembles cost M training sessions serially or standard-training cost when parallelized, while post-training calibration can reduce error substantially at approximately zero computational cost.Source: §6.5.1 · p. 45, §6.6 · pp. 50–51

7. Summary

• The paper introduces a unified SciML UQ framework integrating uncertainty modeling, posterior inference, prior learning, hyperparameter selection, calibration, and evaluation for neural PDEs, SPDEs, and operators.The comparative study covers function approximation and prototype problems with limited or mixed noisy data, physics information, and historical data for cost reduction.Source: §7 · pp. 51–52, §7.3 · p. 54

7.1. Novel contributions

• Original contributions include functional-prior heteroscedastic-noise learning, GP+PI-GAN, PI-GAN-FP, NNPC+, noisy-data operator inference, and metrics for detecting OOD neural-operator inputs.These contributions are supported by results in Figs. 21, 29, D.14, 7, 26, 9, 27, 30–31, and 34, and Table 13.Source: §7.1 · pp. 52–53

7.2. Findings

• Lower-cost MFVI and LA can match more expensive HMC and DEns, especially with learned priors, while post-training calibration reduces calibration error and architecture sensitivity.Performance comparisons require multiple metrics because MCD can outperform HMC on some metrics while producing over-confident OOD predictions.Source: §7.2 · p. 53
• U-NNPC+ outperforms existing methods for mixed SPDEs with noisy stochastic realizations, while PI-GAN-FP outperforms BNN-based UQ methods for mixed PDEs.For forward PDEs, GP+PI-GAN can outperform U-PINNs in some cases but may be over-confident because it does not use the PDE when fitting source data.Source: §7.2 · p. 53
• For operator learning, PA-GAN-FP performs better than PA-BNN-FP on limited noisy inference data, whereas U-DeepONet improves OOD mean predictions and its uncertainty supports OOD detection.PA-BNN-FP incurs no additional computational cost and is more calibrated; detection metrics can flag potentially inaccurate DeepONet outputs.Source: §7.2 · p. 53

7.3. Outlook and future research directions

• Future work targets scalable UQ, cost-effectiveness studies, active learning for rare events, multi-fidelity and augmentation methods, noisy DeepONet pre-training, alternative UQ frameworks, and digital twins.Preliminary results suggest MFVI, LA, and SEns can improve predictions and uncertainty when combined with prior learning, tuning, and calibration.Source: §7.3 · p. 54

A.1. Statistics of the predictive distribution

• The predictive mean requires integration over plausible NN parameters, and total variance combines expected aleatoric variance with parameter-induced epistemic uncertainty.Aleatoric noise can also increase epistemic uncertainty, so the two components are not generally straightforward to disentangle; covariance terms extend analogously beyond the diagonal.Source: §A.1 · p. 67

A.2. Point estimates (standard neural network training)

• MLE infers NN parameters by maximizing p(D|θ), while Gaussian likelihood yields unregularized MSE and alternative likelihoods such as Laplace yield different objectives.In standard function approximation, Laplace likelihood leads to mean absolute error minimization.Source: §A.2 · p. 67
• MAP combines the likelihood with a parameter prior, producing deterministic estimates; with Gaussian likelihood, a Gaussian prior gives regularized MSE or ℓ2 weight decay.Stochastic gradient descent applies MAP updates using mini-batches and a learning rate.Source: §A.2 · p. 68

A.3. Hyperparameters and hyperpriors

• Unknown prior and noise-model parameters should be integrated out Bayesianly rather than treated as fixed, introducing hyperparameters distinct from NN parameters θ.The appendix denotes their prior distributions as hyperpriors.Source: §A.3 · p. 68
• Because exact hyperparameter integration is infeasible, type-II maximum likelihood or empirical Bayes selects hyperparameters by maximizing their marginal likelihood.The same approximation can select among competing NN models, while later sections provide additional metrics for hyperparameter optimization and model design.Source: §A.3 · pp. 68–69

A.4. Posterior tempering for model misspecification

• For overparametrized NNs with finite data, the Bayesian posterior is optimal for model averaging under the PAC-Bayes bound [144] [145], and approximate BMA can outperform point estimates [146].This requires perfect model specification, including a matching NN function space and correct data-noise model.Source: §A.4 · p. 69
• Model misspecification can make BMA worse than point estimates [146], for example when Gaussian likelihood is used for Student-t data or an unsuitable prior restricts the approximation [143].The prior and NN architecture jointly control the function space, so either can produce suboptimal Bayesian results.Source: §A.4 · p. 69
• Posterior tempering samples from p(θ|D)^(1/τ), and reported results indicate that cold posteriors with τ < 1 can perform better [146] [148] [149].Cold posteriors concentrate the posterior by effectively over-counting the data; more informative priors and post-training calibration may reduce or address the effect.Source: §A.4 · p. 69

B.1. Gaussian processes (GPs)

• Gaussian-process regression separates epistemic uncertainty from total uncertainty by adding the observation-noise variance σ2_u to the predictive covariance.Higher σ2_u reduces data-driven epistemic uncertainty; with sufficiently noisy data, the epistemic uncertainty approaches the prior uncertainty because little new information is acquired.Source: §B.1 · p. 70

B.2.1. Markov chain Monte Carlo (MCMC)

• MCMC constructs an ergodic Markov chain whose equilibrium distribution is the posterior p(θ|D), using burn-in and lagged resampling after convergence.Metropolis methods accept or reject proposal states to preserve the posterior as the invariant distribution, whereas Gibbs sampling updates dimensions from conditional distributions when those are available.Source: §B.2.1 · pp. 70–71
• HMC removes discretization bias through Metropolis correction and is used as a high-accuracy reference, with this study tuning its leapfrog step for an acceptance rate near 0.6.SGLD and stochastic-gradient HMC reduce MCMC cost by using data subsets, while uncorrected Langevin dynamics is a cheaper one-step version of HMC but lacks the same correction.Source: §B.2.1 · p. 72

B.2.2. Variational inference (VI)

• Variational inference approximates p(θ|D) with q_ω(θ) by maximizing the ELBO, balancing data explanation against proximity to the prior and enabling direct predictive sampling.Approximation quality depends on the expressiveness of q_ω(θ); the framework uses reparameterized path-wise gradient estimators, mini-batches, and factorized Gaussian or dropout-based variational distributions [168].Source: §B.2.2 · pp. 74–76
• Variational inference separates prior and data uncertainty in KL and likelihood terms, while ELBO-based alternating optimization updates variational parameters and uncertainty hyperparameters.MC-dropout obtains noisy parameter versions through Bernoulli-masked optimal parameters and approximates posterior predictions through repeated stochastic forward passes.Source: §B.2.2 · p. 76

B.2.3. Laplace approximation (LA)

• Laplace approximation fits a Gaussian posterior around the optimized mode, using the inverse Hessian as covariance and cheaper diagonal or block-diagonal approximations when needed.The study uses the full Hessian for performance, while generalized Gauss-Newton, Fisher, KFAC, or last-layer approximations can reduce cost; Hessian-based hyperparameter updates are expensive for high-dimensional parameters.Source: §B.2.3 · pp. 76–77

B.3.1. Deep ensembles (DEns)

• Deep ensembles approximate Bayesian model averaging by independently training M networks from different initializations, yielding M MAP modes rather than posterior samples for standard neural networks.For linear models, deep ensembles can follow the posterior, while proposed modifications target approximate Bayesian inference for neural networks.Source: §B.3.1 · p. 77

B.3.2. Snapshot ensembles (SEns)

• Snapshot ensembles obtain multiple local optima within one training budget by cycling the learning rate, capturing a snapshot before each restart for prediction ensembling.Cosine annealing implements the restarts; the study describes a budget of 25,000 iterations divided into M = 5 cycles as an example.Source: §B.3.2 · p. 78, Figure B.1 · p. 78

B.3.3. Stochastic weight averaging (SWA) and SWA-Gaussian (SWAG)

• SWA averages parameter snapshots for prediction, whereas SWAG fits a Gaussian to snapshots and Multi-SWAG applies this procedure across deep-ensemble trajectories for uncertainty estimation.SWAG estimates covariance diagonally or with a diagonal-plus-low-rank form, using Q = M in this study.Source: §B.3.3 · pp. 78–79

B.4. Functional priors (FPs)

• Functional priors use pre-trained generators to match historical function distributions, enabling faster test-time posterior inference with substantially lower-dimensional parameters.The parameter dimension can be as small as 10-100, and generator outputs are synthetically contaminated with assumed historical noise; test-time datasets may have different noise levels.Source: §B.4 · pp. 79–80

B.5. Solving stochastic PDEs (SPDEs) with U-NNPC(+)

• NNPC+ compares PDE outputs with a decoder trained from low-dimensional PCA inputs, adding expressivity but requiring regularization because performance may deteriorate.NNPC+ is introduced to address information loss from PCA and is evaluated computationally in Section 6.3.Source: §B.5 · p. 81

C.1.1. Secondary metrics evaluating statistical consistency

• For equal RMSCE, sharper predictions are preferred using PIW, but PIW should be interpreted together with calibration rather than alone.PIW measures predictive-interval width averaged across test locations and distinguishes useful sharpness from constant uncertainty that can appear calibrated.Source: §C.1.1 · p. 82
• At fixed RMSCE, larger SDCV is preferred because dispersion in predicted uncertainties may improve robustness away from training data.SDCV is the coefficient of variation of predicted standard deviations across locations; neither SDCV nor PIW is sufficient independently.Source: §C.1.1 · p. 82

C.1.2. Comparison to gold standard

• Gold-standard comparisons use NIP-G for normalized variance agreement and KL-G for divergence between gold-standard and predictive distributions.The gold standard may be an exact solution or an accurate but expensive method such as HMC; NIP-G is bounded between 0 and 1, whereas KL-G compares full distributions.Source: §C.1.2 · p. 82

C.2.1. CDF modification by fitting an auxiliary model

• Auxiliary-CDF calibration transforms predictive CDFs with an estimated calibration map Q, yielding perfect calibration in the idealized construction.Q is estimated from calibration data and can be fitted with isotonic regression; the transformation changes both predicted means and variances.Source: §C.2.1 · p. 83

C.2.2. CRUDE: CDF modification by fitting an empirical distribution

• CRUDE calibrates predictions by fitting an empirical distribution to sorted scaled calibration residuals and combining it with each location’s predicted mean and standard deviation.The resulting empirical predictive distribution modifies both the mean and variance of u at each x.Source: §C.2.2 · p. 83

D.1. Details regarding Fig. 19 for the function approximation problem of Section 6.1

• Fig. 19a compares initial, learned, parameter-level posterior, and stacked posterior distributions for NN parameters θ.Histograms are visualized with kernel density estimation.Source: §D.1 · p. 84
• Fig. 19b and Fig. 19c show prior-versus-posterior distributions for uθ(x = 0) and predicted noise σu, respectively.The function-value distributions use θ samples, while σu values are drawn from Gamma distributions conditioned on the data.Source: §D.1 · p. 84

D.2. Additional results for the function approximation problem of Section 6.1

• Additional results cover both the known homoscedastic-noise case and the unknown Student-t heteroscedastic-noise case.These cases correspond to Sections D.2.1 and D.2.2, respectively.Source: §D.2 · p. 84

D.2.1. Known homoscedastic noise and post-training calibration

• Under known homoscedastic noise, nine UQ methods are evaluated on OOD data using noisy-test metrics, with calibration assessed separately on ID data.Table D.1 notes that final performance depends on the dataset and individual tuning, while conservative OOD uncertainty can worsen MPL; calibration may also degrade OOD performance.Source: §D.2.1 · p. 84, Table D.1 · p. 85
• Post-training calibration compares uncalibrated and calibrated 95% confidence intervals and RMSCEs for GP, HMC, LD, MFVI, MCD, LA, DEns, SEns, and SWAG.The comparisons are reported across Figs. D.1-D.3.Source: §D.2.1 · p. 84, Figure D.1 · p. 85, Figure D.2 · p. 86
• MCD epistemic uncertainty does not increase with larger noise magnitude or smaller dataset size across σu = 0.1, 0.3, 0.5 and N = 32, 16.This behavior is shown through mean and epistemic-uncertainty predictions in Fig. D.4.Source: §D.2.1 · p. 88, Figure D.4 · p. 88

D.2.2. Unknown Student-t heteroscedastic noise

• For unknown Student-t heteroscedastic noise, samples follow ϵu = 0.5|x|ϵt with ν = 5, while the NN predicts both mean uθ(x) and variance σ2u(x).The noise variance is 0.25|x|2ν/(ν−2), and the setup uses h-HMC and h-MFVI with a Gaussian likelihood.Source: §D.2.2 · p. 88
• Because Gaussian likelihoods model Student-t data, h-HMC and h-MFVI are compared with and without posterior tempering under model misspecification.Posterior tempering is varied through the temperature τ to control prediction conservativeness.Source: §D.2.2 · p. 88
• Both methods perform similarly by RL2E, but HMC performs better by MPL, while MFVI uncertainty does not consistently increase for OOD data.The metric comparison may therefore fail to capture MFVI’s disadvantage relative to more accurate techniques such as HMC.Source: Table D.2 · p. 88
• For this misspecified case, a cold posterior improves predictive capacity measured by MPL, consistent with posterior-tempering results across Gaussian and Student-t noise settings.Posterior tempering can be treated as one additional training hyperparameter and applied when the assumed model is correct or misspecified.Source: §D.2.2 · pp. 88–89, Figure D.5 · p. 89, Figure D.7 · p. 91

D.3. Additional results for the mixed PDE problem of Section 6.2

• The additional mixed-PDE study considers standard, large-noise, extrapolation, and steep-boundary-layer cases.The extrapolation case concentrates λ(x) measurements on x ∈ [0, 1], while the steep-boundary-layer case has large gradients near x ≈ −1 and x ≈ 1.Source: §D.3 · p. 92

D.3.1. Standard case: homoscedastic noise and uniform measurements in space and time

• For the standard case, the appendix reports results from MFVI with learned noise and MCD with known noise.The training setting follows the standard case of Section 6.2.1.Source: §D.3.1 · p. 92
• Standard-case predictions for u and f are satisfactory, whereas λ is over-confident in some domain regions.These results are provided in Figs. D.8 and D.9.Source: §D.3.1 · p. 92

D.3.2. Large noise case: homoscedastic noise with large noise scale

• The large-noise case preserves the training-data locations and changes only the noise scale from 0.05 to 0.1.The remaining problem setup and BNN architecture are unchanged from Section 6.2.1.Source: §D.3.2 · p. 92
• HMC yields larger predicted total uncertainty in the large-noise case than in the standard case, as expected.The comparison is illustrated in Fig. D.10.Source: §D.3.2 · p. 92
• For the large-noise case, computational errors for u, f, and λ all fall within the 95% CIs.This result accompanies the increased predicted total uncertainty obtained with HMC.Source: §D.3.2 · p. 92

D.3.3. Extrapolation case: data concentrated on one side of the space domain

• The extrapolation experiment keeps the same u and f training data and λ observation count as Section 6.2.1, but places all observations in x ∈[0, 1].This creates a data-free region on the opposite side of the space domain.Source: §D.3.3 · p. 92

D.3.4. Steep boundary layers case: large gradients close to the boundaries of the space domain

• The steep-boundary-layer experiment concentrates additional f measurements near both boundaries while deleting some measurements away from them.The reference solution and training data are generated using the Matlab PDE toolbox, with u and λ measurement locations unchanged from Section 6.2.1.Source: §D.3.4 · p. 92
• Small fluctuations appear in predicted f near the left boundary, likely because f depends on the second-order derivative of the steep boundary-layer solution u.No corresponding right-side fluctuations are observed.Source: §D.3.4 · p. 92
• The absence of right-side f fluctuations is consistent with denser f measurements near the right boundary and lower noise there than near the left boundary.These measurement choices are part of the steep-boundary-layer experiment, which uses Nf = 13 and σu = σf = σλ = 0.05.Source: §D.3.4 · p. 92

D.3.3. Extrapolation case: data concentrated on one side of the space domain

• Accurate λ predictions extend into the data-free region x ∈[−1, 0], and computational errors for u, f, and λ remain within 95% CIs.The extrapolation is enabled by coupling λ with u and f through the PDE, allowing information from u and f to support λ inference.Source: §D.3.3 · p. 92, Figure D.11 · p. 94

D.3.4. Steep boundary layers case: large gradients close to the boundaries of the space domain

• With steep boundary layers, computational errors for u, f, and λ are mostly within 95% CIs.HMC is evaluated on a reaction-rate configuration that makes u sharper at both boundaries.Source: §D.3.4 · p. 92, Figure D.12 · p. 95

• RMSCE is computed from 300 prediction samples at each test point, combining three ensemble samples with 100 stochastic realizations.For clean test realizations, this measures epistemic uncertainty and stochasticity uncertainty, referred to here as total uncertainty.Source: §D.4 · p. 96
• Calibration reuses 100 prediction realizations sampled at training data locations and varies the number of available left-out realizations from 2 to 50.The calibration data share the training data locations and noise characteristics.Source: §D.4 · p. 96
• A few 2-10 left-out noisy realizations can reduce RMSCE by a factor of 3 across the considered UQ methods.Calibration effects are shown for U-PIGAN, U-NNPC, and U-NNPC+ under different calibration approaches and dataset sizes.Source: Figure D.13 · p. 96, §D.4 · p. 96

D.5. Additional results for the forward PDE problem of Section 6.4

• With Nf = 6, no measurements of f are available at the two boundaries, as illustrated by the forward-PDE comparison.The comparison includes training data, reference functions, and mean and epistemic-uncertainty predictions for u and f.Source: §D.5 · p. 97, Figure D.14 · p. 97
• GP+PI-GAN underestimates u uncertainty relative to U-PINN when only Nf = 6 datapoints of f are available, because f and u are not coupled through the PDE.GP+PI-GAN also predicts larger uncertainty for f at the two unmeasured boundaries than U-PINN.Source: Figure D.14 · p. 97, §D.5 · p. 97

D.6. Details regarding the DeepONet input for the operator learning problem of Section 6.5

• Noisy positive λ data are modeled as log-normal with mean equal to the clean value λc(x, y) and noise proportional to λc(x, y).The transformed quantity ˜λ = log λ is treated as Gaussian under this modeling procedure.Source: §D.6 · p. 98
• The DeepONet input standardizes log-transformed λ separately at each (x, y) using training-data sample means and standard deviations.The standardized datapoints follow a Gaussian distribution, according to the supplied construction.Source: §D.6 · p. 98
• After fitting the approximator on standardized data, predictions for the original quantity λ are recovered through the corresponding inverse transformation.The approximation is fitted using the standardized data rather than the original-scale observations.Source: §D.6 · p. 98
• Because the standardization depends on (x, y), the approximator λθ can only be evaluated on the original grid used to train the DeepONet.Avoiding this limitation requires no standardization or a shared standardization across all points.Source: §D.6 · p. 98

D.7. Additional results for the operator learning problem of Section 6.5

• OOD evaluation reconstructs λ and u from 20 and 10 randomly selected noisy measurements, using a Gaussian-process sample with l = 0.2 and a Matlab PDE-toolbox reference solution.The setup parallels the in-distribution evaluation in Section 6.5 while testing unseen operator-learning inputs.Source: §D.7 · p. 98
• RL2E rises to roughly 40–50% for OOD operator-learning data, while PA-BNN-FP is more calibrated than PA-GAN-FP and covers computational errors more often within its 95% CI.PA-GAN-FP produces confident, over-confident predictions with small epistemic uncertainties for unseen λ, whereas PA-BNN-FP’s larger prior space may explain its broader uncertainty coverage.Source: §D.7 · p. 98, Figure D.15 · p. 99, Figure D.16 · p. 99
• Reducing the post-training calibration dataset to 2–10 noisy datapoints can halve RMSCE in some cases, especially for OOD data and variance re-weighting.Calibration can also worsen predictions in some cases because it calibrates on average; the experiment illustrates trends rather than a cost-effectiveness comparison with active learning.Source: §D.7 · p. 98, Figure D.17 · p. 100

D.8. Hyperparameters used in the comparative study

• The comparative study uses TensorFlow Adam by default, with β1 = 0.5 and β2 = 0.9 for GANs/PI-GANs versus β1 = 0.9 and β2 = 0.999 for standard NNs.The appendix provides a hyperparameter glossary and section-specific settings in Tables D.3–D.14.Source: §D.8 · p. 100, Table D.3 · p. 101
• HMC without Gibbs sampling uses TensorFlow Probability with adaptive step size, whereas HMC with Gibbs sampling fixes and manually tunes step size for an acceptance rate around 0.6.MFVI early stopping is selected using validation negative-log-likelihood, and its initialization and batch size are retained throughout the study.Source: §D.8 · p. 100, Table D.4 · p. 101
• Tables D.4–D.14 list technique-specific hyperparameters for the comparative experiments spanning Sections 6.1–6.5 and several appendix subsections.The tables cover MFVI, dropout, MAP and prior/noise-scale optimization, and the methods used in Sections 6.2.2, 6.3, 6.4, and 6.5.Source: §D.8 · p. 100, Table D.5 · p. 101, Table D.6 · p. 102

Function approximation

• Most BNNs use two hidden layers of 50 neurons with tanh activations and independent standard-normal weight and bias priors.This shared architecture is used for the function-approximation comparison and its BNNs.Source: §D.9 · pp. 103–104

Mixed PDE problem

• In the mixed PDE problem, separate identically structured BNNs approximate u and λ, differing only in input dimension: 2 for space-time u and 1 for spatial λ.Both networks use the same two-layer, 50-neuron-per-layer tanh architecture and standard-normal BNN priors.Source: §D.9 · p. 104

Forward PDE and operator learning

• The forward PDE and operator-learning BNNs reuse the Section 6.1 architecture and priors, while GANs and PI-GANs receive architectures specified in Table D.15.The study uses one generator network when posterior computation is unnecessary and two when posterior estimation benefits from separate Gaussian-noise and coordinate inputs.Source: §D.9 · p. 104, Table D.15 · p. 104

Heteroscedastic noise

• For heteroscedastic noise, an additional two-layer, 20-neuron-per-layer ReLU network maps x to the varying noise scale.This network supplements the model used in Section 6.1.2.Source: §D.9 · p. 104

NNPC and neural operators

• Architectures for NNPC, NNPC+, and DeepONets are reported in Tables D.16 and D.17, with NNPC using separate networks for means and modes of u and λ.Table D.16 specifies the mean-and-mode network arrangement for each predicted quantity.Source: Table D.16 · p. 104, Table D.17 · p. 104