PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial

Venkat Nemani, Luca Biggio, Xun Huan, Zhen Hu, Olga Fink, Anh Tran, Yan Wang, Xiaoge Zhang, Chao Hu

arXiv:2305.04933v2cs.LGcs.AI

TL;DR

Many ML/DL prognostics algorithms do not quantify uncertainty, a limitation for practical use in mission- and safety-critical settings. This tutorial classifies uncertainty, reviews UQ methods and metrics, and compares their performance; it reports that GPR and SNGP generally outperform alternatives in calibration, with SNGP matching GPR on OOD detection.

  • Problem

    Most ML/DL prognostics algorithms did not quantify uncertainty, a limitation for practical use in mission- and safety-critical applications.

    Source: §5.1.3 · p. 54

  • Method

    The tutorial classifies uncertainty, describes state-of-the-art neural-network UQ methods, reviews evaluation metrics, and develops case studies in engineering design and PHM.

    Source: §8 · p. 74

  • Results

    GPR and SNGP generally outperform other UQ methods in calibration, and SNGP achieves OOD-detection performance similar to GPR while scaling better.

    Source: §3.6 · pp. 41–42

  • Takeaways & Limitations

    The comparison can help users select UQ methods for specific applications by weighing calibration, OOD detection, and computational cost.

    Source: §3.6 · pp. 41–42

    GPR can be computationally expensive in high-dimensional problems because training requires optimizing many hyperparameters and handling a large covariance matrix.

    Source: §3.1.2 · p. 25

Abstract

from arXiv · show

On top of machine learning models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Toward this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. Two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage and the remaining useful life prediction of turbofan engines.

Nomenclature · 1. Introduction

The introduction motivates predictive uncertainty quantification (UQ) as a way to improve trust, safety, and risk management for machine-learning models, especially in engineering design and health prognostics. It outlines the tutorial’s coverage of uncertainty sources, UQ methods and assessment, applications, and implementation code.

1. Introduction

  • Neural networks can make large errors on out-of-distribution samples and remain difficult to interpret, motivating UQ that communicates prediction confidence to users.The tutorial argues that UQ and interpretation should ideally be performed together to improve algorithmic transparency and trustworthiness.Source: §1 · p. 3
  • Classification confidence can be expressed as a predicted-class probability, while regression uncertainty is often shown as a confidence interval; narrower intervals indicate lower predictive uncertainty.The example interval 120 ± 15 weeks represents a two-sided 95% confidence interval under a Gaussian prediction assumption.Source: §1 · p. 4
  • UQ can help users judge when predictions warrant caution, supporting safety assurance and risk management in high-stakes decisions.Underestimating uncertainty can create trust issues, whereas overestimating it can lead to overly conservative predictions and reduce ML’s value.Source: §1 · p. 4
  • Work on neural-network UQ grew after high-confidence failures; pioneering approaches included MC dropout and neural-network ensembles, alongside distinctions between aleatory and epistemic uncertainty.Aleatory uncertainty is commonly treated as inherent data randomness, while epistemic uncertainty stems from limited training data; the tutorial notes these categories may encompass broader sources and causes.Source: §1 · pp. 4–5
  • In engineering design, GPR has been used for simulation surrogates and adaptive sampling, while its uncertainty is less often evaluated as a measure of OOD or epistemic uncertainty.Other classical surrogates and DNNs are often used as deterministic functions for uncertainty propagation rather than for predictive UQ.Source: §1 · p. 5
  • UQ has particular importance in PHM because run-to-failure data are costly to collect, unseen operating conditions require extrapolation, and maintenance errors can have serious consequences.Prior PHM applications include Bayesian linear regression, relevance vector machines, and GPR for prognostics.Source: §1 · pp. 5–6
  • The tutorial focuses on predictive UQ for neural networks and contributes a classification of uncertainty types, sources, and causes, plus tutorial-style coverage of methods and uncertainty-assessment metrics.It also explains decomposition of total predictive uncertainty into aleatory and epistemic components and discusses reducing epistemic uncertainty.Source: §1 · pp. 6–7
  • The tutorial reviews emerging UQ applications in engineering design and health prognostics and shares code for a toy regression example and two real-world prognostics case studies.The code is intended to support replication, customization, and testing of UQ methods.Source: §1 · p. 7

2. Types and sources of uncertainty

The section distinguishes irreducible aleatory uncertainty from reducible epistemic uncertainty, explains how predictive uncertainty can be decomposed into these components, and outlines strategies for reducing epistemic uncertainty.

2.1. Aleatory and epistemic uncertainty

  • Aleatory uncertainty reflects irreducible natural variability, whereas epistemic uncertainty arises from knowledge gaps and is reducible.Aleatory uncertainty includes noise and variation in inputs, outputs, or their relationship; epistemic uncertainty includes model-form choices and parameter uncertainty from limited or biased data and training difficulties.Source: §2.1 · pp. 8–9

2.2. Decomposition of predictive uncertainty

  • The law of total variance separates predictive uncertainty into expected conditional variance for aleatory uncertainty and variance of conditional means for epistemic uncertainty.In the linear regression example, epistemic uncertainty depends on parameter uncertainty and input values, not the intrinsic noise variance.Source: §2.2 · p. 11
  • Separating uncertainty components identifies whether uncertainty can be reduced or instead calls for risk-based decisions.The example shows that high epistemic uncertainty can be reduced to improve prediction confidence, while high aleatory uncertainty with low epistemic uncertainty is difficult to reduce and must be accounted for in decisions.Source: Figure 2 · p. 12, §2.2 · p. 13

2.3. Reduction of epistemic uncertainty

  • Parameter uncertainty can be reduced through more training data, physics-informed constraints, and improved training strategies.Clean added data is expected to lower epistemic uncertainty while leaving aleatory uncertainty similar; noisier added data can increase aleatory uncertainty where it is used. Physical constraints narrow feasible parameter space, while transfer learning and optimization strategies may indirectly reduce epistemic uncertainty.Source: §2.3 · pp. 13–14
  • Model-form uncertainty can be reduced by selecting more predictive input features, tuning architectures or kernels, and adding high-fidelity data.Adding high-fidelity data addresses bias-related model-form uncertainty.Source: §2.3 · p. 14
  • In the illustration, adding a relevant input feature reduces model-form uncertainty, and increasing the training set from 50 to 100 samples further reduces parameter uncertainty.The first model used only x1 although the underlying function depended on x1 and x2; the third model used 100 training samples.Source: §2.3 · p. 15, Figure 3 · p. 15

3. Methods for UQ of ML models

This section introduces methods for quantifying uncertainty in ML predictions to support safer decisions under scarce or noisy data. It covers Gaussian process regression and neural-network approaches—including Bayesian inference, MC dropout, ensembles, and deterministic methods centered on SNGP—with attention to uncertainty representations, behavior, limitations, and practical tradeoffs.

3. Methods for UQ of ML models

  • UQ can flag uncertain predictions for cautious review, and the tutorial introduces GPR, Bayesian neural networks, neural network ensembles, and deterministic methods focused on SNGP.The motivation includes scarce and noisy training data and high-stakes decision-making.Source: §3 · p. 16

3.1.1. Basics of Gaussian process regression

  • GPR places a Gaussian process prior on an unknown function, with a kernel encoding dependence; squared exponential covariance declines with input distance, yielding high uncertainty for OOD inputs.ARD assigns separate length scales to input dimensions, with a large learned length scale indicating an input dimension contributes minimally.Source: §3.1.1 · pp. 18, 19, 23
  • GPR prediction requires covariance-matrix inversion, with O(N^3) computation and O(N^2) storage; large datasets and ill-conditioned matrices can require approximations or regularization.The cited remedies include covariance tapering, low-rank approximations, pseudoinversion, and nugget regularization.Source: §3.1.1 · pp. 22–23

3.1.2. UQ capability and some limitations of Gaussian process regression

  • GPR separates aleatory noise from epistemic uncertainty and raises epistemic uncertainty farther from training data, supporting OOD detection, though reliable estimates depend on the test point, function, kernel, and hyperparameters.The posterior variance represents total predictive uncertainty; the noise variance measures aleatory uncertainty, while distance-based covariance drives epistemic uncertainty.Source: §3.1.2 · p. 24
  • GPR can flag distant OOD inputs, but its variance may saturate while extrapolation error grows, and standard training scales poorly with large N and high D.High-D scaling challenges include the many hyperparameters and costly covariance-matrix calculations; proposed approaches include dimensionality reduction, smaller-parameter kernels, and additive kernels.Source: §3.1.2 · pp. 24–25

3.2. Bayesian neural network

  • BNNs treat weights as random variables, update a prior to a data-conditioned posterior, and propagate parameter uncertainty; predictive uncertainty additionally includes observation noise.The posterior-pushforward describes uncertainty in the clean prediction, whereas the posterior-predictive also includes aleatory data uncertainty.Source: §3.2 · pp. 26–27

3.2.1. Markov chain Monte Carlo

  • MCMC samples the posterior with asymptotic convergence guarantees, but poor mixing and dimensional limits make it rarely practical for modern DNNs.Even HMC has been exercised only for parameter vectors of hundreds of dimensions, far below the millions or billions common in modern DNNs.Source: §3.2.1 · p. 27

3.2.2. Variational Inference

  • Variational inference fits a parameterized posterior approximation by minimizing its KL divergence from the target posterior.Mean-field Gaussian VI uses diagonal covariance and cannot capture parameter correlations. Figure 6 contrasts it with MCMC and SVGD, which capture correlated, non-Gaussian structure; SVGD and VI scale better than MCMC, though SVGD is limited by memory requirements.Source: §3.2.2 · pp. 28–29, Figure 6 · p. 29

3.2.3. MC dropout

  • MC dropout estimates uncertainty from repeated predictions of randomized sparse networks, offering easy, low-cost, architecture-agnostic implementation but hyperparameter-sensitive estimates and difficulty detecting OOD cases.It approximates a Bayesian posterior under a specific Bernoulli-mixture variational setup, but does not let users specify priors or likelihoods.Source: §3.2.3 · pp. 30–32

3.3. Neural network ensemble

  • Neural-network ensembles aggregate independently trained same-architecture models, relying on member diversity and often assigning greater uncertainty to OOD inputs.Diversity can come from randomization, such as bootstrap samples, or boosting, which increases the importance of previously mispredicted training instances.Source: §3.3 · p. 32

3.3.1. Aleatory uncertainty: training each network individually

  • A Gaussian output layer predicts each network’s mean and variance to model aleatory observational noise; the simplest homoscedastic assumption assigns equal noise variance across inputs.The observation model represents targets as shifted from the true value by a random noise term, often modeled as zero-mean Gaussian noise.Source: §3.3.1 · pp. 32–33
  • Neural networks can model input-dependent aleatory uncertainty by predicting both target mean and variance, with parameters optimized using negative log-likelihood.A constant noise variance is simpler but may be unrealistic when measurement noise varies across the input space.Source: §3.3.1 · p. 33

3.3.2. Epistemic uncertainty: using an ensemble of independently trained networks

  • An ensemble captures epistemic uncertainty through variation among independently trained networks, while each network’s predicted variance represents aleatory uncertainty.Random initialization and mini-batch sampling yield different local optima; disagreement may be high outside the training distribution, where predictions warrant caution.Source: §3.3.2 · pp. 34–35

3.4.1. Feature collapse and hidden-space regularization

  • Deterministic UQ uses one forward pass to produce probabilistic outputs efficiently, and distance-aware hidden representations help prevent OOD points from collapsing onto in-distribution features.Bi-Lipschitz constraints preserve input-space distances in hidden space: a lower bound counters feature collapse, while an upper bound promotes smoothness.Source: §3.4 · p. 35, §3.4.1 · p. 36

3.4.2. Spectral normalization for distance preservation in hidden space

  • SNGP uses residual connections to enforce a Lipschitz lower bound and spectral normalization to impose an upper bound, but tuning γ trades expressiveness against distance awareness.Small γ can restrict nonlinear learning, while large γ may fail to preserve input distances.Source: §3.4.2 · p. 37

3.4.3. Gaussian process regression output layer for distance-aware prediction

  • SNGP replaces the neural network output layer with GPR on distance-preserving hidden features, so posterior variance rises as a test input moves away from training data.For tractable inference, SNGP uses random Fourier features and a Laplace approximation; DUE instead uses variational inducing points.Source: §3.4.3 · pp. 37–38

3.4.4. Discussion on deterministic UQ methods

  • Deterministic methods are computationally attractive and support OOD detection, but generally cannot separate uncertainty types and may need architectural changes, extra hyperparameters, and calibration improvements.GPR, Bayesian neural networks, and ensembles can separate aleatory and epistemic uncertainty; GPR’s high-dimensional cost and sampling-based inference can hinder scalability and real-time use.Source: §3.4.4 · p. 38, §3.6 · p. 42

3.5. Toy example

  • GPR and SNGP assign high uncertainty to distant OOD samples, while MC dropout, deep ensembles, and DNN-GPR often assign them low uncertainty.MFVI increases uncertainty with distance but assigns nearly identical uncertainty between the training clusters, a deficiency attributed to its Bayesian-inference approximation.Source: §3.5 · pp. 40–41

3.6. Summary

  • GPR and SNGP generally outperform alternatives in calibration; SNGP matches GPR on OOD detection while offering scalability that GPR lacks.The tutorial’s qualitative comparison also notes that GPR and SNGP estimate uncertainty analytically, whereas other methods use Monte Carlo samples.Source: Table 2 · p. 41, §3.6 · pp. 41–42

4. Evaluation of predictive uncertainty

This section describes metrics and diagnostics—including calibration, sparsification, and log-likelihood—for evaluating predictive uncertainty alongside prediction accuracy and its relationship to prediction errors.

4.1. Calibration curves and metrics

  • Calibration curves (reliability diagrams) compare expected with observed confidence; perfect calibration follows the diagonal, while deviations indicate underconfidence or overconfidence.For regression, observed confidence is the fraction of intervals containing outcomes; for safety-critical maintenance, underconfidence is often preferred because overconfidence can prompt actions too early or too late.Source: §4.1 · p. 42, §4.1.1 · pp. 43–44

4.1.1. Calibration curves for regression

  • At 90% confidence, toy GPR coverage was 95% (underconfident), but more OOD samples degraded calibration and increased overconfidence as uncertainty saturated while errors grew.The authors stress that uncertainty may fail to keep pace with prediction error far from the training distribution, making input-space trust boundaries important.Source: §4.1.1 · pp. 45–47, Figure 11 · p. 46

4.1.2. Calibration curves for classification

  • For binary classification, calibration bins confidence and plots predicted against observed class-1 confidence for each bin.The described procedure uses bins of width 1/K before comparing predicted and observed confidence.Source: §4.1.2 · p. 47

4.1.3. Calibration metrics

  • 0.043 was the toy example’s ECE with equal weights; calibration metrics also include miscalibration area, the area between the curve and identity line.The ECE is a weighted average difference between a calibration curve and the ideal line.Source: §4.1.3 · pp. 47–48, Figure 12 · p. 48

4.1.4. Recalibration

  • Temperature scaling was reported the simplest and most effective recalibration approach, while Platt scaling and isotonic regression typically require validation data equal to 20–50% of training data.Platt scaling optimizes two parameters using validation-set NLL; temperature scaling is its single-parameter version, and isotonic regression maps probabilistic predictions to empirical values [186, 187].Source: §4.1.4 · pp. 48–49

4.1.5. Connecting UQ calibration with model validation

  • U-pooling tests whether observations come from their predicted distributions, whereas UQ calibration assesses underconfidence or overconfidence; both can measure predictive-distribution mismatch globally.U-pooling compares the empirical CDF of pooled predictive CDF values with the standard uniform CDF.Source: §4.1.5 · pp. 49–50

4.2. Sparsification plots and metrics

  • Sparsification plots test whether uncertainty ranks prediction errors: error should fall as high-uncertainty samples are removed, while random uncertainty yields a nearly flat curve.The toy GPR curve was close to the oracle error curve, whereas a random-uncertainty curve was nearly flat.Source: §4.2 · pp. 50–51, Figure 13 · p. 51
  • AUSE measures the area between the actual sparsification error curve and its oracle, while higher AURG favors UQ over no UQ when error decreases monotonically.Lower AUSE indicates better representation of prediction error; AURG measures the area between the random curve and the actual error curve.Source: §4.2 · pp. 51–52

4.3. Negative log-likelihood

  • Predictive log-likelihood measures probabilistic model-data fit, with larger values indicating better fit; its negative counterpart, NLL, is commonly used as a loss.NLL can also serve as an indirect calibration measure and is often assessed alongside calibration metrics.Source: §4.3 · p. 52

4.4. Accuracy vs. UQ quality

  • NLL and accuracy can conflict during training: neural networks may gain accuracy while uncertainty quality falls, so evaluation should examine both.The passage notes that overfitting may appear in probabilistic error metrics such as NLL without appearing in classification error or regression errors based on mean predictions; high accuracy may still leave difficulty extrapolating to OOD samples.Source: §4.4 · p. 52

5. UQ of ML models in prognostics

This section explains why prognostics needs reliable uncertainty estimates alongside RUL predictions, reviews uncertainty-aware deep learning and prognostics-specific evaluation metrics, and identifies priorities for trustworthy deployment.

5.1.1. Prognostics and the role of UQ

  • In PHM, prognostics predicts remaining useful life (RUL), and reliable uncertainty estimates are essential to schedule maintenance safely and cost-effectively.Just-in-time maintenance can reduce downtime and costs, while overpredicted RUL risks safety failures and underpredicted RUL can shorten asset life through premature replacement.Source: §5.1.1 · p. 53

5.1.2. The potential of DL for PHM

  • Deep-learning prognostics either maps sensor data directly to RUL or forecasts health indicators to a threshold; both approaches need uncertainty estimates for replacement decisions.Because representative run-to-failure data are difficult to collect when failures are rare, epistemic uncertainty can substantially affect model performance.Source: §5.1.2 · p. 54

5.1.3. Uncertainty-aware DL in prognostics

  • Although point-estimating ML and DL prognostics are widespread, most developed methods do not quantify prediction uncertainty, limiting practical deployment in mission- and safety-critical settings.Data scarcity, model-form uncertainty, and measurement or communication errors affect predictions; UQ can support maintenance planning, user trust, and safety.Source: §5.1.3 · pp. 54–55

5.2. Uncertainty evaluation metrics for prognostics

  • Standard metrics such as NLL, MSE, RMSE, and MAPE are used, but prognostics metrics must account for predictions becoming more certain as end of life approaches.MSE and MAPE do not capture the statistical distribution of RUL predictions; probabilistic metrics also reflect that prediction is harder early in operation and simplifies as fault symptoms become more pronounced.Source: §5.2 · p. 55
  • Saxena et al. introduce Prognostic Horizon, α-λ performance, Relative Accuracy, and Convergence metrics that can also assess DL-based UQ approaches.For probabilistic predictions, Prognostic Horizon requires the probability of falling within a ground-truth confidence region to exceed a prespecified threshold; α-λ performance is binary with bounds that shrink toward end of life. Relative Accuracy is one minus relative error using a central tendency estimate, while Convergence measures how quickly the metrics improve over time.Source: §5.2 · pp. 55–56, Figure 14 · p. 56

5.3. Discussion

  • MC dropout is widely used for neural-network UQ because it is intuitive and needs little architectural change, but its performance is not always satisfactory [218].The tutorial calls for exploring more advanced UQ and Bayesian deep-learning methods and identifies anomaly detection as another application, where high epistemic uncertainty may flag health states unlike nominal training data.Source: §5.3 · pp. 56–57
  • A crucial requirement for prognostics UQ is to disentangle aleatory and epistemic uncertainty, which convey distinct information and must be interpreted separately.Meaningful uncertainty estimates are essential to trustworthy, safe deployment because industrial decisions balance risk against potential economic benefits.Source: §5.3 · pp. 56–57

6. Case studies for benchmarking – Code Sharing on GitHub

The section benchmarks uncertainty-quantification methods on early lithium-ion battery life prediction and turbofan-engine remaining useful life (RUL), using comparable model backbones where applicable. It provides implementation guidance and compares predictive accuracy and uncertainty quality in two safety-critical applications.

6. Case studies for benchmarking – Code Sharing on GitHub

  • The benchmarking is pedagogical rather than a claim of state-of-the-art performance; results could improve with individual hyperparameter optimization, and the code is shared on GitHub.The battery code walk-through limits its neural-network implementation to TensorFlow and Keras; the studies aim to compare methods with similar prediction accuracy.Source: §6 · p. 57, §6.2 · p. 67, §6.2.2 · p. 70

6.1.1. Dataset overview

  • The battery study predicts remaining cycle life from the first 100 cycles of 169 LFP/graphite cells, a challenging early period when most cells show little degradation.The dataset combines 124 Severson cells and 45 Attia cells, with separate training, primary, secondary, and tertiary test subsets; end of life is defined at 80% of nominal capacity.Source: §6.1 · p. 57, §6.1.1 · p. 58

6.1.2. Neural Network Ensemble

  • The battery comparison includes NNE, MC dropout, GPR, and SNGP; NNE predicts a mean and variance with a Gaussian output layer, while MC dropout samples predictions with dropout active.The neural models use a comparable ResNet backbone; NNE and MC dropout each use an ensemble size of 15, selected using an elbow-method study.Source: §6.1 · p. 58, §6.1.2 · p. 59, §6.1.3 · p. 61

6.1.4. Spectral Normalization Gaussian Process (SNGP)

  • SNGP combines spectral normalization in hidden layers with a final Gaussian process layer to preserve distance awareness for uncertainty estimation; the battery GPR uses an RBF kernel with grid-searched hyperparameters.The SNGP implementation uses a ReLU activation and a spectral norm bound of 0.9.Source: §6.1.4 · p. 62, §6.1.5 · p. 62

6.1.6. Evaluation/Results

  • Battery results show NNE slightly more accurate than MC dropout and substantially better at uncertainty quantification, while MC dropout tends to be overconfident and SNGP often overestimates uncertainty.The authors attribute NNE’s uncertainty performance partly to each ensemble member representing aleatory uncertainty.Source: Table 5 · p. 64, §6.1.6 · p. 64
  • Battery predictions align well on training data but miss some test cells, especially the distinct tertiary dataset; calibration results show MC dropout overconfidence and NNE and SNGP underconfidence.NNE and SNGP assign large uncertainty to most tertiary-dataset predictions; underconfidence is described as desirable for safety-critical battery prediction.Source: §6.1.6 · pp. 65–66, Figure 19 · p. 65, Figure 20 · p. 66

6.2.1. Dataset overview

  • The turbofan study uses N-CMAPSS DS02 data from nine engines, with 20-dimensional inputs combining measured properties and operating-condition descriptors; GPR is omitted because of dataset scale.Six engine trajectories are used for training and three for testing, and the data are downsampled by a factor of 500.Source: §6.2 · p. 67, §6.2.1 · p. 68, §6.2.2 · p. 68

6.2.2. Evaluation/Results

  • NNE outperforms MC dropout and SNGP across the reported N-CMAPSS metrics; all capture declining RUL but struggle at degradation onset, where NNE assigns higher uncertainty.The study evaluates nine engines using three test units; the reported comparison uses RMSE, NLL, and ECE.Source: Table 7 · p. 69, §6.2.2 · p. 69

7. Other topics related to UQ of ML models

The section surveys complementary approaches that combine physics, probabilistic learning, interpretable dynamics, and polynomial chaos with uncertainty quantification. It describes their mechanisms, strengths, and limitations for engineering and dynamic-system problems.

7.1. Physics-informed ML and its synergy with UQ and probabilistic ML

  • Physics-informed ML combines physical models or knowledge with data-driven models in seven broad ways, while UQ addresses residual physics and model uncertainty.Examples include physics-informed loss constraints, simulation-data augmentation, transfer learning, hybrid models, model enhancement, physics-model parameter prediction, and physics-specific architectures. Uncertainty remains because physics knowledge is imperfect; MC dropout and BNNs address neural-network uncertainty, while physics-constrained BNNs target uncertainty in PINNs.Source: §7.1 · pp. 70–71

7.2. Probabilistic Learning on Manifolds (PLoM)

  • PLoM generates samples on a learned low-dimensional manifold, capturing data uncertainty while reducing dimension, unlike GPR and BNN’s primary focus on function approximation.PLoM uses PCA, density estimation, an Itô stochastic differential equation, and a diffusion-maps basis to generate samples. It generally needs enough data to reveal distribution geometry, but governing PDE constraints can help in relatively small-data settings; applications include statistical density estimation, optimization under uncertainty, and digital-twin design.Source: §7.2 · pp. 71–72

7.3. Interpretability of ML models for dynamic systems

  • SINDy identifies parsimonious, interpretable governing equations through sparse regression, while UQ-SINDy quantifies coefficient uncertainty from observation errors and limited data.Sparsity-promoting methods include LASSO, sparsifying priors, and hard thresholding with Pareto analysis; sparse representations can generalize to unseen data and support model-selection uncertainty in hybrid systems. UQ-SINDy uses spike-and-slab or regularized horseshoe priors in Bayesian inference.Source: §7.3 · pp. 72–73

7.4. PCE and its relationship with GPR and connection with ML

  • PCE provides a distribution-aware surrogate for expensive simulations, but its parameter and sample requirements can grow exponentially with input dimension.Sparse truncation methods, including Smolyak constructions and compressive-sensing variants, mitigate this curse. PCE can also define the mean trend in PCE-kriging hybrids, combine with other ML techniques, and has been reported to achieve surrogate accuracy on par with several state-of-the-art ML methods.Source: §7.4 · pp. 73–74

8. Conclusion and outlook

The tutorial surveys UQ methods and applications in engineering design and PHM, framing UQ as a safety-assurance layer for high-stakes ML. It identifies research needs spanning benchmarks, scalable methods, engineering design, PHM, and label-free learning.

Conclusion

  • The tutorial classifies ML uncertainty, explains UQ methods and evaluation metrics, and supplies code and two case studies on battery life and turbofan RUL; their hyperparameters were not optimized.The case studies are presented as a user-friendly platform for understanding the methods and encouraging further improvement and new methods.Source: §8 · p. 74

Future research directions

  • A unified, community-acknowledged UQ testbed should assess calibration, decomposition, attribution, interpretation, and behavior on edge cases, beyond conventional prediction and calibration metrics [289].Synthetic data generation could support comprehensive testing and verification across diverse UQ methods.Source: §8 · p. 75
  • UQ methods need stronger theoretical guarantees, scalability, and computational efficiency; SNGP’s distance awareness supports OOD detection, while real-time safety-critical use remains a challenge.The tutorial notes that existing methods commonly lack or have limited theoretical guarantees for detecting OOD instances.Source: §8 · pp. 75–76
  • In engineering design, UQ could enable adaptive DNNs and data-efficient active learning, accelerate large-scale optimization, and account for aleatory and epistemic uncertainty, but DNN surrogates’ potential remains unrealized.The paper also identifies reliability analysis and optimization as applications where active DNNs could reduce training-data needs without sacrificing surrogate accuracy.Source: §8 · p. 76
  • PHM research should develop uncertainty decomposition, routinely evaluate UQ quantitatively, and investigate how UQ and interpretation can jointly improve transparency and trustworthiness.Decomposition may distinguish measurement-noise reduction from needs for more data or model refinement; suggested evaluation metrics include calibration, sparsification, and NLL.Source: §8 · pp. 76–77
  • Label-free learning needs generic UQ methods because missing labels make prediction errors unknown; a proof-of-concept method combines neural networks and GPR but is not extended to time- and space-dependent problems.PINNs are one label-free approach, and the cited method represents systematic error with a mean and random error with a standard deviation.Source: §8 · p. 77

Authors’ contributions · Appendix A. Some further discussions on Gaussian process regression · Appendix A.1. An extended discussion on kernels

The appendix explains how Matérn-kernel smoothness varies with ν, how its special cases span rough to smooth functions, and how an ARD variant models input dimensions separately.

Appendix A.1. An extended discussion on kernels

  • The Matérn family generalizes the squared-exponential kernel, with smoothness parameter ν controlling the smoothness of the resulting function approximation.The appendix also identifies Euclidean distance, the Gamma function, and a modified Bessel function in the covariance definition.Source: Appendix A.1 · p. 112
  • As ν → ∞, the Matérn kernel becomes squared exponential; at ν = 1/2, it equals the absolute-exponential, or Ornstein–Uhlenbeck, kernel [105].Matérn 1/2 yields rough prior and posterior functions, and observations may poorly inform predictions far from observed inputs, impairing generalization.Source: Appendix A.1 · p. 112
  • Matérn 3/2 and 5/2 kernels are respectively once and twice differentiable, offering intermediate smoothness for functions that are smooth but not perfectly smooth.Unlike the squared-exponential kernel, neither is infinitely differentiable.Source: Appendix A.1 · p. 112
  • An isotropic Matérn kernel uses one length scale, while its ARD variant uses D length scales to represent the relevance of each input dimension.For D-dimensional input, the anisotropic kernel has D + 1 hyperparameters: σf and l1 through lD.Source: Appendix A.1 · p. 112
  • A one-dimensional GPR illustration compares kernels using the same eight training observations and five posterior samples, showing squared exponential as smoothest and Matérn 1/2 as roughest.The example uses a sinusoidal noiseless ground truth with Gaussian observation noise; Matérn 1/2 posterior samples correspond to Brownian motion, while the ν → ∞ posterior mean is infinitely differentiable.Source: Appendix A.1 · pp. 112–113, Figure A.24 · p. 113

Appendix A.2. Parametric study on effect of hyperparameter optimization

The appendix illustrates how Gaussian process hyperparameters shape posterior uncertainty in a 1D toy regression example. It shows that observation and function posteriors share a mean, while observation variance is larger because of nonzero noise.

Parametric study on effect of hyperparameter optimization

  • The observation posterior has the same mean as the function posterior but slightly greater variance at each input because of nonzero noise.The figure compares four settings of l, σf, and σε and reports their values and log marginal likelihoods.Source: Appendix A.2 · p. 113
  • A too-small length scale, such as l = 0.1, makes the approximation vary too quickly and widens uncertainty between training points.The length scale controls how quickly correlation between function values decays with distance between input points.Source: Appendix A.2 · p. 113
  • A larger signal amplitude, such as σf = 3, increases the maximum confidence-interval width, while its optimum is difficult to derive from training data alone.Signal amplitude describes the maximum vertical variation of functions or observations drawn from the Gaussian process and is important for quantifying epistemic uncertainty.Source: Appendix A.2 · p. 113
  • A too-small noise standard deviation, such as σε = 0.05, fails to capture observational noise, or aleatory uncertainty.σε controls the amount of input-independent noise in observations.Source: Appendix A.2 · p. 113

Appendix A.3. Connections with neural networks and recent development · Appendix B. UQ of ML models in engineering design

The appendix connects Gaussian process regression (GPR) with neural networks through infinite-width limits and learned feature representations, and describes deep Gaussian processes as a more flexible but computationally costly alternative. These connections combine GPR’s Bayesian uncertainty quantification with neural networks’ capacity to learn nonlinear, high-dimensional representations.

Appendix A.3. Connections with neural networks and recent development

  • An infinitely wide, single-hidden-layer neural network with i.i.d. parameter priors is equivalent to a Gaussian process, enabling exact Bayesian inference through matrix operations.This avoids iterative training algorithms such as gradient descent and approximate Bayesian inference methods for Bayesian neural networks.Source: Appendix A.3 · pp. 114–115
  • Connections between GPR and neural networks have been extended from standard DNNs to convolutional neural networks.This research line builds on early work establishing the infinite-width, single-hidden-layer equivalence.Source: Appendix A.3 · pp. 114–115
  • As hidden-layer width approaches infinity, the network outputs at any finite set of inputs become jointly Gaussian by the multidimensional Central Limit Theorem, defining a Gaussian process.The derivation assumes i.i.d. hidden-to-output weights and bias with zero means; the resulting GP has zero mean.Source: Appendix A.3 · p. 116
  • Infinite-width Bayesian neural networks yield new GPR kernels, and for some activation functions their covariance has an analytic form.With properly scaled initialization, kernel parameters such as prior variances may be the only parameters requiring optimization.Source: Appendix A.3 · p. 116
  • Deep kernel learning uses GPR on neural-network-learned hidden features, jointly optimizing network and kernel parameters end-to-end.These approaches combine GPR distance awareness with neural feature learning to support OOD detection and scalability to high-dimensional inputs.Source: Appendix A.3 · p. 116
  • Deep kernel learning can overfit because of DNN overparameterization, while feature collapse must be addressed to preserve input distances in hidden space.The appendix points to spectral-normalized neural Gaussian process (SNGP) as a representative approach addressing this issue.Source: Appendix A.3 · pp. 116–117
  • Deep Gaussian processes stack GPs to learn complex mappings for small or large datasets, but exact Bayesian inference can be prohibitively expensive.Most approaches therefore use inference techniques based on a small set of inducing points to construct covariance matrices.Source: Appendix A.3 · p. 117

Appendix B.1. Needs of ML models in engineering design

ML supports engineering design by relaxing conventional assumptions, shortening design cycles, and addressing complex systems, especially when simulations are costly. Its design capabilities center on feature extraction, surrogate modeling, and optimization.

Needs of ML models in engineering design

  • ML-enabled engineering design is organized around feature extraction, surrogate modeling, and optimization, with applications spanning materials, reliability, energy systems, and topology.Figure B.27 summarizes these three capability categories and their applications.Source: Appendix B.1 · p. 117, Figure B.27 · p. 118

Feature extraction

  • Deep learning extracts representations from high-dimensional data for design-problem dimension reduction and candidate generation; autoencoders learn compressed codes, while VAEs and GANs support generative design.Generative design learns from existing designs and produces candidates subject to requirements such as performance targets and cost constraints.Source: Appendix B.1 · pp. 117–118

Surrogate modeling

  • Global surrogates target accuracy across the full design domain, requiring domain-wide training data, and can support varied tasks once validated across that domain.Sampling can be denser in nonlinear regions and sparser in smoother ones; representative test data enables UQ calibration metrics to assess accuracy.Source: Appendix B.1 · p. 119
  • Local surrogates concentrate training in purpose-specific regions, such as limit-state boundaries or optima, and represent the simulation accurately only within those regions.Learning or acquisition functions adaptively identify the relevant regions, and accuracy metrics should be tailored to the surrogate’s purpose.Source: Appendix B.1 · pp. 119–120

Optimization

  • Gradient-free ML-based optimization, including Bayesian optimization, addresses costly simulation evaluations that limit conventional optimizers; deep learning can also generate candidate designs directly.Gradient-based methods can struggle to find global optima, while evolutionary approaches may require prohibitively many evaluations of high-fidelity simulations.Source: Appendix B.1 · p. 120

Appendix B.2. Role of UQ of ML models in engineering design

UQ of ML models underpins feature extraction, surrogate modeling, and optimization in engineering design. It supports reliable representation and efficient selection of informative training data and designs.

Appendix B.2. Role of UQ of ML models in engineering design

  • Predictive uncertainty helps ensure extracted features represent original data, address ill-posed inverse problems in generative design, and account for input-feature variability.These are three roles of UQ in ML-enabled feature extraction.Source: Appendix B.2 · p. 121
  • Surrogate refinement uses predictive uncertainty to select informative training points and reduce costly high-fidelity simulation data collection; global refinement often maximizes prediction variance.GPR provides predictive variance directly, while other surrogate models require a separate UQ method.Source: Appendix B.2 · p. 121
  • In local surrogate modeling, UQ underpins learning functions that identify critical input regions and guide iterative refinement, including EFF and the U function.These functions use predictive quantities such as the mean and standard deviation relative to a failure threshold to assess candidate inputs.Source: Appendix B.2 · pp. 121–122
  • Bayesian optimization balances exploration and exploitation through acquisition functions such as probability of improvement, EI, upper confidence bound, and knowledge gradient.For EI, the predictive mean, standard deviation, and current best observed value are needed to define the acquisition function.Source: Appendix B.2 · p. 122
  • Global and local surrogate models can be converted between during construction, and UQ helps identify critical regions when moving from a global to a local model.UQ remains foundational to the paper’s three categories of ML-enabled engineering-design capabilities, whether optimization uses local or global surrogates.Source: Appendix B.2 · p. 123

Appendix B.3. State of knowledge and gaps … Appendix C.3. DL for prognostics

UQ methods offer tools for engineering design across feature extraction, surrogate modeling, and optimization, while prognostics must address varied sources of uncertainty and the limitations of deterministic deep-learning predictions.

Appendix B.3. State of knowledge and gaps

  • Feature extraction studies mostly use neural-network approaches such as autoencoders and GANs, yet few investigate UQ for these models.The reviewed applications include latent representations and generative design; UQ could support dimension reduction or generative design under uncertainty.Source: Appendix B.3 · p. 123
  • GPR dominates local surrogate modeling because its Gaussian predictive uncertainty supports efficient learning-function evaluation, while DNN active learning remains early-stage and less scientifically rigorous.Existing deep-learning UQ methods often lack rigorous calibration, decomposition, and attribution tests, and few provide principled ways to reduce predictive uncertainty.Source: Appendix B.3 · p. 124
  • ML-enabled engineering optimization has mostly used GPR-based Bayesian optimization, with extensions for multifidelity, multi-output, gradient-informed, mixed-variable, heteroscedastic, and non-stationary problems.Applications include material design, reliability design, and additive manufacturing.Source: Appendix B.3 · p. 124
  • UQ for engineering design can extend local surrogate modeling and optimization beyond GPR, but methods also need to account for input-dependent aleatory uncertainty and reduce computational cost.Repeated uncertainty quantification runs can add substantial overhead; the reviewed neural-network UQ methods may help address this efficiency need.Source: Appendix B.3 · p. 125

Appendix C.1. Introduction to prognostics

  • Data-driven RUL prediction either forecasts a health indicator until a threshold is reached or directly maps extracted features or raw measurements to RUL.Prognostics predicts future health from current and past condition and future usage or operating profiles; absent future-profile information, past operation is often assumed to continue.Source: Appendix C.1 · pp. 125–126

Appendix C.2. Sources of uncertainty in prognostics

  • Prognostics uncertainty includes measurement and model uncertainty as well as uncertain future usage, limited representative failure trajectories, variable operating conditions, environmental effects, and incompletely understood failure mechanisms.Source: Appendix C.2 · p. 126

Appendix C.3. DL for prognostics

  • Deep learning can process high-dimensional, heterogeneous, noisy sensor data end-to-end and learn features automatically, but most prognostics methods provide only deterministic point estimates of RUL.Limited training data and sensor noise can create high epistemic uncertainty that must be quantified and communicated to decision makers.Source: Appendix C.3 · pp. 126–127

Appendix C.4. State-of-the-art uncertainty-aware DL approaches for prognostics

Standard deep-learning models in prognostics typically provide point estimates without uncertainty quantification; recent work has developed uncertainty-aware approaches, particularly MC dropout, ensembles, and deep Gaussian processes.

Appendix C.4. State-of-the-art uncertainty-aware DL approaches for prognostics

  • MC dropout is the predominant uncertainty-aware prognostics strategy, used with fully connected networks, CNNs, and RNNs.Reported applications span lithium-ion batteries, turbofan engines, bearings, solenoid valves, hydraulic mechanisms, and circuit breakers.Source: Appendix C.4 · p. 127
  • An adapted MC dropout framework models epistemic and aleatory uncertainty using a final aleatory layer parameterizing a Gaussian or two-parameter Weibull distribution.Sampling from both the dropout weight distribution and the output distribution enables the two uncertainty sources to be disentangled.Source: Appendix C.4 · p. 127
  • ESN ensembles estimate RUL and prediction intervals [385], while deep Gaussian processes predict turbofan RUL and outperform several deep-learning baselines.Deep Gaussian processes combine GPR's probabilistic nature with DNN expressivity and suit prognostics' big-data regime.Source: Appendix C.4 · p. 127

Appendix D. Demonstration of Instability of MC Dropout

The appendix demonstrates that MC dropout uncertainty maps are highly inconsistent across model configurations, despite similar training behavior in the illustrated models. It examines sensitivity to dropout rate, training epochs, and ResNet architecture.

Appendix D. Demonstration of Instability of MC Dropout

  • MC dropout uncertainty maps are highly inconsistent across dropout rates, training epochs, and ResNet architectures.Table D.8 compares these configurations.Source: Appendix D · p. 128, Table D.8 · p. 129
  • The two same-architecture MC dropout models converged around 500 epochs, with no overfitting observed through 10000 epochs.They used the toy-example data and differed in dropout rate.Source: Appendix D · p. 128
  • The appendix examines instability from even slight changes in model size, training epochs, and dropout rate.The authors identify these hyperparameter variations as a source of MC dropout model instability.Source: Appendix D · p. 128

Related papers

Loading 2305.04933v2…