PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial

Venkat Nemani, Luca Biggio, Xun Huan, Zhen Hu, Olga Fink, Anh Tran, Yan Wang, Xiaoge Zhang, Chao Hu

arXiv:2305.04933v2cs.LGcs.AI

TL;DR

ML models can generalize poorly to out-of-distribution data, while high-stakes applications require predictions whose uncertainty supports safety assurance. This tutorial classifies uncertainty, explains and compares neural-network UQ methods, reviews evaluation metrics and applications, and provides code-based case studies. It reports that SNGP achieves OOD-detection performance similar to GPR while offering better scalability, whereas GPR and SNGP generally provide stronger calibration than alternatives.

  • Problem

    Neural networks can produce large prediction errors on out-of-distribution samples, motivating reliable predictive uncertainty for high-stakes ML decisions.

  • Method

    The tutorial classifies uncertainty and reviews, compares, evaluates, and implements emerging UQ methods for neural networks in engineering design and health prognostics.

  • Results

    SNGP achieves similar OOD-detection performance to GPR, while GPR and SNGP generally outperform alternative UQ methods in calibration accuracy.

  • Takeaways & Limitations

    UQ provides a safeguard for detecting OOD samples and supports confidence-aware decisions in engineering and health-prognostics applications.

  • Takeaways & Limitations

    GPR can be unreliable under extreme extrapolation and lacks scalability in high-dimensional input spaces.

Abstract

from arXiv · show

On top of machine learning models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Toward this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. Two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage and the remaining useful life prediction of turbofan engines.

1. Introduction

The tutorial motivates UQ as a safety-assurance layer for ML models whose strong fixed-test accuracy does not guarantee reliable generalization, especially in engineering design and health prognostics. It surveys neural-network UQ methods, evaluation metrics, applications, and implementation resources.

  • 1. Introduction: Neural networks can produce large errors on out-of-distribution samples despite high accuracy on fixed test sets.This generalization problem persists alongside the black-box nature of complex ML models.
  • 1. Introduction: UQ helps users judge when model predictions are trustworthy and when decisions require extra caution.The motivation is strongest when incorrect decisions can cause severe financial losses or life-threatening outcomes.
  • 1. Introduction: The tutorial focuses on quantifying predictive uncertainty in neural networks, distinguishing UQ of ML models from using ML to perform UQ.It presents a comprehensive overview of emerging approaches and applications in engineering design and health prognostics.
  • 1. Introduction: It classifies uncertainty types, sources, and causes; explains epistemic-uncertainty reduction and uncertainty decomposition; and compares emerging UQ approaches quantitatively and qualitatively.The tutorial also covers methods for assessing predictive-uncertainty quality and implementations on case studies.
  • 1. Introduction: Applications are reviewed in engineering design and health prognostics, where increasing ML adoption creates a corresponding need for UQ.The paper includes GitHub code for a toy regression example and two real-world health-prognostics case studies.

2. Types and sources of uncertainty

The paper distinguishes aleatory uncertainty, which is irreducible, from epistemic uncertainty, which is reducible, and explains how their decomposition supports uncertainty-aware decisions. It also identifies practical limits of separating and reducing these components.

  • 2.1. Aleatory and epistemic uncertainty: Aleatory uncertainty arises from natural variability or inherent system noise and is irreducible by collecting additional data.Examples include sensor noise, material-property variability, and stochastic input-output behavior; it is commonly modeled through the likelihood.
  • 2.1. Aleatory and epistemic uncertainty: Epistemic uncertainty reflects knowledge gaps and is reducible in principle through improved models, parameters, data, or assumptions.Its ML sources include model-form choices and parameter uncertainty caused by limited or low-fidelity training data and optimization difficulties.
  • 2.1. Aleatory and epistemic uncertainty: Input, output, or joint aleatory uncertainty may occur, but this tutorial focuses on modeling output aleatory uncertainty during training rather than propagating input uncertainty.Output aleatory uncertainty is more challenging because it must be accounted for during model training.
  • 2.1. Aleatory and epistemic uncertainty: Total predictive uncertainty combines environmental noise with reducible uncertainty from insufficient training data, although clean separation of aleatory and epistemic components can be difficult.The two types often coexist, and complete separation may only be possible in some cases without complicated interactions.
  • 2.2. Decomposition of predictive uncertainty: Variance decomposition separates total predictive variance into aleatory and epistemic components, revealing whether uncertainty arises from noise or knowledge gaps.A linear-regression formulation derives epistemic uncertainty by subtracting aleatory uncertainty from total predictive uncertainty.
  • 2.2. Decomposition of predictive uncertainty: High epistemic uncertainty can motivate uncertainty reduction, whereas high aleatory uncertainty with low epistemic uncertainty calls for risk-based decision making.Aleatory uncertainty is difficult to reduce further, so decisions must account for its irreducible contribution.
  • 2.3. Reduction of epistemic uncertainty: Selecting input features with strong predictive power through domain knowledge, expert opinion, or exploratory analysis can reduce model-form uncertainty.This is presented as one approach to reducing epistemic uncertainty in ML models.

3. Methods for UQ of ML models

The tutorial presents UQ methods for ML models, emphasizing how uncertainty supports safer decisions and how methods differ in representation, scalability, and reliability. It covers GPR, Bayesian inference, MC dropout, ensembles, and related neural-network approaches, then compares their uncertainty behavior and limitations.

  • Gaussian process regression: GPR naturally produces high uncertainty for out-of-distribution samples outside the training distribution.This distance-awareness property supports OOD detection, although extreme extrapolation can still yield unreliable uncertainty magnitudes.
  • Gaussian process regression: GPR represents predictive uncertainty through posterior variance, capturing both aleatory noise and epistemic uncertainty.Aleatory uncertainty is associated with observation noise, while epistemic uncertainty depends mainly on covariance between training points and the query point.
  • Gaussian process regression: GPR uncertainty reliability depends on the test point, underlying function, kernel choice, and hyperparameter optimization.Posterior variance may saturate as inputs move far from the training distribution even while prediction error continues increasing.
  • Gaussian process regression: Standard GPR has O(N^3) training complexity and does not scale well to large datasets or high-dimensional inputs.High-dimensional models require more hyperparameters, larger covariance matrices, and higher-dimensional optimization.
  • Bayesian neural networks: Bayesian neural-network inference uses MCMC, SVGD, or variational inference, trading posterior flexibility against scalability and computational cost.MCMC and SVGD can capture non-Gaussian and correlated posterior structure, whereas mean-field VI provides a Gaussian approximation and scales better to higher parameter dimensions.
  • MC dropout: MC dropout offers a simple, architecture-agnostic approximation to Bayesian prediction but is sensitive to hyperparameters and can struggle with OOD detection.Its uncertainty depends on choices such as dropout rate, dropout layers, training epochs, and network size.

4. Evaluation of predictive uncertainty

Predictive uncertainty must be evaluated alongside prediction accuracy, using calibration, sparsification, and probabilistic metrics to assess whether reported confidence reflects actual errors. The tutorial also highlights that uncertainty quality can diverge from accuracy and may become unreliable farther outside the training distribution.

  • Evaluation principles: Probabilistic models require evaluation of both prediction accuracy and predictive-uncertainty quality.Regression accuracy can use RMSE or mean absolute error, while uncertainty quality concerns the output probability distribution.
  • Calibration curves: Calibration curves compare expected confidence with the empirical fraction of observations contained by predicted confidence intervals.For example, a nominal 90% interval should contain approximately 90% of observed outcomes.
  • Calibration curves: The toy Gaussian process regressor is underconfident at 90%, with 95% observed coverage, indicating higher-than-true predictive uncertainty.The tutorial notes that underconfidence can be preferable to overconfidence in safety-critical applications.
  • Out-of-distribution behavior: On increasingly out-of-distribution inputs, prediction error may grow while predictive uncertainty saturates, making uncertainty difficult to trust beyond established input-space boundaries.The tutorial identifies trustworthy UQ under such distribution shifts as an area needing substantially more work.
  • Recalibration: Calibration methods such as Platt scaling and isotonic regression require a separate validation set that may be unavailable in practice.The cited comparison found temperature scaling to be the simplest and most effective recalibration approach.
  • U-pooling: U-pooling evaluates whether observations arise from predicted distributions, whereas calibration evaluates underconfidence or overconfidence.U-pooling assesses the distribution of probability-integral-transform values against a standard uniform distribution.
  • Sparsification plots: Sparsification plots test whether removing high-uncertainty samples causes prediction error to decrease monotonically toward an oracle curve.A near-flat curve indicates that uncertainty provides little information for identifying samples with large prediction errors.
  • Accuracy versus UQ quality: Uncertainty quality can conflict with accuracy during training, because NLL may worsen while RMSE, mean absolute error, or classification accuracy improves.This divergence means point-prediction performance alone does not establish probabilistic quality.

5. UQ of ML models in prognostics

In prognostics and health management, UQ addresses uncertainty in data-driven predictions while supporting maintenance decisions for safety-critical assets. The tutorial reviews prognostic tasks, evaluation metrics, uncertainty-aware deep learning, and future directions beyond remaining useful life prediction.

  • Prognostics and PHM: Prognostics is the most challenging PHM task and predicts remaining useful life to support just-in-time maintenance.Effective prognostics can reduce maintenance costs and downtime while extending asset lifetime.
  • Deep-learning approaches: Deep-learning prognostics can directly map sensor data to RUL labels or forecast health evolution until a predefined end-of-life threshold.The direct-mapping approach requires datasets linking sensor readings with corresponding RUL targets.
  • Uncertainty-aware deep learning: Uncertainty quantification helps address the interpretability gap between black-box deep-learning models and trustworthy prognostic algorithms.Despite extensive point-estimate RUL research, uncertainty-aware models have not yet significantly affected the field.
  • Sources of uncertainty: Data-driven prognostic predictions are affected by model-form uncertainty, insufficient representative training data, and measurement or communication errors.Most existing ML and DL prognostic algorithms historically did not quantify these associated uncertainties.
  • Evaluation metrics: Standard metrics such as NLL, MSE, RMSE, and MAPE are complemented by prognostics-specific metrics because RUL predictions become more constrained near end of life.These specialized metrics address temporal aspects of prognostic prediction quality.
  • Evaluation metrics: The sequential prognostics metrics include Prognostic Horizon, α-λ performance, Relative Accuracy, and Convergence.The α-λ metric is binary and uses confidence bounds that shrink as the machine approaches end of life.
  • Uncertainty-aware deep learning: MC dropout is widely used because it is intuitive and requires minimal architectural modification, but its uncertainty performance is not always satisfactory.The tutorial identifies more advanced Bayesian and uncertainty-quantification approaches as an important research direction.
  • Future directions: Future prognostics applications could use uncertainty for anomaly detection in addition to RUL prediction.The proposed setting evaluates model confidence at each time step to identify abnormal health states.

6. Case studies for benchmarking – Code Sharing on GitHub

The case studies benchmark neural network ensemble, MC dropout, GPR, and SNGP for battery early-life and turbofan RUL prediction using publicly available datasets and comparable backbones. Across these applications, neural network ensembles generally provide the strongest balance of predictive accuracy, uncertainty quality, and calibration, while model behavior varies across datasets and OOD conditions.

  • Benchmark design: The benchmark evaluates four UQ methods—neural network ensemble, MC dropout, GPR, and SNGP—on two engineering applications.The applications are early life prediction of lithium-ion batteries and RUL prediction of turbofan engines, using publicly available datasets and nearly identical backbones where applicable.
  • 6.1. Case study 1: Battery early life prediction: Battery early-life prediction uses the first 100 cycles of 169 LFP/graphite cells to estimate remaining cycle life with confidence.The task is difficult because most cells show little degradation during the first 100 cycles, making uncertainty estimates important.
  • 6.1. Case study 1: Battery early life prediction: The battery models use ΔQ100−10(V) as input because its variance is significantly correlated with cycle life.The input is defined as ΔQ100−10(V) = Q100(V) − Q10(V).
  • 6.1.6. Evaluation/Results: GPR fits the 41 training points perfectly but generalizes poorly, especially on secondary and tertiary test datasets that differ substantially from training.The paper reports zero RMSE and extremely low NLL on training data, alongside large test prediction errors and high uncertainty.
  • Study scope: The reported comparisons are not fully optimized because each method uses a common simple ResNet backbone rather than individually tuned architectures and hyperparameters.Potential improvements include tuning layers, nodes, dropout rate, ensemble size, and activation functions.
  • 6.1.6. Evaluation/Results: SNGP generalizes better than GPR but often produces unnecessarily large uncertainty, resulting in large NLL and ECE.The neural network ensemble performs slightly better than MC dropout in accuracy and has a substantial UQ advantage because MC dropout is generally overconfident.
  • 6.2.2. Evaluation/Results: For turbofan RUL prediction, the neural network ensemble outperforms MC dropout and SNGP across RMSE, NLL, and ECE, while assigning higher uncertainty near degradation onset.All three models capture the decreasing RUL trend but struggle at the beginning of the trajectory.
  • 6.2.2. Evaluation/Results: Calibration curves indicate overconfident predictions, particularly for unit 11; MC is overconfident across all units, whereas NNE and SNGP are closer to ideal on selected units.NNE performs best on unit 14 and SNGP on unit 15, while NNE’s curve is generally more closely aligned with the ideal and has low ECE.

7. Other topics related to UQ of ML models

The paper surveys additional UQ-related topics spanning physics-informed ML, probabilistic manifold learning, system identification, interpretability, and surrogate modeling. It also highlights limitations and hybrid extensions involving PCE and GPR.

  • Physics-informed ML: Physics-informed ML combines physics-based and data-driven modeling to address training-data scarcity and improve predictive capability.
  • Probabilistic Learning on Manifolds: PLoM learns a generative model on a data-supported low-dimensional manifold, making it suitable for dimension reduction of high-dimensional datasets.It differs from GPR and BNN, which focus more directly on function approximation and regression.
  • Probabilistic Learning on Manifolds: PLoM generally requires enough samples to reveal the underlying distribution geometry, although additional constraints can support performance with relatively small datasets.
  • Interpretability of ML models for dynamic systems: SINDy promotes sparsity in a library of candidate functions to discover parsimonious governing equations and improve model interpretability.Hybrid SINDy can use out-of-sample Akaike information criterion scores to match models to regimes and identify switching points.
  • Interpretability of ML models for dynamic systems: UQ-SINDy extends SINDy with Bayesian inference and sparsifying priors to quantify coefficient uncertainty from observation errors and limited data.The approach also facilitates quantification of prediction confidence.
  • PCE and surrogate modeling: PCE approximates simulation models using polynomial expansions whose coefficients can be computed non-intrusively by projection or regression.
  • PCE and surrogate modeling: PCE faces a curse of dimensionality because model parameters and required simulation training samples increase exponentially with input dimension.Sparse truncation methods, including Smolyak constructions and compressive-sensing variants, are used to alleviate this issue.
  • PCE and surrogate modeling: PCE–kriging hybrids use PCE to represent the Gaussian-process prior mean function and capture the global trend of a computer simulation model.

8. Conclusion and outlook

The tutorial consolidates UQ methods, evaluation metrics, and engineering and PHM case studies into a practical overview of ML uncertainty quantification. It concludes that broader adoption requires principled, scalable evaluation and stronger standards, while noting that the case studies were not hyperparameter-optimized.

  • The tutorial classifies uncertainty, explains emerging UQ techniques, reviews evaluation metrics, and provides source code for engineering design and PHM case studies.The case studies address early-life battery prediction and turbofan-engine RUL prediction.
  • Its examination of UQ methods and case studies addresses future challenges involving scalability, principleness, and uncertainty decomposition.
  • The case studies were not hyperparameter-optimized, and optimization could yield better performance results.Their primary purpose is to provide a user-friendly platform for understanding and improving the analyzed methods.
  • UQ is presented as a safety-assurance layer that supports quantitative risk assessment and management for high-stakes ML applications.
  • The field needs community-acknowledged UQ testbeds covering calibration, decomposition, attribution, interpretation, and edge-case behavior.
  • Existing UQ methods require further development toward principled, scalable, computationally efficient operation, including real-time or near-real-time decision making.The paper notes that deterministic methods such as SNGP exhibit strong OOD detection capability through distance awareness.
  • PHM research needs quantitative UQ assessment using calibration, sparsification, and NLL metrics rather than relying mainly on subjective confidence-interval inspection.

Appendix A.1. An extended discussion on kernels

The appendix explains how Matérn-kernel smoothness controls GPR behavior and compares kernel choices in a one-dimensional toy example. It also introduces isotropic and anisotropic parameterizations.

  • Kernel smoothness: The Matérn family varies the smoothness parameter ν, with larger ν producing smoother approximations; ν →∞ yields the squared-exponential kernel.
  • Kernel smoothness: Matérn 1/2 observations provide limited information for distant input points, leading to poor generalization away from observed locations.
  • Kernel parameterization: The isotropic Matérn kernel uses one length scale, whereas its anisotropic ARD variant uses one length scale for each input dimension.The anisotropic form has D + 1 hyperparameters: σf, l1, …, lD.
  • Kernel comparison: Figure A.24 compares GPR models using multiple kernels, with squared-exponential producing the smoothest and Matérn 1/2 the roughest functions.
  • Kernel smoothness: Matérn 1/2 corresponds to Brownian motion and produces rough GPR prior and posterior samples, while Matérn 3/2 and 5/2 provide intermediate smoothness.

Appendix A.2. Parametric study on effect of hyperparameter optimization

The appendix studies how GPR hyperparameters affect the posterior in a one-dimensional toy regression example. Length scale, signal amplitude, and noise level alter correlation, function variation, and observation uncertainty.

  • Figure A.25 varies l, σf, and σε and reports their values alongside the log marginal likelihood for four posterior cases.
  • The function and observation posteriors have the same mean curve, while observation uncertainty has slightly larger variance because it includes noise.
  • A small length scale, such as l = 0.1, produces rapidly varying approximations and overly wide uncertainty regions between training points.
  • The signal amplitude σf controls the maximum vertical variation of functions and observations sampled from the Gaussian process.

Appendix A.3. Connections with neural networks and recent development

The appendix connects infinitely wide neural networks with Gaussian processes, showing how neural-network priors converge to Gaussian processes and inherit tractable Bayesian inference. It also notes benefits alongside overfitting and feature-collapse limitations.

  • Connections with neural networks: Infinitely wide, single-hidden-layer neural networks with i.i.d. parameter priors are equivalent to Gaussian processes.As hidden-layer width approaches infinity, the network output follows a Gaussian distribution by the Central Limit Theorem.
  • Recent development: Gaussian-process priors enable exact Bayesian inference for infinitely wide neural networks using matrix operations.This avoids iterative gradient-based training and approximate Bayesian inference methods used for Bayesian neural networks.
  • Connections with neural networks: The neural-network output becomes Gaussian because it is a sum of i.i.d. random variables when the hidden-layer width approaches infinity.The resulting prior has zero mean under zero-mean hidden-to-output weights and biases.
  • Recent development: Neural Gaussian-process approaches combine Gaussian-process uncertainty and distance awareness with neural networks’ feature extraction and nonlinear modeling.Certain activation functions, including error-function, one-sided polynomial, and ReLU activations, yield analytic covariance forms.
  • Limitations: Overparameterization can cause overfitting, while feature collapse must be addressed to preserve input distances in hidden representations.The appendix identifies spectral-normalized neural Gaussian process as a representative approach addressing feature collapse.

Appendix B.1. Needs of ML models in engineering design

Machine learning supports engineering design through feature extraction, surrogate modeling, and optimization. These capabilities address high-dimensional data, expensive simulations, localized design objectives, and costly searches for optimal designs.

  • Overview: ML-enabled engineering design is organized around feature extraction, surrogate modeling, and optimization.These categories have been applied to materials, reliability, energy systems, and topology optimization.
  • Feature extraction: Deep learning automatically extracts representations from high-dimensional raw data for dimension reduction and generative design.Autoencoders learn compressed representations through encoder–decoder architectures, while generative design explores many variants toward performance targets.
  • Surrogate modeling: Surrogate models emulate computationally expensive simulations to reduce the cost of evaluating complex engineering systems.High-fidelity simulations predict complicated physical phenomena but require substantial computational effort.
  • Surrogate modeling: Global surrogate models target accuracy across the full design domain, whereas local surrogates focus on purpose-specific regions such as limit-state boundaries.Local regions may be unknown initially and can require adaptive learning functions for identification.
  • Optimization: ML-assisted optimization addresses the high evaluation cost of engineering search, especially when conventional methods require many expensive simulations.Gradient-based methods may miss global optima, while evolutionary methods can require prohibitively many function evaluations.

Appendix B.2. Role of UQ of ML models in engineering design

Uncertainty quantification is foundational to ML-enabled engineering design because it guides feature extraction, adaptive surrogate refinement, local modeling, and optimization. Predictive distributions and uncertainty estimates support learning functions that target informative or critical regions.

  • Feature extraction: UQ helps ensure representative extracted features, reduce ill-posedness in generative design, and account for variability across input features.
  • Surrogate modeling: Surrogate-model refinement begins by quantifying predictive uncertainty because high-fidelity training data are expensive to collect.Refinement strategies seek to improve accuracy while minimizing new simulations and iterations.
  • Surrogate modeling: GPR can directly provide predictive variance, whereas other surrogate models require a separate UQ method.Maximizing prediction variance is a common global-surrogate refinement strategy.
  • Local surrogate modeling: Learning functions for local modeling use the predicted mean, standard deviation, and output distribution to identify critical regions near failure thresholds.The failure threshold separates failure from safe regions, and the critical interval is often set using twice the prediction standard deviation.
  • Optimization: UQ supplies the mean, standard deviation, and predictive distribution required to construct acquisition functions such as expected improvement.These functions support efficient exploration and exploitation during design optimization.

Appendix B.3. State of knowledge and gaps

The reviewed UQ methods address gaps across feature extraction, surrogate modeling, and optimization, while engineering applications still face challenges involving aleatory uncertainty, computational cost, and prognostics data limitations. Neural-network UQ can extend uncertainty-informed methods beyond predominantly GPR-based practice.

  • State of knowledge: Engineering feature-extraction studies mostly use neural networks, while local surrogate modeling and Bayesian optimization remain predominantly GPR-based.GPR’s analytically convenient Gaussian predictive uncertainty helps explain its continued use in local methods.
  • State of knowledge: The reviewed UQ methods provide tools for ML-enabled feature extraction, surrogate modeling, and optimization.Neural-network UQ methods can extend local surrogate and optimization approaches originally developed for GPR models.
  • Research gaps: Most current engineering-design methods lack capability for input-dependent aleatory uncertainty during local modeling or optimization.Neural-network ensemble UQ is identified as an opportunity to address this issue.
  • Research gaps: Computationally efficient UQ is needed because surrogate modeling and optimization may require repeated uncertainty evaluations at many input samples.Expensive UQ can increase overhead and diminish the benefits of ML models.
  • Health prognostics: Prognostics RUL prediction is affected by future usage uncertainty, limited representative failure trajectories, variable operating conditions, and environmental factors.
  • Health prognostics: Deep learning processes heterogeneous noisy sensor data end-to-end, but limited and unrepresentative training data can produce substantial epistemic uncertainty.That uncertainty must be quantified and communicated to decision makers.

Appendix C.4. State-of-the-art uncertainty-aware DL approaches for prognostics

Uncertainty-aware deep-learning prognostics methods add predictive uncertainty to neural-network outputs, with MC dropout as a straightforward strategy and ensembles also used for RUL prediction.

  • Motivation: Standard deep-learning prognostics typically provide point-estimate predictions because neural networks do not naturally incorporate uncertainty quantification routines.
  • MC dropout: MC dropout enables uncertainty quantification by activating dropout layers during inference, making each neural-network forward pass stochastic.Its intuitive rationale and relatively straightforward implementation have made it common in uncertainty-aware prognostics.
  • Alternative approaches: Ensemble methods and deep Gaussian processes have also been applied to prognostics, including RUL prediction with associated prediction intervals.One cited approach ensembles Echo State Networks and estimates residual variance with an additional ESN.
  • MC dropout: Figure D.28 reports training and validation losses for MC dropout models using dropout rates of 0.05 and 0.2.

Appendix D. Demonstration of Instability of MC Dropout

The appendix demonstrates that MC dropout uncertainty is unstable: small hyperparameter changes produce highly inconsistent uncertainty maps, even when training behavior appears converged.

  • Instability demonstration: Small changes in model size, training epochs, or dropout rate can make MC dropout uncertainty estimates unstable.
  • Training behavior: The two examined MC dropout models used the same data and architecture but different dropout rates, and both converged at around 500 epochs.No over-fitting was observed until 10000 epochs in this toy example.
  • Uncertainty maps: The resulting uncertainty maps were highly inconsistent, supporting the appendix’s conclusion that MC dropout is unstable.
  • Uncertainty maps: Table D.8 compares uncertainty-map instability across dropout rate, training-epoch count, and ResNet architecture.
Loading 2305.04933v2…