PaperlayerView exact revision on arXiv

Source-linked AI summary

Information fusion and machine learning for sensitivity analysis using physics knowledge and experimental data

Berkcan Kapusuzoglu, Sankaran Mahadevan

arXiv:2608.17248v1cs.CEcs.LGstat.MEstat.ML

TL;DR

Sensitivity estimates are affected by uncertainty in computational models, while combining available physics knowledge and experimental data remains challenging. The paper develops physics-informed GP and DNN strategies for uncertainty-aware Sobol’ indices, finding that DNN models yield more accurate estimates with smaller uncertainty than GP models in the numerical examples.

  • Problem

    Sensitivity analysis requires computational models despite model uncertainty, while effectively combining physics-based models with limited experimental data remains an open challenge.

  • Method

    The paper fuses physics knowledge and experimental observations through physics-informed GP and DNN models, incorporating constraints or simulation pre-training and uncertainty into Sobol’ indices.

  • Results

    DNN models produced more accurate Sobol’ index estimates with smaller uncertainty and computational effort than GP models in the numerical examples.

  • Takeaways & Limitations

    Physics-informed GP and DNN strategies enabled accurate, physically meaningful Sobol’ indices with smaller amounts of experimental data in the numerical examples.

  • Takeaways & Limitations

    The study used only one Gaussian kernel and requires testing on problems with more input and output dimensions.

Abstract

from arXiv · show

When computational models (either physics-based or data-driven) are used for the sensitivity analysis of engineering systems, the sensitivity estimate is affected by the accuracy and uncertainty of the model. This paper considers global sensitivity analysis (GSA) for situations where both a physics-based model and experimental observations are available, and investigates physics-informed machine learning strategies to effectively combine the two sources of information in order to maximize the accuracy of the sensitivity estimate. Two representative machine learning (ML) techniques are considered, namely, deep neural networks (DNN) and Gaussian process (GP) modeling, and two strategies for incorporating physics knowledge within these techniques are investigated, namely: (i) incorporating loss functions in the ML models to enforce physics constraints, and (ii) pre-training and updating the ML model using simulation and experimental data respectively. Four different models are built for each type (DNN and GP), and the uncertainties in these models are included in the Sobol indices computation. The DNN-based models, with many degrees of freedom in terms of model parameters and training options, are found to result in smaller bounds on the sensitivity estimates when compared to the GP-based models. The proposed methods are illustrated for additive manufacturing and lake temperature modeling examples.

1 Introduction

This paper studies global sensitivity analysis when both a physics-based model and experimental data are available, fusing them through physics-informed machine learning to improve sensitivity estimates. It investigates loss-based and pre-training/updating strategies with DNN and GP models, including model uncertainty in Sobol’ indices.

  • Motivation: Computational and surrogate-model uncertainty can affect system-response predictions and Sobol’ index estimates, motivating GSA methods that account for epistemic uncertainty.Epistemic uncertainty may arise from incomplete physics models, limited data, and surrogate modeling.
  • Problem setting: The paper focuses on the third GSA scenario: combining a physics-based model with available experimental or real-world data.The other scenarios use only a physics-based computational model or only available input-output data.
  • Contributions: The study evaluates how physics knowledge and experimental observations can be fused to maximize the accuracy of sensitivity estimates.The proposed methodology is illustrated through additive manufacturing and lake temperature modeling examples.
  • Methodology: Two physics-informed machine-learning strategies are investigated: physics-constrained loss functions and pre-training on physics-generated data followed by experimental-data updating.The strategies aim to leverage physical laws while improving the generalization of data-driven models.
  • Methodology: Four models are built for each of GP and DNN, with model uncertainties incorporated into Sobol’ indices computation.The study evaluates accuracy, uncertainty, and computational effort across ML and PIML options.

2 Background

This section introduces variance-based global sensitivity analysis, Gaussian process surrogate modeling, and deep neural networks. It defines Sobol indices and outlines how GP and DNN models represent and optimize predictions.

  • Variance-based GSA: Variance-based GSA decomposes output variance into individual-input effects and interaction effects, with Sobol indices obtained by normalizing these contributions by total variance.First-order indices measure individual contributions, while higher-order indices measure interactions among inputs.
  • Variance-based GSA: The first-order index S_i measures X_i’s individual contribution to output variance, whereas the total-effects index S_T includes its interactions with all other inputs.Analytical computation of S_i requires multidimensional integrals, motivating sampling and surrogate-model approaches.
  • Gaussian process modeling: A GP surrogate models y = G(x) as a Gaussian random process with mean function m(x) and covariance function k(x, x′), producing conditional Gaussian predictions.The conditional covariance accounts for training data, prediction inputs, and observation-error variance σ^2_obs.
  • Deep neural networks: A DNN uses multiple hidden layers and combines neurons, activation functions, cost functions, and optimization to map inputs to outputs.Forward-propagated predictions are compared with observations through a loss function, and backpropagation updates weights using layerwise gradients.

3 Proposed methodology

The proposed methodology combines physics knowledge and experimental data for sensitivity analysis through four sequential steps: identifying and implementing PIML strategies, quantifying prediction variance, and computing Sobol’ indices.

  • The methodology begins by identifying physics-informed machine learning (PIML) strategies.
  • The identified PIML strategies are implemented in machine learning models.
  • Prediction variance in the machine learning models is quantified.
  • Sobol’ indices are computed using the variance in machine learning model predictions.

3.1 Identification of PIML strategies

This section identifies two physics-informed machine learning strategies for combining physics knowledge with experimental data: enforcing physical constraints during training and pre-training on physics-model data before experimental updating. The strategies aim to improve model predictions and sensitivity estimates by combining complementary physics-based and data-driven information.

  • Physics-constrained loss: Physics constraints enter training as an additional penalty term weighted by λ_phy, whose value controls the strength of constraint enforcement.The added physics loss promotes physically consistent predictions and helps reduce generalization error.
  • Physics-constrained loss: The physics constraints may be equalities or inequalities involving algebraic relationships or partial differentials of model outputs and other variables.Inequality violations can be penalized with ReLU(x) = max(0, x), including deviations from physically consistent relationships among multiple outputs.
  • Physics-data pre-training: Pre-training uses physics-model data across many input combinations, then updates the ML model with the available experimental data.In the numerical example, pre-training uses 1310 physics-model input combinations, compared with 39 available experiments.
  • Proposed strategies: Two PIML strategies are proposed: incorporating physics-based constraints into the ML loss function, and pre-training with physics-model data before updating with experimental observations.The first strategy trains ML models using experimental data with added physics constraints; the second first trains on physics-generated input-output data and then updates on experimental data.

3.2 Implementation of PIML strategies in ML models

The section defines four physics-informed model variants for each surrogate type by combining experimental training, physics constraints, pre-training, and discrepancy correction. Their implementations differ between Gaussian processes and deep neural networks.

  • Model variants: Four models are constructed for each surrogate type, spanning experimental-data-only training, either PIML strategy alone, or both strategies together.The strategy implementations differ between GP and DNN models.
  • Gaussian-process models: GPLphy enforces physics constraints while maximizing the GP log marginal likelihood during hyperparameter inference.The physics-constraint term is included in the objective function used to train the constrained GP.
  • Gaussian-process models: The second PIML strategy pre-trains a model with physics-model input-output data, then either updates parameters with experiments or learns an additive discrepancy correction.The additive correction follows fHF(X) = fLF(X) + δ(X; θδ).
  • Gaussian-process models: GPupd updates a physics-pre-trained GP with experimental data, whereas GPMF trains a second GP to predict discrepancy between surrogate predictions and actual system responses.GPMF uses one GP for the physics-model surrogate and another for the discrepancy term, which captures measurement, physics, and surrogate-model errors.
  • Deep-neural-network models: DNN uses only experimental data, while DNNLphy incorporates physical knowledge through constraint terms in the neural-network loss function.DNN training optimizes weights, biases, epochs, layers, and neurons; physics-based terms are evaluated at each optimization iteration for DNNLphy.

3.3 Variance of GP and DNN prediction

GP prediction uncertainty is represented through predictive variance, whereas DNN uncertainty arises from uncertain model parameters and can be modeled with Bayesian neural networks. For DNNs, approximate inference such as MC dropout introduces sensitivity estimates that depend on regularization and training choices.

  • GP prediction uncertainty: GP predictions are normal distributions characterized by a mean and variance, and sampling realizations quantifies predictive variance but not bias.Bias requires comparison with validation data.
  • DNN parameter uncertainty: DNN uncertainty concerns neuron-weight estimates and depends on available training data; representing weights probabilistically yields a Bayesian neural network.The Bayesian formulation describes parameter uncertainty with a prior and likelihood.
  • Approximate inference: DNN posterior inference is difficult because the parameter space is high-dimensional and network behavior is highly nonlinear, motivating approximate inference techniques.Variational inference is identified as one such approximation.
  • Approximate inference: MC dropout randomly removes neurons during training, acting as regularization and enabling approximate Bayesian inference.Dropout is intended to prevent over-fitting and reduce generalization error.
  • Sensitivity-estimate limitations: Sensitivity estimates depend on dropout rate, training epochs, and network architecture; over-regularization or incomplete training increases prediction bias and variance.Over-regularization can underfit the data, while insufficient epochs can also cause underfitting.

3.4 Sobol’ indices computation with model uncertainty

The section propagates predictive uncertainty from GP and DNN models into Sobol’ index estimates. GP uncertainty is sampled from predictive distributions, whereas DNN uncertainty uses MC dropout stochastic forward passes, enabling mean and variance estimates across eight PIML models.

  • GP-based uncertainty propagation: GP model uncertainty is propagated to Sobol’ indices by sampling realizations from the Gaussian predictive distribution.Predictions have zero variance at noise-free training points and non-zero variance elsewhere.
  • GP-based uncertainty propagation: The resulting GP sensitivity samples quantify the mean and variance of Sobol’ index estimates.The sample size is NZ, while m denotes the number of Monte Carlo samples.
  • DNN-based uncertainty propagation: DNN model uncertainty is estimated with MC dropout by randomly setting network units to zero and generating predictions with the remaining units.This sampling procedure differs from GP sampling from a multivariate normal distribution.
  • Overall framework: Eight PIML models combine two physics-informed strategies with GP and DNN models to assess sensitivity-estimate accuracy and uncertainty.Validation data assesses accuracy, while Algorithms 1 and 2 and Eqs. 26 to 33 quantify sensitivity-estimate variance.

4 Numerical illustration · 4.1 Illustrative example 1

The additive-manufacturing example evaluates physics-informed GP and DNN models for Sobol sensitivity analysis of porosity using nozzle temperature and speed. Physics-informed DNNs achieve narrower sensitivity-estimate uncertainty and improved accuracy, especially with limited observations, while GP and DNN estimates converge similarly with sufficient data.

  • 4.1.1 Problem setup: The example models FFF-part porosity from nozzle temperature and speed, using transient filament-temperature physics to estimate the printed part’s final mesostructure.The experiments use Ultimaker Black ABS specimens and microscopy-based midpoint porosity measurements.
  • 4.1.2 Training details of the ML models: The eight models combine GP or DNN learning with physics-loss constraints, simulation pre-training followed by experimental updating, or both.Physics penalties enforce nonnegative porosity and predicted final porosity no greater than initial porosity; pre-training uses 1310 simulated input combinations and updating uses 39 experiments.
  • 4.1.3 Comparison of computational effort: Training and Sobol-index estimation are more computationally demanding for physics-informed and updated models than for basic models, while DNN training takes 20–55 sec.The comparison uses 5000 MC samples and n = 39 observations.
  • 4.1.4 Comparison of accuracy: PIML improves DNN accuracy most strongly when observations are scarce, whereas at n=20 and n=30 the basic DNN is as accurate as physics-informed DNNs.At dropout rate 0.05, DNN models are more accurate than GP models, and both pre-training/updating approaches produce similar Sobol estimates.
  • 4.1.5 GSA results using GP models: GP prediction intervals decrease with more experimental data, and all four GP models converge to similar first-order and total-effect estimates for nozzle temperature and speed.GPupd and GPupd,Lphy converge to the 39-observation bounds faster than the first two GP models.
  • 4.1.6 GSA results using DNN models with MC dropout: DNN models likewise converge to similar first-order and total-effect estimates that are consistent with GP results; 20 observations can produce similar performance in this two-input problem.The estimates use 5000 MC samples and 100 stochastic forward passes.
  • 4.1.6 GSA results using DNN models with MC dropout: 0.02-0.03 is the DNN prediction-interval width at n = 39, compared with 0.12-0.15 for GP models; DNN uncertainty is almost negligible above 10 observations.DNN intervals decrease as experimental data increase, with narrower bounds attributed to model flexibility, optimized training epochs, and dropout.

4.2 Illustrative example 2

The lake-temperature example evaluates eight GP and DNN models using eleven meteorological inputs and shows that physics-informed DNNs provide accurate, less uncertain sensitivity estimates with lower computational effort than GP models. Physics-informed updating with a physics loss improves convergence, especially with limited experimental data.

  • Lake temperature modeling: The example models Lake Mendota temperature using 11 meteorological drivers and 13,543 observations collected from 30 April 1980 to 02 November 2015.The General Lake Model produced 662,781 input-output training samples.
  • Computational effort: Using 5000 MC samples and n = 1000 observations, GP models required more computational effort than DNN models for training and Sobol’ index estimation.This comparison covers the eight models in the lake-temperature example.
  • GP sensitivity estimates: GP prediction intervals decreased with more experimental data, while all four GP models converged to similar first-order and total-effect sensitivity estimates.The intervals are reported as 95% prediction intervals.
  • DNN sensitivity estimates: All DNN models converged to sensitivity estimates consistent with GP results, while physics-informed models converged more smoothly than basic ML models.The basic GP and DNN models produced slightly different total-effect estimates for most inputs than the proposed PIML models.
  • Uncertainty and convergence: DNN models produced significantly narrower 95% bounds than GP models, and Models 4 and 8 converged to final mean estimates within 250 observations for almost all variables.Models 4 and 8 are GPupd,Lphy and DNNupd,Lphy; their convergence was faster than that of the other models.
  • Overall findings: Physics-informed ML models achieved higher prediction accuracy than basic ML models, especially when fewer experimental data were available.Across the numerical examples, DNN models also achieved higher accuracy and lower uncertainty than GP models with less training and prediction effort.

5 Conclusion

The paper develops physics-informed machine-learning methodologies that fuse physics knowledge and experimental data for uncertainty-aware Sobol’ sensitivity analysis. The proposed GP- and DNN-based strategies support accurate, physically meaningful estimates with limited experimental data while motivating broader validation and information weighting.

  • Methodology: The methodologies combine physics knowledge and experimental data for variance-based sensitivity analysis while accounting for model uncertainty.Several physics-informed machine-learning models were developed for this purpose.
  • Methodology: Two strategies incorporate physics knowledge: physics constraints in training loss functions, or simulation-data pre-training followed by experimental-data updating.The first strategy does not use the physics model, whereas the second does.
  • Uncertainty quantification: Prediction bounds on Sobol’ index estimates account for uncertainty in trained models, using the proposed estimator for GP models and Monte Carlo dropout for DNN models.Prior work had considered prediction bounds for model outputs, rather than sensitivity-index estimates.
  • Results: PIML strategies for both GP and DNN models enable accurate Sobol’ index computations with smaller experimental datasets while producing physically meaningful results.The approach addresses physics-model approximation by helping fill the ML models’ physics knowledge gap.
  • Future work: Future work should test higher-dimensional input and output problems, examine multiple combinations for convergence, and investigate weighting information sources with different credibility levels.The GP approach could also be extended using different kernels.
Loading 2608.17248v1…