PaperlayerView exact revision on arXiv

Source-linked AI summary

Predictive Uncertainty for Neural CAE Surrogates

Kaustubh Tangsali, Mohammad Amin Nabian, Kelvin Lee, Carmelo Gonzales, Sanjay Choudhry

arXiv:2609.25430v1cs.LGcs.AImath.PRphysics.comp-ph

TL;DR

Neural-surrogate UQ must remain useful across case screening, spatial fields, and engineering quantities, but these needs cannot be assessed with a single calibration score. The paper compares GP-based uncertainty, concrete MC dropout, and deep ensembles across three industrial CAE datasets; method rankings vary by dataset and engineering question.

  • Problem

    CAE uncertainty estimates must support distinct decisions about cases, local field errors, and derived quantities, which a single calibration score cannot assess.

    Source: §1 · pp. 1–2

  • Method

    The study compares GP-based uncertainty, concrete MC dropout, and deep ensembles using a CAE-oriented evaluation protocol across three industrial datasets.

    Source: §1 · p. 2

  • Results

    Method rankings change with dataset and engineering question; GP total variance gives strong raw interval scales in DrivAerStar and crash, while other comparisons differ by task.

    Source: §10 · p. 26

  • Takeaways & Limitations

    UQ methods and evaluation metrics should be selected for the intended downstream CAE decision, with evaluation matched to that use.

    Source: §8 · p. 25

    The GP drag calculation uses pointwise marginals under an independence approximation because the interface does not expose joint field samples; calibration is limited to positive scaling.

    Source: §9 · p. 25

Abstract

from arXiv · show

Neural surrogates can substantially accelerate computer-aided engineering (CAE) workflows, but their use in design requires uncertainty estimates that remain meaningful across varying geometries, spatial prediction fields, and engineering quantities of interest. We investigate how established uncertainty quantification (UQ) approaches behave when adapted to geometry-conditioned neural surrogates. We compare one closed-form and two sampling-based approaches-a Gaussian process (GP)-based method, concrete Monte Carlo (MC) dropout, and deep ensembles-and evaluate them on three large, industry-relevant CAE datasets for external aerodynamics and crash dynamics. We examine whether predicted uncertainties have credible magnitudes, identify locations with larger prediction errors, respond to unfamiliar inputs, and remain informative for derived engineering quantities. On the DrivAerStar dataset, where all three methods are compared, each generally assigns higher uncertainty to locations with larger prediction errors, and validation-based rescaling brings interval coverage close to nominal on a disjoint in-distribution test set. Results on AirFRANS and automotive crash also show useful error ranking and interval estimates, but the relative performance of the methods changes with the dataset and evaluation criterion. UQ methods and evaluation metrics should therefore be selected based on the intended downstream CAE decision.

1 Introduction

Geometry-conditioned surrogates need uncertainty that supports case screening, field-level triage, and engineering decisions. This study compares three UQ approaches across industrial CAE datasets to test how their estimates perform across these uses.

1 Introduction

  • The three evaluation scopes—case, local field, and engineering quantity—require different uncertainty assessments, so calibration alone cannot assess them all.A method may be calibrated yet rank case errors poorly, miss local errors under shift, or overlook dependencies needed for an integrated quantity.Source: §1 · pp. 1–2
  • A controlled comparison evaluates GP-based uncertainty, concrete MC dropout, and deep ensembles across three industrial CAE datasets.DrivAerStar supplies the full three-method comparison; AirFRANS tests angle-of-attack extrapolation, while automotive crash changes the physics and engineering output.Source: §1 · p. 2
  • The paper contributes a CAE-oriented protocol spanning case, local-field, and engineering-quantity evaluation scopes.The motivation is to help engineers decide whether to run a solver, where a predicted field may be unreliable, and how much confidence to place in derived quantities.Source: §1 · pp. 1–2
  • The comparison provides evidence that calibration, error discrimination, distribution-shift response, and usefulness for engineering quantities can yield different method rankings.The study evaluates how these conclusions persist across datasets and scopes.Source: §1 · p. 2

2 Related Work

Prior UQ work develops methods and metrics for predictive distributions, field errors, and engineering quantities, but these are often studied separately. This paper compares three approaches on large, irregular-mesh CAE problems across case, local-field, and engineering-quantity scopes.

2 Related Work

  • Established UQ methods include sampling-based neural approximations, GP predictive variance, and calibration and scoring methods, but these are usually studied separately on large simulation problems.Concrete dropout learns continuous approximations to binary masks, while GP methods use kernels and posterior distributions with practical inducing-point and deep-kernel approaches.Source: §2 · p. 2
  • Calibration, scoring rules, selective prediction, sparsification, and conformal prediction assess complementary properties, but coverage alone does not show field localization, unfamiliar-case detection, or decision usefulness.Conformal coverage relies on calibration and test examples being treated as draws from the same distribution.Source: §2 · pp. 2–3
  • The paper's focus is a comparison of GP-based, concrete MC dropout, and ensemble uncertainty on large, irregular-mesh CAE problems across three evaluation scopes.Related work includes uncertainty methods for scientific computing and CFD, neural operators and transformers for complex geometries [29] [30], and function-valued predictions that retain spatial dependencies.Source: §2 · p. 3

3 Predictive Uncertainty in CAE Workflows

The paper organizes predictive uncertainty by where it is used in CAE workflows: screening cases, locating field errors, or informing derived engineering quantities. It distinguishes epistemic uncertainty from the GP's learned residual-variance term, which represents remaining surrogate errors for deterministic simulations.

3.1 Uncertainty scopes

  • The three workflow scopes are case screening, local-field error assessment, and uncertainty in a derived engineering quantity.They are parallel uses rather than a hierarchy, and each calls for different evaluation criteria.Source: Figure 2 · p. 3, §3.1 · pp. 3–4
  • Case-level uncertainty can aggregate a predicted field to prioritize inputs for review or recomputation, but unfamiliar inputs need not have high uncertainty.Case-level scores should be tested on both in-distribution and deliberately shifted cases before serving as guardrails.Source: §3.1 · pp. 3–4
  • Local-field uncertainty can support error triage, while epistemic uncertainty is generally relevant to active learning because additional data may reduce it.The association between local uncertainty and prediction error must be evaluated rather than assumed.Source: §3.1 · p. 4
  • Engineering-quantity uncertainty depends on spatial and cross-channel dependencies, so credible field-level uncertainty does not ensure credible uncertainty in the final quantity.Pointwise standard deviations cannot generally be combined independently to estimate quantities such as drag, peak stress, or integrated force.Source: §3.1 · p. 4

3.2 Variance Decomposition

  • For the evaluated GP, epistemic variance is posterior variance of the latent function, while the likelihood-fitted residual variance represents remaining surrogate error on deterministic simulations.The paper reports the residual term separately and includes it in total predictive variance; it is interpreted as learned model-discrepancy variance rather than physical randomness.Source: §3.2 · p. 4

4 Uncertainty Quantification Methods

This section contrasts a GP-based predictive distribution with empirical distributions from concrete MC dropout and deep ensembles. The methods differ in how they construct uncertainty and which variance components they estimate.

4.1 Gaussian Process-Based Method

  • Independent per-output variational GPs provide posterior epistemic variance, while learned residual variance is added to obtain total predictive variance.For deterministic simulation data, the residual component is interpreted as model-discrepancy variance; it may fail to signal unfamiliar inputs if transformed features overlap training features, and independent output kernels do not model cross-channel covariance.Source: §4.1 · pp. 4–5

4.2 Concrete MC dropout

  • Concrete MC dropout estimates epistemic disagreement from stochastic passes, with learned layer-specific dropout probabilities rather than a single shared rate.The evaluated implementation omits residual variance, and its disagreement is not guaranteed to increase for unfamiliar inputs.Source: §4.2 · p. 6

4.3 Deep Ensembles

  • Deep ensembles average predictions from independently trained members and use their spread to estimate epistemic disagreement.Correlated members can produce narrow uncertainty despite shared inaccuracy; the evaluated implementation omits residual variance, so its spread is not automatically total predictive uncertainty or a calibrated interval.Source: §4.3 · p. 7

5 Experimental Design

The experiments compare UQ across aerodynamics and crash datasets with distinct geometry, boundary-condition, and transient-response challenges. Metrics and validation-based rescaling separate uncertainty properties and test calibration transfer.

5 Experimental Design

  • Three CAE datasets cover external aerodynamics and crash dynamics; DrivAerStar compares all three approaches, while AirFRANS and crash compare the GP-based method and concrete MC dropout.Source: §5 · p. 7

5.1 Datasets and Distribution Shift Definitions

  • DrivAerStar tests held-out body styles after Fastback-only training, AirFRANS extrapolates beyond fitted angles of attack, and crash evaluates held-out designs within the sampled design space.DrivAerStar uses categorical geometry shift, AirFRANS tests boundary-condition extrapolation, and the crash study is not an explicit out-of-distribution test.Source: §5.1 · pp. 7–8

5.2 Metrics

  • Metrics separately assess interval magnitude, error ranking, case-level OOD separation and shift response, while drag uncertainty is estimated both marginally and from sampled whole fields.The marginal drag estimate assumes independence across surface points and output channels; the sampled-field estimate preserves dependencies represented in the predictions. AUSE compares uncertainty-based case removal with oracle error-based removal.Source: §5.2 · p. 10

5.3 Post-Hoc Calibration Protocol

  • Post-hoc calibration rescales standard deviations by output channel for aerodynamics or time step for crash, fitting on validation data and applying factors unchanged to disjoint evaluation cases.The protocol tests whether in-distribution calibration transfers to held-out body styles, angle-of-attack extrapolation, or held-out crash runs.Source: §5.3 · pp. 10–11

5.4 Implementation and Reproducibility

  • Implementations use a common distribution interface for consistent UQ benchmarking, with shared within-dataset data and evaluation settings.Crash training uses method-specific point-sampling budgets; three-seed DrivAerStar variability was small and did not alter the principal method comparisons.Source: §5.4 · p. 11

6 Controlled Comparison on DrivAerStar

DrivAerStar compares three UQ approaches across prediction accuracy, interval magnitude, error discrimination, geometric shift, drag, solver review, and computational cost. Method rankings vary with the uncertainty property and evaluation criterion.

6.1 Predictive Accuracy

  • Across 36 matched field-and-class comparisons, mean relative L2-error differences from the deterministic reference average 6.4%, with no method consistently most accurate.Because the models were trained separately and the sampling methods average predictions, individual differences cannot be attributed solely to the UQ mechanism.Source: §6.1 · pp. 12–13

6.2 Magnitude Credibility

  • Before recalibration, GP total variance is closest to nominal scale; after independent Fastback calibration, all three methods reach approximately nominal magnitude and coverage on the disjoint Fastback test set.The GP epistemic standard deviation is approximately 9.6% of its total standard deviation, while the sampling methods require larger scale corrections.Source: §6.2 · pp. 13–14

6 Controlled Comparison on DrivAerStar

  • All methods provide useful local error ranking, but calibration, geometric-shift detection, and drag-error ranking favor different methods or remain inconclusive.For example, GP total uncertainty has the strongest local rank correlations, while the ensemble is most stable after calibration under body-style shifts; drag uncertainty does not consistently rank drag errors.Source: §6.3 · p. 14, §6.2 · p. 14, §6.4 · p. 16

6.3 Error Discrimination

  • Local uncertainty meaningfully ranks error for all methods, with Spearman correlations around 0.55–0.73; elevated uncertainty aligns with the dominant roof-trailing-edge error region.The GP total uncertainty has the highest correlation on each body style, while the GP epistemic component is less discriminative than total variance.Source: §6.3 · p. 14, Figure 6 · p. 15

6.4 Response to Geometric Shift

  • Uncertainty grows less than prediction error for all methods under geometry shift; Estateback is easier to detect than Notchback, with ensemble AUROC 0.883 versus values near chance for Notchback.After recalibration, the ensemble has the most stable interval magnitude under body-style shifts and the strongest Estateback detection.Source: §6.4 · p. 16

6.5 Engineering Quantities of Interest

  • Drag uncertainty does not consistently rank absolute drag error, and sample-wise propagation changes mean drag standard deviation by about 1.1× for dropout and 15–22× for the ensemble.The marginal rule omits spatial and cross-channel covariance; sample-wise propagation retains dependencies represented by sampled fields, while the evaluated GP interface lacks joint field samples.Source: §6.5 · p. 17

6.6 Uncertainty-Guided Solver Review

  • Uncertainty offers modest value for prioritizing cases with large pressure errors, while overlapping 95% intervals make small AUSE differences indecisive.Figure 7 compares uncertainty-ranked review curves with oracle ordering.Source: Figure 7 · p. 17, §6.6 · p. 18

6.7 Computational Trade-Offs

  • The GP concentrates complexity in joint training but returns deterministic variance in one inference; dropout spends inference passes, while ensembles multiply training, storage, and evaluations.With 20 dropout passes and five ensemble members, idealized standard-deviation sampling errors are approximately 16% and 35%, respectively.Source: §6.7 · p. 18

7 Additional CAE Case Studies

AirFRANS and automotive crash serve as supporting case studies that vary the shift axis, physical system, and engineering use of uncertainty. The included comparisons use the GP-based method and concrete MC dropout, with numerical results compared only within each dataset.

7 Additional CAE Case Studies

  • The supporting studies vary the shift axis, physical system, and engineering use of uncertainty relative to the DrivAerStar comparison.Only the GP-based method and concrete MC dropout are included, so numerical comparisons are within each dataset.Source: §7 · p. 18
  • Cross-case analysis focuses on qualitative findings supported by measurements compatible across the datasets.Numerical values are compared only within each dataset.Source: §7 · p. 18

7.1 AirFRANS

  • AirFRANS evaluates continuous angle-of-attack extrapolation using the study defined in Section 5.Table 10 reports mean relative L2 error across four predicted channels and pressure error.Source: §7.1 · p. 19, Predictive accuracy · p. 19

Predictive accuracy

AirFRANS tests angle-of-attack extrapolation, while the crash study evaluates held-out designs and intrusion probes. In both settings, interval calibration, error discrimination, and predictive accuracy do not always align.

Predictive accuracy

  • The AirFRANS GP mean error increases from 0.085 in distribution to 0.150 under extrapolation, while the deterministic model increases from 0.082 to 0.098.Concrete MC dropout has higher error both in and outside the fitting range, so its near-one OOD-to-ID error ratio partly reflects high in-distribution error.Source: Predictive accuracy · p. 19
  • After in-distribution scaling, both AirFRANS methods become overconfident under extrapolation, and uncertainty grows less than prediction error for every method.OOD z-RMS rises to 4.657 for GP and 2.046 for concrete MC dropout; growth ratios are 0.701 for GP total, 0.707 for GP epistemic, and 0.768 for dropout.Source: Predictive accuracy · p. 19

7.2 Automotive Crash

  • At crash mesh level, GP total has the more credible raw interval magnitude, whereas concrete MC dropout has stronger spatial error ranking and a more accurate mean.After validation-based scaling, the three z-RMS values are similar; Figure 10 shows error and calibrated uncertainty most pronounced around the frontal impact structure.Source: §7.2 · p. 21, Table 12 · p. 21
  • On held-out crash probes, calibration brings both methods near nominal interval magnitude, but concrete MC dropout has lower RMSE and calibrated NLPD than the GP-based method.Concrete MC dropout reaches RMSE 30.5 mm versus 59.5 mm for GP; calibrated NLPD is 4.70 versus 5.13, respectively.Source: §7.2 · p. 22, Table 13 · p. 22
  • Within seven held-out crash runs, uncertainty remains approximately constant or decreases on designs with larger errors, despite calibration.Two rebound-dominated runs have 3.54 times the GP error of the other five, but uncertainty ratios of 0.99 raw and 1.00 calibrated.Source: §7.2 · p. 22, Figure 11 · p. 23
  • Crash-model dropout probabilities continue changing beyond the epoch-30 accuracy-selected checkpoint, so checkpoint selection does not imply their convergence.The geometry-encoder rates approach 0.5 early, while transformer-block rates evolve at different speeds; the probabilities parameterize stochastic masks, not layerwise predictive variance.Source: §7.2 · p. 22, Figure 12 · p. 23

8 Cross-Case Synthesis and Practical Guidance

Across datasets, methods, and engineering questions, uncertainty methods have different relative strengths; no single criterion establishes usefulness for another. Calibration, error ranking, distribution-shift response, and derived-quantity usefulness each need evaluation suited to the intended decision.

8 Cross-Case Synthesis and Practical Guidance

  • Method rankings change across datasets and criteria, and the cross-case comparison is qualitative because targets, splits, aggregation, and available methods differ.Table 14 summarizes the cross-case findings.Source: §8 · p. 24, Table 14 · p. 24
  • Fastback calibration transfers approximately nominal coverage to disjoint Fastbacks but differently to other body styles; AirFRANS scaling leaves both methods overconfident under angle-of-attack extrapolation.AirFRANS uncertainty growth ratios remain below one.Source: §8 · p. 24
  • Sample-wise drag propagation changes mean drag standard deviation by approximately 1.1× for concrete MC dropout and 15–22× for the ensemble versus marginal propagation.Spatial and cross-channel dependencies can materially alter the reported uncertainty scale.Source: §8 · p. 24
  • For crash probes, GP total variance has more credible raw interval magnitude, but RMSE is 59.5 mm versus 30.5 mm for concrete MC dropout.Methods also differ in trained-model counts, stored checkpoints, and per-case evaluations; uncertainty does not compensate for an inaccurate predictive mean.Source: §8 · p. 25
  • Method selection should match the decision: assess coverage and sharpness for intervals, error ranking for triage, shift detection and growth for OOD guardrails, and error reduction for active learning.Table 15 presents method-selection guidance based on downstream decision, prediction interface, and computational budget.Source: §8 · p. 25, Table 15 · p. 25
  • Validate implementation choices: test sampling convergence, GP representation and kernel sensitivities, and UQ checkpoint metrics during training.The best UQ checkpoints need not coincide with those selected for mean accuracy.Source: §8 · p. 25

9 Limitations

The comparison is constrained by method availability across datasets, differences in uncertainty components, and specific propagation and sampling choices. These boundaries limit which cross-dataset comparisons and uncertainty estimates the results support.

9 Limitations

  • The complete three-method matrix is available only on DrivAerStar; AirFRANS and crash compare GP-based and concrete MC dropout without a deep ensemble.Bootstrap intervals capture finite test-set sensitivity, while training-run sensitivity covers only GP-based and concrete MC dropout on DrivAerStar.Source: §9 · p. 25
  • GP uncertainty includes learned residual variance as well as posterior epistemic variance, whereas sampling configurations report epistemic disagreement only.Predictive intervals use GP total variance, while epistemic-task comparisons use GP posterior variance against sampling disagreement.Source: §9 · p. 25
  • GP drag uncertainty uses pointwise marginals under an independence approximation because the interface does not expose joint field samples.Concrete MC dropout and the ensemble instead use sample-wise propagation across all 300 cases.Source: §9 · p. 25
  • Concrete MC dropout uses 20 passes on DrivAerStar and AirFRANS and 32 on crash; the DrivAerStar ensemble has five members.Sampling estimates therefore contain finite-sample variation; GP estimates avoid inference-time sampling noise but are sensitive to configuration choices.Source: §9 · p. 26

10 Conclusion

Across three physically distinct CAE problems, the experiments find that uncertainty properties are separate and method rankings depend on dataset and engineering question. The appendix records configuration details that affect interpretation and reproduction.

10 Conclusion

  • The evaluation spans vehicle aerodynamics, airfoil flow under angle-of-attack extrapolation, and transient crash dynamics, with complex geometries, irregular meshes, and practical engineering quantities.The study examines whether uncertainty remains useful across geometry variation and aggregation in CAE workflows.Source: §10 · p. 26
  • Calibration, error discrimination, shift response, and engineering-quantity usefulness are separate properties; success on one does not guarantee success on another.In-distribution scaling corrects aggregate interval magnitude but cannot make uncertainty grow as quickly as error under shift.Source: §10 · p. 26
  • GP and sampling-based methods both remain useful, but their relative ordering changes by dataset and engineering question.GP total variance provides strong raw interval scale on DrivAerStar and crash; GP epistemic variance gives stronger AirFRANS epistemic OOD detection, while concrete MC dropout ranks crash mesh errors more strongly.Source: §10 · p. 26
  • Future work will study conformal prediction and structured-shift coverage, larger industrial datasets, and workflows where uncertainty guides design or simulation acquisition.The stated aim is to test whether uncertainty improves an engineering process, beyond post-prediction scoring.Source: §10 · p. 26

Appendix A Evaluated Configuration Details

  • Appendix A records UQ-specific settings and dataset-specific backbones, including GP heads with independent sparse variational GPs and a separate residual-variance network.The configurations vary in inputs, output fields, mesh handling, inducing-point counts, and parameter counts across DrivAerStar, AirFRANS, and crash.Source: Appendix A · pp. 29–30

Appendix B DrivAerStar Reproducibility Across Training Runs

Across three independent training runs, the DrivAerStar field-level conclusions persist, while drag predictions and marginal uncertainty rankings remain consistent. The association between drag uncertainty and absolute drag error is less stable.

Appendix B DrivAerStar Reproducibility Across Training Runs

  • Across three runs, concrete MC dropout retains lower pressure error, while GP-based UQ retains stronger field-error discrimination and lower total AUSE.Separate calibration brings both methods close to unit z-RMS on Fastback, but interval scale degrades on held-out body styles, more strongly for GP.Source: Appendix B · p. 30, Table 17 · p. 30
  • Predicted-versus-true drag Spearman correlation exceeds 0.86 for both methods across every body style and run; between-run predicted-drag and marginal drag-uncertainty rankings exceed 0.97 and 0.89, respectively.The uncertainty-to-absolute-drag-error association is less consistent because small changes in integrated field residuals can reorder absolute drag errors.Source: Appendix B · p. 31, Table 18 · p. 31
Loading 2609.25430v1…