PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantitative model validation techniques: new insights

You Ling, Sankaran Mahadevan

arXiv:1206.5014v1physics.data-anstat.ME

TL;DR

The paper addresses unresolved validation questions, including how to handle different experimental-data and prediction settings, directional bias, and metric thresholds. It formulates Bayesian interval testing and examines validation metrics; among its reported results, z-test and reliability-based failure percentages coincide under a specified threshold choice.

  • Problem

    Model-validation practice leaves unclear how to handle characterized versus partially characterized data, deterministic versus stochastic predictions, directional bias, and metric thresholds.

    Source: §1 · p. 2

  • Method

    The paper formulates Bayesian interval hypothesis testing to assess the accuracy of predicted means and standard deviations.

    Source: §6 · p. 36

  • Results

    The z-test and reliability-based metric give the same failure percentages when rth is selected based on the z-test significance level α.

    Source: §5.4 · p. 35

  • Takeaways & Limitations

    Model-selection risk can be addressed through a risk-based threshold and subsequent model averaging.

    Source: §6 · p. 36

    The area-based metric lacks a direct interpretation for its results.

    Source: §6 · p. 36

Abstract

from arXiv · show

This paper develops new insights into quantitative methods for the validation of computational model prediction. Four types of methods are investigated, namely classical and Bayesian hypothesis testing, a reliability-based method, and an area metric-based method. Traditional Bayesian hypothesis testing is extended based on interval hypotheses on distribution parameters and equality hypotheses on probability distributions, in order to validate models with deterministic/stochastic output for given inputs. Two types of validation experiments are considered - fully characterized (all the model/experimental inputs are measured and reported as point values) and partially characterized (some of the model/experimental inputs are not measured or are reported as intervals). Bayesian hypothesis testing can minimize the risk in model selection by properly choosing the model acceptance threshold, and its results can be used in model averaging to avoid Type I/II errors. It is shown that Bayesian interval hypothesis testing, the reliability-based method, and the area metric-based method can account for the existence of directional bias, where the mean predictions of a numerical model may be consistently below or above the corresponding experimental observations. It is also found that under some specific conditions, the Bayes factor metric in Bayesian equality hypothesis testing and the reliability-based metric can both be mathematically related to the p-value metric in classical hypothesis testing. Numerical studies are conducted to apply the above validation methods to gas damping prediction for radio frequency (RF) microelectromechanical system (MEMS) switches. The model of interest is a general polynomial chaos (gPC) surrogate model constructed based on expensive runs of a physics-based simulation model, and validation data are collected from fully characterized experiments.

1. Introduction

The paper addresses unresolved choices in quantitative model validation across experimental-data characterization, deterministic or stochastic predictions, directional bias, and metric thresholds. It investigates extensions to validation methods and their relationships, with a MEMS damping model as an application.

1. Introduction

  • The paper identifies four unresolved practice issues: characterization of experimental data, deterministic versus stochastic predictions, directional bias, and validation-metric thresholds.These issues motivate examining validation across different data and model settings.Source: §1 · p. 2
  • Under certain conditions, some validation metrics are mathematically related, potentially informing metric and acceptance-threshold choices.The paper investigates relationships among classical hypothesis-testing, Bayesian hypothesis-testing, and reliability-based metrics.Source: §1 · p. 4
  • The study extends Bayesian hypothesis testing to assess predicted means and standard deviations or entire probability distributions, for fully or partially characterized experiments.The formulations are presented for both experiment types; other validation methods are also considered with both types of data.Source: §1 · p. 4
  • Bayesian interval testing, a modified reliability-based method, and the area metric-based method are investigated for accounting for directional bias.Directional bias is defined as bias whose direction remains unchanged as model and experimental inputs vary.Source: §1 · pp. 3–4
  • The numerical example validates a generalized polynomial chaos surrogate for MEMS switch squeeze-film damping, replacing expensive microscale fluid simulations.The surrogate is used to expedite probabilistic analysis of the MEMS device.Source: §1 · pp. 4–5

2. Quantitative validation of model prediction

This section frames quantitative validation as comparing model predictions with experimental observations under uncertainty. It distinguishes fully and partially characterized experiments, describes the data-quality scope, and outlines method selection and threshold-based decisions.

2. Quantitative validation of model prediction

  • Fully characterized experiments report measured model or experimental inputs as point values, whereas partially characterized experiments leave some inputs unmeasured or report intervals.For partially characterized data, the paper represents epistemic uncertainty about inputs probabilistically and treats model and system outputs as distributions.Source: §2 · p. 5
  • Experimental-observation uncertainty is separated into uncertainty in the physical-system response and measurement uncertainty, with YD = Y + εD.The paper restricts its discussion to cases where uncertainty in measured inputs and outputs is limited; poor-quality data can provide little useful information about model validity.Source: §2 · p. 6
  • The paper considers classical hypothesis testing, Bayesian interval and equality testing, reliability-based validation, and an area metric-based method.The table summarizes method applicability across the validation scenarios discussed in the section.Source: §2 · p. 6, Table 1 · p. 6
  • After selecting a method and computing its metric, model acceptance or rejection depends on the metric and a chosen threshold.The section presents a flowchart as a systematic procedure for quantitative model validation.Source: §2 · p. 6, Figure 1 · p. 7

3. Hypothesis testing-based methods

This section examines classical and Bayesian hypothesis-testing methods for validating model predictions as model-selection decisions, with hypotheses and acceptance criteria grounded in domain knowledge. It covers classical p-value and confidence-interval approaches and Bayesian tests based on distribution-parameter intervals or equality of predicted and observed distributions, including directional bias and fully or partially characterized experiments.

3. Hypothesis testing-based methods

  • Domain knowledge is needed to formulate H0 and H1, choose an acceptance threshold, and decide whether correctness concerns selected moments or the full output distribution.For stochastic model predictions, the intended physical meaning of “model being correct” must be specified.Source: §3 · p. 7

3.1. Classical hypothesis testing

  • Classical testing derives a statistic’s null and alternative distributions, computes its p-value from validation data, and rejects the model when the p-value is below α.The significance level α limits the probability of a Type I error; decisions also depend on the decision maker’s acceptable Type II error probability.Source: §3.1 · p. 8
  • Failing to reject H0 indicates acceptable accuracy, not proof that the model is true, and p-values tend to shrink with larger samples.The paper warns that over-interpreting p-values and significance testing can mislead model validation.Source: §3.1 · pp. 8–9
  • For mean-based validation, the t-test requires at least two samples; with one sample, the z-test assumes the output standard deviation equals the model-prediction standard deviation.Both test the null hypothesis that the output mean equals the model prediction.Source: §3.1 · pp. 9–10
  • For partially characterized experiments, classical tests compare a sample set sharing the same input intervals or distributions with the model’s unconditional output mean.The unconditional distribution is obtained by propagating input uncertainty to the output; fully characterized experiments allow direct comparison at given inputs.Source: §3.1 · p. 11

3.2. Bayesian hypothesis testing

  • Bayesian tests compare intervals on prediction and observation means and standard deviations, or directly test equality of their probability distributions.The parameter-interval formulation can also be used when either quantity is deterministic; the PDF-equality formulation applies only when both are stochastic.Source: §3.2 · pp. 12–13
  • Two one-sided interval tests capture directional bias: the model fails the combined test if either null hypothesis fails.Together, the tests cover cases where the model mean is below or above the observed mean, increasing the chance of failure when directional bias exists.Source: §3.2 · pp. 14–15
  • In the PDF-equality test, the alternative may use a non-informative uniform distribution, but its bounds affect the estimated Bayes factor.The bounds should therefore be selected carefully using available information.Source: §3.2 · p. 15
  • For partially characterized experiments, input uncertainty is represented by an input PDF and propagated to obtain the model prediction distribution.For fully characterized experiments, the tests instead use predictions conditioned on the measured point-value inputs.Source: §3.2 · p. 16
  • Appropriate selection of the Bayes-factor threshold can reduce Bayes risk; Bth = 1 follows when error costs and prior probabilities are equal.Bayes risk sums the costs of decision errors, and the threshold choice is part of the decision-making process.Source: §3.2 · p. 17
  • Using posterior probabilities for both hypotheses in Bayesian model averaging carries validation uncertainty into model predictions and helps avoid Type I and Type II errors.Rather than completely accepting one model, the approach combines the predictive PDFs under the null and alternative hypotheses.Source: §3.2 · p. 18

3.3. Relationship between p-value and Bayes factor

  • For fully characterized experiments with normally distributed predictions and negligible measurement noise, Bayes factors relate to t-test and z-test p-values, and matched thresholds yield the same validation decision.The noise condition is that measurement-noise variance is negligible compared with the variance of the model prediction.Source: §3.3 · pp. 18–20

4. Non-hypothesis testing-based methods

The section presents reliability- and area-based alternatives to binary hypothesis testing for assessing agreement between model predictions and experimental observations. It relates reliability to classical test metrics and describes how both approaches can account for directional bias, with distinct interpretability and applicability constraints.

4.1. Reliability-based metric

  • The reliability metric r measures the probability that the prediction–observation difference falls within a specified tolerance interval.It is described as a direct measure of prediction quality and is relatively easy to compute.Source: §4.1 · p. 21
  • Under stated assumptions, r relates mathematically to the t- or z-test statistic and p-value, and choosing its threshold from p = α yields the same validation decision.The relationship depends on the assumptions used to derive the metric from the test statistic.Source: §4.1 · pp. 21–22
  • Directional bias can be assessed by splitting the tolerance interval into [0, ϵ] and [−ϵ, 0] and comparing each reliability value with r_th/2.The model fails if either interval’s value is below that half-threshold.Source: §4.1 · pp. 22–23

4.2. Area metric-based method

  • The area metric compares the empirical CDF of transformed observations with the standard uniform CDF; a smaller discrepancy indicates closer agreement.Under the probability integral transform, observations sampled from the model-output distribution produce uniform u-values.Source: §4.2 · p. 23
  • The area metric can reflect directional bias because consistently low or high observations shift the model-output CDF values away from uniformity.For Gaussian model outputs, observations below the corresponding means yield u-values below 0.5.Source: §4.2 · p. 24
  • The area metric lacks a direct interpretation for setting an acceptance threshold and cannot test computational models with deterministic output.The authors also note that a confidence-interval method they discuss is unclear to apply to multivariate models with discrete test combinations because it requires integration over test inputs.Source: §4 · p. 20, §4.2 · pp. 23–24

5. Numerical example

The numerical example applies the validation methods to MEMS-switch damping prediction. Its fully characterized validation data show directional bias between the model predictions and observations.

5. Numerical example

  • The example applies the validation methods to damping prediction for MEMS switches, treating the damping coefficient as a random variable because of limited physical-model understanding.The computational model is also stochastic.Source: §5 · p. 24
  • The example represents epistemic uncertainty in the damping coefficient with a subjective probability distribution.Source: §5 · p. 24
  • The validation experiments are fully characterized, and the model predictions show directional bias relative to the validation data.The passage identifies the existence of bias but does not specify its direction here.Source: §5 · p. 24

5.1. Damping model and experimental data

The example constructs a gPC surrogate for squeeze-film damping prediction and compares it with fully characterized experiments. The graphical comparison reports better performance at middle pressures and a consistent overprediction at the lowest pressure.

5.1.1. Uncertainty quantification in micro-scale squeeze-film damping prediction

  • The surrogate is a third-order gPC model with Legendre polynomial bases, constructed from Navier–Stokes Slip Jump model runs to reduce the cost of repeated simulations.Membrane thickness, gap height, and frequency are identified as the main sources of damping-coefficient variability.Source: §5.1.1 · pp. 25–26
  • The model’s damping-coefficient prediction at a given input is represented as a Gaussian random variable.The gPC residual is treated as random under the Gauss–Markov assumption.Source: §5.1.1 · p. 26
  • The example assesses only the surrogate model because the original model is inaccessible, and surrogate validity does not guarantee validity of the original model.Source: §5.1.1 · p. 26

5.1.2. Experimental data for validation

  • The validation set contains 140 observations from seven devices and four pressures, with inputs recorded for every observation, making the experiments fully characterized.There is one observation per test combination because the 140 input combinations differ.Source: §5.1.2 · p. 27
  • The gPC model performs better at the middle two pressures, while at 18798.45 Pa its mean predictions are consistently larger than the experimental data.This is the directional bias reported for the lowest pressure.Source: §5.1.2 · p. 27, Figure 4 · p. 28

5.2. Validation based on binary hypothesis testing

The numerical validation applies a z-test and two Bayesian hypothesis tests to the gPC models. The reported comparisons include pressure-specific performance differences and a threshold choice that makes Bayesian failure counts coincide with the z-test.

5.2.1. Classical hypothesis testing

  • Because each experimental combination has one observation, the classical analysis uses a z-test rather than a t-test and marks p-values below α = 0.05 as failures.The paper notes that a more rigorous test should include the probability of Type II error.Source: §5.2.1 · p. 29

5.2.2. Bayesian hypothesis testing

  • The interval-based Bayesian test sets distribution-parameter intervals to reflect the selected validation strictness and the under-damped device setting.For illustration, the authors specify ϵ_µ = 0.025, ϵ_σ1 = −0.005, and ϵ_σ2 = 0.005.Source: §5.2.2 · p. 30
  • The equality-based Bayesian test evaluates gPC performance using a Bayes factor with threshold B_th = 1.Its reported results are shown in Figure 7 and Table 4.Source: §5.2.2 · p. 31, Table 4 · p. 32
  • Raising the Bayes-factor threshold from 1 to 2.88 makes the Bayesian failure counts match those of the z-test.The cited threshold is calculated using p = 0.05; the second model’s performance remains unchanged as the threshold rises.Source: §5.2.2 · p. 31
  • The interval- and equality-based Bayesian tests both show worse performance for the 18798.45 Pa model in the first test, while the other three pressure models have similar failure percentages across tests.The passage links the worse first-test performance to directional bias at 18798.45 Pa.Source: §5.2.2 · pp. 32–33

5.3. Validation using non-hypothesis testing-based methods

The reliability-based metric matches z-test failure-percentage decisions under a threshold based on the significance level, while directional bias worsens reliability- and area-based results.

5.3.1. Reliability-based metric

  • The reliability-based metric gives the same failure-percentage decisions as the z-test when its decision threshold is based on the z-test significance level.The comparison uses r against r_th; comparing r1 and r2 with r_th/2 reveals a higher failure percentage for the 18798.45 Pa gPC model because of directional bias.Source: §5.3.1 · p. 33

5.3.2. Area metric-based method

  • The gPC model at 18798.45 Pa has the highest area value and performs worst, reflecting directional bias between mean predictions and experimental data.The area results are presented in Figure 9 and Table 6.Source: §5.3.2 · p. 33

5.4. Discussion

The gPC model agrees better with observations at the middle two pressures than at the lowest and highest pressures. The discussion also relates test decisions under selected thresholds and shows that bias-sensitive methods penalize directional bias more strongly.

5.4. Discussion

  • The gPC model agrees better with observations at 28664.31 Pa and 43596.41 Pa than at 18798.45 Pa and 66661.19 Pa.The validation example uses 140 fully characterized experimental data points.Source: §5.4 · p. 34
  • The reliability-based metric gives the same failure-percentage results as the z-test when its threshold is selected using the z-test significance level.The discussion also reports that classical and Bayesian hypothesis testing can give the same result with a specific threshold.Source: §5.4 · p. 35
  • Directional bias is reflected by Bayesian interval testing, the reliability method with modified intervals, and the area metric; these methods rate biased models worse than classical and PDF-equality tests.The comparison is stated for models with directional bias.Source: §5.4 · p. 35

6. Conclusion

The conclusion presents extensions to Bayesian hypothesis testing and evaluates classical, Bayesian, reliability-based, and area-based validation methods. It reports risk-reduction strategies for Bayesian testing, special-case metric relationships, and differing sensitivity to directional bias.

6. Conclusion

  • Bayesian interval hypothesis testing evaluates predicted means and standard deviations while accounting for directional bias, and a second formulation tests the full prediction PDF.The two formulations apply to fully and partially characterized experiments.Source: §6 · p. 36
  • Bayesian hypothesis testing can reduce model-selection risk through a risk-based acceptance threshold and subsequent model averaging with posterior probabilities.The stated risk includes Type I and Type II errors.Source: §6 · p. 36
  • Under some conditions, classical-test p-values are mathematically related to the Bayes factor and reliability-based metric.The conclusion specifies p-values from z-tests or t-tests.Source: §6 · p. 36
  • The area metric and reliability-based method are sensitive to directional bias, and biased models perform worse under them than under classical or PDF-equality testing.Bayesian model-validation results and reliability metrics can be incorporated into long-term device failure and reliability analysis, explicitly accounting for model uncertainty.Source: §6 · p. 36
  • The area-based metric lacks a direct interpretation for its results.Source: §6 · p. 36
  • Bayesian model validation can be extended to validation data given as intervals through its use of a likelihood function.The conclusion cites prior demonstrations of this extension.Source: §6 · p. 36
Loading 1206.5014v1…