Source-linked AI summary
Efficient Calibration for Imperfect Computer Models
Rui Tuo, C. F. Jeff Wu
TL;DR
The paper studies calibration of imperfect computer models when physical observations contain random error, extending earlier deterministic analyses. It proposes L2 calibration, establishes its efficiency under normal errors, and finds that ordinary least squares is consistent but inefficient, with numerical examples favoring the proposed method.
Problem
Calibration methods for imperfect computer models can yield unreasonable estimates, while prior theoretical analysis focused on deterministic physical responses rather than stochastic observations.
Method
The paper extends L2 calibration to stochastic physical data using nonparametric regression and studies ordinary least squares for comparison.
Results
The L2 calibration is semiparametrically efficient under normal measurement errors, while ordinary least squares is consistent but not efficient.
Takeaways & Limitations
The authors recommend L2 calibration because its higher estimation efficiency can reduce the physical trials needed for a given estimation efficiency.
Takeaways & Limitations
For non-normal error distributions, the defined L2 calibration is not semiparametrically efficient, although a parametric-error modification can restore efficiency.
Abstract
from arXiv · showhide
Many computer models contain unknown parameters which need to be estimated using physical observations. Kennedy and O'Hagan (2001) shows that the calibration method based on Gaussian process models proposed by Kennedy and O'Hagan (2001) may lead to unreasonable estimate for imperfect computer models. In this work, we extend their study to calibration problems with stochastic physical data. We propose a novel method, called the $L_2$ calibration, and show its semiparametric efficiency. The conventional method of the ordinary least squares is also studied. Theoretical analysis shows that it is consistent but not efficient. Numerical examples show that the proposed method outperforms the existing ones.
1. Introduction.
The paper addresses calibration when computer models are imperfect and physical observations are stochastic, extending prior theory beyond deterministic responses. It proposes an efficient L2 calibration method and contrasts it with ordinary least squares.
- Calibration motivation: Calibration estimates unknown computer-model parameters by matching simulated outputs with physical responses across model settings.The paper describes calibration as running the computer model and physical experiments, then finding parameters whose outputs match observed responses.
- Model uncertainty: Imperfect models cannot generally fit physical responses exactly because their simplifying assumptions may not hold in reality.This model uncertainty motivates methods that account for discrepancy between computer output and the physical process.
- Research gap: Prior Kennedy–O’Hagan calibration can produce unreasonable estimates for imperfect computer models, while existing theory largely assumed deterministic physical responses.The paper identifies stochastic physical experiments as the needed extension because real physical observations commonly contain measurement or observational error.
- Contributions: The proposed L2 calibration is semiparametrically efficient under normally distributed measurement error, whereas ordinary least squares is consistent but not efficient.The article presents both methods within a general calibration framework using nonparametric regression for physical outputs.
- Paper scope: The paper develops stochastic-system theory, studies ordinary least squares, and evaluates the proposed method in two numerical examples.The sections cover L2 calibration, asymptotic behavior, ordinary least squares, numerical examples, and concluding remarks.
2. L2 Projection for Systems with Stochastic Physical Experiments.
The paper defines calibration for stochastic physical experiments through an L2 projection that targets the closest computer-model response surface. It replaces deterministic interpolation with smoothing-based nonparametric regression and minimizes discrepancy between the fitted physical surface and an emulator of the computer code.
- Stochastic physical experiments: Stochastic physical experiments are modeled with random measurement or observational errors, unlike the deterministic setting used in earlier calibration theory.The setup assumes independent errors with zero mean and finite variance, with normality relaxed in parts of the analysis.
- L2 projection: The target calibration parameter θ* is the L2 projection minimizing the distance between the physical response surface and computer outputs.This definition addresses the unidentifiability of the “true” calibration parameter in the Kennedy–O’Hagan formulation.
- Nonparametric regression: The method estimates the physical response surface with a regularized nonparametric regressor because interpolation can overfit noisy observations.The smoothing parameter can be selected by criteria such as generalized cross-validation.
- Calibration construction: L2 calibration minimizes the discrepancy between the fitted physical response surface and an emulator of the deterministic computer code.The computer-code emulator may use radial basis functions, Gaussian processes, or polynomial chaos approximations.
- Calibration construction: The framework permits flexible computer-code emulators provided they approximate the simulator well.The paper lists radial basis function, Gaussian-process, and polynomial-chaos approximations as examples.
3. Asymptotic Results for L2 Calibration.
The L2 calibration estimator is shown to be consistent, asymptotically normal, and semiparametrically efficient under normal measurement errors, despite slower nonparametric estimation. These results rely on regularity conditions for the model, function class, tuning sequence, and emulator approximation.
- 3. Asymptotic Results for L2 Calibration: The analysis uses empirical-process weak-convergence tools because interpolation theory does not apply when physical responses contain random error.The proof framework includes Donsker properties and asymptotic equicontinuity for relevant function classes.
- 3. Asymptotic Results for L2 Calibration: The asymptotic theory assumes independent random design and errors, regularity conditions on the calibration model, controlled nonparametric estimation, and negligible emulator error.The emulator approximation is assumed smaller than the physical-experiment measurement-error estimation error.
- 3.2. Asymptotic Results for θ̂L2: The L2 calibration estimator converges at rate Op(n^-1/2), and its scaled error is asymptotically normal under the stated conditions.This is nontrivial because the nonparametric component generally converges more slowly than n^-1/2.
- 3.3. Semiparametric Efficiency: Under normal measurement errors, L2 calibration is semiparametrically efficient.The result follows because its asymptotic expression matches the maximum-likelihood estimator for the corresponding parametric model.
- 3.3. Semiparametric Efficiency: For non-normal error distributions, ordinary L2 calibration is not semiparametrically efficient, although a likelihood-based modification can recover efficiency under regularity conditions.The paper distinguishes the normal-error result from the modified procedure for parametric error models.
4. Ordinary Least Squares.
This section studies ordinary least squares calibration for imperfect computer models, establishing consistency while comparing its efficiency and computational trade-offs with L2 calibration.
- OLS calibration is consistent even when the computer code is imperfect.
- OLS calibration is computationally more efficient and requires no tuning parameter, unlike L2 calibration.L2 calibration requires selecting the tuning parameter λ.
- The asymptotic variance of OLS does not attain the semiparametric lower bound achieved by the L2 analysis.
- OLS and L2 calibration have equal asymptotic distributions only when a perfect computer model exists under the stated derivative condition.
- Because practical discrepancies between the physical process and optimal computer output can be large, OLS is generally less efficient than L2 calibration.
5. Numerical Studies.
The numerical studies compare L2, OLS, and KO calibration in perfect and imperfect computer-model settings. In the imperfect setting, L2 and OLS outperform KO, while L2 has lower variability than OLS.
- 5.1. Example 1: perfect computer model: For the perfect computer model, all three methods give good estimation results, consistent with KO consistency under perfect models.Table 1 reports mean values and MSEs over 1000 simulations for σ2 = 0.1 and σ2 = 1.
- 5.2. Example 2: imperfect computer model: The imperfect-model example defines the L2 target as the minimizer of an explicit discrepancy function, numerically θ∗≈−0.1789.
- 5.2. Example 2: imperfect computer model: L2 and OLS calibration outperform KO calibration for the imperfect computer model.This comparison is reported from 1000 simulations in Table 2.
- 5.2. Example 2: imperfect computer model: L2 calibration has smaller standard deviation than OLS calibration in the imperfect-model simulations.The result agrees with the theoretical conclusion that L2 calibration is more efficient than OLS calibration for imperfect computer models.
- 5.2. Example 2: imperfect computer model: The mean value of the KO estimator changes substantially as σ2 changes, unlike the desired stability of a good estimator under large samples.
6. Concluding Remarks and Further Discussions.
The work extends calibration theory to stochastic physical systems, establishes efficiency results for L2 calibration, and compares it with OLS. It also identifies scope limits involving the choice of norm and random-design assumptions.
- The framework is extended from deterministic to stochastic physical systems using weak-convergence tools, including empirical-process limiting theory.
- L2 calibration is asymptotically normal and semiparametrically efficient, whereas OLS is consistent but inefficient.The authors recommend L2 calibration despite its higher computational cost because it can reduce the number of physical trials needed for a given estimation efficiency.
- Calibration is defined as finding the L2 projection, the parameter minimizing discrepancy between the true process and computer output under the L2 norm.The identified calibration value therefore depends on the chosen norm.
- The main results extend to norms equivalent to L2, but the proof does not apply to nonequivalent norms such as the L∞ norm.The authors state that convergence rates of O(n^-1/2) may not exist for such norms and leave this issue for future work.
- The reported asymptotic results assume random designs with independently uniform-sampled inputs, leaving fixed-design calibration for further investigation.