Source-linked AI summary
A theoretical framework for calibration in computer models: parametrization, estimation and convergence properties
Rui Tuo, C. F. Jeff Wu
TL;DR
Calibration parameters may be unmeasurable in physical experiments, and their identifiability and estimation require a theoretical framework. The paper analyzes KO calibration, finds its simplified form L2-inconsistent, and proposes a consistent L2 calibration method with optimal convergence rate.
Problem
Calibration parameters can be unmeasurable in physical experiments, while the KO approach lacked a theoretical framework for studying their identifiability and estimation.
Source: §1 · pp. 1–2Method
The paper defines L2-consistency for calibration, analyzes simplified KO estimation, modifies its correlation scale, and proposes least L2 distance calibration.
Source: §1 · pp. 2–3, §5.1 · p. 13Results
Simplified KO calibration is asymptotically L2-inconsistent, while least L2 distance calibration is proven L2-consistent with optimal convergence rate for cheap codes and extended to expensive codes at a slower rate.
Source: §1 · p. 3Takeaways & Limitations
L2-consistency provides the paper's justification criterion for calibration methods, and its proposed L2 calibration meets that criterion with an optimal rate for cheap codes.
Source: §1 · pp. 2–3Modified KO calibration requires a prespecified scale sequence, with no theoretical guidance for choosing its value at a fixed sample size.
Source: §5.1 · p. 14
Abstract
from arXiv · showhide
Calibration parameters in deterministic computer experiments are those attributes that cannot be measured or available in physical experiments. Kennedy and O'Hagan \cite{kennedy2001bayesian} suggested an approach to estimate them by using data from physical experiments and computer simulations. A theoretical framework is given which allows us to study the issues of parameter identifiability and estimation. We define the $L_2$-consistency for calibration as a justification for calibration methods. It is shown that a simplified version of the original KO method leads to asymptotically $L_2$-inconsistent calibration. This $L_2$-inconsistency can be remedied by modifying the original estimation procedure. A novel calibration method, called the $L_2$ calibration, is proposed and proven to be $L_2$-consistent and enjoys optimal convergence rate. A numerical example and some mathematical analysis are used to illustrate the source of the $L_2$-inconsistency problem.
1. Introduction.
The paper develops a theoretical framework for calibration when computer-code inputs cannot be measured in physical experiments, addressing identifiability and estimation. It shows that simplified KO calibration can be L2-inconsistent and presents modifications and a new method with consistency guarantees.
1. Introduction.
- Calibration parameters are unknown or unmeasured computer-code inputs, so their values cannot be estimated from physical responses alone.Computer runs at selected parameter values are combined with physical data to make inferences about them.Source: §1 · pp. 1–2
- The spot-welding example treats load and current as control variables and unmeasurable contact resistance as a calibration parameter used in the ANSYS code.Nugget size is the response used as a measure of weld strength.Source: §1 · p. 2
- The framework studies parametrization, estimation, and modeling in the influential Kennedy–O’Hagan formulation, for which the paper identifies a lack of prior theoretical research.The approach had been applied across fields including hydrology, spot welding, and climate prediction.Source: §1 · p. 2
- Simplified KO calibration is asymptotically L2-inconsistent, while modifying its correlation function can restore L2-consistency when its scale parameter grows at a specified rate.The paper also uses a numerical example and mathematical analysis to illustrate the inconsistency and its source.Source: §1 · pp. 2–3
- Least L2 distance calibration is proven L2-consistent with optimal convergence rate for cheap codes and is extended to expensive codes, where its rate is slower.For expensive codes, the function associated with the code must be estimated from data.Source: §1 · p. 3
2. Preliminaries.
The preliminaries introduce kernel interpolation as a functional approximation method for scattered designs and relate it to Gaussian process prediction. They describe common kernel choices and note that cross-validation can guide selection.
2. Preliminaries.
- Kernel interpolation approximates a smooth function from observed values and is suited to scattered design points without regular structure.The construction begins by choosing a symmetric positive definite kernel.Source: §2 · p. 3
- Gaussian and Matérn kernels are common choices, and cross-validation is a common way to select a suitable kernel.Stationary kernels depend on the difference between their inputs; radial basis kernels depend on the distance between them.Source: §2 · pp. 3–4
- The interpolation coefficients have a unique solution because the kernel matrix is positive definite, and the resulting function interpolates the observed values.The coefficients are given by u = Φ^-1Y.Source: §2 · pp. 3–4
- The kernel interpolator has the same form as the predictive mean of a zero-mean Gaussian process with the corresponding covariance function.The paper presents this as a known relationship between kernel interpolation and Gaussian process prediction.Source: §2 · p. 4
3. Calibration Problem.
The section defines calibration settings for physical systems and computer codes, then examines KO modeling and parameter identifiability. Because the KO parameter is unidentifiable, it defines an L2 projection as a target and reports that KO calibration is L2-inconsistent.
3. Calibration Problem.
- Calibration uses physical responses to adjust computer outputs when code inputs represent system attributes that cannot be controlled or measured physically.The code takes control variables and calibration variables; a code is called cheap when its functional form is known and expensive when it must be estimated from runs.Source: §3 · p. 4
3.1. Kennedy-O’Hagan Method.
- In the KO formulation, the physical response is modeled through the computer output and a discrepancy function, with Gaussian processes used for the code and discrepancy in the original approach.The original KO method was proposed as a Bayesian framework for stochastic systems with expensive codes.Source: §3.1 · p. 5
3.2. L2 Distance Projection.
- From a frequentist perspective, the KO “true” parameter is unidentifiable because the parameter and discrepancy cannot be uniquely determined even when both response functions are known.The analysis treats the physical system as deterministic.Source: §3.2 · p. 5
- The paper defines θ* as the parameter minimizing the L2 discrepancy between physical and computer observations, assuming a unique global minimizer.This projection minimizes average predictive error, and its value depends on the distance norm used.Source: §3.2 · p. 6
- An estimator is L2-consistent when it converges to θ* as the physical or joint physical-and-computer designs become dense.For expensive codes, the estimator uses observed code values on a separate computer design.Source: §3.2 · p. 6
- The theoretical analysis finds KO calibration L2-inconsistent and gives examples in which it may produce unreasonable answers.Source: §4.1 · p. 7
4. Frequentist Properties of the Kennedy-O’Hagan Model.
The simplified frequentist KO procedure can converge to a parameter different from the L2 projection, making it L2-inconsistent; examples show likelihood-based code choices disagreeing with L2 comparisons. Estimating the correlation parameter does not remove the illustrated misranking, though relatively large values give the correct L2-discrepancy order.
4.1. Simplified KO calibration.
- Because θ′ generally differs from θ*, simplified KO calibration is not L2-consistent under the paper’s L2-projection definition.The estimator converges to θ′, rather than the L2 projection θ*.Source: §4.2 · p. 9
- Functions can have arbitrarily small L2 norm while their native-space norm stays bounded away from zero, allowing KO results far from the L2 projection.For Matérn kernels, the native-space norm is equivalent to a Sobolev norm that can penalize wild oscillation despite small L2 discrepancy; the paper says this issue is more serious for Gaussian kernels.Source: §4.2 · pp. 9–11
- 12.594 (code 1), 57.908 (code 2), and 17978.65 (code 3) are the PSS values, but the L2 projection selects code 3.The example says code 3 has smaller pointwise error than the first two, while likelihood-based calibration selects code 1.Source: §4.2 · pp. 9–10
4.3. Numerical Study on Kennedy-O’Hagan Method with Estimated Correlation Func-
- Even when the correlation function is estimated, the frequentist KO method retains the Example 1 problem and selects ε1 in the continued example.The maximum log-likelihood occurs at l(1, 1).Source: §4.3 · p. 12
- At very small φ, the correlation matrix becomes nearly singular, preventing likelihood calculation; unbounded likelihood as φ decreases is only conjectured.The authors state they cannot prove the likelihood is unbounded.Source: §4.3 · p. 12
- For φ > 4, likelihood values give the correct order of L2 discrepancy in the example.The authors report that this is not occasional and say a modified KO model will be proved L2-consistent.Source: §4.3 · p. 13
5. Asymptotic Results: Cheap Code.
For cheap computer code, the section studies asymptotic calibration properties and addresses the L2-inconsistency of fixed-correlation KO calibration. It introduces a modified KO procedure and least L2 distance calibration, comparing their consistency and convergence guarantees.
5. Asymptotic Results: Cheap Code.
- The convergence-rate analysis in this section assumes cheap code, meaning the computer code can be evaluated without cost.It follows the paper’s initial asymptotic result.Source: §5 · p. 13
5.1. Modified KO Calibration.
- Modified KO calibration is L2-consistent when its scale sequence φn increases to infinity while φn h(Dn) tends to zero, under the stated regularity conditions.The correlation function uses a scale parameter to approach the Dirac delta needed for L2-consistency.Source: §5.1 · pp. 13–14
- Modified KO calibration requires a prespecified φn sequence, with no theoretical guidance for choosing φ at fixed sample size; least L2 calibration avoids choosing a kernel function.The section presents least L2 distance calibration as the more practical procedure.Source: §5.1 · p. 14
5.3. Least L2 Distance Calibration.
- Least L2 distance calibration converges to the L2 projection θ* at the optimal rate under Theorem 5.4’s conditions.The result assumes θ* is the unique solution and an interior point of Θ, among other conditions.Source: §5.3 · pp. 14–15
- Under similar conditions, KO and L2 calibration converge at O(h^2k(Dn)); L2 calibration also converges under less restrictive conditions, at an optimal but slower rate.The slower rate matches the convergence rate of the interpolator.Source: §5.3 · p. 15
6. Least L2 Distance Calibration for Expensive Code.
For expensive computer code, least L2 distance calibration replaces the unavailable true computer output with an interpolant. The resulting convergence analysis is slower than for cheap code because the computer output must also be estimated.
6. Least L2 Distance Calibration for Expensive Code.
- For expensive code, the L2 calibration uses an interpolant of the computer output in place of the true output.The computer input space is Ω×Θ, and the surrogate is built from a design G.Source: §6 · p. 15
- The interpolant-based target θ*(G) can be viewed as an L2 projection, and its gap from θ* has a slower rate than in Theorem 5.4.The calibration result applies under Theorem 5.4’s conditions; the expensive-code analysis also bounds the target gap.Source: §6 · pp. 15–16
- The expensive-code convergence result is stated under the assumptions of Theorems 5.4 and 6.1.Theorem 6.1’s assumptions include smoothness conditions on Ψ and a function-space condition on ys.Source: §6 · p. 16
7. Further Discussions and Remarks.
The paper develops a theoretical framework for calibration and discusses its scope, extensions, and alternatives. It explains the KO method’s L2 inconsistency, compares other calibration approaches, and describes convergence-rate implications.
7. Further Discussions and Remarks.
- The framework defines L2 consistency using θ* as the true calibration parameter, while other positive-definite metrics or Lp norms could define alternatives.A Sobolev norm may suit settings where predictor oscillation or system energy matters, but efficient estimation in that framework requires separate investigation.Source: §7 · p. 16
- The KO method can still predict the physical response well despite its L2-inconsistent calibration, because the discrepancy function it approximates may be easier to estimate.The paper relates this to the smaller native-space norm of the discrepancy function, even when its pointwise absolute values are greater.Source: §7 · p. 16
- Extensions to expensive-code KO calibration and the original Bayesian formulation remain unresolved; the paper only heuristically suggests the prior has no asymptotic effect when θ′ lies in its support.The authors also note that relaxing the known-Φ assumption requires more rigorous analysis; extensions are deferred to future work.Source: §7 · pp. 16–17
- For infinitely differentiable kernels such as the Gaussian covariance kernel, convergence can be faster than any power of fill distance, and Gaussian kernels can yield an exponential rate.The paper attributes this to the interpolation error estimate for Gaussian kernels in.Source: §7 · p. 17
- The analysis assumes deterministic physical responses; noisy observations may undermine Theorem 4.2, and the paper says deterministic experiments are necessary for its KO asymptotic theory.The cited stochastic-system work uses different tools and proposes smoothing-spline calibration with semiparametric efficiency.Source: §7 · p. 17
- OLS is L2-consistent for independent uniform random designs or quasi-Monte Carlo designs, but does not attain the L2 calibration method’s convergence rate.For physical experiments with measurement error, a modified L2 calibration is also more efficient than OLS.Source: §7 · p. 17
Appendix A. Reproducing Kernel Hilbert Spaces.
The appendix collects reproducing-kernel Hilbert space properties used in the paper’s analysis, including extension, restriction, Fourier representations, and interpolation error results. It also identifies a Matérn RKHS with a Sobolev space and introduces the technical proofs in Appendix B.
Appendix A. Reproducing Kernel Hilbert Spaces.
- The appendix presents kernel-interpolation error estimates under smoothness assumptions, with fill distance used to characterize the design points.Source: Appendix A · p. 18
- For continuous, integrable stationary kernels, the RKHS can be represented using Fourier transforms of the kernel.Source: Appendix A · pp. 18–19
- For Matérn correlation functions, the RKHS equals the Sobolev space H^(ν+d/2)(Ω) with equivalent norms when [ν+d/2] > d/2.Source: Appendix A · p. 19
Appendix B. Technical Proofs.
- Appendix B provides formal proofs for Theorems 4.2, 5.2, 5.3, 5.4, and 6.1.Source: Appendix B · p. 19
B.1. Proof of Theorem 4.2.
The proof reduces Theorem 4.2 to showing that the kernel interpolator’s L2 error vanishes uniformly over calibration parameters. The resulting bound tends to zero as the design grows.
B.1. Proof of Theorem 4.2.
- It is sufficient to establish that the kernel interpolator for ε under design D_n has vanishing L2 norm, as set up in the proof.Source: §B.1 · p. 19
- The proof uses the identity involving the native-space inner product to bound the interpolator error by a term that tends to zero independently of θ.Source: §B.1 · p. 20
B.2. Proof of Theorem 5.2.
The proof establishes uniform approximation of the discrepancy function by a scale-indexed kernel operator, then combines that approximation with interpolation bounds. Under the stated regularity conditions and φ_n h(D_n)→0, the resulting error vanishes.
B.2. Proof of Theorem 5.2.
- The native-space norm comparison for φ>1 supports the bounds used to control the kernel approximation.Source: §B.2 · p. 20
- As φ grows, the kernel operator κ_φ approximates ε uniformly over θ, providing the central approximation step in the proof.Source: §B.2 · p. 22
- The interpolation bound yields the limiting result under conditions A2 and A3 when φ_n h(D_n)→0.Source: §B.2 · pp. 22–23
B.3. Proof of Theorem 5.3.
The proof uses an intermediate parameter converging to θ′ and bounds that vanish as h(Dn) tends to zero to establish its desired result.
B.3. Proof of Theorem 5.3.
- ˜θn lies between θ′ and ˆθKO(Dn) and converges to θ′ by Theorem 4.2.Source: §B.3 · p. 24
- NΦ(Ω) is invertible under assumption (5.4) once ˜θn converges to θ′.Source: §B.3 · p. 24
- The proof obtains its desired result by combining bounds whose limiting terms vanish when h(Dn) tends to zero.The argument invokes (B.13)–(B.15), then combines (B.16), (B.18), and (B.19).Source: §B.3 · pp. 24–25
B.4. Proof of Theorem 5.4.
The proof of Theorem 5.4 establishes L2-consistency and derives a convergence rate; the proof of Theorem 6.1 uses a parallel argument with bounds as h(Gn) tends to zero.
B.4. Proof of Theorem 5.4.
- ˆθL2(Dn) is L2-consistent because h(Dn) → 0 yields uniform convergence, with compactness of Ω giving the required L2 bounds.Source: §B.4 · p. 25
- Theorem 5.4’s convergence-rate proof uses an invertible matrix and combines its intermediate bounds to obtain (5.6) and (5.7).The argument follows a similar approach to Theorem 5.3.Source: §B.4 · pp. 25–26
B.5. Proof of Theorem 6.1.
- Theorem 6.1’s proof uses convergence of θ*(Gn) to θ* and an argument parallel to Theorems 5.3 and 5.4, with bounds relying on h(Gn) → 0.Source: §B.5 · pp. 26–27