Sequential design of computer experiments for the estimation of a probability of failure

Julien Bect; David Ginsbourger; Ling Li; Victor Picheny; Emmanuel Vazquez

Source-linked AI summary

Sequential design of computer experiments for the estimation of a probability of failure

Julien Bect, David Ginsbourger, Ling Li, Victor Picheny, Emmanuel Vazquez

arXiv:1009.5177v2Sep 27, 2010stat.COstat.AP

TL;DR

The paper studies estimating the probability mass of a threshold excursion set when model evaluations are expensive and simulation budgets are limited. It derives Bayesian decision-theoretic SUR strategies using Gaussian-process models, then compares them with existing kriging-based strategies. The paper presents numerical illustrations and empirical comparisons of these sampling criteria.

Problem
The paper asks how to estimate a system’s failure probability, represented by the probability mass of f’s excursion set above a fixed threshold, when evaluations are expensive and limited.
Method
The paper derives sequential SUR strategies from a Bayesian decision-theoretic framework and uses Gaussian-process and kriging models to guide evaluations of f.
Results
The paper provides a unified review of kriging-based strategies and compares them numerically with the proposed SUR strategies.
Takeaways & Limitations
The paper supplies a Bayesian framework and implementable sequential strategies for estimating probability of failure with limited evaluations of an expensive model.
Takeaways & Limitations
The approach assumes expensive deterministic-model evaluations and does not consider stochastic simulators, which raise specific modeling and computational issues.

Abstract

from arXiv · show

This paper deals with the problem of estimating the volume of the excursion set of a function $f:\mathbb{R}^d \to \mathbb{R}$ above a given threshold, under a probability measure on $\mathbb{R}^d$ that is assumed to be known. In the industrial world, this corresponds to the problem of estimating a probability of failure of a system. When only an expensive-to-simulate model of the system is available, the budget for simulations is usually severely limited and therefore classical Monte Carlo methods ought to be avoided. One of the main contributions of this article is to derive SUR (stepwise uncertainty reduction) strategies from a Bayesian-theoretic formulation of the problem of estimating a probability of failure. These sequential strategies use a Gaussian process model of $f$ and aim at performing evaluations of $f$ as efficiently as possible to infer the value of the probability of failure. We compare these strategies to other strategies also based on a Gaussian process model for estimating a probability of failure.

1 Introduction

The paper addresses estimating the probability mass of a function’s excursion set above a threshold when evaluations are expensive and limited. It develops Gaussian-process-based sequential strategies, including SUR strategies derived from a Bayesian decision-theoretic framework, and compares them with existing kriging-based approaches.

Problem: The target is the probability mass, under P_X, of the excursion set where f exceeds a fixed threshold u.In robust design, this quantity can represent a system’s probability of failure.
Problem: Expensive computer simulations make estimating this probability difficult because only a restricted number of evaluations of f is feasible.The paper notes that classical Monte Carlo estimation is generally excluded in this setting.
Problem: 50000 evaluations are required in the illustration with α = 2×10−3 and δ = 0.1, corresponding to about 35 days at one evaluation per minute.The example motivates avoiding basic Monte Carlo estimation when failure probabilities are small.
Related approaches: Existing low-evaluation approaches use parametric approximations of f or the excursion-set boundary, but their accuracy depends on the actual shape of f or ∂Γ.The introduction also notes that these methods do not provide statistically consistent estimators of the target quantity.
Contributions: The paper introduces a Bayesian decision-theoretic framework from which an optimal strategy’s theoretical form can be derived.It then proposes numerically implementable one-step lookahead SUR strategies.
Contributions: The paper unifies existing kriging-based strategies and compares them numerically with the proposed SUR strategies.The comparison is presented through illustrations and empirical sampling-criterion evaluations.

2 Bayesian decision-theoretic framework

The framework formulates probability-of-failure estimation as a sequential Bayesian decision problem under a limited evaluation budget. Strategies use observed evaluations to select future points, with optimal and lookahead policies approximated through sampling criteria, while high-dimensional state spaces limit direct optimization.

Problem formulation: The objective is to obtain a numerical approximation of the probability of failure when the computer model is expensive and evaluations are limited.The failure probability is tied to the excursion-set indicator, and pointwise evaluations provide information about both the function and this probability.
Sequential strategies: A sequential non-randomized algorithm begins with a fixed point and chooses each subsequent evaluation from the accumulated input-output history.The terminal estimator depends measurably on the full history of evaluations.
Bayesian decision formulation: The strategy is selected by defining approximation error through a loss function and seeking a policy that performs well in expected Bayesian risk.The unknown function is modeled as a sample path of a stochastic process representing uncertain initial knowledge.
Optimal and lookahead strategies: Dynamic programming characterizes the optimal policy, but its state space has dimension n(d + 1), so direct numerical optimization beyond a few steps suffers from the curse of dimensionality.The policy selects the next evaluation by minimizing an information-dependent sampling criterion.
Optimal and lookahead strategies: k-step lookahead strategies truncate the dynamic-programming expansion after k terms and replace the exact future risk with a surrogate, producing an iterative sampling rule.The paper restricts attention to one-step lookahead strategies, while whether k ≥2 yields significant improvement remains open.
Initial design: Sequential strategies are intended to follow an initial space-filling design, but choosing its size n0 under the global budget N is an important open question.A rule of thumb cited in the paper recommends an initial sample size proportional to the input dimension d, such as n0 = 10 d.

3 Stepwise uncertainty reduction

The paper derives stepwise uncertainty reduction strategies by treating future evaluations as opportunities to reduce Bayesian estimation uncertainty, then makes the resulting criteria numerically tractable.

3.1 Principle: A myopic strategy evaluates each candidate as if it were the final evaluation, applying Bayesian risk reduction to choose the next point.When the Bayesian risk measures estimation error or uncertainty, the paper calls the resulting strategy SUR.
3.1 Principle: The authors derive SUR criteria from upper bounds and discretization of an intractable sampling criterion, varying the estimator and bounding technique.The estimators include the posterior mean and a plug-in estimator.
3.2 Upper bounds of the SUR sampling criterion: Four SUR criteria are summarized in Table 1, each using a conditional expectation of an integral criterion involving posterior misclassification uncertainty.The criteria include one previously proposed criterion and others described as new by the authors.
3.4 Implementation: The SUR implementation discretizes integrals over X and the predictive distribution, then minimizes the resulting criterion over candidate evaluation points.The implementation uses Monte Carlo sampling for PX and Gauss-Hermite quadrature for the one-dimensional Gaussian integral.
3.4 Implementation: The discrete candidate search is simpler but sub-optimal because points outside the sampled set can sometimes provide more information.The paper retains the discrete search and reports satisfactory results, while noting that continuous global optimization is possible.

4 Other strategies proposed in the literature

The paper reviews alternative Gaussian-process sampling strategies for probability-of-failure estimation and related objectives, emphasizing exploration near the uncertain failure boundary.

4.1 Estimation of a probability of failure and closely related objectives: The paper distinguishes estimating the excursion region, its boundary, and its probability under PX as closely related objectives.All require informative evaluations near the unknown level-set boundary.
4.1 Estimation of a probability of failure and closely related objectives: The compared strategies may receive different performance rankings when evaluated against objectives other than probability-of-failure estimation.The paper explicitly limits its comparison to performance on the fourth objective.
4.2 Targeted IMSE: The targeted IMSE criterion weights kriging variance more heavily near regions where the prediction is close to the failure threshold.Unlike ordinary IMSE, targeted IMSE emphasizes exploration near the frontier of the failure domain.
4.2 Targeted IMSE: Targeted IMSE trades global uncertainty reduction against threshold-region exploration through a weight-function parameter σε that controls the window around u.For large σε, the criterion behaves approximately like ordinary IMSE.
4.3 Other strategies: Other criteria from Ranjan, Bichon, and Echard use marginal posterior distributions and are therefore cheaper to compute than integral criteria such as SUR and tIMSE.The paper presents these methods as alternatives for sequentially improving probability-of-failure estimation.
4.3 Other strategies: Ranjan's and Bichon's criteria can be expressed through kriging variance and posterior misclassification probability, targeting points where the event is uncertain.The associated criterion is maximal when the predicted event probability is one-half.

Appendix B.

Appendix B records that the paper’s closed-form criterion agrees with Bichon et al. for δ = 1 and corrects a computation in Ranjan et al. for δ = 2.

Appendix B: For δ = 1, the paper’s result coincides with the expression reported by Bichon et al.The correspondence is identified with Bichon et al.'s Eq. (17).
Appendix B: For δ = 2, the authors report correcting mistakes in Ranjan et al.'s Eq. (8) and Appendix B.The correction concerns the computations underlying the corresponding criterion.

5 Numerical experiments

The numerical experiments compare Gaussian-process-based sequential criteria for estimating failure probabilities, using structural-reliability benchmarks and Gaussian-process sample paths. SUR performs particularly well in refinement and overall comparisons, while performance depends on the strategy and tuning parameters.

5.2 An example in structural reliability: The experiments evaluate sequential sampling criteria on a structural-reliability benchmark whose failure probability is approximately α = 0.4%.The benchmark uses n0 = 10 initial points and a Monte Carlo sample of size m = 30000, giving a coefficient of variation of approximately 9%.
5.2 An example in structural reliability: SUR variants have similar performance across m0 = 10 and m0 = 500, with J_SUR^2 variants slightly outperforming J_SUR^4.The comparison considers four SUR variants with Q = 12.
5.2 An example in structural reliability: SUR is slightly slower during the initial rough-estimation phase but performs very well during refinement.The paper distinguishes n0.1 as initial-phase performance from n0.03 and n0.01 as refinement-phase performance.
5.2 An example in structural reliability: tIMSE performs best with σ²ε ≈ 0 and is almost as good as SUR, losing about one evaluation on average in refinement.With σ²ε = 1, the required accuracy was not reached after 200 iterations in 17% of runs, and the strategy behaved like a space-filling design.

6 Concluding remarks

The paper develops SUR strategies from a Bayesian decision-theoretic framework for estimating failure probabilities with Gaussian-process models. Experiments report better average performance than existing strategies, while highlighting computational costs and limitations in covariance-parameter estimation and high-dimensional settings.

The paper presents a Bayesian decision-theoretic framework that derives the theoretical form of an optimal strategy for estimating a probability of failure.
Because the corresponding dynamic-programming problem is not numerically tractable, the paper derives one-step lookahead sub-optimal strategies called SUR strategies.These strategies replace exact risk with a substitute risk accounting for expected information gain about the failure probability.
Numerical experiments show that SUR strategies perform better on average than other strategies proposed in the literature.
The improved performance comes at a higher computational cost than strategies based only on marginal distributions, and tIMSE also has high computational complexity.The tIMSE convergence rate appears comparable to SUR when σ2ε ≈ 0.
The methods’ usefulness depends on simulation budget and the ability to elicit an appropriate prior, especially as dimension increases and evaluations remain expensive.The paper notes that covariance-parameter uncertainty is ignored, ML estimation is not guaranteed to work well under localized designs, and the initial-design size is crucial.
The paper identifies fully Bayesian procedures and more robust covariance-parameter treatment as directions needed to make Gaussian-process sampling strategies usable industrially.

A The Mat´ern covariance

The paper uses stationary covariance models, including the Matérn class, to model real-valued functions in computer experiments. An anisotropic Matérn covariance represents coordinate-specific correlation lengths and uses a log-parameterized covariance vector.

The exponential and Matérn covariance functions are conventional stationary covariance choices for design and analysis of computer experiments.
The Matérn class adjusts process regularity at the origin through a single parameter ν > 0.
The anisotropic Matérn covariance models coordinate-specific dependence through positive range parameters ρ_i, interpreted as characteristic correlation lengths.
The covariance variance parameter satisfies kθ(x, x) = σ2, and the positive parameters σ2, ν, and ρ_i are represented using logarithms for practical parameterization.

B Proof of Proposition 4

The proof uses the Gaussian standardization of the kriging prediction error and established closed-form criteria to justify subsequent probability calculations. It also identifies the criteria’s closed-form expressions as the next step.

Under Pn, the standardized kriging error U := (ξ(x) − bξn(x))/σn(x) follows a standard normal distribution.
The proof obtains the stated result by substituting the standardized normal variable into equation (31).
The closed-form expressions for the criteria of Ranjan et al. and Bichon et al. are established in the following sections.

B.1 A preliminary decomposition common to both criteria

The appendix introduces a common decomposition for both criteria before evaluating an integral separately for δ = 1 and δ = 2. The notation uses t, t+, and t− derived from the excursion probability.

The proof defines t = Φ−1(1 − p), with t+ = t + κ and t− = t − κ, before the common decomposition.
The integral A is computed separately for δ = 1 and δ = 2 using elementary intermediate results.

B.2 Case δ = 1

The section computes integral A for δ = 1 and indicates how the resulting expression is substituted to recover equation (33).

The value A1 of integral A is computed for δ = 1.
Equation (38) is used to obtain the final result for this computation.
Substituting equation (40) into equation (37) yields equation (33).