PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey of multifidelity methods in uncertainty propagation, inference, and optimization

Benjamin Peherstorfer, Karen Willcox, Max Gunzburger

arXiv:1806.10761v1math.NAstat.COstat.ME

TL;DR

Repeated high-fidelity evaluations make outer-loop applications computationally expensive, while substituting low-fidelity models can leave the final accuracy unknown. This survey organizes multifidelity methods that combine explicit low-fidelity models with high-fidelity recourse, reviewing adaptation, fusion, and filtering across uncertainty propagation, inference, and optimization. It highlights broad methodological coverage and especially substantial speedup opportunities in computationally intensive uncertainty quantification.

  • Problem

    Outer-loop applications require many model evaluations, making exclusive use of computationally expensive high-fidelity models exceed available resources while simple low-fidelity substitution yields unknown accuracy.

  • Method

    The survey categorizes explicit multifidelity methods by low-fidelity model type and by adaptation, fusion, and filtering strategies that retain high-fidelity recourse.

  • Results

    The literature spans many low-fidelity models and management strategies, with uncertainty quantification offering especially substantial opportunities for computational speedups because Monte Carlo and MCMC sampling are computationally burdensome.

  • Takeaways & Limitations

    Multifidelity methods can reduce runtime where possible while preserving the outer-loop accuracy or convergence guarantees associated with high-fidelity evaluation.

  • Takeaways & Limitations

    Most existing methods assume the high-fidelity model represents truth, although every model, including the high-fidelity model, is itself an approximation of reality.

Abstract

from arXiv · show

In many situations across computational science and engineering, multiple computational models are available that describe a system of interest. These different models have varying evaluation costs and varying fidelities. Typically, a computationally expensive high-fidelity model describes the system with the accuracy required by the current application at hand, while lower-fidelity models are less accurate but computationally cheaper than the high-fidelity model. Outer-loop applications, such as optimization, inference, and uncertainty quantification, require multiple model evaluations at many different inputs, which often leads to computational demands that exceed available resources if only the high-fidelity model is used. This work surveys multifidelity methods that accelerate the solution of outer-loop applications by combining high-fidelity and low-fidelity model evaluations, where the low-fidelity evaluations arise from an explicit low-fidelity model (e.g., a simplified physics approximation, a reduced model, a data-fit surrogate, etc.) that approximates the same output quantity as the high-fidelity model. The overall premise of these multifidelity methods is that low-fidelity models are leveraged for speedup while the high-fidelity model is kept in the loop to establish accuracy and/or convergence guarantees. We categorize multifidelity methods according to three classes of strategies: adaptation, fusion, and filtering. The paper reviews multifidelity methods in the outer-loop contexts of uncertainty propagation, inference, and optimization.

1. Introduction.

The paper defines multifidelity methods for outer-loop applications as explicit low-fidelity models combined with high-fidelity evaluations to reduce computational cost while preserving accuracy or convergence guarantees. It surveys these methods across uncertainty propagation, statistical inference, and optimization, organizing model types and management strategies.

  • Outer-loop applications: Outer-loop applications repeatedly evaluate models at changing inputs to obtain a final result, such as an optimal design.Examples include optimization, uncertainty propagation, and statistical inference.
  • Motivation: Using only the high-fidelity model can exceed available computational resources, whereas replacing it with a low-fidelity model sacrifices accuracy with typically unknown quality.This trade-off motivates multifidelity methods.
  • Multifidelity methods: Multifidelity methods use explicit low-fidelity models that approximate the same output quantity as the high-fidelity model, distinguishing them from approximations embedded within numerical solvers.Examples of embedded approximations include quasi-Newton and Krylov subspace methods.
  • Low-fidelity models: Low-fidelity models can be simplified, projection-based, or data-fit models, with heterogeneous costs and approximation errors.The survey includes coarse-grid, reduced, interpolation, regression, machine-learning-based, and other simplified models.
  • Model management: Model management balances evaluations among models and seeks to preserve the outer-loop accuracy obtained with the high-fidelity model.The survey classifies management strategies as adaptation, fusion, and filtering.
  • Guarantees: The high-fidelity model remains in the loop so multifidelity methods can establish accuracy and/or convergence guarantees, trading some speedup for guarantees even when low-fidelity quality is unknown.This recourse distinguishes the surveyed methods from approaches that provide only error bounds or estimates.

2. Multifidelity model management strategies.

Multifidelity model management categorizes how low- and high-fidelity models are used during outer-loop computations into adaptation, fusion, and filtering strategies.

  • Overview: These strategies manage model evaluations by using low-fidelity models for speedup while retaining high-fidelity evaluations for accuracy or convergence.The strategies therefore differ in whether they modify the low-fidelity model, combine outputs, or screen evaluations.
  • Adaptation: Adaptation enhances the low-fidelity model with high-fidelity information as computation proceeds.Examples include additive or multiplicative corrections, Gaussian-process calibration, and adaptive model reduction.
  • Fusion: Fusion evaluates both models and combines their outputs or information.Control variates use few high-fidelity evaluations and many low-fidelity evaluations to obtain unbiased, lower-variance estimators.
  • Filtering: Filtering invokes the high-fidelity model after a low-fidelity evaluation determines whether further evaluation is warranted.Examples include two-stage MCMC, multifidelity stochastic collocation, and multifidelity importance sampling.

3. Multifidelity model management in uncertainty propagation.

The high-fidelity model maps uncertain inputs to outputs, with the input uncertainty represented by a random variable having an absolutely continuous distribution.

  • Setting: The uncertainty-propagation setting uses a high-fidelity model fhi: Z → Y and a random input Z with density p.The input distribution is assumed absolutely continuous, including non-Gaussian random variables.

3.1. Uncertainty propagation and Monte Carlo simulation.

Monte Carlo uncertainty propagation estimates output statistics by evaluating the high-fidelity model at independent random inputs. Its RMSE converges at O(m^-1/2), while variance reduction can improve pre-asymptotic performance through correlated auxiliary variables.

  • Monte Carlo estimator: Monte Carlo draws m i.i.d. realizations of Z and estimates the expectation from the corresponding high-fidelity model outputs.The estimator is formed from model evaluations at sampled inputs.
  • Monte Carlo estimator: The Monte Carlo estimator is unbiased for the expectation of the high-fidelity output.The supplied passages state E[estimator] = E[fhi].
  • Monte Carlo accuracy: The RMSE convergence rate is O(m^-1/2).The rate is independent of integrand smoothness and input dimension, making Monte Carlo suitable for high-dimensional inputs.
  • Variance reduction: In the pre-asymptotic regime, output variance dominates the Monte Carlo estimator’s RMSE.Variance-reduction methods reformulate the estimation problem around a lower-variance function.
  • Variance reduction: Multifidelity variance reduction constructs an auxiliary random variable from low-fidelity models and exploits its correlation with the high-fidelity output.The paper discusses multifidelity methods based on control variates.

3.2. Multifidelity uncertainty propagation based on control variates.

Multifidelity Monte Carlo uses low-fidelity outputs as control variates for high-fidelity statistics, allocating evaluations and coefficients to minimize error under a computational budget. Its estimator is unbiased, and efficiency depends on correlations, model costs, and interactions among all models.

  • Control variates: Low-fidelity model outputs serve as control variates for estimating high-fidelity statistics, while both model types are fused into one estimator.The approach uses many low-fidelity evaluations together with high-fidelity evaluations to estimate statistics of the high-fidelity model.
  • Examples: Numerical experiments combine high-fidelity finite element models with projection-based, data-fit, and support vector machine low-fidelity models.These examples demonstrate applicability across several low-fidelity model types.
  • Estimator properties: The multifidelity estimator is unbiased for E[fhi].Because of unbiasedness, minimizing its mean-squared error is equivalent to minimizing its variance.
  • Model management: The estimator’s control variate coefficients and model-evaluation counts are selected to minimize mean-squared error for a fixed computational budget.The optimization jointly determines coefficients and the numbers of high- and low-fidelity evaluations.
  • Efficiency: The variance-reduction ratio compares equal-cost multifidelity and high-fidelity-only Monte Carlo estimators.The ratio aggregates model costs and correlations, so the contribution of one low-fidelity model depends on the other models included.
  • Efficiency: Correlation and model costs are both critical to making the multifidelity estimator cheaper than high-fidelity-only Monte Carlo.The paper explicitly identifies both quantities as conditions for estimator efficiency.

3.3. Multifidelity uncertainty propagation based on importance sam-

Multifidelity importance sampling uses a cheap low-fidelity model to construct a biasing distribution, then evaluates the high-fidelity model under that distribution to estimate event probabilities. With suitable support coverage, the resulting estimator is unbiased, and an accurate low-fidelity model can reduce construction runtime.

  • Importance sampling: Importance sampling constructs a biasing density and reweights samples so event probabilities under the nominal distribution can be estimated.The density ratio p/q compensates for sampling from the biasing distribution rather than the nominal one.
  • Multifidelity importance sampling: The multifidelity method uses the low-fidelity model to construct the biasing distribution and the high-fidelity model to derive the event-probability estimate.This separates inexpensive density construction from high-fidelity estimation.
  • Biasing distribution: A Gaussian mixture biasing model is fitted to inputs where the low-fidelity indicator predicts the event of interest.The low-fidelity model is evaluated at many realizations, and expectation-maximization fits the mixture model to selected inputs.
  • Estimator properties: Under support coverage by the biasing density, the high-fidelity importance sampling estimator is unbiased for the event probability.The support of q must include the support of the nominal density p.
  • Efficiency: If the low-fidelity model is sufficiently accurate, constructing the biasing distribution can yield significant runtime savings.The savings arise because the low-fidelity model is cheap to evaluate at many realizations.

3.4. Other model management strategies for probability estimation and limit state function evaluation.

Other probability-estimation strategies use filtering or adaptation to reduce high-fidelity evaluations near failure boundaries. Filtering evaluates the high-fidelity model only when low-fidelity predictions are near the boundary, while adaptation updates or learns boundary approximations.

  • Scope: The surveyed strategies target probability estimation and limit-state function evaluation using different forms of low- and high-fidelity model management.The section places filtering and adaptation alongside probability-estimation approaches based on importance sampling.
  • Filtering: Filtering combines low- and high-fidelity outputs to approximate the high-fidelity indicator function.The method uses a threshold γ to decide when the low-fidelity prediction is sufficiently separated from the failure boundary.
  • Filtering: When the low-fidelity output lies outside the threshold region, the indicator is decided without a high-fidelity evaluation.High-fidelity evaluation is reserved for inputs satisfying |flo(z)| ≤γ.
  • Adaptation: Adaptation-based methods estimate failure boundaries or failure probabilities while computation proceeds.Examples include support vector machines that construct low-fidelity limit-state models and decompose the input space using decision boundaries.

3.5. Stochastic collocation and multifidelity.

Multifidelity stochastic collocation combines low-fidelity sampling with selected or corrected high-fidelity information, while multilevel collocation uses model hierarchies to accelerate convergence. These methods span adaptation, filtering, and fusion strategies in uncertainty propagation and inference.

  • Adaptation: A discrepancy model can correct low-fidelity collocation outputs additively, multiplicatively, or through a weighted combination.The discrepancy accounts for differences between high- and low-fidelity models.
  • Filtering: Filtering-based collocation evaluates the low-fidelity model at many collocation points, then selects a small subset greedily for high-fidelity evaluation.High-fidelity state solutions at selected points are used to construct an approximation space.
  • Fusion: Multilevel stochastic collocation uses a hierarchy of coarse-grid models to accelerate convergence.Complexity reduction can be shown when model errors decay faster than model costs increase.
  • Statistical inference: In Bayesian inference, multifidelity methods include two-stage MCMC schemes that screen candidate samples with a low-fidelity model before high-fidelity evaluation.The high-fidelity model makes the final accept-or-reject decision after low-fidelity screening.

4. Multifidelity model management in statistical inference.

Multifidelity inference methods retain the high-fidelity Bayesian formulation while reducing costly model evaluations through low-fidelity screening, staged sampling, and adaptive approximations.

  • 4.1. Bayesian framework for inference.: Bayesian inference estimates uncertain inputs from noisy observations using a prior, a high-fidelity forward model, and a likelihood based on data misfit.The observational noise is modeled as zero-mean Gaussian with covariance Σϵ.
  • 4.1. Bayesian framework for inference.: MCMC explores the posterior, while sampling efficiency can improve by increasing effective sample size or generating more iterations within a fixed budget.The paper identifies adaptive MCMC and two-stage MCMC as examples of these two routes.
  • 4.2. Two-stage Markov chain Monte Carlo.: Two-stage delayed-acceptance MCMC evaluates candidates with a low-fidelity likelihood first, forwarding only accepted candidates to the high-fidelity stage.This filtering strategy avoids high-fidelity evaluations for candidates rejected early.
  • 4.2. Two-stage Markov chain Monte Carlo.: Delayed rejection retries rejected candidates with different proposal distributions, including independence, random-walk, or progressively localized proposals.Such proposals can also be derived from models with different fidelities and costs.
  • 4.2. Two-stage Markov chain Monte Carlo.: Because MCMC rejection rates are typically high, delayed acceptance can save many high-fidelity evaluations while retaining analyzed convergence properties.The method’s efficiency comes from skipping the second stage for candidates rejected by the low-fidelity model.
  • 4.2. Two-stage Markov chain Monte Carlo.: Adaptive two-stage MCMC can build a data-driven low-fidelity model from observations and yield unbiased Monte Carlo estimators.The low-fidelity model uses information from yobs rather than only prior information.
  • 4.3. Other multifidelity inference methods.: Bayesian approximate error modeling corrects low-fidelity errors in applications including diffuse optical tomography with coarse-grid or simplified-physics models.The cited applications combine lower-cost approximations with fine-grid or high-fidelity models.

5. Multifidelity model management in optimization.

Multifidelity optimization seeks high-fidelity minimizers while using cheaper models during iterative searches, with adaptation balancing exploration and exploitation.

  • 5.1. Optimization setting.: Unconstrained multifidelity optimization seeks an input z∗ minimizing the high-fidelity objective fhi(z).Optimization generally constructs a sequence of inputs intended to converge to an optimum.
  • 5.1. Optimization setting.: Local optimization searches near an initial point for a local optimum, whereas global optimization searches the entire feasible domain for a minimizer.Global methods typically do not require high-fidelity gradients and can therefore handle black-box models.
  • 5.1. Optimization setting.: Penalty formulations can convert constrained multifidelity optimization into an unconstrained problem, while local methods generally handle constraints more readily than global methods.Some multifidelity methods instead provide more specialized constraint handling.
  • 5.1. Optimization setting.: Many global multifidelity methods adapt a low-fidelity model using high-fidelity evaluations so that its minimizer is also a minimizer of the high-fidelity model.Their central management challenge is balancing exploitation of the current approximation with exploration that improves it.

5.2. Global multifidelity optimization.

Global multifidelity optimization adaptively searches with low-fidelity surrogates, using high-fidelity evaluations to refine, fuse, or filter candidate points while supporting global search.

  • 5.2. Global multifidelity optimization.: Global methods balance exploitation of the current low-fidelity minimizer against exploration that adapts the model using high-fidelity information.This balance is a central model-management task in adaptive global optimization.
  • 5.2.1. Efficient global optimization (EGO).: EGO selects a new point through expected improvement, evaluates the high-fidelity model there, and updates the kriging low-fidelity model iteratively.The expected improvement uses the kriging model’s predicted mean and uncertainty.
  • 5.2.1. Efficient global optimization (EGO).: EGO can be globally convergent and derivative-free, but its performance is sensitive to the initial points used to construct the kriging model.An initially exhaustive search may be needed before broader exploration begins.
  • 5.2.2. Other global methods.: Fusion approaches combine multiple kriging models or form weighted averages of low-fidelity models using weights derived from model errors.These methods integrate information across a multifidelity model ensemble.
  • 5.2.2. Other global methods.: Active-learning Gaussian-process methods fit local models in partitioned input regions, supporting settings where high-fidelity sampling involves large parallel computations.Other Gaussian-process approaches address constrained optimization through augmented-Lagrangian formulations.
  • 5.2.3. Filtering methods.: Filtering-based pattern search uses low-fidelity models for additional search directions while preserving convergence to a high-fidelity minimizer.The low-fidelity model guides search without replacing the high-fidelity convergence target.
  • 5.2.4. Adaptive methods.: Local-model adaptation evaluates the high-fidelity model at the current low-fidelity minimizer and uses that output to update the low-fidelity model.This creates an iterative feedback loop between surrogate improvement and optimization.

5.3. Local multifidelity optimization.

Local multifidelity optimization uses restricted low-fidelity subproblems, consistency corrections, trust-region updates, and gradient-free alternatives to guide high-fidelity descent.

  • 5.3.1. Trust-region methods.: Trust-region methods optimize the low-fidelity model only within a bounded neighborhood around the current point.The region limits the step based on the local approximation’s trustworthiness.
  • 5.3.1. Trust-region methods.: Multifidelity trust regions impose first-order consistency by matching low- and high-fidelity values and gradients at the trust-region center.Additive or multiplicative corrections can adjust arbitrary low-fidelity models to satisfy this requirement.
  • 5.3.1. Trust-region methods.: Each trust-region iteration constructs a corrected low-fidelity model, proposes a step, accepts it based on high-fidelity decrease, and expands or contracts the region.The algorithm compares actual and estimated high-fidelity decreases to manage the trust-region size.
  • 5.3.1. Trust-region methods.: Projection-based reduced models such as TRPOD are adapted within trust-region optimization, while error bounds can guide refinement for interpolatory reduced models.These methods update the reduced model as optimization proceeds.
  • 5.3.2. Gradient-free methods.: When gradients are unavailable or too costly, gradient-free trust-region frameworks use radial basis function interpolants as low-fidelity models.The framework builds on gradient-free trust-region and tailored radial-basis modeling methods.
  • 5.3.3. Optimization under uncertainty.: Optimization under uncertainty embeds an uncertainty-quantification loop inside each optimization iteration, with objectives based on statistics such as expected performance and standard deviation.Examples include Monte Carlo sampling and stochastic collocation.

5.4. Multifidelity optimization under uncertainty.

Optimization under uncertainty is highlighted as an important future target for multifidelity methods because it is computationally demanding and important to engineering design. The broader literature indicates multifidelity methods are already affecting diverse outer-loop applications.

  • Optimization under uncertainty is identified as an important target area for future multifidelity methods.The paper describes it as computationally demanding and critically important to areas such as engineering design.
  • The computational demands of optimization under uncertainty motivate continued development of multifidelity methods.The passage specifically characterizes this process as computationally demanding.
  • Multifidelity methods have begun to affect diverse outer-loop applications in computational science and engineering.

6. Conclusions and outlook.

The survey organizes a broad multifidelity literature by low-fidelity model types and model-management strategies, while identifying substantial speedup opportunities and unresolved scope challenges. Its outlook emphasizes model inadequacy and the need to incorporate broader information sources and input-location decisions.

  • Conclusions and outlook: Multifidelity methods have been used for over two decades in optimization, while uncertainty quantification offers greater potential speedups because Monte Carlo and MCMC are computationally burdensome.
  • Conclusions and outlook: The survey categorizes low-fidelity models as simplified, projection-based, or data-fit, and management strategies as adaptation, fusion, or filtering.It presents these categories as a framework for comparing similarities, differences, and relative advantages across methods.
  • Conclusions and outlook: Existing methods generally treat the high-fidelity model as truth, although high-fidelity outputs remain approximations and model relationships may be richer than linear hierarchies.
  • Conclusions and outlook: A further challenge is expanding multifidelity methods beyond models to experiments, expert opinions, lookup tables, and other information sources.
  • Conclusions and outlook: Managing multiple information sources requires deciding both when and where to evaluate them, with experimental design and value-of-information analysis providing relevant foundations.
Loading 1806.10761v1…