PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrate, Emulate, Sample

Emmet Cleary, Alfredo Garbuno-Inigo, Shiwei Lan, Tapio Schneider, Andrew M Stuart

arXiv:2001.03689v1stat.CO

TL;DR

Expensive, potentially noisy forward models that are difficult to differentiate make Bayesian parameter estimation and uncertainty quantification costly. The paper combines ensemble Kalman calibration, GP emulation, and MCMC sampling; its central claim is that the ensemble supplies a useful design with few forward evaluations.

  • Problem

    Bayesian inversion is challenging when forward models are expensive or noisy and derivatives are unavailable, while ensemble Kalman methods alone do not provide systematic uncertainty quantification.

    Source: §1 · p. 2, §1.1 · p. 3

  • Method

    The calibrate-emulate-sample framework uses ensemble Kalman methods to calibrate parameters and guide GP emulation, then uses MCMC with the emulator for approximate Bayesian inversion.

    Source: §2.1 · p. 6, §6 · p. 22

  • Results

    EKS provides an experimental design suited to Bayesian inference with few forward-model evaluations, including in high-dimensional parameter spaces.

    Source: §6 · p. 23

  • Takeaways & Limitations

    Because MCMC evaluates the cheap emulator rather than the costly forward model, the framework aims to approximate the true posterior at low evaluation cost.

    Source: §2.1 · p. 6

    High-dimensional performance is not assured across the full pipeline: GP regression and standard MCMC generally do not scale well with high-dimensional inputs.

    Source: §1.1 · p. 3

Abstract

from arXiv · show

Many parameter estimation problems arising in applications are best cast in the framework of Bayesian inversion. This allows not only for an estimate of the parameters, but also for the quantification of uncertainties in the estimates. Often in such problems the parameter-to-data map is very expensive to evaluate, and computing derivatives of the map, or derivative-adjoints, may not be feasible. Additionally, in many applications only noisy evaluations of the map may be available. We propose an approach to Bayesian inversion in such settings that builds on the derivative-free optimization capabilities of ensemble Kalman inversion methods. The overarching approach is to first use ensemble Kalman sampling (EKS) to calibrate the unknown parameters to fit the data; second, to use the output of the EKS to emulate the parameter-to-data map; third, to sample from an approximate Bayesian posterior distribution in which the parameter-to-data map is replaced by its emulator. This results in a principled approach to approximate Bayesian inference that requires only a small number of evaluations of the (possibly noisy approximation of the) parameter-to-data map. It does not require derivatives of this map, but instead leverages the documented power of ensemble Kalman methods. Furthermore, the EKS has the desirable property that it evolves the parameter ensembles towards the regions in which the bulk of the parameter posterior mass is located, thereby locating them well for the emulation phase of the methodology. In essence, the EKS methodology provides a cheap solution to the design problem of where to place points in parameter space to efficiently train an emulator of the parameter-to-data map for the purposes of Bayesian inversion.

1. INTRODUCTION

The introduction addresses Bayesian inversion when forward models are expensive, potentially noisy, and difficult to differentiate. It proposes using ensemble Kalman calibration to guide emulation, then sampling an approximate posterior with the emulator.

1. INTRODUCTION

  • EKS calibration provides parameter–model pairs concentrated near posterior support, creating an experimental design suited to training the emulator for Bayesian inversion.The introduction presents this as the paper’s primary innovation.Source: §1 · p. 2

1.1 Literature Review

  • Ensemble Kalman inversion can estimate parameters with typically O(10^2) forward evaluations, whereas MCMC often requires O(10^5), making direct MCMC infeasible for some models.The introduction cites climate models as an example of this constraint.Source: §1.1 · p. 3
  • High-dimensional scaling is uneven across the pipeline: calibration scales well with appropriately chosen priors, while GP emulation and standard MCMC generally do not scale well.Alternative emulators and non-traditional sampling methods are identified as possible routes to better scaling.Source: §1.1 · pp. 3–4

1.2 Our Contribution

  • The calibrate-emulate-sample framework uses ensemble Kalman methods to design GP emulation and then performs approximate Bayesian inversion with MCMC.The approach is intended for expensive, difficult-to-differentiate maps whose evaluations may be noisy.Source: §1 · p. 2, §1.2 · p. 4
  • The paper demonstrates the methodology on a linear inverse problem, Darcy flow, and the Lorenz ’63 and ’96 models.For time-dependent models, finite time-averaged data are treated as noisy infinite-time averages, with GP emulation estimating the map and induced noise.Source: §1.2 · p. 4

2. CALIBRATE-EMULATE-SAMPLE

The method calibrates parameters with ensemble Kalman sampling, uses its posterior-focused evaluations to train a Gaussian-process emulator, and samples an approximate posterior with MCMC. This design limits costly forward-model evaluations while requiring no derivatives of the forward model.

2.1 Overview

  • The method assumes Gaussian observation noise and a Gaussian prior on θ, with θ and the noise independent a priori.For statistical aggregates of varying quantities, the paper regards Gaussian noise as reasonable; constrained parameters can be transformed to ones for which Gaussian priors make sense.Source: §2.1 · p. 5
  • EKS places design points in regions of high posterior probability, making them well-positioned to train an emulator for approximate Bayesian inversion.The paper uses the final EKS iteration as its design, with M = J, although other subsets of the JN evaluations are possible.Source: §2.1 · p. 5, §2.3 · p. 7
  • EKS evaluations train a GP emulator, which replaces the costly forward model during MCMC sampling from an approximate posterior.The workflow uses the expensive forward model only during calibration; MCMC evaluates the cheaper emulator.Source: §2.1 · p. 6

2.2 Calibrate – EKI And EKS

  • Time-discretized EKI drives particles toward consensus and data fit, yielding derivative-free least-squares optimization; EKS adds prior-related damping and parameter-dependent noise to approximate posterior sampling.For linear forward models, the resulting interacting particle system is invariant with respect to the posterior; more generally, it concentrates close to it.Source: §2.2 · p. 7

2.3 Emulate – GP Emulation

  • The GP emulator models each output coordinate independently and allows noisy input-output pairs, with hyperparameters learned by an empirical Bayes approach.The emulator may incorporate both observation noise and GP noise, with the smaller source sometimes neglected.Source: §2.3 · pp. 7–8

2.4 Sample – MCMC

  • MCMC starts at the final EKS ensemble mean and uses a random-walk Gaussian proposal with covariance from the EKS ensemble.This initializes sampling with approximate posterior information from EKS.Source: §2.4 · p. 9

3. LINEAR PROBLEM

Section 3 tests CES on a linear Gaussian inverse problem with a known posterior, using EKS calibration, GP emulation, and GP-based MCMC. The GP emulation improves with more training data, and sampling captures the true posterior with ensembles of 16 or more.

3.1 Linear Inverse Problem

  • The test uses a two-parameter linear Gaussian forward map and Gaussian prior, for which the posterior is itself Gaussian [see 24, Part IV].The synthetic data use θ† = [−1, 2]⊤ and Gaussian noise with covariance Γ = 0.12I.Source: §3.1 · p. 9

3.2 Numerical Results

  • At iteration 20, EKS samples mismatch the true posterior, although the mean-field limit of EKS exactly reproduces the invariant measure in this linear case.The passage attributes the mismatch to time-discretization error without metropolization and finite particle counts.Source: §3.2 · p. 10, Figure 2 · p. 10
  • For training sets of size J ≥16, the GP emulates the linear map accurately with low uncertainty in the posterior's main support.The emulator uses EKS input-output pairs; its accuracy increases as training data increases.Source: §3.2 · p. 10, Figure 3 · p. 11
  • With ensemble size 16 or more, GP-based MCMC captures the true posterior.The figure compares posterior estimates from GP-based MCMC with the true posterior.Source: §3.2 · p. 11, Figure 4 · p. 11

4. DARCY FLOW

Section 4 applies CES to infer permeability parameters from noisy pressure measurements in a Darcy-flow problem. With 512 ensemble members, CES accurately reproduces the true posterior; forward uncertainty results are similar to the true-model MCMC reference except for the 128-member pressure-exceedance case.

4.1 Elliptic Inverse Problem

  • The problem infers permeability parameters from noisy pressure observations in a Darcy-flow PDE, using a log-normal field parameterization and a 10-term truncation of 256 coefficients.The 10-term truncation avoids using the same model to generate and solve the inverse problem.Source: §4 · p. 11, §4.1 · pp. 11–12, §4.2 · p. 12
  • Pointwise pressure measurements are used without mollification, although in two dimensions the associated functionals are not elements of the dual space H1_0(D; R).The text states that narrow-kernel mollification does not affect results of this type.Source: §4.1 · p. 12

4.2 Numerical Results

  • The experiment uses Gaussian observation noise and ensembles of 128 or 512 members, initialized from a centered Gaussian prior with covariance Γθ = 10^2 × I_p′.The true field has 256 coefficients, while inference uses p′ = 10 terms.Source: §4.2 · p. 12, §4.1 · p. 12
  • CES calibrates with EKS, trains a linear-mean GP with a squared-exponential kernel, then samples using a random-walk procedure and a Gaussian transition matched to the final EKS ensemble moments.The likelihood is selected for a deterministic forward map with additive data noise.Source: §4.2 · pp. 12–13
  • With M = J = 512, CES accurately reproduces the true posterior; with M = J = 128, some components deviate systematically, though the posterior captures the true values.The comparison is against gold-standard MCMC using the true forward model, which requires tens of thousands of evaluations.Source: §4.2 · p. 13, Figure 5 · p. 13
  • For forward uncertainty, the 128-member pressure-exceedance samples differ statistically from the true-model distribution, while other tested distributions show no evidence of difference.The comparison uses a nonparametric k-sample Anderson–Darling test at 5% significance.Source: §4.2 · p. 14, Figure 6 · p. 14

5. TIME-AVERAGED DATA

The section develops Bayesian inversion from finite time-averaged data, where initial conditions and finite-window fluctuations make forward evaluations noisy. Lorenz ’63 and Lorenz ’96 experiments examine EKS-guided GP emulation and sampling, including how output decorrelation affects posterior estimates.

5.1 Inverse Problems From Time-Averaged Data

  • The inverse problem uses time-averaged observations of a function of the dynamical state, with a spinup period and a finite averaging horizon.The operator defining the average also depends on the trajectory’s initial condition.Source: §5.1 · p. 15
  • Finite time averages depend on the initial condition and have approximately Gaussian errors whose covariance depends on θ, unlike infinite-horizon averages.The setup assumes a perfect model and notes that computational cost limits feasible averaging windows.Source: §5.1 · p. 15

5.2 Numerical Results – Lorenz ’63

  • For Lorenz ’63, first- and second-moment observations over windows of T = 10 produced a noisy misfit landscape that may challenge standard optimization and MCMC.The experiment recovers r and b while fixing σ at its true value; Γobs is estimated from averages over τ = 360 after a spinup of T0 = 30.Source: §5.2 · p. 17
  • The Lorenz ’63 emulator used independent GPs for nine outputs trained on EKS input-output pairs, focusing runs in regions of high posterior probability.The initial diagonal treatment did not learn dependencies among simulator outputs, motivating output decorrelation strategies.Source: §5.2 · p. 17
  • In Lorenz ’63, Φm smoothed the noisy misfit, while ΦGP initially missed posterior contour orientation and eccentricity when output dependencies were not learned.Decorrelating outputs allowed variance-informed ΦGP to overcome problems associated with diagonal emulation; the comparison is shown in Fig. 8.Source: §5.2 · p. 17, Figure 8 · p. 19
  • For Lorenz ’63, GP-based Metropolis acceptance was about 41% with Φm and 47% with ΦGP, versus 16% using the true forward model.The paper attributes the higher acceptance to emulator speed and smoothing of the noisy log-likelihood.Source: §5.2 · p. 18, Figure 9 · p. 20

5.3 Numerical Results – Lorenz ’96

  • For Lorenz ’96, EKS used 54 iterations with 100 members, and GP training used 10 ensemble snapshots to form 1,000 points with lengthscale priors informed by pairwise distances.The priors were chosen to let the kernel decay away from training data and avoid spurious short-term variations.Source: §5.3 · pp. 20–21
  • For Lorenz ’96, Φm gave scatter plots closest to EKS, while ΦGP produced an overly dispersed posterior and no gold-standard MCMC comparison was feasible.The authors report this experiment as confirming CES viability when forward-model use is prohibitively expensive.Source: §5.3 · p. 22

6. CONCLUSIONS

The conclusion presents CES as a framework for Bayesian inversion with expensive, derivative-free, and potentially noisy forward models. Its central design idea is to use EKS both for calibration and to locate training points for GP emulation.

6. CONCLUSIONS

  • CES combines calibration with ensemble Kalman methods, GP emulation, and MCMC sampling for Bayesian inversion with expensive forward models and no derivative information.The framework is also intended to handle noisy forward-model evaluations.Source: §6 · p. 22
  • EKS provides parameter estimates and an experimental design suited to Bayesian inference, using a small number of forward-model evaluations even in high-dimensional parameter spaces.The paper identifies this design as a source of computational efficiency.Source: §6 · pp. 22–23
  • Future directions include interleaving emulation with EKS, developing theory for posterior-focused designs, exploring alternatives to GPs for high-dimensional inputs, and applying CES to expensive legacy models.Source: §6 · p. 23

A.1 Time variability decorrelation

This appendix strategy decorrelates forward-model outputs using the observed-noise covariance before fitting uncorrelated GPs.

A.1 Time variability decorrelation

  • The method diagonalizes Γobs with an orthogonal matrix Q and an invertible diagonal matrix, then transforms both observations and model outputs into those coordinates.The transformed model-output components are emulated as uncorrelated GPs.Source: §A.1 · p. 26
  • The transformed observations and emulator outputs are used to calculate the misfit in decorrelated output coordinates.The emulator can also be transformed back to the original output coordinates.Source: §A.1 · p. 26
  • The same decorrelation approach can be used when evaluating the alternative misfit expressions.Source: §A.1 · p. 26

A.2 Parameter variability decorrelation

This appendix strategy uses the simulator training outputs, rather than the observation-noise covariance, to construct uncorrelated coordinates for GP emulation.

A.2 Parameter variability decorrelation

  • The method centers the M-by-d matrix of training outputs and applies SVD to construct orthogonal output coordinates.The transformed coordinates represent uncorrelated outputs, while the diagonal matrix contains unscaled standard deviations and the orthogonal matrix contains proportional loadings.Source: §A.2 · p. 27
  • Training outputs and observations are projected using the SVD-derived mean, loadings, and scale, and the projected model outputs train the emulator.Source: §A.2 · p. 27
  • The resulting emulator is transformed back to original output coordinates to compute the misfit and its alternative forms.Source: §A.2 · p. 27
Loading 2001.03689v1…