PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive construction of surrogates for the Bayesian solution of inverse problems

Jinglai Li, Youssef M. Marzouk

arXiv:1309.5524v1stat.COcs.NAmath.NA

TL;DR

Costly forward-model evaluations make Bayesian inverse problems difficult to solve by posterior sampling, while globally accurate surrogates can be unnecessary when posterior mass concentrates. The paper adaptively constructs polynomial surrogates using stochastic optimization, and PDE demonstrations report order-of-magnitude gains in efficiency and accuracy over prior-based methods.

  • Problem

    Posterior sampling for Bayesian inverse problems requires repeated forward-model evaluations, while globally accurate surrogates for nonlinear models can be costly to construct and unnecessary when posterior mass concentrates on a small part of the prior support.

    Source: §1 · pp. 1–2

  • Method

    The method uses stochastic optimization to adaptively build polynomial surrogates over a sequence of data-determined measures, seeking distributions close to the posterior in Kullback-Leibler divergence.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 5

  • Results

    PDE demonstrations show order-of-magnitude increases in computational efficiency, measured by forward solves, and accuracy, measured by posterior moments and information divergence from the exact posterior.

    Source: §5 · p. 16

  • Takeaways & Limitations

    The adaptive surrogates make sampling-based Bayesian solutions to inverse problems more computationally tractable.

    Source: §5 · p. 16

    The approach is best suited to problems where data are informative enough for the posterior to concentrate relative to the prior.

    Source: §5 · p. 16

Abstract

from arXiv · show

The Bayesian approach to inverse problems typically relies on posterior sampling approaches, such as Markov chain Monte Carlo, for which the generation of each sample requires one or more evaluations of the parameter-to-observable map or forward model. When these evaluations are computationally intensive, approximations of the forward model are essential to accelerating sample-based inference. Yet the construction of globally accurate approximations for nonlinear forward models can be computationally prohibitive and in fact unnecessary, as the posterior distribution typically concentrates on a small fraction of the support of the prior distribution. We present a new approach that uses stochastic optimization to construct polynomial approximations over a sequence of measures adaptively determined from the data, eventually concentrating on the posterior distribution. The approach yields substantial gains in efficiency and accuracy over prior-based surrogates, as demonstrated via application to inverse problems in partial differential equations.

1. Introduction.

Bayesian inference addresses noisy, incomplete inverse problems, but sampling can be prohibitively expensive when each sample requires costly forward-model evaluations. The paper proposes adaptively concentrating surrogate accuracy on regions of appreciable posterior probability.

1. Introduction.

  • Noisy, limited observations make Bayesian inference useful for quantifying uncertainty, but MCMC can make PDE-based inverse problems computationally prohibitive through repeated forward-model solves.The posterior is generally not available in analytical or standard parametric form.Source: §1 · p. 1
  • Prior-weighted surrogate convergence can support posterior convergence under assumptions, yet globally accurate surrogates may be formidable to construct for high-dimensional, nonlinear models.The paper notes this global-construction challenge in practical problems.Source: §1 · pp. 1–2
  • Because posterior mass often occupies a smaller part of parameter space than the prior, focusing surrogate accuracy there can improve efficiency while supporting accurate posterior sampling and expectations.The posterior reflects information gain relative to the prior.Source: §1 · p. 2
  • The CE-inspired algorithm seeks a distribution close to the posterior in Kullback-Leibler divergence through intermediate candidate distributions, accelerating each optimization step with local surrogates.Candidate distributions are selected from a parameterized family.Source: §1 · p. 2
  • The method’s total cost was much lower than building a globally accurate surrogate of comparable or lesser accuracy, and its final distribution sometimes outperformed adaptive random-walk samplers as an MCMC proposal.The comparison is reported for numerical examples.Source: §1 · p. 2

2. Bayesian inference and polynomial chaos surrogates.

This section reviews Bayesian inference and polynomial chaos surrogates, which approximate forward-model outputs so sampling-based posterior calculations require fewer costly model evaluations.

2. Bayesian inference and polynomial chaos surrogates.

  • Because the posterior generally has no closed form, estimating posterior moments requires sampling methods such as MCMC, which need many forward-model evaluations.The likelihood combines the data with the forward model and a statistical model for measurement noise and model error.Source: §2 · pp. 2–3
  • Prior-based stochastic spectral methods construct polynomial approximations of the forward model over the prior support, converging in a prior-weighted L2 norm for use as inexpensive sampling surrogates.The approach propagates prior uncertainty through the forward model.Source: §2 · p. 3
  • Assuming independent input priors, the method represents each output component with a finite polynomial chaos expansion in orthogonal polynomial basis functions.The expansion uses multi-indices, coefficients, and basis functions.Source: §2 · p. 3
  • The prior marginals determine the orthogonal polynomial types, while total-order truncation is an exposition choice rather than a restriction on admissible index sets.Examples include Hermite polynomials for Gaussian distributions, Jacobi for beta, and Laguerre for gamma.Source: §2 · p. 3
  • A non-intrusive coefficient method needs finitely many deterministic forward simulations; quadrature estimates coefficients, with sparse and Smolyak rules offering efficient point selection while guarding against aliasing.The Smolyak construction combines constituent full-tensor approximations and can exploit weak coupling and anisotropy.Source: §2 · p. 4

3. Adaptive surrogate construction.

Because posterior mass may occupy only a small part of the prior support, the paper proposes building surrogates focused on that region rather than maintaining prior-wide accuracy. Its adaptive procedure uses cross-entropy optimization and a sequence of distributions to approach the posterior while constructing local surrogates.

3. Adaptive surrogate construction.

  • Posterior-focused surrogates target the region of appreciable posterior measure, which can make construction more efficient than prior-based surrogates when the posterior is concentrated.The paper notes that concentration can also reduce the forward model’s nonlinearity over the region of interest.Source: §3 · p. 4

3.1. Minimizing cross entropy.

  • The cross-entropy objective selects a candidate distribution by minimizing its K-L divergence from the posterior, equivalently maximizing an expected log density under the posterior.The chosen direction of divergence tends to yield a broader approximation, which the paper considers desirable for encompassing the posterior.Source: §3.1 · pp. 4–5

3.2. Adaptive algorithm.

  • The adaptive algorithm uses tempering, importance sampling, and local surrogates in a sequence of optimization problems approaching the original objective.Each biasing distribution supports importance sampling and supplies the input measure for its local surrogate; the method does not require one specific distribution family.Source: §3.2 · pp. 5, 7

3.3. Convergence analysis.

  • The algorithm reaches λ = 1 in finitely many steps, recovering the original optimization problem; the convergence analysis assumes L2_q approximation convergence and a bounded, uniformly continuous likelihood.The proposition is stated under Assumption 3.1; the analysis is not restricted to total-order polynomial chaos expansions.Source: §3.3 · pp. 7–8

3.4. Independence sampler MCMC.

  • When the final biasing distribution is close to the posterior, it can serve as an independence-sampler proposal, allowing larger jumps than a standard random-walk proposal.The paper also permits standard random-walk or other valid MCMC samplers to explore the posterior induced by the final surrogate.Source: §3.4 · pp. 8–9

4. Numerical examples.

The numerical examples test adaptive surrogate construction on a low-dimensional contaminant source inversion and a time-dependent inverse heat conduction problem. The heat-conduction example evaluates posterior distributions and moments, autocovariance, and MCMC mixing.

4.1. Source inversion.

  • The source-inversion example infers contaminant-source location from noisy measurements at nine sensors and two times, with a uniform prior over the two-dimensional domain.Source: §4.1 · p. 9
  • The adaptive source-inversion posterior is essentially identical to the true posterior while requiring an order of magnitude fewer model evaluations than the prior-based surrogate.The prior-based method uses 417 evaluations, compared with 36 for the adaptive method.Source: Figure 4.2 · p. 11, §4.1 · p. 12

4.2. Inverse heat conduction.

  • The heat-conduction example infers time-varying boundary heat flux from interior temperature measurements, using a four-mode Fourier parameterization and a sensor at x = 0.4.The resulting inverse problem is nine-dimensional.Source: §4.2 · p. 13
  • Three orders of magnitude greater accuracy in low-frequency modes and at least an order of magnitude in high-frequency modes are reported for the adaptive surrogate, despite over six times as many model evaluations for the prior-based surrogate.Accuracy is measured by K-L divergence from the exact posterior for two-dimensional marginals.Source: Table 4.1 · p. 15, §4.2 · p. 15
  • The adaptive surrogate also outperforms the prior-based surrogate when reconstructing posterior mean, variance, and skewness of the heat flux.The reported nonzero skewness indicates that the posterior is non-Gaussian.Source: §4.2 · pp. 15–16
  • The adaptive surrogate's posterior autocovariance of heat flux agrees well with values computed from the true model.Source: §4.2 · p. 16
  • The independence sampler's autocorrelation decays considerably faster than the adaptive random-walk sampler's, suggesting the final biasing distribution can be a good MCMC proposal.The comparison uses the adaptive surrogate.Source: §4.2 · p. 16

5. Conclusions.

The paper develops adaptive surrogates that make sampling-based Bayesian inverse solutions more tractable by focusing accuracy on regions of high posterior probability. PDE demonstrations show order-of-magnitude gains in efficiency and accuracy over prior-based surrogates.

5. Conclusions.

  • Order-of-magnitude gains in computational efficiency and accuracy over prior-based surrogates are reported in PDE examples, measured by forward solves, posterior moments, and information divergence from the exact posterior.The adaptive approach focuses forward-model accuracy on regions of high posterior probability.Source: §5 · p. 16
  • The algorithm builds local forward-model approximations along a finite sequence of biasing distributions, whose final member minimizes Kullback-Leibler divergence from the posterior.Convergence to that minimizer is assured as internal importance-sampling size and local-surrogate accuracy increase; the final distribution can also propose MCMC exploration.Source: §5 · p. 16
  • The adaptive approach is best suited to problems where data make the posterior concentrate relative to the prior, with greater potential benefit under stronger nonlinearity and tighter concentration.In ill-posed problems, concentration may vary by direction; the inverse heat-conduction example is less informative about higher-frequency parameter variations.Source: §5 · p. 16
  • The algorithm does not require forward-model derivatives; when derivatives and an efficiently found posterior mode are available, a Laplace approximation could initialize it, while EM is another proposed optimization alternative.Localized surrogates could also accelerate computations within an EM approach.Source: §5 · pp. 16–17
  • Although the numerical demonstrations use polynomial-chaos approximations and Gaussian biasing distributions, the algorithm is presented as general, with other approximation and distribution families proposed for future exploration.Proposed extensions include mixture biasing distributions and reusing earlier full-model evaluations when constructing later local surrogates.Source: §5 · p. 17
Loading 1309.5524v1…