PaperlayerView exact revision on arXiv

Source-linked AI summary

Multilevel Gaussian Processes with Applications to Inverse Problems

Stjepan Salatovic, Aretha L. Teckentrup, Sebastian Krumscheid

arXiv:2610.23202v1math.NAcs.NA

TL;DR

Expensive functions may be accessible only through hierarchies of approximations with increasing accuracy and computational cost. The paper models level increments with independent Matérn-kernel GPs and develops convergence analysis and adaptive sample allocation; its results include computational savings over single-level surrogates and matching asymptotic rates for adaptive criteria.

  • Problem

    The paper addresses approximating expensive functions when evaluations are available only through increasingly accurate and costly approximations.

    Source: §1 · p. 1

  • Method

    The framework models level increments with independent Matérn-kernel GPs and proposes adaptive, cost-adjusted sample allocation across levels.

    Source: §3 · p. 8, §7 · p. 24

  • Results

    The adaptive forward- and posterior-oriented criteria achieve the non-adaptive multilevel method’s asymptotic rate, while the posterior-oriented criterion reaches a given accuracy at a smaller constant.

    Source: §7 · p. 24

  • Takeaways & Limitations

    For Bayesian inverse problems, multilevel cost reduces the single-level product of sample count and PDE-solve cost to their maximum, with either PDE solves or regression coming for free depending on rate relationships.

    Source: §7 · p. 24

    The posterior variance described for the finest-level surrogate quantifies sampling error for fL, not error relative to the true function f.

    Source: §3.1 · p. 11

Abstract

from arXiv · show

We present a multilevel Gaussian process (GP) regression framework for approximating functions that are accessible only through a hierarchy of approximations of increasing accuracy and computational cost. One example is the response surface of a parametrized partial differential equation, where the hierarchy arises from the resolution of the numerical solver. Modeling level increments independently with Mat\'ern-kernel GPs and allocating samples flexibly across discretization levels yields substantial computational savings over single-level GP regression. We derive $L^2$ convergence rates for the posterior mean and variance, accounting for maximum likelihood estimation of the marginal variance and for kernel misspecification. When increment regularity or solver cost is unknown a priori, an adaptive sequential-design algorithm with cost-adapted active learning criteria allocates samples across levels. We apply the surrogate to Bayesian inverse problems, for which Hellinger distance bounds between the true and approximate posteriors are known. Numerical experiments illustrate the theoretical rates and the resulting computational savings.

1. Introduction.

The paper addresses expensive function approximation when evaluations are available only through hierarchies of increasingly accurate and costly approximations. It introduces a multilevel GP framework, analyzes its properties, applies it to Bayesian inverse problems, and proposes adaptive sample allocation when problem-dependent rates are unknown.

1. Introduction.

  • Expensive PDE response surfaces motivate surrogates because each evaluation requires solving the governing PDE, while only approximate evaluations may be available.The approximation hierarchy can arise from solver resolution, iterative-method tolerance, or physical-model fidelity.Source: §1 · p. 1
  • Multilevel methods combine information across discretizations, doing most work on cheap coarse levels and using few expensive fine-level samples.The paper situates its approach among prior multilevel methods for expectations, other statistical quantities, function approximation, and regression.Source: §1 · p. 2
  • The framework models level differences with independent GPs, unlike autoregressive multi-fidelity models, and analyzes both the surrogate mean and its predictive variance.The paper also accounts for maximum-likelihood estimation of marginal variance and kernel smoothness misspecification.Source: §1 · p. 2
  • For settings where problem-dependent rates are unknown beforehand, the paper proposes adaptive sample allocation across levels.The paper also applies the surrogate to Bayesian inverse problems using known Hellinger-distance bounds and illustrates its theoretical rates and computational savings numerically.Source: §1 · p. 3

2. Gaussian process regression.

The section establishes GP regression as an interpolating surrogate with a posterior mean and a model-based uncertainty measure, then gives its assumptions and convergence behavior. For Matérn kernels, the rates depend on target and kernel regularity, design geometry, and estimation of the marginal variance.

2. Gaussian process regression.

  • In the noiseless setting, the posterior mean interpolates exact evaluations and the posterior variance vanishes at sampled points; the uncertainty is model-based and depends on the kernel and design.The deterministic discretization error is treated separately from observational noise.Source: §2 · pp. 3–4

2.1. Mat´ern kernels and reproducing kernel Hilbert spaces.

  • For a Matérn kernel, length scale λ controls correlation decay with distance and smoothness ν determines sample-path differentiability; regression accuracy also depends on design-point distribution, with quasi-uniform fill distance and separation radius scaling as N^-1/d [41].A small separation radius indicates nearly identical points and leads to ill-conditioned kernel matrices.Source: §2.1 · pp. 4–5, §2.2 · p. 5

2.2. Convergence of GP regression.

  • For Matérn GP regression, posterior mean error, posterior variance, and marginal-variance MLE have separate bounds; the MLE-scaled posterior standard deviation converges at the same rate as the mean error.Under quasi-uniform designs, that rate is min(τ, τf)/d; kernel smoothness exceeding target regularity can reduce rates, and the MLE scale need not decay under strong misspecification.Source: §2.2 · pp. 5–7

3. Multilevel Gaussian Processes.

The framework builds a surrogate by independently modeling approximation-level increments and combining their Gaussian-process posteriors. It analyzes mean and variance errors and the computational cost of allocating samples across levels.

3. Multilevel Gaussian Processes.

  • The multilevel construction combines many inexpensive coarse evaluations with fewer costly fine evaluations as discretization accuracy and evaluation cost increase.The analysis assumes GP construction costs are negligible relative to PDE evaluations.Source: §3 · pp. 7–8, §3.2 · p. 11
  • Independent level GPs model telescoping increments, and summing their posterior means and covariances yields the multilevel surrogate and its predictive uncertainty.Each increment uses a Matérn kernel with level-dependent marginal variance.Source: §3 · pp. 8–9

3.1. Error analysis.

  • Under quasi-uniform level designs and maximum-likelihood variance estimation, the multilevel posterior mean and covariance satisfy the error bounds in Proposition 3.1.The common convergence rate is µ = min(τ, τ∆)/d; the root mean squared error combines mean bias and posterior variance.Source: §3.1 · pp. 10–11
  • The posterior variance measures sampling error for the finest discretized response, not the discretization error between that response and the true function.Accuracy therefore requires balancing the finest resolution and per-level sample sizes against the available computational budget.Source: §3.1 · p. 11

3.2. Complexity.

  • Multilevel complexity reduces the single-level cost exponent from the product of PDE-solve and GP-regression rates to their maximum, with a logarithmic factor when β = γµ.When β > γµ, cost is coarsest-level dominated; when β < γµ, it is fine-level dominated.Source: §3.2 · pp. 13–14
  • The framework is applied to inverse problems by replacing the potentially expensive forward model with a multilevel GP surrogate.The presented case is scalar; vector-valued outputs could be approximated componentwise with independent multilevel GPs.Source: §3.2 · p. 14

4. Application to Bayesian Inverse Problems.

The section carries multilevel-GP approximation into Bayesian inverse problems by marginalizing over surrogate uncertainty and bounding the resulting posterior error. It also shows how posterior-weighted error and observation noise qualify design and accuracy.

4.1. GP based approximation.

  • A plug-in posterior mean treats the surrogate as exact and ignores its error, a concern when higher-level increments have few samples and substantial posterior uncertainty.Source: §4.1 · p. 14
  • Marginalizing the GP posterior over the likelihood yields a deterministic approximate posterior, with predictive variance inflating the observation variance.Source: §4.1 · pp. 14–15
  • Posterior-weighted bounds discount surrogate errors where posterior density is very small, motivating designs that prioritize accuracy on the posterior’s support.Source: §4.1 · p. 15
  • The marginalized multilevel-GP posterior has Hellinger-error bounds tied to forward mean-and-variance estimates, and its complexity carries over from the forward problem.Source: §4.1 · pp. 15–16
  • The posterior-error bound’s constant typically grows as observation noise decreases, reflecting greater posterior sensitivity to surrogate error.Source: §4.1 · p. 16

5. Adaptive multilevel construction.

The adaptive construction adds samples and levels incrementally when rates are unknown, selecting level-point pairs by information gain per evaluation cost. Forward- and posterior-oriented criteria direct that allocation toward different accuracy goals.

5. Adaptive multilevel construction.

  • When rates are unknown, the method incrementally refines level designs and adds levels as needed, using an error-to-cost acquisition criterion; this heuristic is not covered by the theory.Source: §5 · pp. 16–17
  • The gain-to-cost ratio selects the level-point pair with the largest information gain per evaluation cost, with costs estimated online from measured evaluation times.Source: §5 · p. 17

5.1. Forward-oriented criteria.

  • The forward ALM criterion ranks variance per cost, shifts samples toward coarse levels relative to the stated error-minimizing allocation, and yields complexity consistent with the non-adaptive method in experiments.Source: §5.1 · p. 18

5.2. Posterior-oriented criteria.

  • Posterior-oriented criteria target posterior accuracy rather than uniform surrogate accuracy, concentrating designs in regions of high posterior mass instead of producing quasi-uniform designs.Source: §5.2 · pp. 18–19
  • The SN gain measures expected Bayes-risk reduction; maximizing it selects points whose conditioning makes the marginalized posterior closest to the true likelihood in the stated L2 sense.Source: §5.2 · p. 18

6. Numerical experiment.

The numerical experiment tests rate assumptions and compares rate-driven and adaptive multilevel surrogates on a Bayesian inverse problem. The multilevel construction tracks prescribed tolerances, while adaptive methods attain its asymptotic rate and the posterior-oriented criterion reaches a given accuracy at lower cost.

6. Numerical experiment.

  • The test discretizes a one-dimensional elliptic boundary-value problem on grids refined by a factor of two, with a first-order upwind solver.The quantity of interest is the integral of the solution; although a closed form exists, the response is treated as solver-accessible only.Source: §6 · p. 19

6.1. Verification of the rate assumptions.

  • The measured discretization error and evaluation time are consistent with rates α = 1 and γ = 1, respectively.The cost-rate fit is visible from finer levels onwards, roughly n_l = 2000.Source: §6 · p. 20, Figure 1 · p. 20

6.2. Multilevel construction.

  • Observed bias, GP approximation error, posterior standard deviation, and Hellinger distance all decay proportionally to the target tolerance ε.The a priori sample allocation attains prescribed accuracy up to a moderate constant; the integrated posterior standard deviation overestimates posterior-mean error by at most a factor of 2.5 in this example.Source: §6.2 · p. 21, Figure 2 · p. 21

6.3. Complexity comparison.

  • At the finest tolerances, multilevel methods save around two orders of magnitude in computational work over single-level GP regression.Figure 3 compares Hellinger distance against computational work and time.Source: Figure 3 · p. 22, §6.3 · p. 23
  • Both adaptive criteria attain the rate-driven method’s asymptotic rate, while the posterior-oriented criterion reaches a given accuracy at smaller cost.Its design concentrates evaluations in regions of significant posterior mass, unlike the forward-oriented criterion’s quasi-uniform coverage.Source: §6.3 · pp. 23–24, Figure 4 · p. 23
  • The time comparison includes GP-system and acquisition costs, whose overhead becomes less significant relative to PDE solves as ε decreases.The theoretical rates predict computational time better at smaller tolerances.Source: Figure 3 · p. 22, §6.3 · p. 23

7. Conclusion.

The conclusion presents independent Matérn-kernel GPs for level increments, with convergence bounds that extend to smoother, misspecified kernels and support posterior complexity results. Adaptive sampling matches the non-adaptive multilevel asymptotic rate, while dimensionality remains a challenge.

7. Conclusion.

  • Independent Matérn-kernel GPs for level increments yield L2 convergence rates for the posterior mean and MLE-scaled posterior standard deviation, including a reduced rate under kernel misspecification.The mean squared error, accounting for predictive uncertainty, has the same guaranteed rate as the posterior mean alone.Source: §7 · p. 24
  • Combining the surrogate bounds with a Hellinger estimate gives a complexity result for Bayesian inverse-problem posteriors, with multilevel cost reducing to the maximum of regression and PDE-solve costs.Depending on increment decay relative to solver-cost growth, either PDE solves or regression comes for free.Source: §7 · p. 24
  • When rates and increment regularity are unknown, adaptive sampling across levels attains the non-adaptive method’s asymptotic rate; the posterior-oriented criterion reaches a given accuracy at a smaller constant.That criterion concentrates its design in regions of significant posterior mass.Source: §7 · p. 24
  • The rates degrade as min(τ, τ_Δ)/d, making the multilevel method difficult to apply in larger parameter dimensions.Sparse or additive kernels that exploit anisotropic structure are noted as a possible remedy; using posterior variance as a stopping criterion is left for future work.Source: §7 · p. 24
Loading 2610.23202v1…