Fast Bayesian experimental design: Laplace-based importance sampling for the expected information gain

1. Introduction

• Classical DLMC typically needs many samples because of its nested sample-average structure, and its inner loop can suffer arithmetic underflow, especially with small sample sizes.The paper also describes Laplace-based MCLA as reducing the computation to a single integral, but introducing bias unless the approximation is exact.Source: §1 · p. 2
• Prior self-normalized importance sampling reduces inner samples, but its normalization-constant approximation can underflow when posteriors are concentrated, such as with small measurement error or many repetitions.Source: §1 · p. 2
• DLMCIS uses a Laplace approximation at MAP estimates as the inner-loop sampling distribution, reducing inner samples by orders of magnitude without adding bias and mitigating underflow risk.The MAP estimates are obtained by solving an optimization problem.Source: §1 · p. 2
• The method selects outer and inner sample counts and, for PDEs, the discretization parameter to minimize average computational cost for a desired error tolerance.Source: §1 · p. 2
• Three numerical examples compare robustness and computational efficiency: a linear model, a nonlinear model, and EIT sensor placement to recover laminate-composite inter-ply delamination parameters.Source: §1 · p. 3

2.1. Problem setting

• The data model uses a deterministic response with independent, identically distributed zero-mean Gaussian measurement errors, and treats the unknown true parameter through a prior distribution.The model accounts for parametric uncertainty, but not structural uncertainty such as model error in the forward function.Source: §2.1 · p. 3

2.2. Expected information gain

• Expected information gain averages the Kullback-Leibler divergence between prior and posterior over possible data, with larger divergence indicating a more informative dataset about the unknown parameter.Source: §2.2 · p. 4

2.3. Fast numerical estimators for expected information gain

• The estimator settings are chosen to minimize average computational work subject to a specified tolerance and confidence level, balancing bias and statistical-error tolerances.The framework requires estimates of work, variance, and bias; settings are considered both with and without mesh discretization.Source: §2.3 · p. 5
• The section reviews DLMC, MCLA, and DLMCIS estimators and derives their optimal parameter settings within the cost-minimization framework.Source: §2.3 · p. 5

3.1. Double-loop Monte Carlo (DLMC) estimator

• DLMC estimates expected information gain with outer and inner sample averages, whose sample counts and forward-model discretization determine its computational work.Source: §3.1 · pp. 5–6

3.2. Optimal setting for the DLMC estimator

• The DLMC parameter settings are optimized for tolerance using estimates of bias and variance, including discretization bias and inner-averaging error arising through the logarithm.The section notes that reusing inner and outer samples can reduce forward-model evaluations from O(NM) to O(N), but generally introduces extra bias.Source: §3.2 · pp. 6–7

3.3. Arithmetic underflow

• Underflow occurs when every inner-loop likelihood falls below machine precision, making the likelihood average—the denominator in the estimator's logarithm—numerically zero.The risk rises when the prior poorly covers the posterior; lower measurement noise concentrates the posterior, while likelihood decreases exponentially with data size qNe.Source: §3.3 · p. 7
• Figure 1 illustrates underflow when sampling the inner integral with conventional DLMC.Source: Figure 1 · p. 8

4. Monte Carlo with the Laplace method

• MCLA replaces the nested inner integral with a normal-posterior approximation, leaving a single integral over the prior space.The approximation introduces bias that decreases as the number of measurements increases.Source: §4 · pp. 8–9

4.1. Laplace approximation

• For sufficiently large Ne, MCLA uses the approximation ˆθ ≈ θt.The Laplace-approximation bias decreases as the number of repetitive experiments grows.Source: §4.1 · p. 9, §4 · p. 9

4.2. Monte Carlo with the Laplace approximation (MCLA) estimator

• The Gaussian approximation makes the Kullback-Leibler divergence analytic, leaving a Monte Carlo integral whose error includes Laplace and discretization bias.The total bias includes a Laplace term proportional to 1/Ne and a numerical-discretization term.Source: §4.2 · p. 10

4.3. Optimal setting for the MCLA estimator

• For a given tolerance, optimized MCLA has a better work rate than DLMC, but cannot achieve accuracy below the Laplace bias.The bias floor is the Laplace-approximation bias proportional to 1/Ne.Source: §4.2 · p. 10, §4.3 · p. 11

5. Double-loop Monte Carlo with Laplace-based importance sampling

• Laplace-based importance sampling changes the inner-loop measure without adding extra bias, substantially reduces inner samples, and mitigates underflow risk.The proposal uses the Laplace approximation on MAP estimates.Source: §5 · p. 11

5.1. Double-loop Monte Carlo with importance sampling (DLMCIS) estimator

• DLMCIS changes the evidence-term sampling density to a Laplace-based proposal, drawing θ_n from the prior and ˜θ_n,m from that proposal.Source: §5.1 · pp. 11–12
• Centering the proposal at θ_n is cheaper than centering it at ˆθ_n, but can be inefficient when the two estimates are too far apart to concentrate on the posterior.The authors report this discrepancy-based inefficiency from numerical verification.Source: §5.1 · p. 12

5.2. Optimal setting for the DLMCIS estimator

• DLMCIS preserves DLMC's asymptotic work order, while importance sampling typically makes the first three leading constants several magnitudes smaller; the third remains unchanged.The passage attributes the smaller constants to variance reduction.Source: §5.2 · p. 12

5.3. Effect of the change of measure on the arithmetic underflow

• Changing the measure mitigates DLMC's risk of arithmetic underflow.Source: §5.3 · p. 13

6.1.2. Expected information gain

• For ξ ∈[10, 30] at TOL = 0.01, DLMCIS and MCLA estimates of expected information gain agree, but MCLA has larger estimator variance.Source: §6.1.2 · p. 16, Figure 6 · p. 17

6.2.1. Optimality and error convergence

• At accuracy 10^-3, DLMCIS uses M ∗= 5 inner samples versus M ∗= 105 for DLMC, but incurs about 30 extra forward solves per outer sample.With Ne = 10, MCLA reaches lower tolerances than with Ne = 1, constrained by Laplace bias O (1/Ne).Source: §6.2.1 · p. 17, Figure 7 · p. 18
• For TOL = 0.01, DLMCIS is about hundred times faster than DLMC; MCLA performs similarly but cannot be computed below its Laplace bias.All three methods’ absolute errors are below tolerance over [10−2, 1] with at least 97.5% probability; MCLA’s optimal κ∗ decreases quadratically toward zero as TOL approaches its bias.Source: §6.2.1 · pp. 18–19, Figure 8 · p. 18, Figure 10 · p. 19

6.3.3. Bayesian experimental design formulation

• The EIT example uses a complete electrode model for a two-ply laminate, with two fiber orientations as unknowns, 10 electrodes, and a finite-element mesh of 1600 × 160 linear elements.Sharp potential changes at electrode boundaries require fine meshes; the Bayesian model uses Gaussian measurement error and a uniform prior for the orientation parameters.Source: §6.3.1 · p. 20, §6.3.3 · pp. 22–24

6.3.4. Optimality and error convergence

• At TOL ∼0.01, DLMCIS needs M ∗= 1 inner sample versus M ∗∼103 for DLMC, while MCLA cannot reach accuracies better than TOL ∼1.The Laplace approximation for importance sampling requires about 30 forward model evaluations per outer sample to estimate ˆθn; κ∗ is less than 0.5 for all methods.Source: Figure 15 · p. 26, §6.3.4 · p. 27
• DLMCIS reaches TOL = 0.1 without uncontrolled bias, whereas Laplace bias limits MCLA to TOL = 1; its estimated error agrees with tolerance at 97.5% confidence.The optimal mesh size is capped for TOL = 10 and TOL = 1, and potential-field convergence has η ≈1.15 with work rate γ ≈3.5.Source: §6.3.4 · pp. 25–27, Figure 16 · p. 27

6.3.5. Expected information gain

• For EIT expected-information-gain calculations, the design variables are ξ = (esh, esp), varied over [0, 2] × [0.05, 0.5], with measurement error ϵ ∼N(0, 0.01).A Gauss-Legendre quadrature reference solution targets relative error 0.01; all top-boundary electrodes inject current and bottom electrodes act as outlets.Source: §6.3.5 · pp. 27–28
• The potential field is smoother than in the experimental setup because current is injected at the top and leaves at the bottom.Figure 18 shows potential profiles at x2 = 0 and x2 = 2.Source: §6.3.5 · p. 28, Figure 18 · p. 29
• Expected information gain oscillates as either bottom-electrode set shifts, and is lower when electrode spacing is larger.Shifting the set changes the main current flow from one electrode pair to another.Source: §6.3.5 · p. 29, Figure 19 · p. 30
• Maximum expected information gain occurs at ξ = (2.0, 0.6), versus the worst design at ξ = (0.5, 2.0); optimal electrode spacing is about a quarter electrode size.The best design suggests that further increasing the electrode shift, esh, could improve results further.Source: §6.3.5 · p. 30, Figure 19 · p. 30, Figure 20 · p. 31

Conclusion

• DLMCIS reduces inner-loop sample counts by orders of magnitude and improves DLMC running time by several orders while preserving asymptotic unbiasedness and mitigating underflow.Its inner sampling distribution is based on the Laplace approximation at MAP estimates.Source: Conclusion · p. 31
• The authors derive optimal settings for DLMCIS, DLMC, and MCLA by minimizing average computational work for a given tolerance.Source: Conclusion · p. 31
• DLMCIS matches MCLA's work rate at higher tolerances, while MCLA is inapplicable at lower tolerances because of its inherent bias.Source: Conclusion · p. 31
• The authors demonstrate DLMCIS for optimal sensor placement in an EIT problem using a high-fidelity finite-element composite-material model.Source: Conclusion · p. 31

Appendix A. Estimation of the bias and variance of the DLMC estimator

• The DLMC bias analysis separates numerical discretization bias from the bias caused by inner averaging within the logarithmic estimator.The inner-averaging component is associated with the sample count M.Source: Appendix A · p. 33
• The numerical bias from mesh discretization is O(h^η), matching the convergence order of the discretized forward model.Source: Appendix A · p. 33
• The variance expression is obtained using the law of total variance and second-order Taylor expansions of its terms.Source: Appendix A · pp. 34–35

Appendix B. A leading-order expansion of the likelihood with respect to Ne

• The appendix defines the likelihood and expresses the inner-loop likelihood in terms of the parameter prior and data likelihood.Source: Appendix B · p. 36
• The derivation invokes Kolmogorov’s inequality for a random variable X, with k greater than 1.Source: Appendix B · p. 36
• The noise terms are treated as chi-squared distributed, with mean σ_ϵj Ne and variance 2σ²_ϵj Ne.Source: Appendix B · p. 37

Appendix C. Laplace approximation of posterior distributions

• The negative logarithm of the posterior is used as the expansion objective, whose gradient vanishes at the MAP estimate.Source: Appendix C · p. 38
• The Laplace covariance is the inverse Hessian of the negative log posterior evaluated at the MAP estimate.Source: Appendix C · p. 38
• The appendix recasts moments in terms of the forward model’s Jacobian and Hessian, then uses probabilistic rates with respect to Ne to approximate them.Source: Appendix C · pp. 38–39

• A Gaussian approximation of the posterior density is used to rewrite the Kullback–Leibler divergence.Source: Appendix D · pp. 39–40
• The logarithmic error from replacing the posterior density with its Gaussian approximation is denoted by εla.Source: Appendix D · p. 40
• The derivation expresses expected information gain as the expectation of the Kullback–Leibler divergence.Source: Appendix D · pp. 39–40

• The derivation examines the effect of Laplace-based importance sampling on the inner-loop computation as the number of repetitive experiments, Ne, varies.It follows closely the derivation in Appendix B.Source: Appendix E · p. 41
• The Gaussian distribution used in the derivation is centered at θ̂n and has covariance ˆΣ(θ̂n).Source: Appendix E · p. 41
• The derivation applies a logarithm to obtain a log-likelihood expression.Source: Appendix E · p. 42