Likelihood-informed dimension reduction for nonlinear inverse problems

Tiangang Cui; James Martin; Youssef M. Marzouk; Antti Solonen; Alessio Spantini

Source-linked AI summary

Likelihood-informed dimension reduction for nonlinear inverse problems

Tiangang Cui, James Martin, Youssef M. Marzouk, Antti Solonen, Alessio Spantini

arXiv:1403.4680v2Mar 19, 2014stat.COmath.NAstat.ME

TL;DR

High-dimensional nonlinear Bayesian inverse problems require efficient ways to identify which parameter directions are informed by data rather than dominated by the prior. The paper constructs a likelihood-informed subspace, samples only its low-dimensional posterior while treating the complement as Gaussian, and reduces estimator variance through Rao-Blackwellization. Across two numerical examples, the approach closely matches full-space inference while substantially improving MCMC efficiency, although its reported scope remains tied to Gaussian-prior nonlinear inverse problems and adaptive subspace construction can omit some data-informed directions.

Problem
High-dimensional inverse problems make posterior sampling costly, while prior-to-posterior changes may be concentrated in a lower-dimensional subspace.
Method
The method identifies a likelihood-informed subspace from prior-relative likelihood influence, approximates the posterior as LIS and complementary-prior factors, and uses lower-dimensional MCMC with Rao-Blackwellization.
Results
Across elliptic PDE and GOMOS examples, the reduced approach produced results similar to full-space MCMC with improved mixing and computational cost; in GOMOS, CPU time fell from 3160 to 2560 seconds for 10^6 iterations.
Takeaways & Limitations
The LIS framework provides an efficient way to exploit nonlinear inverse-problem structure for large-scale Bayesian inference and reduce posterior-estimate variance.
Takeaways & Limitations
The analysis targets nonlinear inverse problems with Gaussian priors, and adaptive LIS construction might ignore some directions that are also data-informed.

Abstract

from arXiv · show

The intrinsic dimensionality of an inverse problem is affected by prior information, the accuracy and number of observations, and the smoothing properties of the forward operator. From a Bayesian perspective, changes from the prior to the posterior may, in many problems, be confined to a relatively low-dimensional subspace of the parameter space. We present a dimension reduction approach that defines and identifies such a subspace, called the "likelihood-informed subspace" (LIS), by characterizing the relative influences of the prior and the likelihood over the support of the posterior distribution. This identification enables new and more efficient computational methods for Bayesian inference with nonlinear forward models and Gaussian priors. In particular, we approximate the posterior distribution as the product of a lower-dimensional posterior defined on the LIS and the prior distribution marginalized onto the complementary subspace. Markov chain Monte Carlo sampling can then proceed in lower dimensions, with significant gains in computational efficiency. We also introduce a Rao-Blackwellization strategy that de-randomizes Monte Carlo estimates of posterior expectations for additional variance reduction. We demonstrate the efficiency of our methods using two numerical examples: inference of permeability in a groundwater system governed by an elliptic PDE, and an atmospheric remote sensing problem based on Global Ozone Monitoring System (GOMOS) observations.

1. Introduction

Bayesian inverse problems can be computationally difficult because posterior sampling costs grow with parameter dimension, motivating a likelihood-informed reduction for nonlinear problems. The proposed approach confines sampling to a data-informed subspace while treating its complement with the prior, enabling variance reduction.

Motivation: Posterior sampling in high-dimensional inverse problems can suffer slower MCMC convergence, longer mixing times, and higher posterior-estimate variance.These issues arise even though some infinite-dimensional MCMC methods avoid dimension-dependent scaling problems.
Proposed approach: The paper identifies a likelihood-informed subspace relative to the prior for dimension reduction with nonlinear forward operators and Gaussian priors.The strategy reflects prior smoothing, observation informativeness, and forward-operator smoothing.
Proposed approach: The posterior is approximated by a low-dimensional distribution on the LIS multiplied by the prior marginalized onto its complementary subspace.The complement is treated as independent of the data in this approximation.
Relation to prior work: Unlike prior approaches based on parameter representations or proposal distributions, this method confines posterior sampling itself to a lower-dimensional space.The approach is not tied to a particular MCMC algorithm and can also support other posterior exploration or integration methods.
Construction: A low-rank approximation of the posterior expectation of the prior-preconditioned Hessian yields a projection operator for constructing the LIS.The resulting projection supports lower-dimensional MCMC and Rao-Blackwellization of posterior estimates.

2. Bayesian formulation for inverse problems

The Bayesian formulation represents unknown parameters probabilistically and conditions their distribution on observed data. The posterior combines prior knowledge about parameters with a likelihood describing the data under each parameter value.

Problem formulation: The inverse problem estimates parameters x from data y generated through a known parameter-to-observable map with additive noise or model error.The parameter space is treated as finite-dimensional after discretization or restriction.
Posterior distribution: Bayesian inference characterizes parameters through their posterior distribution conditioned on an observed realization of the data.Posterior expectations provide point estimates and uncertainty summaries.
Posterior distribution: The posterior density is proportional to the product of the prior density π0(x) and likelihood π(y|x).The prior encodes knowledge before observing data, while the likelihood describes the probability of the data for each parameter value.
Gaussian assumptions: The prior is modeled as a multivariate Gaussian with covariance Γpr and precision matrix Γpr^-1.The additive observation noise is modeled as a zero-mean Gaussian distribution with covariance Γobs.

3. Methodology

The methodology identifies likelihood-informed directions relative to a Gaussian prior and uses them to approximate and sample nonlinear inverse-problem posteriors in reduced dimension. It combines posterior-informed subspace construction with analytical treatment of the complementary Gaussian prior and Rao–Blackwellized estimators.

Linear dimension reduction: For linear Gaussian inverse problems, the optimal rank-r projector is built from leading eigenvectors of the prior-preconditioned Hessian and minimizes posterior-covariance error in Förstner distance.The projector is optimal among rank-r negative semidefinite updates of the prior covariance.
Linear dimension reduction: The likelihood-informed directions maximize data informativeness relative to prior influence, with smoothing priors and weak forward sensitivity producing small Rayleigh quotients.The basis vectors are orthogonal with respect to the prior-precision inner product.
Nonlinear LIS construction: For nonlinear models, local likelihood-informed subspaces are obtained from prior-preconditioned Gauss–Newton Hessians, retaining eigenvectors whose eigenvalues exceed a threshold.The local Hessian is constructed from the forward-model Jacobian, and thresholds below 1 typically retain directions where likelihood influence is not negligible relative to the prior.
Nonlinear LIS construction: A global LIS blends local likelihood-informed directions over the posterior by eigendecomposing the posterior-expected prior-preconditioned Gauss–Newton Hessian.The expected Hessian is approximated with a Monte Carlo estimator.
Posterior approximation: The posterior approximation factors into a reduced posterior on the LIS and a Gaussian prior on the complementary subspace, whose factorization follows from a prior-precision self-adjoint projector.This decomposition separates likelihood-informed coordinates from complementary coordinates while preserving a full-rank posterior covariance.
Posterior approximation: Sampling only the reduced posterior enables posterior estimates using analytically known complement-prior properties, while combining complement-prior samples with reduced-posterior samples can approximate full-posterior draws.The latter samples would require importance weighting or a Metropolis correction, an approach not pursued further here.
Reduced-variance estimation: Subspace MCMC and Rao–Blackwellization address high-dimensional exploration by increasing effective sample size per iteration and removing complement-space sampling from variance-reduced calculations.The procedure is presented as useful when the prior-to-posterior update is reasonably low rank.
Computational benefits: Subspace MCMC also reduces sample-storage requirements and can lower costs for methods requiring operations with the square root of the full prior covariance.These advantages are identified as additional computational benefits over direct full-posterior MCMC.

4. Example 1: Elliptic PDE

The elliptic PDE example evaluates LIS construction under mesh refinement and varying observational data, then tests reduced-space MCMC and Rao–Blackwellized posterior estimates. The LIS stabilizes at about 20 dimensions, while subspace MCMC produces less-correlated samples and closely matching posterior moments at lower computational cost.

Problem setup: The example uses an elliptic PDE to test LIS construction, posterior mean and variance estimation, mesh-refinement effects, and changes with observational data.The permeability field is inferred from pressure observations governed by a PDE model.
Problem setup: Keeping 99% of prior covariance energy requires 2427 eigenmodes, making prior-only truncation inefficient for this problem.The slow covariance-spectrum decay means high-frequency prior modes would require a large retained basis.
LIS construction: After adaptive construction, LIS dimensions are 21 on the 60 × 20 grid and 20 on both the 120 × 40 and 180 × 60 grids.The dimension changes rapidly during the first 100 iterations before stabilizing; the coarser-grid difference may reflect larger discretization errors.
LIS construction: The generalized-eigenvalue spectra and leading LIS basis vectors have similar values and shapes across all three grid refinements.Refinement adds slightly more structure to the fifth basis vector, while overall mode shapes remain close.
Posterior estimation: Subspace MCMC yields faster autocorrelation decay than full-space MCMC for the log-likelihood and projections onto the first, third, and fifth LIS vectors.The comparison uses 10^6 iterations with the first half discarded as burn-in.
Posterior estimation: 2.1 × 10^4 versus 2.6 × 10^5 seconds of CPU time makes subspace MCMC faster than full-space MCMC for 10^6 iterations, including LIS construction.LIS construction costs roughly 2.2 × 10^4 seconds, or 8.5% of the full-space runtime; the resulting samples are also less correlated.
Posterior estimation: The combined Rao–Blackwellized variance estimate is nearly the same as the full-space MCMC estimate, while full-space and subspace MCMC produce very similar mean and variance estimates.Variance in the LIS is estimated by MCMC, whereas variance in the complementary space is evaluated analytically.
Data dependence: With 24, 34, 50, and 83 observations, generalized eigenvalues decay more slowly as the amount of data increases.The passage attributes this to more directions in which the likelihood dominates the prior.

5. Example 2: atmospheric remote sensing

The GOMOS example applies LIS dimension reduction to infer atmospheric gas concentrations from star-occultation transmission measurements. A 200-dimensional problem is reduced to a 22-dimensional LIS, producing results similar to full-space MCMC with faster mixing and lower computational cost.

Problem: GOMOS infers atmospheric gas concentrations from wavelength-dependent absorption measurements of starlight.The inverse problem is ill-posed because some gases and altitudes are strongly informed by the data while others are effectively described by the prior.
GOMOS model: The measurement model forms transmission spectra by dividing atmospheric observations by a reference spectrum and represents the atmosphere as spherical layers.The discretized model combines gas cross-sections, layer geometry, and unknown gas densities, with independent Gaussian measurement errors.
Prior and discretization: The analysis estimates log gas profiles under smooth Gaussian-process priors for four gases across 50 atmospheric layers.The resulting inverse problem has total dimension 200, and the prior is designed to promote smooth profiles while representing plausible density magnitudes.
Inversion results: A 22-dimensional LIS subspace approach yields posterior gas profiles very similar to full-space MCMC.Gas 1 is dominated by the likelihood, gas 4 is determined by the prior, and gases 2 and 3 transition from likelihood-dominated lower altitudes to more prior-influenced higher altitudes.
Inversion results: The first LIS basis vectors primarily represent gas 1, include lower-altitude features of gases 2 and 3, and exclude gas 4.This basis structure reflects the differing degrees to which the data inform the four gas profiles.
Computational performance: Subspace MCMC mixes faster and costs less than full-space MCMC for the GOMOS problem.For 10^6 iterations, subspace MCMC used about 2560 CPU seconds versus 3160 for full-space MCMC; LIS construction required about 136 seconds.

6. Conclusions

The paper reduces nonlinear Bayesian inverse problems by isolating a likelihood-informed subspace and treating its complement as Gaussian. Across two examples, this improves MCMC efficiency while preserving accuracy, although the approach remains an approximation and motivates further reduced-order modeling.

Conclusions: The method divides parameter space into a likelihood-informed subspace for non-Gaussian inference and a complementary subspace dominated by the Gaussian prior.MCMC explores the LIS, while expectations over the complement can be integrated analytically through Rao–Blackwellization.
Conclusions: The global LIS combines local likelihood-informed subspaces across the posterior support and is adaptively enriched until a weighted convergence criterion is met.This generalizes the linear-Gaussian construction to nonlinear problems with parameter-dependent Hessians.
Numerical examples: In the elliptic PDE example, parameter dimensions from 1200 to 10800 yielded LIS dimensions of around 20, with similar mean and variance fields to full-space MCMC.The LIS was empirically discretization-invariant, while LIS MCMC had better mixing and lower computational cost.
Numerical examples: In the GOMOS atmospheric remote sensing example, dimension reduction reduced the 200-dimensional problem to a 22-dimensional LIS without sacrificing accuracy.The reduced method again significantly improved MCMC mixing properties.
Conclusions: The approach is presented as an efficient way to exploit nonlinear inverse-problem structure for large-scale Bayesian inference affected by the curse of dimensionality.The authors identify reduced-order and surrogate modeling within the LIS as a direction for extending the approach to larger problems.