Source-linked AI summary
Analyzing Inverse Problems with Invertible Neural Networks
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W. Pellegrini, Ralf S. Klessen, Lena Maier-Hein, Carsten Rother, Ullrich Köthe
TL;DR
The paper addresses ambiguous inverse problems where measurements may correspond to multiple hidden parameter settings. It uses invertible neural networks with latent variables to learn the forward process and generate the full posterior, reporting practical success across synthetic, astrophysical, and medical problems. The method also reveals multimodalities, correlations, and unrecoverable parameters.
Problem
Inverse problems can be ambiguous and ill-posed, while existing point-estimate methods may not express the full uncertainty and ambiguity needed for scientific analysis.
Method
Invertible neural networks jointly learn the well-defined forward mapping and an inverse mapping conditioned on measurements and Gaussian latent variables.
Results
The authors show that INNs estimate full inverse posteriors theoretically and on synthetic, astrophysical, and medical data, comparing favourably with ABC and conditional VAEs.
Takeaways & Limitations
INNs provide a tool for identifying unrecoverable parameters, parameter correlations, and multimodal posterior structure while generating posterior samples cheaply.
Takeaways & Limitations
Finite training can leave residual dependence between measurements and latent variables, causing the learned posterior to deviate from the true posterior.
Abstract
from arXiv · showhide
In many tasks, in particular in natural science, the goal is to determine hidden system parameters from a set of measurements. Often, the forward process from parameter- to measurement-space is a well-defined function, whereas the inverse problem is ambiguous: one measurement may map to multiple different sets of parameters. In this setting, the posterior parameter distribution, conditioned on an input measurement, has to be determined. We argue that a particular class of neural networks is well suited for this task -- so-called Invertible Neural Networks (INNs). Although INNs are not new, they have, so far, received little attention in literature. While classical neural networks attempt to solve the ambiguous inverse problem directly, INNs are able to learn it jointly with the well-defined forward process, using additional latent output variables to capture the information otherwise lost. Given a specific measurement and sampled latent variables, the inverse pass of the INN provides a full distribution over parameter space. We verify experimentally, on artificial data and real-world problems from astrophysics and medicine, that INNs are a powerful analysis tool to find multi-modalities in parameter space, to uncover parameter correlations, and to identify unrecoverable parameters.
1 Introduction
Inverse problems require recovering hidden parameters from measurements even when the forward mapping is well defined but the inverse is ambiguous. The paper proposes invertible neural networks that learn the forward process while representing the full conditional posterior through latent variables.
- Inverse problems are often ill-posed because the forward process loses information, making multiple parameter settings consistent with one measurement.
- Estimating the complete posterior p(x | y) exposes uncertainty, multimodality, parameter degeneracies, and unrecoverable parameters.
- Invertible networks provide bijective, efficiently computable forward and inverse mappings with tractable Jacobians.
- Additional latent variables z capture information lost from measurements y, allowing unique parameter pairs [y, z] and posterior sampling through x = g(y, z).
- The approach avoids directly specifying a supervised loss for an ambiguous posterior by training primarily on the better-understood forward process.
- The authors report posterior estimation on synthetic, astrophysical, and medical data, with improved finite-data results from combining forward and unsupervised backward training.
2 Related work
Related approaches include Bayesian sampling, variational inference, conditional adversarial and variational models, and normalizing flows. Their trade-offs concern computational cost, posterior-family restrictions, diversity, and practical invertibility.
- Exact Bayesian inference is usually intractable for real-world problems, while approximate Bayesian computation embeds a known forward model in rejection sampling.
- Variational methods are efficient but can fail when the true posterior is too complex for their restricted distribution families, including multimodal cases.
- Conditional GANs can lack satisfactory diversity, whereas conditional VAEs are used as a closer baseline for these tasks.
- Normalizing flows transform simple densities into data distributions using bijective mappings, but common flow inverses can be too costly for practical bidirectional training.
- Earlier invertible architectures were applied to memory efficiency, overfitting analysis, adversarial autoencoding, and generative modeling, but were rarely studied as inverse-problem solvers.
3 Methods
The method models an ambiguous inverse posterior with an invertible network that maps parameters to measurements plus Gaussian latent variables. Bidirectional losses train the forward fit, latent distribution, prior consistency, and practical convergence.
- 3.1 Problem specification: The problem is to approximate p(x | y) when observations y are generated from hidden parameters x through an information-losing forward model.
- 3.1 Problem specification: The posterior model samples Gaussian z and deterministically maps measurement-latent pairs to parameters as x = g(y, z; θ).
- 3.1 Problem specification: A single invertible network jointly implements the forward map f and inverse map g, with [y, z] = f(x; θ) and f = g^-1.
- 3.1 Problem specification: Invertibility requires matching nominal and intrinsic dimensions; when necessary, the network pads inputs with zeros to satisfy the dimensional constraint.
- 3.2 Invertible architecture: Reversible affine coupling blocks provide tractable inverses and Jacobians, while permutations, padding, and fully connected subnetworks increase representational capacity.
- 3.3 Bi-directional training: Forward training minimizes supervised Ly for predicting y, while Lz shapes z as Gaussian and independent of y; Lx matches backward predictions to the prior p(x).
- 3.3 Bi-directional training: If Ly and Lz reach zero, inverse sampling returns the true posterior p(x | y*); with finite training, residual y-z dependence motivates Lx to accelerate convergence.
- 3.4 Maximum mean discrepancy: MMD compares sample-accessible distributions and is used for unsupervised losses because it is comparatively easy, inexpensive, and stable to train.
4 Experiments
Experiments on synthetic inverse problems and real medical and astrophysical applications evaluate whether INNs can recover full, potentially multimodal posterior distributions. Across these settings, INNs recover posterior structure and achieve strong conditioning and calibration, while revealing unrecoverable parameters and physical correlations.
- Artificial data: Synthetic experiments test INNs on an eight-component Gaussian mixture and a continuous inverse-kinematics problem with multimodal posteriors.The inverse-kinematics example uses an articulated arm with four parameters and compares INNs with cVAE methods.
- Artificial data: Both INN and cVAE recover the two symmetric inverse-kinematics modes, but cVAE samples deviate more from the target end point.The cVAE learns the forward process only implicitly, whereas the INN is trained using the explicit forward mapping.
- Real-world applications: Medical experiments simulate multispectral tissue measurements involving oxygen saturation, blood volume fraction, scattering amplitude, anisotropy, and layer thickness.The study compares INNs with ablations, cVAEs, dropout-based methods, a classical neural network, and ABC.
- Real-world applications: In the medical evaluation, INNs match or outperform competing methods on point-estimate error and lead on calibration error, the main measure of posterior-shape correctness.Calibration error summarizes the correctness of the whole posterior distribution in one number.
- Real-world applications: In astrophysics, an INN recovers a multimodal posterior with strong parameter correlations for star clusters inferred from multispectral measurements.The posterior distinguishes alternatives such as young clusters with large expansion velocity and older systems expanding slowly.
- Real-world applications: INN posteriors identify anisotropy g and layer thickness d as unrecoverable, while revealing a strong correlation between blood volume fraction vhb and scattering amplitude amie.The unrecoverability agrees with ABC results; the correlation has a physical explanation involving increased absorption and compensating scattering.
5 Conclusion
The paper presents INNs as a generic tool for estimating full inverse-problem posteriors, with applications in medicine and astrophysics. The method offers computational and representational advantages over alternative approaches.
- INNs estimate the full posterior of inverse problems theoretically and practically on problems from medicine and astrophysics.
- INNs can help application experts better interpret data and models and improve experimental setups.
- The coupling-layer restriction does not noticeably reduce the network's expressive power in the reported results.
- Learning the forward process provides the more complicated inverse process for free, unlike methods that focus on learning the inverse.
- Unlike classical variational methods, INN posteriors are not restricted to a particular parametric form.
- Compared with ABC and related Bayesian methods, INN posterior generation is computationally very cheap.
1 Proof of correctness of generated posteriors
The correctness proof shows that an INN trained with zero supervised and unsupervised losses generates the true posterior when a measurement is fixed and latent variables are sampled from their prescribed distribution.
- The proof uses the inverse-function theorem: Jacobians of the forward and inverse mappings cancel, preserving the target density under inversion.
- If both supervised loss Ly and unsupervised loss Lz reach zero, inverse sampling returns the true posterior p(x | y*) for any measurement y*.
- For a fixed measurement y*, the forward-transformed posterior has measurement marginal δ(y−y*) and latent marginal pZ(z).
- Independence between y and z makes the joint output distribution equal to the fixed-measurement distribution combined with the prescribed latent distribution.
- Inverting repeatedly with y* fixed and z sampled from pZ(z) therefore produces samples from p(x | y*).
- Because correct conditional posteriors marginalize to the prior distribution, the unsupervised reverse loss Lx is also zero.
2 Artificial data – Gaussian mixture
The Gaussian-mixture experiment compares INNs with alternative posterior-generating methods on a multimodal inverse problem. INNs explicitly learn the forward mapping and use invertibility and latent variables to recover posterior modes.
- Other baselines: A cGAN with two noise dimensions collapses each label’s modes into nearly one-dimensional structures, even after pre-training.This makes the cGAN’s recovered distributions inadequate for the multimodal toy problem.
- Other baselines: A larger 2M-parameter cGAN with 128 latent dimensions and variance-matching regularization recovers the posteriors reasonably well.The earlier cGAN used 10K parameters and two latent dimensions.
- cVAE and cVAE-IAF: The standard cVAE training setup resembles the INN’s encoder-decoder structure but lacks an explicit forward-process loss and reconstruction loss.The INN is bijective by construction, whereas the cVAE learns x → y only indirectly.
- cVAE and cVAE-IAF: cVAE-IAF generates correct Gaussian-mixture posteriors on par with the INN.The comparison uses a conditional variational autoencoder augmented with Inverse Autoregressive Flow.
- Other baselines: Dropout sampling cannot produce multimodal outputs by construction and fails on this task.The limitation follows from using learned error terms with dropout sampling.
- INN latent space: For a fixed label y∗, the INN’s latent space allocates samples across modes according to their expected probability fractions.Latent positions are evaluated by the closest x-space mode, with probability-mass regions marked in the visualization.
3 Artificial data – inverse kinematics
The inverse-kinematics experiment evaluates INNs on a continuous four-parameter arm model. Quantitative evaluation reports an advantage for the INN in both re-simulation and calibration error.
- Task setup: The forward model computes endpoint coordinates from the arm configuration using three segment lengths.The specified lengths are l1 = 0.5, l2 = 0.5, and l3 = 1.0.
- Results: The INN has a clear advantage in both re-simulation error and calibration error.Because simulation is cheap, re-simulation error is averaged over the whole posterior rather than only the MAP estimate.
- Results: The evaluation uses quantitative posterior-quality metrics computed over the test set.The reported metrics are re-simulation error and calibration error.
4 Multispectral measurements of biological tissue
The INN is applied pixel-wise to multispectral endoscopic footage to estimate tissue oxygenation and uncertainty. The figure reports lower oxygenation in clipped connecting tissue and concentrated uncertainty near some boundaries and specularities.
- Measurement and uncertainty: The method estimates oxygenation sO2 and a 68% confidence interval from multispectral endoscopic footage.The INN trained in the earlier experiment is applied pixel-wise to the footage.
- Measurement and uncertainty: Clips on connecting tissue correspond to lower oxygenation in the small intestine.The figure encodes lower oxygenation in blue.
- Measurement and uncertainty: Uncertainty is low in crucial regions and high at some edges and specularities.The uncertainty is represented through the predicted interval.
5 Star cluster spectral data
The star-cluster application uses synthetic emission-line maps from a time-evolving physical model to infer system parameters. For one observation, the INN recovers multimodal posteriors and strong parameter correlations linked to physical degeneracies.
- Data and model: The astrophysical setting concerns star formation in molecular clouds shaped by competing physical agents.BPT diagnostic diagrams use emission lines including H+, O++, O+, and S+ for analysis.
- Data and model: The data are generated from WARPFIELD 1D simulations with radiative-transfer calculations producing synthetic emission-line maps.The simulations follow cloud evolution until destruction and provide training observations at each timestep.
- Results: The recovered posterior for one observation is strongly multimodal and visibly differs from the prior.The same posterior also exhibits strong correlations among system parameters.
- Results: The observation is compatible with either a young cluster expanding rapidly or an older system expanding slowly.These alternatives exemplify ambiguities in the posterior and degeneracies in the underlying model.
6 Calibration curve for tissue parameter estimation
The section evaluates calibration for four methods by plotting calibration error against confidence. It also describes approximate Bayesian computation procedures used to obtain posterior samples, including a predictable-runtime quantile strategy for the medical application.
- Calibration: Figure 11 compares calibration curves for all four methods, plotting calibration error q_inliers − q against confidence q.Negative calibration-error values indicate overconfidence, while positive values indicate the opposite.
- Approximate Bayesian computation: ABC rejection sampling accepts prior samples whose simulated measurements are within a threshold ϵ of the target measurement, but smaller ϵ requires more simulations.This threshold approach is used for Sec. 4.1, where millions or billions of forward simulations are affordable.
- Approximate Bayesian computation: The quantile ABC strategy accepts the N samples closest to the target after exactly N/q simulations, providing more predictable runtime when simulations are costly.The medical application in Sec. 4.2 uses q = 0.005.