Source-linked AI summary
Likelihood-free inference with emulator networks
Jan-Matthis Lueckmann, Giacomo Bassetto, Theofanis Karaletsos, Jakob H. Macke
TL;DR
The paper addresses Bayesian inference for simulation-based models whose likelihoods are intractable and for which conventional ABC can be difficult to scale. It learns probabilistic neural emulator networks as synthetic likelihoods and actively chooses simulations using uncertainty-aware local or global acquisition rules. Across synthetic and biophysical examples, the approach supports accurate and efficient inference without user-defined rejection thresholds or distance functions, including high-dimensional settings.
Problem
Simulation-based models often have intractable likelihoods, while conventional ABC scales poorly and requires ad-hoc thresholds, distances, or summary statistics.
Method
The method trains probabilistic neural emulator networks to approximate simulator distributions, uses their tractable likelihoods for posterior inference, and adaptively selects simulations for local or global learning.
Results
The approach enables accurate and efficient inference across synthetic examples and a biophysical neuron model, including high-dimensional problems, without user-defined rejection thresholds or distance functions.
Takeaways & Limitations
Emulators and active-learning rules provide a resource-efficient basis for Bayesian data analysis with simulators in the loop.
Takeaways & Limitations
The current acquisition rules may be improved: MaxVar does not predict whether targeted uncertainty will decrease, while alternatives such as ExpIntVar can be computationally expensive.
Abstract
from arXiv · showhide
Approximate Bayesian Computation (ABC) provides methods for Bayesian inference in simulation-based stochastic models which do not permit tractable likelihoods. We present a new ABC method which uses probabilistic neural emulator networks to learn synthetic likelihoods on simulated data -- both local emulators which approximate the likelihood for specific observed data, as well as global ones which are applicable to a range of data. Simulations are chosen adaptively using an acquisition function which takes into account uncertainty about either the posterior distribution of interest, or the parameters of the emulator. Our approach does not rely on user-defined rejection thresholds or distance functions. We illustrate inference with emulator networks on synthetic examples and on a biophysical neuron model, and show that emulators allow accurate and efficient inference even on high-dimensional problems which are challenging for conventional ABC approaches.
1. Introduction
Simulation-based models often make Bayesian inference difficult because their likelihoods are intractable, while conventional ABC can scale poorly and depend on ad-hoc choices. The paper addresses these limitations with flexible neural conditional density estimators that support likelihood-free inference without user-specified rejection thresholds or distance functions.
- Intractable likelihoods prevent conventional statistical inference for many stochastic simulation models.
- Classical ABC scales poorly to high-dimensional, non-Gaussian data and depends on rejection thresholds, distance functions, and summary statistics.
- Synthetic-likelihood approaches still require summary statistics, distance functions, and assumptions such as homoscedastic Gaussian processes.
- The paper targets multivariate, potentially non-Gaussian and heteroscedastic data using neural-network conditional density estimators called emulator networks.
- The proposed algorithms avoid user-defined rejection thresholds, distance functions, and restriction to a small number of summary statistics.
2. Likelihood-free inference with emulator networks
The method learns a probabilistic emulator of a simulator's conditional data distribution, uses its tractable likelihood as a synthetic likelihood, and actively selects simulations to reduce simulator calls. Local emulators target a given observation, whereas global emulators learn across observations using uncertainty-aware acquisition rules.
- Core approach: The emulator approximates p(x|θ), and its likelihood q(x = xo|θ; φ) is combined with the prior to form an approximate posterior.
- Core approach: Active learning repeatedly selects new parameter locations, simulates data, updates the emulator, and limits calls to an expensive simulator.
- Emulator representation: Neural emulators can use output distributions suited to the simulator, including binomial and categorical distributions rather than only multivariate normals.
- Uncertainty representation: Deep ensembles represent uncertainty about emulator parameters and guide acquisition of additional training data.
- Local and global emulators: Local learning targets likelihood accuracy near observed data, while global learning approximates p(x|θ) across θ and can be reused for any later observation.
- Acquisition rules: MaxVar selects locations with high posterior uncertainty for local learning, whereas MaxMI uses mutual information to improve a global emulator.
- Posterior inference: HMC draws samples from the approximate posterior defined by the emulator-based synthetic likelihood.
- Empirical results: In the Gaussian-noise example, MaxVar converges faster than uniform sampling and BOLFI, with the same qualitative ordering in two dimensions.
3. Results
The paper demonstrates emulator-network inference across low-dimensional, high-dimensional, and neuroscience examples. Emulators compete with state-of-the-art methods, accelerate global learning through active acquisition, and recover posterior distributions for a biophysical neuron model.
- Across three examples, emulator networks support likelihood-free inference for Gaussian observations, high-dimensional images, and a biophysical neuron model.The examples cover competitive low-dimensional inference, amortized modeling of high-dimensional observations, and posterior inference for Hodgkin-Huxley parameters.
- i) Low-dimensional example: Simulator with Gaussian observations: Emulators are competitive with state-of-the-art methods on an example with Gaussian observations.The comparison uses BOLFI and evaluates total variation between true and approximate posteriors as acquisitions increase.
- ii) High-dimensional observations: Inferring the location and contrast of a blob: The high-dimensional image example demonstrates inference without restricting the observations to summary statistics.The forward model maps displacement and contrast parameters to per-pixel activation probabilities and 8-bit pixel values.
- ii) High-dimensional observations: Inferring the location and contrast of a blob: MaxMI acquisitions learn the blob-image emulator faster than uniform random acquisitions, eventually converging to the held-out test-set log-likelihood.The global emulator models per-pixel binomial observations for 32 × 32 images from three blob parameters.
- iii) Scientific application: Hodgkin-Huxley model: For the Hodgkin-Huxley model, emulator networks infer posterior probabilities over ion-channel conductances from observed spike counts.The learned mapping is checked by comparing posterior distributions with simulations producing the corresponding numbers of spikes.
4. Discussion
The discussion presents emulator networks with active learning as a likelihood-free inference framework that avoids several conventional ABC choices while supporting local and global likelihood approximation. It also identifies computational and modeling limitations that motivate future improvements.
- Discussion: Emulator networks provide tractable synthetic likelihoods for simulation-based models, which can then be used in Bayesian inference.The approach learns a probabilistic model consistent with the simulator and plugs its likelihood into Bayesian methods.
- Discussion: The method scales to high-dimensional observations, does not require user-defined distance functions or acceptance thresholds, and is not limited to Gaussian observations.These properties address challenges faced by conventional ABC approaches.
- Discussion: Global emulators approximate p(x|θ) across parameter space, allowing likelihood approximation for any observed data after the emulator is learned.This can amortize the cost of learning the emulator across multiple observations.
- Discussion: The approach remains limited by density-estimator expressiveness and acquisition costs, including computationally expensive evaluation of rules such as ExpIntVar.The authors suggest more expressive estimators and acquisition functions sensitive to the relative costs of simulations and proposals.
- Discussion: The global acquisition rule uses information gain and combines predictive entropy with expected conditional entropy to select informative simulations.The maximum mutual information rule targets inputs with uncertain predictions but relatively certain outputs conditional on emulator parameters.
- Discussion: In the synthetic evaluation, the method was compared with BOLFI in one- and two-dimensional problems using total variation between approximate and ground-truth posteriors.The experiments used 10 initial samples in 1D, 25 in 2D, and 100 acquisitions after initialization.
D.3. Network architecture and training
The image-model experiments generate 32 × 32 pixel observations with offset and contrast parameters under specified uniform priors, while fixing the blob width.
- D.1. Model: Images are generated from a binomial model with horizontal offset, vertical offset, contrast, and width-related parameters.The free parameters are xoff, yoff, and γ, while σ controls the width.
- D.1. Model: The experiments use 32 × 32 pixel images with uniform priors of [−16, 16] for xoff and yoff and [0.25, 5] for γ.The width parameter is fixed at σ = 2.
E.2. Evaluation
The evaluation tracks test-set log-likelihood throughout active-learning acquisitions to compare how acquisition methods improve emulator performance.
- E.2. Evaluation: Performance is measured by the log-likelihood of a 5000-pair test set during acquisitions.The experiments begin with an initial sample of 50 parameter-image pairs.
E.3. Network architecture and training
The neuron application models membrane dynamics with ionic currents and gating variables, using conductance parameters as the free quantities under uniform priors.
- E.3. Network architecture and training: The neuron model evolves membrane potential and gating variables through dynamic equations involving ionic currents and externally applied input.The ionic currents include sodium, potassium, M-type, and leak currents.
- E.3. Network architecture and training: The gating variables m, h, n, and p each evolve according to first-order kinetics.These variables describe the dynamic state of the neuron model.
- E.3. Network architecture and training: The free model parameters are the maximum sodium and potassium conductances, with uniform priors for ḡNa between 0.5 and 60 and ḡK between 0.5 and 10.The model is driven by a step current input.
F.2. Evaluation
The evaluation runs the emulator for 250 acquisitions from an initial dataset of 30 samples and checks posterior predictions across a parameter-space grid.
- A posterior predictive check compares simulator outputs with the posterior across a grid spanning the parameter space.
F.3. Network architecture and training
The emulator uses an ensemble of neural networks for categorical prediction, while BOLFI convergence depends on additional acquisitions and the threshold hyperparameter.
- The emulator outputs a categorical distribution with K = 6 classes using an ensemble of M = 25 neural networks.Each network has two hidden layers of 200 units with ReLU activations and is trained with Adam at a learning rate of 0.001.
- BOLFI converges with additional acquisitions, but inferred-posterior quality strongly depends on its threshold hyperparameter.The figure reports performance up to 100 acquisitions, marked by a dotted line.
Appendix H. Posteriors and samples for image example
The image example presents observed images alongside their inferred posteriors and posterior samples for two observations.
- The figure pairs each observed image with inferred posteriors and posterior samples.Panels a–c cover one observation, while panels d–f cover another.
Appendix I. MaxMI acquisition for Hodgkin-Huxley model
For the Hodgkin–Huxley example, the MaxMI acquisition surface evolves over parameter space and proposes samples at posterior decision boundaries.
- The acquisition surface is shown over θ as additional samples are acquired.
- The acquisition rule proposes datapoints at the decision boundaries of the posterior.