Source-linked AI summary
BayesFlow: Learning complex stochastic models with invertible neural networks
Stefan T. Radev, Ulf K. Mertens, Andreas Voss, Lynton Ardizzone, Ullrich Köthe
TL;DR
Likelihood-free Bayesian inference is challenging because standard methods cannot evaluate an unavailable likelihood. BayesFlow uses simulations to jointly train learned summaries and an invertible neural posterior estimator, then applies the trained model feed-forward across datasets. Across examples, it provides excellent parameter estimation, posterior contraction, and correct-posterior sampling under perfect convergence.
Problem
Likelihood-free models can make numerical likelihood evaluation impossible, preventing standard statistical inference and requiring approximate posterior estimation.
Method
BayesFlow jointly trains a learned summary network and invertible inference network on simulated data from a known forward model for globally amortized Bayesian inference.
Results
BayesFlow produced excellent parameter estimates across models and domains, with posterior contraction observed throughout the examples and correct-posterior sampling guaranteed under perfect convergence.
Takeaways & Limitations
The method is especially advantageous when an intractable model must be fit to multiple independent datasets because its learned data–parameter relationship is reusable across the model family.
Takeaways & Limitations
The theoretical guarantee may not be achieved in practice, so each application requires calibration of point estimates and estimated joint posteriors.
Abstract
from arXiv · showhide
Estimating the parameters of mathematical models is a common problem in almost all branches of science. However, this problem can prove notably difficult when processes and model descriptions become increasingly complex and an explicit likelihood function is not available. With this work, we propose a novel method for globally amortized Bayesian inference based on invertible neural networks which we call BayesFlow. The method uses simulation to learn a global estimator for the probabilistic mapping from observed data to underlying model parameters. A neural network pre-trained in this way can then, without additional training or optimization, infer full posteriors on arbitrary many real datasets involving the same model family. In addition, our method incorporates a summary network trained to embed the observed data into maximally informative summary statistics. Learning summary statistics from data makes the method applicable to modeling scenarios where standard inference techniques with hand-crafted summary statistics fail. We demonstrate the utility of BayesFlow on challenging intractable models from population dynamics, epidemiology, cognitive science and ecology. We argue that BayesFlow provides a general framework for building amortized Bayesian parameter estimation machines for any forward model from which data can be simulated.
1 Introduction
Likelihood-free Bayesian inference is difficult because the posterior must be approximated when likelihood values cannot be computed. BayesFlow addresses this with globally amortized inference learned from simulations, jointly learning summaries and an invertible posterior estimator for rapid reuse across datasets.
- Motivation: Likelihood-free models define likelihoods implicitly through stochastic simulators, making numerical likelihood evaluation—and therefore standard inference—infeasible.Such models can arise from stochastic differential equations, Monte Carlo simulations, or complicated algorithms.
- Amortized inference: Case-based inference reruns the entire posterior-estimation procedure from scratch for every observation sequence, whereas amortized inference separates upfront training from fast evaluation.The upfront training cost is amortized over repeated datasets from the same model family.
- Inference: BayesFlow performs inference feed-forward, without additional training or optimization, after global upfront simulation-based training.The method is intended to amortize training over arbitrarily many observed datasets from a shared model family.
- Method: BayesFlow jointly trains a summary network and an invertible inference network on simulated data from the forward model.The summary network maps variable-size observations to learned fixed-size statistics, while the inference network learns the posterior from those summaries.
- Method: Learned summary statistics replace manually selected statistics and are reported to be more informative for inference than hand-crafted alternatives.This design targets applications where manually constructed summaries may be sub-optimal.
- Evaluation: The paper demonstrates BayesFlow on tractable toy models and challenging intractable models from population dynamics, cognitive science, epidemiology, and ecology.The evaluations consider speed, parameter-recovery accuracy, and probabilistic calibration.
2 Methods
BayesFlow learns posterior estimators with conditional invertible neural networks, optionally paired with a learned summary network for variable-size datasets. Under perfect convergence and sufficient summaries, inverse transformations of Gaussian latent samples produce true posterior samples, while practical errors arise from simulation, summary, and network limitations.
- Learning the Posterior: The method reparameterizes the approximate posterior with a conditional invertible neural network mapping parameters θ to Gaussian latent variables z.The network is trained to minimize the KL divergence between the true posterior and the model-induced posterior across possible datasets.
- Learning the Posterior: Training uses simulated datasets to estimate the objective with Monte Carlo optimization, including a Gaussian latent-density term and a Jacobian volume-change term.The Jacobian term accounts for the nonlinear transformation from θ to z, while minimizing the loss enforces a unit-Gaussian latent distribution.
- Learning the Posterior: Under perfect convergence, the latent output becomes independent of the conditioning data, so samples drawn from the unit Gaussian and transformed backward follow the true posterior.This result relies on the objective reaching its global minimum.
- Learning the Posterior: For datasets with arbitrary observation counts, a summary network hψ(x1:N) is trained jointly with the cINN to produce a fixed-size, maximally informative representation.The construction assumes sufficient summary statistics exist and seeks to preserve the information about θ contained in the observations.
- Limitations: In practice, incorrect posteriors can result from Monte Carlo error, insufficient summaries, absent sufficient statistics, or an insufficiently expressive invertible network.More simulations can reduce Monte Carlo error, whereas the latter two sources are harder to detect and alleviate systematically.
- Summary Networks: The summary-network architecture should match data structure, with recurrent or convolutional choices for time series and other architectures for variable-size i.i.d. samples.Alignment with the observed data’s probabilistic symmetry is presented as an architectural design principle.
3 Experiments
Across synthetic and intractable models, BayesFlow recovered complex posteriors and parameters, including multimodal and uninformative cases, while performance generally improved with more data. Its amortized inference was especially efficient for repeated LFM datasets, though comparisons depend on potentially sub-optimal implementation choices.
- Multivariate Normal Distribution: The 5-D and 50-D MVN experiments reached essentially zero KL divergence, while the 500-D model reached 0.37 with only 0.05% nats extra entropy.These results indicate near-perfect recovery in lower dimensions and decent approximation in the high-dimensional case.
- Ricker Model: BayesFlow recovered Ricker-model parameters well across metrics, with r and ρ broadly recoverable and σ hardest to estimate.BayesFlow maintained low calibration error, indicating that approximate posterior shapes closely matched the true posteriors.
- Ricker Model: Only BayesFlow and ABC-NN recovered the Ricker model’s uninformative posterior for the dummy noise variable u.The Ricker posterior for σ often spread across the prior range, highlighting substantial uncertainty in those estimates.
- Lévy-Flight Model: BayesFlow produced sharper LFM posteriors than SMC-MMD and achieved NRMSEs from 0.008 to 0.048 and R2 values from 0.972 to 0.99 at the maximum trial count.Performance remained good across trial counts, improved with more trials, and showed no systematic marginal-posterior deviations in SCB histograms.
- Lévy-Flight Model: For LFM inference, BayesFlow’s upfront training became worthwhile after 5 datasets versus SMC-MMD, while SNPE-C/APT reached break-even after 75 datasets.After a day of training, BayesFlow inferred 500 datasets in less than 5 seconds, whereas 500 SMC-MMD runs would take more than half a year.
- Lévy-Flight Model: Performance comparisons should be treated cautiously because the authors acknowledge that their choices for manually finding meaningful and efficiently computable summary statistics might be sub-optimal.For only a few dozen datasets, case-based inference might be preferable to SNPE-C/APT on break-even grounds.
- SIR Epidemiology Model: On the SIR model, BayesFlow achieved NRMSE around 0.03 and R2 around 0.99 at T = 500, with posterior variance shrinking as T increased.The approximate posteriors were well calibrated, although the posterior mean of β slightly overestimated lower true parameter values.
4 Discussion
BayesFlow is a globally amortized Bayesian inference method using invertible neural networks and simulation-trained mappings from data to parameters. The discussion highlights its flexibility, posterior properties, computational advantages, and practical limitations.
- Method: BayesFlow uses invertible neural networks and simulations to learn a globally amortized mapping between observed data and model parameters.The method is intended for forward models implemented as computer simulations.
- Flexibility: A separate summary network produces fixed-size, data-driven representations that support arbitrary observed-data shapes or sizes and can replace inadequate hand-crafted statistics.If sufficient statistics are available, they may instead be supplied directly to the invertible network.
- Posterior properties: Under perfect convergence, BayesFlow generates samples from the correct posterior without assuming a particular posterior shape, while experiments show mostly excellent posterior-mean estimates.The method also exhibits posterior contraction and improved recovery as observations increase.
- Computation: After training, BayesFlow efficiently computes posteriors for new datasets, shifting most computational cost to an upfront phase and supporting parallel GPU computation.Its learned weights represent relationships across the hidden-parameter space rather than memorizing precomputed summaries.
- Limitations: Theoretical posterior guarantees may not hold in practice, so each application requires calibration of point estimates and joint posteriors.Amortization makes validation of a trained architecture easier.
- Limitations: Practical performance depends on suitable summary and inference-network designs, and users may need to tune many neural-network and optimization hyperparameters.The implementational burden is non-trivial, motivating development of user-friendly software.
- Outlook: The authors hope BayesFlow will accelerate model-based inference across fields and prove useful beyond the demonstrated examples.This conclusion is framed as a future expectation rather than an established result.
A Learned vs. Hand-Crafted Summaries: The Lotka-Volterra Population Model
The Lotka–Volterra experiment compares BayesFlow with learned LSTM summaries against the same invertible-network architecture using hand-crafted summaries. Learned summaries produce better parameter recovery and posterior calibration.
- Model: The Lotka–Volterra model describes predator–prey dynamics using two first-order nonlinear differential equations.Its interaction parameters are θ = (α, β, γ, δ).
- Experimental setup: Training uses initial conditions u0 = 10 and v0 = 5 over 15 discrete time units with T = 500 samples per time series.Each observation is a two-dimensional vector containing prey and predator counts.
- Experimental setup: Two invertible networks are compared: one jointly trained with a 9-dimensional LSTM summary and one using nine hand-crafted summary statistics.Both use the same cINN architecture with 5 ACBs.
- Results: Learned summary statistics yield notably better recovery of true parameter estimates than hand-crafted summaries.The comparison is reported for the Lotka–Volterra model in Figure S1.
- Results: Approximate posteriors are better calibrated when conditioned on the nine learned summary statistics.The result supports learned summaries when sufficient hand-crafted statistics are unavailable.
Normalized Root Mean Squared Error
This section defines normalized recovery metrics for comparing estimated and true parameters across differing numerical scales, alongside the coefficient of determination.
- Normalized Root Mean Squared Error: NRMSE is scale-independent because normalization by θmax − θmin permits recovery comparisons across parameters with different numerical ranges.It equals zero when estimates exactly match the true values.
- Coefficient of Determination: The coefficient of determination R2 measures the proportion of variance in true parameter samples explained by estimated parameter samples.R2 = 1 indicates perfect reconstruction of the true parameters.
- Comparison: Figure S1 compares recovery and calibration using learned versus hand-crafted summary statistics for the Lotka–Volterra model.Its panels include SBC, parameter recovery, and example full posteriors.
Re-simulation Error
Re-simulation error evaluates whether posterior-based parameter estimates can reproduce the observed data distribution. It uses maximum mean discrepancy, where lower values indicate closer distributions.
- Procedure: The re-simulation procedure estimates parameters by the posterior mean, then simulates a dataset from those parameters using the mathematical model.The simulated dataset is generated as xs1:N = g(θ̃, ξ).
- Metric: Maximum mean discrepancy compares the observed and simulated datasets by comparing all distributional moments.It is a kernel-based metric between samples from two distributions.
- Interpretation: A low MMD indicates that the simulated-data distribution is close to the observed-data distribution, whereas a high MMD indicates separation.The reported summary is the median MMD across validation datasets.
Calibration Error
Calibration error measures how closely an approximate posterior’s coverage matches the coverage of the unknown true posterior. Simulation-based calibration tests posterior sampling through prior self-consistency and rank-statistic uniformity.
- Calibration Error: Calibration error Errcal is the median absolute deviation between approximate-posterior coverage αθ and nominal coverage α across 100 equally spaced α values.Perfect calibration requires αθ = α for every α ∈(0, 1).
- Calibration Error: Kullback-Leibler divergence quantifies approximation error when a closed-form target distribution is available, despite being asymmetric and therefore not a proper distance metric.Closed forms are given for multivariate and diagonal Gaussian distributions.
- Calibration Error: Simulation-based calibration detects systematic bias by checking whether posterior sampling preserves the prior under the Bayesian joint distribution.It samples parameters from the prior, generates data from the forward model, and evaluates posterior samples.
- Calibration Error: A rank-statistic histogram should be uniform when approximate posterior samples are exact; deviations indicate interpretable posterior-sampling error.The procedure repeats prior sampling and simulation, then inspects the rank statistics for uniformity.
The Ricker Model
The Ricker example summarizes raw time-series data with a bidirectional LSTM and trains using broad uniform parameter priors. Broad priors improve coverage of the training space but may reduce parameter recovery at extreme values.
- The Ricker Model: A bidirectional LSTM summarizes raw Ricker time-series data while capturing long-term dependencies and supporting variable-length sequences.The architecture is intended for temporal or spatially autocorrelated datasets.
- The Ricker Model: Training uses broad uniform priors over the Ricker model parameters, including values that can generate implausible datasets.The authors retain these broad priors for training despite potential recovery degradation at the extremes.
- The Ricker Model: Draws from the prior generate varied simulated Ricker time-series datasets.These simulations are depicted in Figure S2.
The Lévy-Flight Model
The Lévy-Flight example uses a permutation-invariant network for independent reaction-time observations, with priors chosen to cover realistic RT distributions. Prior draws produce varied simulated RT distributions.
- The Lévy-Flight Model: A permutation-invariant neural network summarizes independent reaction-time observations so dataset permutations yield the same parameter estimates.The architecture reflects the assumed independence among responses in each RT dataset.
- The Lévy-Flight Model: Uniform priors over the LFM parameters are selected to cover realistic reaction-time distributions in empirical choice RT scenarios.The stated motivation is breadth across the realistic RT range.
- The Lévy-Flight Model: Prior draws generate different simulated reaction-time distributions for the model.These examples are shown in Figure S3.
The Stochastic SIR Model
The stochastic SIR example maps variable-size time-series data into fixed-size vectors with a computationally efficient 1D convolutional network. Its priors are empirically plausible, while divergent Lotka–Volterra simulations are removed during online learning.
- The Stochastic SIR Model: A 1D fully convolutional neural network embeds raw SIR time-series into fixed-size vectors and handles variable input sizes.Convolutions are chosen for computational efficiency and for multidimensional time-series utility instead of an LSTM.
- The Stochastic SIR Model: Uniform priors over the stochastic SIR rate parameters are chosen according to the empirical plausibility of generated time-series.The prior ranges are explicitly based on plausible simulated SIR behavior.
- The Stochastic SIR Model: Prior draws generate varied simulated SIR time-series datasets.These examples are depicted in Figure S4.
- The Stochastic SIR Model: The LV example uses a bidirectional LSTM for raw time-series, as in the Ricker example.Some divergent simulations from broad uniform priors are removed during online learning.
D Example Posteriors on Ricker Datasets
Ten validation datasets simulated from the Ricker model produce widely different marginal posterior shapes, underscoring the need to handle arbitrary posterior forms.
- Figure S5 depicts marginal posteriors from ten validation datasets simulated from the Ricker model.
- The ten datasets yield widely different posterior shapes.
- These varied shapes highlight the importance of working with arbitrary posterior shapes.