PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Simulation-Based Inference

Jan-Matthis Lueckmann, Jan Boelts, David S. Greenberg, Pedro J. Gonçalves, Jakob H. Macke

arXiv:2101.04653v2stat.MLcs.LG

TL;DR

Simulation-based inference lacks a public benchmark suitable for comparing likelihood-free algorithms and identifying their strengths and weaknesses. The paper introduces an open benchmark of algorithms, metrics, and inference tasks with reference posteriors, finding that metric choice matters, sequential estimation improves sample efficiency, neural methods generally perform better, and no algorithm is uniformly best.

  • Problem

    A public benchmark with appropriate performance metrics for likelihood-free algorithms was lacking, making comparisons of their strengths and weaknesses difficult.

  • Method

    The paper benchmarks distinct SBI algorithms across ten tasks and multiple metrics, using evaluable likelihoods to provide reference posteriors.

  • Results

    Metric choice is critical; sequential estimation generally improves sample efficiency; neural-network approaches outperform classical ABC methods for small and moderate simulation budgets; no algorithm dominates all tasks.

  • Takeaways & Limitations

    The benchmark provides practical guidance for selecting algorithms and can diagnose shortcomings while supporting transparent, reproducible SBI research.

  • Takeaways & Limitations

    The initial benchmark omits several SBI families, including active-learning and Bayesian-optimization methods, gray-box algorithms, and Gaussian-process density-estimation alternatives.

Abstract

from arXiv · show

Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods. However, a public benchmark with appropriate performance metrics for such 'likelihood-free' algorithms has been lacking. This has made it difficult to compare algorithms and identify their strengths and weaknesses. We set out to fill this gap: We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms including recent approaches employing neural networks and classical Approximate Bayesian Computation methods. We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency. Neural network-based approaches generally exhibit better performance, but there is no uniformly best algorithm. We provide practical advice and highlight the potential of the benchmark to diagnose problems and improve algorithms. The results can be explored interactively on a companion website. All code is open source, making it possible to contribute further benchmark tasks and inference algorithms.

1 Introduction

Simulation-based inference addresses likelihood-intractable models, but rapid algorithmic development has outpaced systematic comparison. This benchmark provides an open framework of tasks, metrics, and algorithms, revealing performance differences and practical guidance for SBI.

  • Motivation: Likelihood-free inference performs Bayesian inference when numerical likelihood evaluation is intractable or impractical.SBI generally does not require a differentiable simulator or access to its internal random variables.
  • Motivation: Rapid SBI development has made theoretical and empirical relationships among approaches difficult to assess, especially across disciplines.Closely related algorithms have developed independently in different fields.
  • Impact: An extensible benchmark can support transparent comparison, reproducible research, community contributions, and algorithm development in SBI.The framework is intended to expand through additional tasks and algorithms.
  • Benchmark framework: The benchmark combines distinct SBI algorithms, multiple performance metrics, and ten inference tasks with evaluable likelihoods and reference posteriors.The reference posteriors enable rapid evaluation of approximate SBI solutions.
  • Initial findings: The initial benchmark finds that metric choice is critical, some tasks leave substantial room for improvement, and sequential estimation generally improves sample efficiency.These findings are presented as initial benchmark insights rather than universal algorithmic conclusions.
  • Initial findings: Neural-network approaches outperform classical ABC methods for small and moderate simulation budgets, but algorithm rankings remain task-dependent.The benchmark therefore finds no single algorithm that dominates across tasks.

2 Benchmark

The benchmark compares diverse SBI algorithms, acquisition strategies, and posterior representations against reference posteriors using performance metrics whose applicability depends on available ground-truth and algorithm outputs.

  • Benchmark design: The benchmark evaluates approximate posteriors or posterior samples against reference posteriors using performance metrics.Given a prior, simulator, and observation, an algorithm returns q(θ|xo) or samples from it; the result is tested against p(θ|xo).
  • Algorithms: Sequential algorithms adaptively acquire informative simulations to improve sample efficiency, but require additional algorithmic steps and hyperparameter choices.The benchmark includes sequential and non-sequential algorithms to assess whether empirical efficiency gains justify this added burden.
  • Algorithms: Algorithms span classical ABC and neural approaches that approximate likelihoods, posteriors, or density ratios.The initial selection includes REJ-ABC, SMC-ABC, neural likelihood estimation, neural posterior estimation, neural ratio estimation, Random Forest ABC, and Synthetic Likelihood.
  • Performance metrics: Posterior output representations constrain which metrics can be used for comparison.Methods may return posterior samples, samples with evaluable unnormalized densities, or directly evaluable and sampleable posterior densities.
  • Performance metrics: The benchmark emphasizes metrics comparing inferred and reference posteriors, while commonly used alternatives have important limitations.Probability of θo requires averaging over many prior-sampled observations, PPCs can pass for a good MAP estimate despite a poor posterior, and MMD is sensitive to kernel hyperparameters for complex multimodal posteriors.
  • Software: The accompanying web interface supports interactive exploration of benchmark results.The authors provide the interface alongside the code.

3 Results

The benchmark shows that metric choice materially affects conclusions, sequential methods improve sample efficiency, and neural density- or ratio-estimation methods generally outperform classical ABC, without a universally best algorithm. It also demonstrates that benchmark comparisons can diagnose implementation problems and guide practical algorithm selection.

  • Performance metrics: Metric choice is critical: MMD, MEDDIST, and KSD can disagree with C2ST or suffer sensitivity and numerical problems.MEDDIST can favor a good point estimate despite a poor posterior, while MMD depends on kernel choices and KSD encountered gradient-calculation problems on Two Moons.
  • Overall performance: C2ST indicated that several benchmark tasks remained unsolved within the specified simulation budgets, leaving substantial room for stronger algorithms.SLCP and Lotka-Volterra were examples where no algorithm solved the task under the tested budget.
  • Sample efficiency: Sequential algorithms generally outperformed non-sequential methods, especially on nontrivial tasks, but their gains diminished as simulation budgets increased.The sequential advantage was small for linear Gaussian cases and more pronounced on most other tasks.
  • Algorithm comparisons: Neural density- or ratio-estimation methods generally outperformed REJ-ABC and SMC-ABC, although classical ABC retained competitiveness for low-dimensional problems with cheap simulators.Classical rejection-based methods require far less computation because they avoid network training.
  • Algorithm comparisons: No single sequential method was consistently preferable because SNLE, SNRE, and SNPE have distinct strengths that depend on problem structure.The benchmark therefore motivates practical guidance for choosing algorithms based on the application.
  • Diagnostics and improvement: Reference posteriors enabled diagnosis and improvement of MCMC-related implementation issues in (S)NLE and (S)NRE.Changing chain initialization and sampling strategy, and transforming bounded parameters to unbounded ones, improved performance and speed on benchmark tasks.

4 Limitations

The benchmark has important scope, representativeness, tuning, computational, and generalization limitations. These constrain how broadly its comparisons and recommendations should be interpreted.

  • Algorithm coverage: The benchmark omits active-learning, Bayesian-optimization, gray-box, Gaussian-process, and many other SBI algorithms.The initial selection focused on neural-network density estimation, simpler sequential acquisition schemes, and black-box settings.
  • Task coverage: The tasks exclude high-dimensional spatial structure such as images, where algorithms must learn summary statistics while exploring data structure.Such tasks remain an active research area and may pose different challenges than those represented in the benchmark.
  • Evaluation choices: Benchmark results may reflect the authors’ areas of expertise despite extensive investigation of tuning choices and implementation differences.This qualification affects interpretation of algorithm comparisons.
  • Computational cost: The reported performance focuses on simulation calls, although computation times differ substantially across algorithms.(S)ABC was much faster than methods requiring network training, while sequential neural algorithms had the longest runtimes.
  • Generalization: Because the benchmark uses problems with computable reference posteriors, its insights may not generalize fully to real-world simulators.Nevertheless, these simpler problems reveal differences and limitations among SBI approaches.
  • Practical use: Algorithm choice also depends on difficult-to-quantify factors such as available problem information, inference goals, and simulator speed.The paper therefore includes practical considerations and recommendations in Box 1.
  • Benchmark use: Benchmark overfitting can create an illusion of progress and encourage attention to implementation details that may not generalize.The authors also note that simulators could enable excessive task-specific hyperparameter tuning unlike typical practice.

5 Discussion

The paper presents the initial SBI benchmark as a tool for transparent, quantitative comparison and improvement. It supplies resources intended to support exploration, reproducibility, and future contributions.

  • Discussion: The benchmark is intended to clarify and expedite progress in simulation-based inference when used sensibly.The authors position quantitative evaluation, comparison, and improvement as central to progress in machine learning.
  • Discussion: Precomputed reference posteriors, an interactive website, and open-source code support rapid algorithm comparison and community contributions.The website is sbi-benchmark.github.io and the code is available at github.com/sbi-benchmark/sbibm.

A.1 Rejection Approximate Bayesian Computation (REJ-ABC)

The appendix describes classical ABC and neural SBI approaches used in the benchmark, emphasizing their sampling mechanisms, sequential variants, and practical limitations.

  • REJ-ABC: Rejection ABC replaces likelihood evaluation with a distance-based comparison between observed and simulated data.It accepts parameters whose simulations lie within threshold ϵ of the observation.
  • REJ-ABC: Rejection ABC approaches the true posterior as ϵ →0 and N →∞, but accuracy trades off against simulation budget and rejection strictness.With linear growth in data dimensionality, maintaining accuracy requires an exponential increase in simulation budget.
  • REJ-ABC: The benchmark used quantile-based rejection, retaining 100 parameters with the smallest distances for budgets of 1k, 10k, and 100k simulations.A KDE fitted to accepted parameters supplied 10k samples for metric computation.
  • SMC-ABC: SMC-ABC constructs intermediate sample populations that gradually approach the final approximate posterior.The benchmark used the ABC-PMC scheme of Beaumont et al. (2009) as its SMC-ABC implementation.
  • SMC-ABC: SMC-ABC can improve sampling efficiency over rejection ABC but requires tuning the distance measure, perturbation kernel, and ϵ-schedule.It also suffers from the curse of dimensionality.
  • NLE: NLE trains a conditional neural density estimator qψ(x|θ) on prior-sampled simulations and uses MCMC with the learned likelihood approximation.The estimator approximates the likelihood within the prior support when simulations are sufficient and model flexibility is adequate.
  • Sequential neural methods: Sequential SBI focuses simulations through proposal distributions because prior-wide simulations may contain little information about the posterior for one observation.SNRE refines a classifier across rounds using proposal-weighted positive and negative examples, at the cost of training new classifiers for different observations.
  • RF-ABC: RF-ABC applies only to one-dimensional parameters or multidimensional settings where the posterior factorizes, ignoring posterior correlations.This restriction limited its applicability to only a few benchmark tasks.

B Benchmark

The benchmark constructs reference posteriors, implements algorithms and metrics, and evaluates them across ten tasks. Extensive cross-task hyperparameter exploration supports systematic comparison and robust algorithm development.

  • Reference posteriors: Reference posteriors were generated analytically for several Gaussian tasks and with custom or likelihood-based procedures for more complex tasks.The Gaussian and Two Moons tasks use task-specific constructions, while SLCP, SIR, and Lotka-Volterra use proposal sampling followed by density estimation.
  • Reference posteriors: The reference-posterior construction produced indistinguishable results across two independent runs under C2ST.This sanity check was performed across all tasks and observations.
  • Benchmark framework: The benchmark evaluates approximate posteriors against reference posteriors using algorithms, performance metrics, and inference tasks.It includes reference posterior samples and tooling for running and analysing results at scale.
  • Code and integration: The framework is open, modular, and designed to integrate algorithms from different SBI toolboxes.It can compare implementations across toolboxes rather than serving as a benchmark for one specific toolbox.
  • Additional analyses: Additional evaluations cover ABC adjustments, RF-ABC, synthetic likelihood, metric correlations, and MMD behaviour on Two Moons.These analyses expose method assumptions, computational requirements, and metric sensitivity.
  • Hyperparameter selection: Benchmark analyses examine hyperparameter selection and sensitivity, using cross-task tuning rather than task-specific optimization.More than 10,000 runs explored general settings intended to avoid overfitting individual tasks.

H.1 REJ-ABC

The study tunes classical ABC and neural likelihood methods across tasks, emphasizing general settings and implementation choices. Sequential algorithms require careful MCMC and density-estimation decisions, while estimator preferences differ by method.

  • REJ-ABC: REJ-ABC uses an l2 distance and quantile-based acceptance tolerances varied across simulation budgets and tasks.The implementation retains samples with the smallest distances to the observation.
  • REJ-ABC: The 0.1, 0.01, 0.001 quantile schedule performed best across tasks and budgets, corresponding to the top 100 samples at each budget.Adding KDE improved performance, especially on Gaussian tasks, so the main-paper configuration uses the top 100 samples and KDE.
  • SMC-ABC: SMC-ABC performed best with population size 100, kernel covariance scale 0.5, and epsilon quantile 0.2.The two implementations were qualitatively similar, while version A was slightly better on Gaussian tasks.
  • (S)NLE and (S)NRE: Neural likelihood methods use MCMC, and modified Slice Sampling schemes improved performance and speed.Multiple chains help sample multimodal posteriors, transformations avoid bounded-support failures, and vectorization enables parallel likelihood evaluations.
  • Density estimators: For (S)NLE, MAFs were selected over NSFs because NSFs generally performed worse across tasks.NSFs significantly reduced performance on SIR and Lotka-Volterra, indicating that their added flexibility was unnecessary there.

H.5 Density estimator for (S)NPE

For (S)NPE, neural spline flows generally outperform masked autoregressive flows in C2ST across tasks. The advantage is especially evident on tasks with complex or multimodal posterior structure.

  • Complex posterior structure: NSF flexibility improved direct posterior estimation on complex multimodal tasks such as Two Moons and SLCP.Artifacts from density transformation visible in Two Moons posteriors disappeared when using NSFs.
  • Novelty: Results on (S)NPE with NSFs had not previously been published to the authors’ knowledge.
  • Density estimator comparison: NSFs generally performed better than MAFs for (S)NPE, especially on Gaussian Mixture, Two Moons, and SIR.The benchmark therefore reports (S)NPE results using NSFs.
  • Related architecture choice: ResNets generally outperformed MLPs for (S)NRE classification, particularly on Two Moons and SIR at low to medium simulation budgets.The main paper therefore uses the ResNet architecture.

M Metrics

The benchmark compares several posterior-quality metrics and finds that their reliability depends strongly on estimation details and posterior structure. C2ST is therefore used for reported performance, while other metrics serve narrower diagnostic roles.

  • NLTP: In the infinite-observation limit, average NLTP converges to an average KL divergence plus an entropy term shared across algorithms.The KL term compares true and approximate posteriors over observations generated from the prior.
  • Metric cost: SBC and NLTP require inference over many observations, making them prohibitively expensive for most benchmark algorithms.The benchmark therefore does not rely on SBC, and avoids probability-of-true-parameter evaluation across hundreds of observations.
  • SBC: SBC checks whether data-averaged posterior samples follow the prior when the posterior approximation is exact.It is only a consistency check and cannot detect an approximate posterior equal to the prior.
  • MEDDIST: MEDDIST compares posterior predictive samples with observed data but is treated as a check because it may not test posterior structure.The benchmark uses 10000 posterior predictive samples, requiring additional simulator evaluations.
  • MMD: MMD with simple Gaussian kernels can have low test power for multimodal posteriors and is sensitive to kernel length-scale choices.On Two Moons, the median heuristic contradicted visual and C2ST assessments, whereas a manually chosen length scale agreed with them.
  • C2ST: C2ST trains a classifier to distinguish true from approximate posterior samples, with chance-level accuracy equal to 0.5 when samples are indistinguishable.The benchmark uses a two-layer neural network and five-fold cross-validation.

R Runtimes

The benchmark records algorithm runtimes under a standardized single-CPU setup because simulation generation was optimized and simulations are not the dominant cost in this benchmark. ABC methods are fastest, while sequential neural methods take longer because they train networks across rounds.

  • Runtime measurement: Runtime comparisons used single CPU cores across algorithms and tasks, without GPU training for neural networks.ABC and neural-network runs used different CPU hardware, with less than 100 ms runtime difference when ABC algorithms were run on the neural-network hardware.
  • Algorithm comparisons: REJ-ABC and SMC-ABC had short runtimes because they require neither neural-network training nor MCMC.
  • Algorithm comparisons: Sequential versions of all three neural-network algorithms took longer than non-sequential versions because they involve 10 rounds of neural-network training.
  • Algorithm comparisons: Among sequential algorithms, SNPE had the longest runtimes, while MAFs tended to run faster than NSFs.For SNPE on SLCP at 100k simulations, the passage begins reporting a MAF-versus-NSF difference but does not provide the complete value.

T Tasks

The benchmark spans analytical, mixture, distractor, generalized-linear, and dynamical-system inference tasks with varied posterior structure and observation designs. These tasks include Gaussian models, nonlinear and multimodal posteriors, epidemiological dynamics, and ecological interactions.

  • Gaussian tasks: The Gaussian tasks infer the mean of a 10-dimensional model with fixed covariance under either a conjugate Gaussian prior or a uniform prior.
  • Nonlinear and distractor tasks: The SLCP task uses four two-dimensional observations from a Gaussian likelihood whose mean and variance are nonlinear functions of five parameters, producing a complex posterior.
  • Nonlinear and distractor tasks: A related task adds uninformative dimensions to the observation, with 100 reordered observation dimensions.
  • Generalized linear tasks: The benchmark includes Bernoulli GLM tasks with 10 parameters, comparing sufficient-statistic observations with raw observations and using smoothness-encouraging Gaussian priors.Both GLM variants use duration T = 100.
  • Mixture and multimodal tasks: The benchmark includes mixture and multimodal tasks, including a two-dimensional Gaussian mixture with unequal covariances and a posterior combining bimodality with crescent-shaped local structure.

W Website

The companion website supports interactive comparison of benchmark results and inspection of posterior samples. These views were useful for selecting hyperparameters and diagnosing implementation issues.

  • Website capabilities: The companion website allows interactive comparisons across all metrics and inspection of posterior samples from every run.
  • Website capabilities: Posterior inspection helped with hyperparameter selection and diagnosis of implementation issues.
  • Website screenshots: Figure 13 shows C2ST classification accuracy for selected sequential algorithms on SLCP and an SNLE posterior for x(1).
Loading 2101.04653v2…