PaperlayerView exact revision on arXiv

Source-linked AI summary

Approximate Bayesian computation with the Wasserstein distance

Espen Bernton, Pierre E. Jacob, Mathieu Gerber, Christian P. Robert

arXiv:1905.03747v1stat.ME

TL;DR

Likelihood-free inference is useful when generative models can simulate data but their likelihoods cannot be evaluated, yet summary-based ABC can lose information. The paper replaces summaries with Wasserstein distances between empirical distributions, develops scalable and time-series variants, and shows convergence and concentration properties alongside empirical performance that can match or exceed summary-based ABC in some examples.

  • Problem

    Likelihood-free ABC must compare observed and simulated data when likelihoods are unavailable, but summary-based comparisons can lose information and require a difficult trade-off with simulation variability.

  • Method

    The paper uses Wasserstein distances between empirical distributions in ABC, with Hilbert-curve approximations, time-series extensions, and combinations with summaries.

  • Results

    The WABC posterior converges to the posterior as ε →0 and, in some examples, is at least as close to the posterior as state-of-the-art summary-based ABC, while being computationally cheaper in the toggle-switch example.

  • Takeaways & Limitations

    Wasserstein ABC can identify high-posterior-density regions for both independent and dependent data and can be combined with summaries to identify remaining parameters.

  • Takeaways & Limitations

    Asymptotic dependence of WABC on observation dimension remains insufficiently analyzed, and under misspecification its concentration target can differ from the standard posterior target.

Abstract

from arXiv · show

A growing number of generative statistical models do not permit the numerical evaluation of their likelihood functions. Approximate Bayesian computation (ABC) has become a popular approach to overcome this issue, in which one simulates synthetic data sets given parameters and compares summaries of these data sets with the corresponding observed values. We propose to avoid the use of summaries and the ensuing loss of information by instead using the Wasserstein distance between the empirical distributions of the observed and synthetic data. This generalizes the well-known approach of using order statistics within ABC to arbitrary dimensions. We describe how recently developed approximations of the Wasserstein distance allow the method to scale to realistic data sizes, and propose a new distance based on the Hilbert space-filling curve. We provide a theoretical study of the proposed method, describing consistency as the threshold goes to zero while the observations are kept fixed, and concentration properties as the number of observations grows. Various extensions to time series data are discussed. The approach is illustrated on various examples, including univariate and multivariate g-and-k distributions, a toggle switch model from systems biology, a queueing model, and a Lévy-driven stochastic volatility model.

1 Introduction

The paper replaces summary-based ABC discrepancies with Wasserstein distances between empirical distributions, aiming to reduce information loss while retaining likelihood-free inference. It develops scalable approximations, time-series extensions, and theoretical guarantees for the resulting WABC posterior.

  • Motivation: ABC simulates parameters and synthetic data, retaining parameters whose data resemble observations, but summary statistics can systematically lose information.Using more summaries can reduce information loss while increasing distance variability across simulations.
  • Wasserstein ABC: WABC compares observed and synthetic empirical distributions with the Wasserstein distance, bypassing the choice of summaries.In one dimension, this generalizes matching sorted order statistics; in arbitrary dimensions, it uses optimal transport distances.
  • Theory: The WABC posterior converges to the standard posterior as ε →0 under stated conditions, while its large-sample behavior can differ under misspecification.The asymptotic target is the parameter minimizing Wasserstein distance to the limiting data distribution, rather than necessarily the KL minimizer.
  • Computation and extensions: Hilbert space-filling-curve distances approximate Wasserstein distances in low dimensions while reducing computational cost, and two strategies extend the approach to time series.Curve matching augments observations with time; reconstructions transform series so empirical distributions retain parameter information.
  • Problem setting: The paper defines generative-model inference when data can be simulated for each parameter but the likelihood cannot be numerically evaluated.The framework does not assume a particular mechanism for generating model data.

2 Wasserstein ABC

Wasserstein ABC replaces the generic ABC discrepancy with an empirical Wasserstein distance. Its theoretical behavior includes convergence to the standard posterior at vanishing threshold and concentration around a Wasserstein projection as sample size grows.

  • Definition: The WABC posterior is the ABC posterior obtained by replacing the discrepancy D with Wp between observed and synthetic data.The experiments use p = 1, requiring minimal assumptions on data-generating moments.
  • Fixed-data theory: As ε →0, the WABC posterior converges to the true posterior under the paper’s stated conditions.This parallels the corresponding result for ABC based on Euclidean distance.
  • Large-sample theory: As n →∞ with a slowly decreasing threshold, WABC concentrates around θ⋆ := argminθ∈H Wp(µ⋆, µθ), which can differ from the posterior target under misspecification.In well-specified models, θ⋆ coincides with the data-generating parameter; in misspecified models, it typically differs from the KL minimizer.

2.1 Sampling sequentially from the WABC posterior

The paper samples sequentially from WABC using an SMC algorithm that gradually lowers the threshold while preserving particle diversity. The implementation is parallelizable and uses configurable rejuvenation kernels and tuning parameters.

  • Sequential sampling: The SMC sampler starts at ε0 = +∞, where WABC equals the prior, then selects each next threshold to retain at least αN unique particles.Particles are propagated and rejuvenated as the threshold decreases.
  • Computation: The algorithm is parallelizable across N particles and their corresponding model simulations and distance calculations.This parallel structure applies equally to the simulation and discrepancy-evaluation workload.
  • Implementation: The implementation permits any MCMC kernel for rejuvenation and uses r-hit proposals with default settings N = 2,048 and α = 50%.The paper uses two hits and a five-component multivariate Normal proposal mixture by default.

2.2 Illustration on a Normal location model

A bivariate Normal location example compares Wasserstein-based ABC with Euclidean and summary-statistic alternatives. The SMC-based Wasserstein method approximates the posterior nearly as well as sufficient statistics, while Euclidean ABC performs poorly.

  • Model setup: 100 bivariate Normal observations are modeled with an unknown mean vector and covariance having diagonal 1 and off-diagonal 0.5.The mean has a centered Normal prior with variance 25 on each component.
  • Methods compared: The experiment compares Wasserstein ABC, Euclidean-distance ABC, and ABC based on Euclidean distance between sample means using SMC sampling.The summary means are sufficient statistics for this model, and all methods use a budget of 10^6 model simulations.
  • Posterior accuracy: The SMC-based Wasserstein and sufficient-statistic ABC methods both approximate the posterior accurately.The posterior approximations are evaluated against independently sampled exact posterior draws using the Wasserstein distance.
  • Posterior accuracy: Wasserstein ABC performs almost as well as sufficient-statistic ABC but requires more model simulations for comparable accuracy.The comparison uses Wasserstein distances between 2,048 ABC samples and 2,048 exact posterior samples.
  • Posterior accuracy: Euclidean-distance ABC struggles to approximate the posterior accurately, seemingly requiring billions of simulations to match the other methods.The figure plots the discrepancy against the number of model simulations on a log-log scale.
  • Computational cost: Per-sample distance evaluation averages 6.4 × 10^-5s for summary statistics, 3.8 × 10^-4s for Euclidean distance, and 1.2 × 10^-2s for Wasserstein distance.These timings motivate the paper’s discussion of faster Wasserstein approximations.

2.3 Computing and approximating the Wasserstein distance

The section develops exact and approximate strategies for computing Wasserstein distances in ABC. It covers sorting-based univariate computation, multivariate assignment methods, Hilbert and swapping distances, regularized Sinkhorn methods, and subsampling trade-offs.

  • Exact computation: Univariate Wasserstein distance computation costs O(n log n) because the optimal assignment is obtained by sorting the samples.The assignment pairs observations according to their sorted orders.
  • Exact computation: Multivariate Wasserstein computation can cost O(n^3) with the Hungarian algorithm, while other methods have lower empirical or conditional costs.A short-list method is used in the numerical experiments and has no guaranteed polynomial running time.
  • Regularized approximations: Entropic regularization enables Sinkhorn iterations costing O(n^2) per iteration and interpolates between Wasserstein distance and maximum mean discrepancy.As ζ → 0 the dual-Sinkhorn divergence approaches Wasserstein distance; as ζ → ∞ it approaches maximum mean discrepancy.
  • Hilbert distance: The Hilbert distance sorts multivariate samples along a Hilbert space-filling curve, yielding an empirical-distribution distance computable in O(n log n).The construction uses a one-to-one mapping into a univariate ordering and defines a valid distance for empirical distributions of equal size.
  • Hilbert distance: For small observation dimension, Hilbert distance accurately approximates Wasserstein distance and has similar parameter-discrimination level sets, but this degrades as dimension grows.The Hilbert-sorting coupling is feasible for the assignment problem, so its distance is at least the Wasserstein distance.
  • Swapping distance: The swapping distance starts from Hilbert sorting, improves assignments through pairwise swaps, and lies between Wasserstein and Hilbert distances.Each sweep costs O(n^2), and optimality is not guaranteed; the distance may approximate Wasserstein more accurately as dimension increases.
  • Acceleration: Subsampling m < n observations reduces computation time but increases distance variance, creating a computation–variance trade-off.Other multiscale approaches can also accelerate distance evaluation.

3 Theoretical properties

The paper establishes fixed-data posterior convergence as ε→0 and concentration results as n grows while the threshold shrinks. These guarantees depend on assumptions about empirical-distribution convergence, model approximation, prior mass, separation, and identifiability.

  • Behavior as n increases and ε decreases: The theoretical assumptions are typically difficult to verify for complex models with intractable likelihoods, and the stated concentration-rate bound may not be optimal.The paper identifies dimension dependence of Wasserstein convergence as an open research issue.
  • Behavior as ε→0 for fixed observations: As ε→0 with observations fixed, the ABC posterior converges strongly to the posterior under continuity and zero-discrepancy conditions.The result applies to discrepancies that distinguish observed data up to permutation or exactly, as specified by the proposition’s alternatives.
  • Behavior as ε→0 for fixed observations: The Wasserstein distance on unmodified data satisfies the required zero-discrepancy condition up to permutation, so the fixed-data convergence result applies.For time series, the paper discusses separate discrepancies designed to satisfy the alternative condition.
  • Behavior as n increases and ε decreases: For fixed ε>0 and n→∞, the WABC posterior converges to parameters whose model distributions lie within the Wasserstein threshold, rather than to a Dirac mass.The paper notes that this behavior can make WABC less sensitive to perturbations under model misspecification.
  • Behavior as n increases and ε decreases: With ε_n→0 at suitable rates and empirical Wasserstein convergence, the WABC posterior concentrates around parameters minimizing Wp(µθ, µ⋆).The required rate sequence must be slower than both the empirical-data convergence rate and the simulated-model convergence rate.
  • Behavior as n increases and ε decreases: Under additional separation and identifiability assumptions, the concentration rate is bounded by δ_n = K(4ε_n/3 + f_n^-1(ε_n^L/R))^α.The bound depends on observation-space dimension, Wasserstein order p, and model misspecification through α.

4 Time series

For time series, the paper extends Wasserstein ABC with curve matching and delay reconstruction to retain temporal dependence that marginal empirical distributions can miss. These methods improve parameter identification in cosine and AR(1) examples, while their accuracy depends on distance settings and reconstruction choices.

  • Curve matching: Curve matching augments each observation with its time index and uses a Wasserstein distance that interpolates between Euclidean curve distance and marginal-distribution distance.The parameter λ controls this compromise: smaller λ permits matching observations at different times, while λ approaching zero ignores time.
  • Curve matching: For any λ > 0, the curve-matching WABC posterior converges to the standard posterior as ε →0.The distance can be computed exactly or approximated and combined with the SMC algorithm.
  • Curve matching: In the cosine model, curve matching estimates the noise standard deviation σ better than Euclidean distance between time series after 10^6 simulations.The comparison uses a Metropolis–Hastings approximation based on the last 50,000 samples of a 100,000-step chain.
  • Delay reconstruction: Delay reconstruction forms vectors from current and lagged observations, allowing Wasserstein comparison of empirical distributions that capture more dependency structure than marginal distributions.The paper contrasts this approach with summarizing lagged pairs through autocovariances alone.
  • Delay reconstruction: Ignoring temporal dependence in an AR(1) model leaves φ and σ^2 unidentified because the marginal distribution depends only on σ^2/(1 −φ^2).The marginal-distribution WABC posterior concentrates around a ridge of parameter values with the same ratio.

5 Numerical experiments

The numerical experiments show that Wasserstein ABC can approach target posteriors across several models, while Hilbert-based distances reduce computation and can be combined with summaries for time series.

  • Univariate g-and-k: WABC posterior marginals appeared closer to the target than semi-automatic ABC, especially for a, b, and k in the univariate g-and-k example.Neither method captured g well, although WABC was more concentrated in the region of interest.
  • Univariate g-and-k: The joint W1-distance decreased from around 10 to around 0.06 during the univariate g-and-k SMC run.The comparison used 2,048 posterior samples and 2,048 SMC particles at each step.
  • SMC behavior: ε decreased to 0.07, while the number of model simulations per SMC step increased as smaller thresholds became harder to reach.The increasing simulation count resulted from the r-hit kernel requiring more attempts for the desired number of hits.
  • Bivariate g-and-k: All four distances produced ABC posteriors increasingly close to the true posterior, but all remained significantly different at the tested simulation budgets.The MMD thresholds are not directly comparable with Wasserstein thresholds because they use different scales.
  • Computational cost: The Hilbert distance required 0.002s per dataset, compared with 0.22s for exact Wasserstein distance at n = 500.The same comparison reported 0.01s for MMD and 0.03s for the swapping distance.
  • Lévy-driven stochastic volatility: In the stochastic-volatility model, combining Hilbert distance with summaries concentrated WABC posteriors closer to data-generating values, particularly for λ.The second run used an additional 6.6 × 10^5 model simulations.

6 Discussion

The discussion presents Wasserstein ABC as a summary-free approach with scalable approximations, theoretical guarantees under specified regimes, and important limitations for fixed thresholds and time-series tuning.

  • Scope and contribution: Wasserstein ABC bypasses summary selection and provides a way to infer parameters in generative models, including deterministic and dependent-data settings.The paper reports posterior approximations at least as close to the posterior as summary-based ABC in some examples.
  • Empirical findings: The toggle-switch example produced more concentrated posterior approximations around data-generating parameters than sophisticated case-specific summaries while being computationally cheaper.The paper also demonstrates combining summaries and transport distances.
  • Time-series scope: Time-series approaches require tuning parameters such as the curve-matching λ or delay-reconstruction lags.The discussion points to prior work for guidance on choosing these parameters.
  • Theory: For fixed observations, WABC converges to the posterior as ε goes to zero under stated conditions, whereas with growing n it can concentrate in regions different from the posterior.The concentration rates depend on the Wasserstein order, observation-space dimension, and model misspecification.
  • Limitations: ABC posteriors at fixed ε need not provide conservative statements about the posterior and can have little overlap with it despite concentrating away from the prior.The discussion identifies this as a limitation for fixed thresholds and datasets.
  • Scalability: The Hilbert distance is computable in order n log n, while the swapping distance it can initialize costs order n^2.This addresses the super-quadratic scaling of Wasserstein calculations with observation count.

A Proofs

The proofs establish convergence and concentration properties by bounding ABC posterior probabilities through empirical-measure approximation and Wasserstein-distance inequalities.

  • Posterior bounds: The proof treats the WABC posterior probability of parameters whose population Wasserstein distance from the target exceeds δ.It keeps the observed empirical distribution fixed while bounding numerator and denominator terms.
  • Posterior bounds: The triangle inequality decomposes the discrepancy between target and model distributions into observed-sample error, synthetic-sample error, and the ABC threshold.This decomposition underlies the subsequent probability bounds.
  • Empirical-measure control: The proof restricts attention to observed datasets whose empirical measure lies within ε/3 of the target measure.The event A(n, ε) is used to control empirical-measure deviations.
  • Empirical-measure control: Under the assumed convergence-rate conditions, the numerator is bounded using the probability that the synthetic empirical measure deviates from its population measure.The resulting bounds are combined after selecting ε and ζ appropriately.
  • Asymptotic conclusion: The proof concludes that the relevant high-probability bounds hold as n grows because the observed data fall in A(n, ε_n) with probability tending to one.This supports the asymptotic concentration argument.
  • Vanishing-threshold limit: For fixed observations and vanishing ε, continuity and metric properties identify the limiting acceptance sets needed for convergence of the quasi-likelihood.The proof uses permutation symmetry in one case and the exact observed vector in another.
Loading 1905.03747v1…