PaperlayerView exact revision on arXiv

Source-linked AI summary

Likelihood-free inference by ratio estimation

Owen Thomas, Ritabrata Dutta, Jukka Corander, Samuel Kaski, Michael U. Gutmann

arXiv:1611.10242v6stat.MLstat.COstat.ME

TL;DR

Likelihood-free inference is needed when simulator-based models can be sampled but their likelihoods are too costly to compute. The paper introduces LFIRE, using logistic-regression ratio estimation to approximate posteriors and automatically select summaries, demonstrating selection from 291 candidates.

  • Problem

    Simulator-based inference remains difficult when likelihood evaluation is prohibitively costly or unavailable despite the ability to generate model samples.

  • Method

    LFIRE approximates posterior distributions through density-ratio estimation with logistic regression and regularization that selects relevant summary statistics.

  • Results

    Linear LFIRE selected an average of 17.3, 23.9, and 30.5 summary statistics from 291 as simulations increased from 50 to 100 and 150.

  • Takeaways & Limitations

    The method can remove irrelevant summaries, adaptively include more as computational resources increase, and improve inference interpretability.

  • Takeaways & Limitations

    Synthetic-likelihood approaches generally require transforming summary statistics to satisfy their multivariate Gaussianity assumption.

Abstract

from arXiv · show

We consider the problem of parametric statistical inference when likelihood computations are prohibitively expensive but sampling from the model is possible. Several so-called likelihood-free methods have been developed to perform inference in the absence of a likelihood function. The popular synthetic likelihood approach infers the parameters by modelling summary statistics of the data by a Gaussian probability distribution. In another popular approach called approximate Bayesian computation, the inference is performed by identifying parameter values for which the summary statistics of the simulated data are close to those of the observed data. Synthetic likelihood is easier to use as no measure of `closeness' is required but the Gaussianity assumption is often limiting. Moreover, both approaches require judiciously chosen summary statistics. We here present an alternative inference approach that is as easy to use as synthetic likelihood but not as restricted in its assumptions, and that, in a natural way, enables automatic selection of relevant summary statistic from a large set of candidates. The basic idea is to frame the problem of estimating the posterior as a problem of estimating the ratio between the data generating distribution and the marginal distribution. This problem can be solved by logistic regression, and including regularising penalty terms enables automatic selection of the summary statistics relevant to the inference task. We illustrate the general theory on canonical examples and employ it to perform inference for challenging stochastic nonlinear dynamical systems and high-dimensional summary statistics.

1 Introduction

The paper addresses posterior inference for simulator-based models when likelihood evaluation is intractable but simulation is possible. It introduces LFIRE, which estimates density ratios by logistic regression and supports automatic summary-statistic selection while including synthetic likelihood as a special case.

  • Problem setting: Likelihood-free inference is needed when the model likelihood is too costly or unavailable in closed form, despite the ability to generate simulated data.The setting permits highly complex simulators without imposing particular conditions on the data space.
  • Existing methods: Existing ABC methods compare summary-statistic distances, whereas synthetic likelihood assumes Gaussian summary statistics and both depend on suitable summaries.ABC similarity is typically assessed after reducing observed and simulated data to summary statistics.
  • Method: The framework uses logistic regression to estimate density ratios and introduces linear LFIRE as a practical method for posterior approximation.Logistic regression is used because it is closely related to probabilistic classification and has favourable asymptotic properties.
  • Contributions: LFIRE includes synthetic likelihood as a special case and naturally enables automatic selection of summary statistics relevant to the inference task.The paper later interprets linear LFIRE as an exponential-family approximation, which includes Gaussian distributions.
  • LFIRE framework: LFIRE estimates the ratio between the data-generating distribution p(x|θ) and marginal distribution p(x), yielding posterior and likelihood estimates without direct likelihood computation.The denominator p(x) is independent of θ, enabling the ratio-based likelihood-free framework.

2 Posterior estimation by logistic regression

LFIRE estimates the ratio p(x|θ)/p(x) by logistic regression, producing posterior and likelihood estimates without evaluating likelihoods directly. The framework supports classification across θ values and can specialize the log-ratio model using summary statistics.

  • 2 Posterior estimation by logistic regression: LFIRE estimates r(x, θ) by logistic regression, yielding estimates of the posterior and likelihood function.The ratio is estimated by classifying data from p(x|θ) against data from the marginal p(x).
  • 2 Posterior estimation by logistic regression: The classification distinguishes samples from p(x|θ) and p(x), with class-size correction ν = n_m/n_θ and larger h(x) indicating greater probability of origin from p(x|θ).The loss is minimized on independently generated conditional and marginal samples.
  • 2 Posterior estimation by logistic regression: For large n_m and n_θ, the minimizing function h* equals the log-ratio between p(x|θ) and p(x).This result holds for every θ represented in the classification data.
  • 2 Posterior estimation by logistic regression: Averaging the loss over θ, including under the prior p(θ), performs classification in joint x, θ space and enables amortization of computation.The prior-based construction classifies samples from p(x|θ)p(θ) against p(x)p(θ).
  • 2 Posterior estimation by logistic regression: Different classifier families yield different LFIRE assumptions, including a parametric family spanned by summary statistics.Non-parametric families and deep architectures are also possible.

3 Exponential family approximation

Restricting the ratio model to a linear basis of summary statistics yields linear LFIRE, where coefficients are estimated by logistic regression. The resulting implicit model belongs to the exponential family, which includes Gaussian synthetic likelihood as a special case.

  • 3 Exponential family approximation: Linear LFIRE restricts h(x) to a linear combination of b summary statistics and estimates the ratio through logistic regression.The basis functions map data x to R, with coefficient vector β.
  • 3 Exponential family approximation: The posterior and likelihood estimates are proportional to exp(β̂(θ)⊤ψ(x0)), multiplied by the prior for the posterior.Summary statistics may incorporate the observed data x0, thereby influencing the logistic-regression stage.
  • 3 Exponential family approximation: The estimated coefficients β̂ depend on θ because the simulated training data depend on θ.The ratio estimate is obtained by minimizing J(β, θ) with respect to β.
  • 3 Exponential family approximation: The implicit estimated model density belongs to the exponential family, with ψ(x) as sufficient statistics and β̂(θ) as natural parameters.The density is implicit because the marginal pdf p(x) is not explicitly estimated.
  • 3 Exponential family approximation: Gaussian synthetic likelihood is a special case of this exponential-family approximation when its summary statistics φ correspond to the individual basis statistics ψ.This also includes the approximation used by Leuenberger and Wegmann (2010).

4 Data-driven selection of summary statistics

Linear LFIRE uses L1-regularised logistic regression to select relevant summary statistics jointly with posterior estimation. This data-driven selection supports large candidate sets and improves interpretability, although cross-validation adds computational cost.

  • Selection: Regularisation enables automatic, data-driven selection from a large list of candidate summary statistics.This makes posterior inference less dependent on subjective user input.
  • Interpretability: The selected summaries may be few enough for statisticians and model experts to examine, substantially increasing interpretability.The approach is presented as making posterior inference more robust and less dependent on subjective user input.
  • Procedure: Linear LFIRE estimates the posterior by fitting the penalised logistic regression, selecting λ by ten-fold cross-validation, and using the resulting coefficients in the density-ratio estimate.The procedure jointly performs summary-statistic selection and posterior estimation.
  • Regularisation: The L1 penalty can force some regression coefficients exactly to zero, identifying which summary statistics contribute to the posterior approximation.The coefficients determine the extent to which each summary statistic contributes.
  • Limitations: Cross-validation adds computational cost, and θ-dependent λmin can complicate theoretical analysis; fixing λ can reduce cost or facilitate analysis.The paper gives fixed regularisation as an alternative for these purposes.

5 Validation on canonical low-dimensional problems

The proposed linear LFIRE method is validated on canonical low-dimensional inference problems, where it selects relevant summaries and accurately estimates posteriors. It outperforms synthetic likelihood on a non-Gaussian dynamical system and remains robust to irrelevant summaries.

  • Gaussian mean: The estimated Gaussian-mean posterior matched the true posterior up to random fluctuations, performing comparably to synthetic likelihood for Gaussian data.
  • ARCH(1): Average symmetrised Kullback-Leibler divergence decreased with more simulated data sets for LFIRE, unlike synthetic likelihood, reflecting better handling of non-Gaussian summaries and relevant-summary selection.
  • ARCH(1): 82% without and 83% with irrelevant summaries of the ∆sKL distribution lay on the negative side, with Wilcoxon signed-rank p-values < 10−10 favoring LFIRE.The results indicate superior performance over synthetic likelihood and robustness to perturbing irrelevant summary statistics.

6 Bayesian inference for nonlinear dynamical systems

The study applies linear LFIRE to Bayesian inference in the intractable-likelihood Ricker and Lorenz nonlinear dynamical systems, comparing it with synthetic likelihood. Across both applications, LFIRE generally improves posterior estimation and yields clearer gains for short-term weather prediction.

  • Models and comparison: Linear LFIRE is evaluated on the ecological Ricker model and the stochastic Lorenz weather model, both of which have intractable likelihood functions.The comparison uses synthetic likelihood as the reference approach.
  • Ricker model: The Ricker analysis estimates log r, σ, and φ from a length-50 population time series using 13 summary statistics, their pairwise combinations, and a constant.The observed data were generated with θ0 = (3.8, 0.3, 10) under uniform parameter priors.
  • Lorenz model: The Lorenz analysis estimates two closure parameters from 40 slow weather variables observed over twenty days, using six summary statistics and their pairwise combinations.The same summary-statistic construction was used for synthetic likelihood and LFIRE to make the comparison fair.
  • Inference procedure: 10,000 prior samples were used for importance sampling, with weights computed by Algorithm 1; synthetic likelihood used a robust variance-covariance estimator.The importance-sampling procedure is equivalent to one generation of the SMC algorithm.
  • Posterior estimation: The empirical distributions of Δrel−error were tilted toward negative values for all parameters in both models, indicating generally better performance for LFIRE than synthetic likelihood.Negative Δrel−error means the proposed method has the smaller relative error.
  • Weather prediction: LFIRE produced a clear average improvement in Lorenz prediction performance over the first forecast days, while the gain became smaller for longer-term forecasts.The prediction horizon covered ten days, from t ∈ [4, 6], and the reduced long-term gain reflects the difficulty of forecasting chaotic time series.

7 Inference with high-dimensional summary statistics

LFIRE enables inference for a stochastic cell-spreading model using 291 candidate summary statistics by selecting those most informative for the posterior. With more training simulations, posterior characterization and MAP estimates improve, while the selected statistic set expands.

  • Model and summaries: LFIRE inferred motility and proliferation parameters, Pm and Pp, from 14 summary statistics based on cell counts and image-grid Hamming distances.The model used the total final cell count and Hamming distances between cell-population image grids at each simulation time point.
  • Inference method: Linear LFIRE with lasso-type regularisation selected summary statistics most informative for characterising the posterior.Inference used true values Pm = 0.35 and Pp = 0.001, with nθ = nm ∈{50, 100, 150}.
  • Inference results: As nθ = nm increased from 50 to 150, the posterior became more stable and MAP estimates clearly improved.The results were consistent with earlier synthetic-likelihood results, which used nθ ∈{2500, 5000, 10000}.
  • Summary-statistic selection: 17.3, 23.9, and 30.5 summary statistics were selected on average at nθ = nm = 50, 100, and 150, respectively, from 291 candidates.The number of selected statistics increased as the classifier received more training data.

8 Computational aspects

LFIRE requires more computation per posterior estimate than synthetic likelihood because it adds reference-data construction and penalised logistic regression, whereas synthetic likelihood inverts a covariance matrix. However, when simulations have non-trivial computational demands, simulation dominates total runtime for both methods.

  • Computational costs: Both methods require simulating X_θ, while LFIRE additionally constructs X_m and performs penalised logistic regression, potentially with costly cross-validation.Synthetic likelihood instead requires covariance-matrix inversion for each θ.
  • Timing results: 1 second simulations were used for a hypothetical inexpensive model, with assumed likelihood-proxy evaluation times of 0.09 seconds for synthetic likelihood and 10 seconds for LFIRE.These assumptions extend the timing comparison beyond the ARCH and cell-spreading models.
  • Timing results: For very cheap ARCH simulations, LFIRE spends most of its computational time on posterior estimation.For moderately more expensive cell-spreading simulations, posterior-estimation costs remain dominant but become more balanced with simulation costs.
  • Timing results: Despite higher per-estimate cost, LFIRE and synthetic likelihood both spend most of their runtime generating data when simulators have non-trivial computational demands.The comparison uses relative simulation and posterior-estimation times rather than platform-specific absolute times.

9 Discussion

The discussion presents likelihood-free inference by ratio estimation as a logistic-regression density-ratio framework for posterior estimation when likelihoods are unavailable but model sampling is possible. It emphasizes automated summary-statistic selection, effective handling of high-dimensional summaries, and computational costs dominated by simulation.

  • The method frames posterior estimation as density-ratio estimation solved by nonlinear logistic regression, connecting the approach to classification and contrastive learning.
  • Sampling several model datasets is the main computational cost, although simulations can be performed perfectly in parallel.
  • Automated selection and combination of summary statistics from a large candidate pool improves interpretability and can remove irrelevant statistics or add relevant ones as computational resources increase.
  • The approach handles high-dimensional summary statistics, while likelihood-free inference over high-dimensional parameter spaces remains challenging without a generally accepted solution.
  • A linear basis expansion with logistic regression already generalizes synthetic likelihood and typically provides more accurate estimation, while richer regressions and loss functions remain possible.
Loading 1611.10242v6…