Source-linked AI summary
Non-linear regression models for Approximate Bayesian Computation
M. G. B. Blum, O. Francois
TL;DR
The paper tackles the curse of dimensionality that limits rejection-based ABC when many summary statistics are used. It fits a nonlinear conditional heteroscedastic regression and adaptively improves estimation with importance sampling, reporting reduced computational burden in population-genetic and queueing examples.
Problem
Rejection-based ABC suffers from the curse of dimensionality as the number of summary statistics increases, limiting approximate Bayesian inference based on summaries.
Method
The method estimates parameters from summary statistics with a nonlinear conditional heteroscedastic regression, then adaptively refines estimation using importance sampling.
Results
The NCH and ANCH algorithms reduce computational burden compared with ABC rejection and ABC local linear adjustment in population-genetic and queueing examples.
Takeaways & Limitations
Neural-network-based nonlinear regression reduced computational generation costs in three examples of implicit statistical models.
Takeaways & Limitations
Increasing the number of summary statistics has a dramatic effect, and accurate posterior estimation can come at the expense of computational cost.
Abstract
from arXiv · showhide
Approximate Bayesian inference on the basis of summary statistics is well-suited to complex problems for which the likelihood is either mathematically or computationally intractable. However the methods that use rejection suffer from the curse of dimensionality when the number of summary statistics is increased. Here we propose a machine-learning approach to the estimation of the posterior density by introducing two innovations. The new method fits a nonlinear conditional heteroscedastic regression of the parameter on the summary statistics, and then adaptively improves estimation using importance sampling. The new algorithm is compared to the state-of-the-art approximate Bayesian methods, and achieves considerable reduction of the computational burden in two examples of inference in statistical genetics and in a queueing model.
1 Introduction
The paper addresses the curse of dimensionality in rejection-based ABC by modeling the relationship between summary statistics and parameters, then adaptively refining posterior estimation. In examples from population genetics and queueing, the approach reduces computational burden relative to rejection and local-linear ABC.
- Approximate Bayesian computation: ABC simulates datasets from prior-drawn parameters, compares their summary statistics with observed summaries, and retains sufficiently close parameters as an approximate posterior sample.This likelihood-free approach is useful when likelihoods are analytically or computationally intractable.
- Limitation of rejection methods: Rejection-based ABC methods suffer from the curse of dimensionality as the number of summary statistics increases.These methods sample from a small multidimensional neighborhood around the observed summaries, making acceptance difficult in higher dimensions.
- Proposed approach: The proposed method constructs a functional relationship between generated summary statistics and model parameters so posterior sampling can use the extended simulated set rather than only retained values.This machine-learning perspective is intended to address the dimensionality problem.
- Proposed approach: A flexible nonlinear conditional heteroscedastic model estimates the relationship between summary statistics and parameters while reducing dimension and accounting for correlations among summaries.The first stage uses flexible regression models, including neural networks, for implicit statistical models.
- Proposed approach: The adaptive NCH algorithm uses importance sampling to iteratively limit the discrepancy between the sampling and posterior distributions, especially when the prior is vague.This forms the second stage of the proposed procedure.
- Results: The NCH and ANCH algorithms reduce computational burden compared with ABC using rejection and ABC using local linear adjustment in population-genetic and queueing examples.The paper reports evidence for this result across two historical demographic-inference examples and one queueing-process example.
2 Method
The method develops ABC regression adjustments from local-linear models to nonlinear conditional heteroscedastic regression, then iteratively improves estimation through adaptive importance sampling. These changes address dimensionality-related limitations and reduce computational cost through adaptive posterior approximation.
- Approximate Bayesian computation: ABC replaces the likelihood with an approximation based on the distance between observed and simulated summary statistics, using the prior distribution for parameter inference.When summary statistics are sufficient, the approximate posterior converges to the true posterior as δ approaches zero; for large δ, it approaches the prior.
- Local-linear ABC: The Nadaraya-Watson weighting approach suffers from the curse of dimensionality because convergence deteriorates sharply as the summary-statistic dimension increases.The local-linear ABC model instead fits φi = α + (si − s)^Tβ + ζi, with independent zero-mean errors sharing a common variance.
- Local-linear ABC: Increasing the tolerance δ lowers variance by enlarging the regression-fitting sample but increases bias when the relationships depart from linearity and homoscedasticity.This trade-off motivates modeling both the conditional location and scale of the response parameter.
- Nonlinear conditional heteroscedastic model: The proposed nonlinear conditional heteroscedastic model represents φi as m(si) + σ(si) × ζi, estimating the conditional mean and variance with separate regressions.The variance regression uses log-squared residuals, and feed-forward neural networks are used for both regression functions.
- Nonlinear conditional heteroscedastic model: Feed-forward neural networks can reduce the summary-statistic dimension through internal projections onto lower-dimensional hidden-unit representations.The hidden-unit count H is typically smaller than the summary-statistic dimension D, so the network reduces the initial input dimension.
- Iterated importance sampling: The algorithm converts regression-based ABC into an iterative multistage procedure, implemented here with two stages that adaptively refine the posterior approximation.The adaptive step estimates the support of an initial sample, proposes parameters from the prior conditional on that support, and can pool agreeing empirical distributions.
3 Examples of implicit statistical models
The examples evaluate regression-based ABC methods on implicit models, including a coalescent model with computationally intractable likelihood. Across these examples, NCH and ANCH provide accurate posterior estimation with reduced sensitivity to tolerance rates and summary-statistic dimensionality.
- Examples and setup: The examples assess LocL, NCH, and ANCH regression-based ABC algorithms on implicit statistical models, including population-genetic coalescent models.The first example estimates the effective mutation rate θ from segregating sites under the infinitely-many-sites model.
- Examples and setup: The coalescent example uses simulations because its likelihood is usually computationally intractable.The summary statistic is the number of segregating sites, which is not sufficient for θ.
- First example: Rejection-based posterior distributions deviated significantly from the empirical posterior for tolerance rates above 10%, while LocL was accurate only for rates ≤20%.LocL performance deteriorated as the tolerance rate increased.
- First example: NCH performance was less sensitive to tolerance rates, and ANCH achieved superior performance for tolerance rates ranging between 0 and 90%.The adaptive algorithm remained close to the optimum achieved by the algorithms across the evaluated tolerance rates.
- Population-genetic results: For three parameters, NCH and ANCH produced marginal posterior quantiles similar to each other, whereas LocL showed larger deviations at small tolerance rates.LocL performance decreased as tolerance increased above 20%, with substantial bias in ancestral-size estimation.
4 Discussion
The discussion presents nonlinear regression-based ABC as a way to reduce computational burden and accommodate higher-dimensional summary statistics. Neural networks provide dimension reduction and probabilistic outputs, while predictive averaging and regularization can address estimator variability.
- Motivation: Nonlinear regression-based ABC is presented as an alternative to methods that can require heavy computational loads for accurate posterior approximation.The discussion contrasts linear regression-based ABC and rejection-based approaches with the proposed nonlinear method.
- Results: In three examples, neural networks reduced computational generation costs.The discussion reports this as an observed benefit of the neural-network approach.
- Mechanism: Neural-network first layers can nonlinearly project many summary statistics into a lower-dimensional subspace before regression.This projection provides a heuristic explanation for performance when the number of summary statistics is large.
- Limitations: Increasing summary-statistic dimension inflates the variability of conditional mean and variance estimators, and consequently the variances of estimated posterior distributions.The discussion identifies this as a limitation and motivates variance-reduction strategies.
- Variance reduction: Variance can be reduced by training many feed-forward neural networks for each conditional regression and averaging across replicate networks.The discussion also notes that Bayesian neural-network theory supplies regularization rules, making the method relatively automatic.
- Extensions: The algorithms could be modified to include support-vector-machine regression without changing their underlying spirit.The discussion places this possibility alongside related regression approaches such as projection pursuit regression.
- Model choice: Feed-forward neural networks provide probabilistic outputs that support a unified Bayesian treatment of model choice.The model itself can be treated as an additional parameter, extending the regression framework to candidate-model inference.
- Implications: Improved ABC models are intended to help address increased model complexity and the need for higher-dimensional summary-statistic vectors.The discussion frames ABC as potentially useful across ecology, evolution, epidemiology, and social science.