Source-linked AI summary
Approximate Bayesian Computation: a nonparametric perspective
Michael Blum
TL;DR
ABC addresses likelihood-free inference when likelihood evaluation is computationally intractable for stochastic generating mechanisms. The paper analyzes standard rejection and linear-adjustment estimators, introduces quadratic adjustment, and studies transformations of summary statistics. It finds that adjustment is especially beneficial when residuals are independent of the summaries, while the analysis does not address the full-to-partial posterior approximation.
Problem
Likelihood evaluation can be computationally intractable for stochastic models involving unobserved high-dimensional variables, while theoretical results for ABC remain lacking.
Method
The paper derives asymptotic bias and variance for rejection and linear-adjustment estimators, introduces quadratic adjustment, and proposes summary-statistic transformations.
Results
Regression adjustment is more beneficial when residuals are independent of summary statistics, and quadratic adjustment has fewer asymptotic bias terms than linear adjustment.
Takeaways & Limitations
Transforming summary statistics to make the parameter relationship more homoscedastic can improve the potential performance of adjustment methods.
Takeaways & Limitations
The paper does not investigate the consequences of replacing the full posterior with the partial posterior.
Abstract
from arXiv · showhide
Approximate Bayesian Computation is a family of likelihood-free inference techniques that are well-suited to models defined in terms of a stochastic generating mechanism. In a nutshell, Approximate Bayesian Computation proceeds by computing summary statistics s_obs from the data and simulating summary statistics for different values of the parameter theta. The posterior distribution is then approximated by an estimator of the conditional density g(theta|s_obs). In this paper, we derive the asymptotic bias and variance of the standard estimators of the posterior distribution which are based on rejection sampling and linear adjustment. Additionally, we introduce an original estimator of the posterior distribution based on quadratic adjustment and we show that its bias contains a fewer number of terms than the estimator with linear adjustment. Although we find that the estimators with adjustment are not universally superior to the estimator based on rejection sampling, we find that they can achieve better performance when there is a nearly homoscedastic relationship between the summary statistics and the parameter of interest. To make this relationship as homoscedastic as possible, we propose to use transformations of the summary statistics. In different examples borrowed from the population genetics and epidemiological literature, we show the potential of the methods with adjustment and of the transformations of the summary statistics. Supplemental materials containing the details of the proofs are available online.
1. INTRODUCTION
ABC provides likelihood-free inference for stochastic generating mechanisms by conditioning on simulated summary statistics. This paper addresses missing theoretical results by analyzing estimator behavior and proposing adjustment and transformation strategies.
- Motivation: Likelihood evaluation can be computationally intractable when stochastic models involve unobserved high-dimensional variables and integrals over them.ABC instead works with a computer-based data-generating mechanism.
- Motivation: ABC replaces the full posterior g(Θ|D) with the partial posterior g(Θ|sobs), unless the summary statistics are sufficient.The summaries are computed from observed data and determine the conditioning target.
- ABC framework: ABC simulates parameter values from the prior and summary statistics conditional on them, turning partial-posterior estimation into conditional density estimation.The simulated pairs (Θi, si) are used to estimate g(Θ|sobs).
- Contributions: The paper derives asymptotic bias and variance for Nadaraya-Watson, linear-adjustment, and quadratic-adjustment estimators.Quadratic adjustment is introduced as an original estimator whose bias contains fewer terms than linear adjustment.
- Scope: The paper does not investigate the consequences of replacing the full posterior with the partial posterior.It refers readers to work on approximate sufficiency and informative-summary selection.
- Contributions: Regression adjustment is most beneficial when residuals are independent of summary statistics, motivating transformations that make the relationship more homoscedastic.Applications are shown in population genetics and epidemiology.
2. PARAMETER INFERENCE IN ABC
ABC estimates a partial posterior by smoothing simulated summary statistics around the observed summaries. Regression adjustment modifies simulated parameters using local conditional-mean models, including a proposed quadratic form.
- 2.1 Smooth rejection: The smooth-rejection estimator weights simulated parameter values by kernel evaluations of their summaries relative to sobs.The multivariate kernel uses a diagonal bandwidth matrix B=bD.
- 2.1 Smooth rejection: ABC conditional density estimation uses simulated parameter-summary pairs and a univariate kernel for the parameter coordinate.The analysis is univariate, although multivariate density estimation can be implemented similarly.
- 2.1 Smooth rejection: The original rejection method discards parameters whose simulated summaries are too different from the observed summaries, while smooth kernels replace this binary decision.Smooth-kernel estimation was proposed by Beaumont et al. (2002).
- 2.2 Regression adjustment: Linear regression adjustment estimates a local conditional mean, forms residuals εi=θi−m̂1(si), and adjusts parameters using the observed summaries.The adjusted values are then used in conditional density estimation.
- 2.2 Regression adjustment: The adjusted samples are inserted into the kernel density estimator, producing the linear- and quadratic-adjustment estimators.The linear estimator is identified as Beaumont et al. (2002, equation (9)).
- 2.2 Regression adjustment: Quadratic adjustment extends the local regression model with quadratic summary-statistic terms before recomputing adjusted parameters.Its coefficients are obtained by weighted least squares, with a symmetric quadratic coefficient matrix.
3. ASYMPTOTIC BIAS AND VARIANCE IN ABC
The paper derives asymptotic bias and variance for three ABC partial-posterior estimators and compares their behavior under different conditional structures. Quadratic adjustment has fewer bias terms than linear adjustment, while adjustment is most favorable in nearly homoscedastic settings.
- Main theorem: Theorem 1 gives asymptotic bias and variance expressions for rejection, linear-adjustment, and quadratic-adjustment estimators.The result assumes a diagonal bandwidth matrix B = bD and regularity conditions (A1):(A5).
- Curse of dimensionality: n^-4/(d+5) is the minimal MSE order when both bandwidths are of order n^-1/(d+5).The convergence rate worsens as the summary-statistic dimension d increases.
- Effective local size and design: The estimators' variance increases when simulations are sparse near s_obs, because effective local sample size is approximately n|D|p(s_obs)b^d.Sequential Monte Carlo and likelihood-free MCMC methods are cited as approaches that target simulations closer to s_obs.
- Bias decomposition: Rejection-estimator bias reflects design and conditional-density curvature, increasing with greater curvature of g(·|s) at s_obs.The design term is large when gradients of the simulation density and conditional density are collinear.
- Comparing adjustments: Quadratic adjustment has the bias of an estimator with the conditional mean known exactly, whereas linear adjustment adds a conditional-mean-curvature term.For general partial posterior distributions, the three biases cannot be universally ranked; under ideal homoscedastic models with linear conditional mean, adjusted estimators have smaller bias than rejection.
4. CHOOSING A REGRESSION MODEL
The paper selects among local regression models and transformations by minimizing cross-validated prediction error. It considers logarithmic and square-root summary-statistic transformations, with parameter transformations chosen according to prior support.
- Transforming summary statistics: Logarithmic and square-root transformations of summary statistics are selected by minimizing the weighted sum of squared residuals.An uniform kernel makes the weighted residual sums comparable across transformations.
- Model-search size: There are 3^d regression models to consider, so a greedy algorithm can be used when d is large.The candidate count reflects transformation choices across d summary statistics.
- Transforming parameters: Positive parameters are log-transformed before adjustment and returned to the original scale exponentially, while finite-interval parameters use a logit transformation.The parameter transformations are chosen to respect the support of the prior.
- Choosing the regression model: The regression model is chosen by minimizing cross-validated prediction error among local regressions of degrees 0, 1, and 2.Leave-one-out cross-validation accounts for the different numbers of predictors in the candidate models.
5. EXAMPLES
The examples apply ABC with transformed summary statistics and regression adjustments to Gaussian, coalescent, and epidemiological models. Performance depends on the model: adjustments help substantially in the Gaussian example, while effects are smaller in the coalescent and epidemiological examples.
- Gaussian example: Log-transforming the empirical variance makes the regression setting accurate, with the WSSR criterion selecting the logarithmic transformation in all 100 replicates.The transformation is selected because it minimizes the WSSR criterion.
- Gaussian example: With the mean and log empirical variance, both linear and quadratic adjustment estimate σ2 well, whereas the method without adjustment performs worse.Cross-validation selects linear adjustment 74 times and quadratic adjustment 26 times, never selecting no adjustment.
- Coalescent model in population genetics: For the coalescent model, ABC estimates the TMRCA from segregating sites S and the ρ statistic because likelihood evaluation requires integrating over a high-dimensional ancestral tree.The TMRCA is the sum of inter-coalescence times, and DNA mutations are simulated along the tree.
- Coalescent model in population genetics: For the coalescent data, WSSR selects log ρ and S, while cross-validation selects quadratic adjustment; linear and quadratic prediction errors are almost identical.The transformations and adjustments do not greatly alter the estimated posterior distribution.
- Epidemiological example: In the epidemiological example, transformations and adjustments usually change posterior estimates little, except for R0; genotype data are more informative for transmission rate and doubling time.The posterior 95% credibility interval for R0 is hardly narrower than its prior interval.
6. CONCLUSION
The paper develops theoretical and adjusted estimators for ABC and examines when transformations and regression adjustment improve posterior inference. Applications show that these methods can have little effect in some examples, while data informativeness determines whether posterior estimates differ meaningfully from priors.
- The quadratic-adjustment estimator has an asymptotic bias involving fewer terms than Beaumont et al.’s linear-adjustment estimator.
- Regression adjustment is more beneficial when residuals are independent of summary statistics, corresponding to a more homoscedastic regression relationship.
- Transformations of summary statistics are proposed to make the regression relationship as homoscedastic as possible during adjustment.
- In the tuberculosis example, transformations and regression adjustment greatly changed posterior distributions for transmission rate and doubling time but not for R0.
- The tuberculosis data did not convey enough information for confident estimation of R0, whose posterior did not importantly differ from its prior.
- In two other population-genetics and epidemiological examples, adjustment and summary-statistic transformations had little effect, while the examples demonstrated ABC’s potential for models with computationally difficult likelihoods.
APPENDIX A. HYPOTHESES OF THEOREM 1
The theorem assumes regularity of the kernels, density and conditional mean, support points, and bandwidth sequences needed for the asymptotic analysis.
- Kernel conditions: The multivariate kernel has a finite second-order moment, while its first-order moments vanish.The zero first-moment condition is satisfied by spherically symmetric and product kernels built from symmetric univariate kernels.
- Kernel conditions: The univariate kernel is symmetric and has a finite second-order moment.
- Smoothness and support: The observed summary statistics lie in the interior of the support of p, where p has continuous second-order derivatives.
- Smoothness and support: The parameter θ lies in the support of the partial posterior, and the partial posterior has continuous second-order derivatives at (θ, sobs).
- Smoothness and support: The conditional mean m(s) exists and is finite near sobs, with continuous second-order derivatives.
- Bandwidth conditions: The bandwidth matrices B are nonsingular, and 1/(n|B|b′), each entry of BᵀB, and b′ tend to zero as n → ∞.
APPENDIX B. PROOF OF THEOREM 1
The appendix derives asymptotic expansions for the three posterior estimators by linearizing their ratio form and applying Taylor expansions under the theorem’s assumptions.
- Asymptotic analysis: The three partial-posterior estimators ĝ_j(·|sobs), j = 0, 1, 2, are analyzed through their asymptotic bias and variance.
- Proof strategy: The Nadaraya–Watson estimator is difficult to analyze because it is a ratio of two random variables.
- Proof strategy: The estimators are linearized to compute their biases and variances, using expansions of numerator and denominator terms.
- Intermediate expansions: Lemma 2 expands the relevant expressions around ϵ = θ − m(sobs), with terms involving bandwidth traces, kernel moments, and derivatives.
- Theorem application: Theorem 2 states the resulting bias and variance formulas for all three estimators under assumptions (A1)–(A5).
- Variance derivation: The variance derivation uses Taylor expansions and expectation calculations involving covariance and variance remainder terms.