PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation

M. G. B. Blum, M. A. Nunes, D. Prangle, S. A. Sisson

arXiv:1202.3819v3stat.MEstat.CO

TL;DR

ABC needs low-dimensional summaries that retain information while avoiding inefficient high-dimensional smoothing. The paper reviews and compares three dimension-reduction classes, introducing AIC/BIC selection and ridge regression, with performance varying across examples and no single best method. Regression adjustment alone can yield substantial gains, while further dimension reduction sometimes helps and sometimes worsens performance.

  • Problem

    ABC requires summary-statistic vectors instead of full data, creating a need for low-dimensional summaries that minimize information loss while avoiding high-dimensional kernel-smoothing inefficiency.

  • Method

    The paper reviews best subset selection, projection, and regularization methods, introduces AIC/BIC selection and ridge regression, and compares them across three challenging models and data sets.

  • Results

    No single dimension-reduction method was best across examples; regression adjustment alone produced an RSSE of −25%, with further gains sometimes obtained and sometimes lost after dimension reduction.

  • Takeaways & Limitations

    Method choice should be evaluated for the specific ABC analysis, including whether expensive dimension reduction improves on simple regression adjustment.

  • Takeaways & Limitations

    The entropy approach is mostly suited to models with relatively diffuse prior distributions, and expensive dimension-reduction methods may offer barely any improvement in some examples.

Abstract

from arXiv · show

Approximate Bayesian computation (ABC) methods make use of comparisons between simulated and observed summary statistics to overcome the problem of computationally intractable likelihood functions. As the practical implementation of ABC requires computations based on vectors of summary statistics, rather than full data sets, a central question is how to derive low-dimensional summary statistics from the observed data with minimal loss of information. In this article we provide a comprehensive review and comparison of the performance of the principal methods of dimension reduction proposed in the ABC literature. The methods are split into three nonmutually exclusive classes consisting of best subset selection methods, projection techniques and regularization. In addition, we introduce two new methods of dimension reduction. The first is a best subset selection method based on Akaike and Bayesian information criteria, and the second uses ridge regression as a regularization procedure. We illustrate the performance of these dimension reduction techniques through the analysis of three challenging models and data sets.

1. INTRODUCTION

ABC replaces intractable likelihood evaluation with comparisons between simulated and observed summary statistics, making the choice of low-dimensional, informative summaries central to accuracy and efficiency.

  • ABC targets an approximate posterior when the likelihood is unavailable or computationally intractable but simulation from the model is feasible.
  • The first approximation replaces full data with lower-dimensional summaries, while the second uses kernel weighting to avoid intractable likelihood terms.
  • Smaller summary dimensions improve kernel-smoothing efficiency because convergence deteriorates as the number of summary statistics increases.
  • Choosing summaries involves trading information retention against the inefficiency of smoothing in high dimensions.
  • Dimension reduction methods seek summaries that are both low-dimensional and highly informative for the parameters.
  • The paper reviews three method classes, introduces AIC/BIC and ridge-regression techniques, and compares them across three challenging models and data sets.

2. CLASSIFICATION OF ABC DIMENSION REDUCTION METHODS

ABC summary-statistic dimension reduction is organized into best subset selection, projection, and regularization, with methods differing in whether they select, combine, or shrink candidate summaries.

  • Candidate summaries often greatly outnumber model parameters, motivating procedures that reduce their dimension while minimizing information loss.
  • The most suitable summary set may depend on the data set because summary-statistic information can vary across the parameter space.
  • Best subset selection evaluates and ranks candidate subsets using information-based criteria, including sufficiency, entropy, and the paper’s AIC and BIC criteria.
  • Projection techniques reduce dimension through linear or nonlinear combinations of summaries using a regression layer that predicts parameters from summaries.
  • Regularization uses ridge regression to shrink coefficients toward zero, giving uninformative summaries weaker contributions without explicitly selecting a subset.

2.1 Regression Adjustment in ABC

ABC regression adjustment models the relationship between simulated parameters and summaries, then adjusts posterior draws to reduce the effects of summary-statistic discrepancies.

  • Regression adjustment addresses ABC’s dimensionality problem by explicitly modeling the discrepancy between simulated summaries and observed summaries.
  • The homoscedastic approach estimates a conditional mean through weighted least squares, weighting simulations by their kernel distance from the observed summaries.
  • The kernel scale parameter ε controls a bias-variance trade-off: increasing it raises effective sample size but can increase bias.
  • Heteroscedastic adjustment extends the model by estimating conditional variance through a second regression on log squared residuals.
  • Regression-adjusted samples can approximate adjusted expectation and variance for a Bayes linear analysis.

2.2 Best Subset Selection Methods

Best subset methods select informative summary statistics using sufficiency, entropy, or information criteria, but their computational cost and assumptions constrain practical use.

  • Best subset selection methods: Exhaustive best-subset search evaluates 2^p −1 combinations, becoming infeasible beyond moderate p and requiring greedy or stochastic alternatives.
  • A sufficiency criterion: The ε-sufficiency method adds a statistic when it changes the posterior beyond a user-specified threshold, producing a choice dependent on ε.
  • A sufficiency criterion: The ε-sufficiency approach assumes posterior changes from added statistics are beneficial, even when a noninformative statistic can increase bias.
  • A sufficiency criterion: Evaluating the sufficiency criterion across parameter values is computationally considerable and appears largely restricted to univariate parameters.
  • An entropy criterion: Entropy-based selection minimizes approximate-posterior entropy as a proxy for informativeness, then can assess posterior error using RSSE when the truth is known or simulated.
  • An entropy criterion: The entropy approach is mainly suited to relatively diffuse priors because adding informative statistics may otherwise increase posterior diffuseness.
  • AIC and BIC criteria: AIC and BIC are introduced as information-criterion techniques for best subset selection in ABC.

2.3 Projection Techniques

Projection techniques reduce summary-statistic dimension by forming linear or nonlinear combinations that are informative about model parameters. The reviewed approaches include partial least squares, neural networks, and decision-theoretic regression summaries.

  • Projection techniques: Projection techniques combine summary statistics through linear or nonlinear transformations to construct lower-dimensional, informative summaries.Unlike subset selection, this can preserve information distributed across several statistics.
  • Partial least squares regression: Partial least squares constructs orthogonal components with high variance and correlation with the response parameter.The summaries may be transformed before fitting, for example using Box–Cox transformations.
  • Partial least squares regression: The number of partial least squares components is chosen using leave-one-out cross-validation of parameter RMSE and minimum-gradient-change inspection.RMSE is evaluated for each candidate number of components.
  • Partial least squares regression: Partial least squares may be unsuitable when the global prior-predictive relationship differs from the relationship near the observed summaries.The resulting components can then be inappropriate for local ABC inference.
  • Neural networks: Neural networks reduce summary dimension to H hidden-unit statistics and use regularized nonlinear regression to model conditional means and variances.The hidden units are nonlinear transformations of the original summaries; H may be fixed, such as H = q.
  • Minimum expected posterior loss: Decision-theoretic regression estimates posterior-mean summaries for each parameter using least-squares models fitted to prior-predictive draws.The fitted summaries are β̂_k^⊤f(s), and the resulting q-dimensional vector is used in ABC.

2.4 Regularization Approaches

Regularization reduces overfitting in ABC regression by penalizing model complexity. The review motivates this approach because standard regression adjustment can over-adjust parameters toward uninformative summaries and introduces ridge regression adjustment.

  • Regularization approaches: Regularization penalizes model complexity to reduce overfitting in ABC regression adjustment.The paper identifies over-adjustment toward uninformative summary statistics as a relevant risk.
  • Regularization approaches: The review introduces ridge regression adjustment as a regularization procedure for ABC.It presents ridge regression as a response to the over-adjustment problem in standard regression adjustment.

2.5 Other Methods

Other ABC approaches either use alternative constructions of summary statistics or avoid explicit dimension reduction. Their applicability can depend on model structure and simulation capabilities.

  • Alternative dimension-reduction methods: Indirect-inference methods identify ABC summary statistics using a computationally tractable model designed to resemble the original model.The review notes similarity in spirit to pilot-ABC posterior-mean summaries.
  • Methods avoiding dimension reduction: Some approaches model the full vector of initial statistics or the full data set rather than reducing its dimension.Examples include mixture modeling of joint parameter-summary samples and sequential approximations to p(s|θ).
  • Methods avoiding dimension reduction: The expectation-propagation approach is not suitable for all analyses because it assumes conditional simulation of y_i is available for every component.Other alternatives exploit hidden Markov structure or claim dimension-robust post-processing convergence.

3. NEW DIMENSION REDUCTION METHODS

The paper introduces AIC/BIC-based subset selection and ridge regression as two new ABC dimension-reduction methods. The criteria trade posterior precision against model complexity, while their use has scope and implementation constraints.

  • New methods: The new methods are an AIC/BIC-based best-subset procedure and a ridge-regression modification of local linear ABC adjustment.Both are implemented in the freely available R package abc.
  • AIC and BIC criteria: AIC and BIC are adapted to ABC because likelihood evaluation and maximum-likelihood estimation are unavailable or challenging in this framework.The criteria are constructed under the local linear regression adjustment.
  • AIC and BIC criteria: The effective simulation count is defined using positive kernel weights so AIC scores remain comparable across summary-statistic subsets.This definition is suitable only for kernels with compact support.
  • AIC and BIC criteria: AIC and BIC select summary-statistic combinations that minimize the product of marginal variances of the regression-adjusted posterior, subject to complexity penalties.This trades posterior precision against model complexity.
  • AIC and BIC criteria: AIC and BIC identify informative statistics only when those statistics vary with parameters within the local regression range.A statistic informative outside that range may not be selected.
  • Ridge regression adjustment: Ridge regression shrinks coefficients toward zero, implicitly reducing the contribution of uninformative summary statistics and reducing over-adjustment risk.It can also provide a unique solution when multicollinearity makes the standard least-squares system singular.

4. A COMPARATIVE ANALYSIS

Across three ABC analyses, dimension-reduction methods produced highly variable performance, with the strongest results depending on the model, parameter target, and regression setting. Entropy-based selection, AIC/BIC, partial least squares, and posterior-loss methods were among the strongest approaches, while multicollinearity exposed weaknesses in standard least-squares adjustment.

  • Study design: The comparison covered coalescent recombination, drug-resistant tuberculosis, and clean-steel particle-inclusion analyses, each using 1,000,000 prior-based simulations.Performance was assessed across multiple parameter combinations and relative RSSE measures.
  • Example 1: Coalescent analysis: For the coalescent analysis, entropy-based selection improved performance most uniformly, while partial least squares outperformed adjustment using all six statistics.AIC, AICc, and BIC were comparable to using all six statistics; ridge regression provided no improvement.
  • Example 2: Tuberculosis fitness cost: For drug-resistant tuberculosis, AIC/BIC achieved mean relative RSSE of −17%, followed by the entropy procedure at −15%.The standard least-squares adjustment, ε-sufficiency, and partial least squares were weaker in the reported ranking.
  • Example 3: Clean-steel quality control: The clean-steel comparison was limited to a few candidate subsets, because exhaustive enumeration of all subsets was computationally untenable.Alternative subsets might have performed substantially better, but this possibility was not evaluated exhaustively.

5. DISCUSSION

The review finds no universally best dimension-reduction method: performance varies across examples and depends on statistical structure, computational cost, and tuning requirements. Regression adjustment can perform strongly alone, while selected dimension-reduction methods may provide further gains but can also worsen performance.

  • −21% was the overall mean relative RSSE for AIC/BIC, while ridge regression and neural networks achieved −19%.Both performed better on average than standard least squares regression with no dimension reduction, whose overall mean relative RSSE was −17%.
  • The entropy-based and minimum expected posterior loss methods were the top average performers, with an overall mean relative RSSE of −25%.
  • Regression adjustment alone can yield substantial gains, but combining it with dimension reduction can improve performance further or worsen it, depending on the method.The ε-sufficiency technique and partial least squares sometimes reduced performance relative to standard least squares regression.
  • Entropy-based and AIC/BIC subset-selection methods can be computationally expensive because they may require evaluating all 2^p potential models.For the third example, computationally expensive methods barely improved over cheap least squares adjustment, so their benefits should be assessed before full implementation.
  • Regularization and projection techniques can help when summary statistics are multicollinear, whereas complex neural networks may become overparametrized with many statistics.In the clean-steels example, p = 113 summary statistics made simpler least squares or ridge regression alternatives more effective than nonlinear methods.
  • The best method depended on the example: posterior loss and entropy led example 3, AIC/BIC ranked first in example 2, and partial least squares outperformed posterior loss in example 1.

SUPPLEMENTARY MATERIAL

The supplement provides detailed error comparisons for dimension-reduction methods across three examples. It serves as a comprehensive comparison resource accompanying the article.

  • The supplement reports error comparisons for different dimension-reduction methods.
  • These comparisons cover each of the article’s three examples.
  • The supplement is provided as supporting material for the comparative review.
Loading 1202.3819v3…