Source-linked AI summary
Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Network
Bai Jiang, Tung-yu Wu, Charles Zheng, Wing H. Wong
TL;DR
ABC requires effective summary statistics because likelihoods may be intractable and summary-statistic choice affects approximation quality. The paper addresses automatic construction using DNNs and identifies posterior means as theoretically strong summaries, while noting that the DNN choice is only heuristically justified.
Problem
ABC can involve intractable likelihoods, and the choice of summary statistic is crucial for approximation quality.
Method
The paper considers DNN-based automatic construction of summary statistics and uses the posterior mean Eπ[θ|X] as a theoretically motivated summary statistic.
Results
Using Eπ[θ|X] as the summary statistic maximizes point-estimation accuracy, while S∗ produces the highest-quality ABC posterior.
Takeaways & Limitations
The approach targets low-dimensional, informative ABC summaries with minimal expert knowledge.
Takeaways & Limitations
The choice of DNNs for constructing approximate summaries is justified only heuristically.
Abstract
from arXiv · showhide
Approximate Bayesian Computation (ABC) methods are used to approximate posterior distributions in models with unknown or computationally intractable likelihoods. Both the accuracy and computational efficiency of ABC depend on the choice of summary statistic, but outside of special cases where the optimal summary statistics are known, it is unclear which guiding principles can be used to construct effective summary statistics. In this paper we explore the possibility of automating the process of constructing summary statistics by training deep neural networks to predict the parameters from artificially generated data: the resulting summary statistics are approximately posterior means of the parameters. With minimal model-specific tuning, our method constructs summary statistics for the Ising model and the moving-average model, which match or exceed theoretically-motivated summary statistics in terms of the accuracies of the resulting posteriors.
1. Introduction
ABC enables approximate inference when likelihoods are intractable, but its accuracy and efficiency depend critically on choosing an effective summary statistic. The paper develops a more automated approach using deep neural networks to construct low-dimensional summaries from raw data.
- Approximate Bayesian Computation: ABC uses simulated parameter-data pairs and accepts proposals whose simulated data are sufficiently close to the observed data.For continuous data, exact matching has probability zero, so ABC relaxes acceptance to a tolerance-based distance criterion.
- Approximate Bayesian Computation: Smaller tolerance thresholds reduce approximation error but also lower acceptance probability, creating an efficiency–accuracy trade-off.
- Summary Statistics: Summary statistics are crucial because mapping high-dimensional data to low-dimensional summaries introduces information loss, while higher-dimensional summaries make rejection sampling inefficient.Effective summaries must balance these competing approximation errors.
- Summary Statistics: Low-dimensional sufficient statistics are attractive but generally exist only in special cases and may still be unavailable or difficult to construct.
- DNN Approach: The paper proposes automatically learning informative summaries with deep neural networks rather than relying on expert-designed candidate statistics or basis expansions.The approach uses raw high-dimensional data and learns nonlinear transformations, increasing automation relative to linear-regression-based methods.
- DNN Approach: The DNN approach is motivated by approximating Eπ[θ|X], which can serve as an ABC summary statistic and has theoretically favorable point-estimation accuracy.
2. Methods
The method constructs a low-dimensional summary statistic for high-dimensional data by training a deep neural network to approximate the posterior mean of the parameters, then uses that estimator in ABC.
- The approach assumes independent data can be simulated from the model despite the intractability of p(X|θ), using prior draws of θ.
- ABC seeks a low-dimensional, informative summary statistic for high-dimensional X to obtain accurate posterior approximations.
- The DNN estimator ˆθ(X) is used as the ABC summary statistic after training on generated parameter–data pairs.Training minimizes squared error between θ(i) and ˆθ(X(i)) using backpropagation and stochastic gradient descent.
- Posterior mean as summary statistic: As ϵ → 0, the ABC posterior mean matches the exact posterior mean, while a basis-function extension gives weak posterior convergence as K →∞ at an appropriate rate.
- Posterior mean as summary statistic: The posterior mean Eπ[θ|X] is theoretically motivated as an ABC summary statistic because ABC point estimates are sample means of the ABC posterior.Theorem 1 states that this summary is well defined when Eπ[|θ|] < ∞ and maximizes point-estimation accuracy.
- Deep neural network construction: DNNs are selected because their layered nonlinear transformations can approximate the highly nonlinear, smooth mapping from X to Eπ[θ|X].The network uses hidden layers with weight matrices, bias vectors, and tanh activations; overfitting is addressed with large training sets, early stopping, and regularization.
3. Example: Ising Model
The Ising-model experiment evaluates DNN-learned summary statistics against a sufficient statistic and semi-automatic alternatives for ABC inference. The DNN summary tracks the sufficient statistic more closely than the semi-automatic construction, while posterior comparisons assess performance across several observed parameters.
- Model: The Ising model places binary spins on an m × m lattice, with inverse temperature θ controlling neighbor interactions.The model’s probability mass function is intractable except in small cases because its normalizing constant requires exponential-time computation.
- Experimental design: The experiment simulates training, validation, and testing data with Metropolis sampling, then trains a DNN to predict θ from X.The training, validation, and testing sets contain 10^6, 10^5, and 10^5 instances, respectively, from a 10 × 10 lattice.
- Summary statistics: The sufficient statistic S∗ is univariate and produces the highest-quality ABC posterior, so it serves as the gold-standard comparison.The paper compares the DNN-based and semi-automatic summaries against S∗ and uses ϵ = 0 for the sufficient-statistic ABC procedure.
- Results: The DNN-based summary statistic approximates an increasing function of S∗, whereas the semi-automatic construction fails to capture the relevant nonlinearity.The semi-automatic summary is a linear combination of X_j and performs poorly around the prior mean θc = 0.4406.
- Results: Above the critical point θc, the ABC posteriors have a similar shape to the prior because strongly aligned spins make different θ values difficult to distinguish.All three ABC posteriors demonstrate this behavior in the highlighted θ = 0.8 case.
4. Example: Moving Average of Order 2
The moving-average order-2 experiment compares DNN-learned summaries with auto-covariance and semi-automatic summaries against the exact posterior. Across repeated observed datasets, the DNN summary yields more accurate posterior approximations and captures posterior correlation.
- Methods: A 3-layer DNN is trained to predict (θ1, θ2) from X, while the semi-automatic summary uses linear regression on polynomial bases of the observations.A single-hidden-layer FFNN is also evaluated as a comparison method.
- Experimental design: The experiment compares ABC summaries based on auto-covariances, a DNN, and a semi-automatic construction against the numerically computed exact posterior.Posterior accuracy is evaluated using posterior means and covariance-related quantities.
- Results: The DNN learns a better prediction rule than the semi-automatic method and FFNN, but requires more training time; regularization does not improve performance.These comparisons are reported alongside the training results for the moving-average experiment.
- Results: The DNN-based summary produces a more accurate ABC posterior than either the auto-covariance or semi-automatic summaries.The comparison is made using posterior draws for an observed datum generated with θ = (0.6, 0.2).
- Results: The DNN-based ABC posterior correctly captures posterior correlation and better approximates posterior moments across 100 different observed datasets.The repeated comparison reports better approximation of posterior moments than the auto-covariance and semi-automatic methods.
5. Discussion
The paper constructs ABC summary statistics with deep neural networks, targeting posterior means and requiring minimal expert knowledge. Experiments indicate strong performance, while also identifying overfitting and limited theoretical justification as boundaries.
- Contribution: The automatically constructed summaries require minimal expert knowledge and target low-dimensional, informative representations for ABC.
- Approach: The method uses DNNs to approximate posterior means as ABC summary statistics, leveraging their representational power.The posterior mean is desirable but generally intractable; the network learns an approximation from simulated data.
- Training and limitations: The choice of DNNs is justified only heuristically, despite promising empirical results.
- Empirical results: The Ising model provides a stringent comparison because its univariate sufficient statistic is ideal and yields the best achievable ABC posterior.
- Empirical results: DNN-based summaries approximate an increasing function of the sufficient statistic in the experiments.
- Empirical results: In the MA(2) model, the DNN-based summary statistic outperforms the semi-automatic construction and auto-covariances.Auto-covariances can be transformed into consistent parameter estimates and are widely used in the literature.
- Training and limitations: Training-data overfitting is mitigated in ABC because arbitrarily many simulated training samples can be generated, and training and testing losses show little discrepancy.The experiments use 10^6 training samples versus 10^4 neural-network parameters.
- Training and limitations: The three-hidden-layer DNN outperforms the single-hidden-layer FFNN, while regularization does not significantly improve performance.
Supplementary Materials
The supplementary materials extend the theoretical result for posterior-expectation summaries and provide implementation details for training the deep neural networks.
- Theory: The supplementary materials extend Theorem 1 and establish convergence of posterior expectations under ABC using S_b(X) = E_π[b(θ)|X].
- Theory: The extension provides a global approximation to the posterior distribution.
- Implementation: Training details cover backpropagation and stochastic gradient descent for deep neural networks.
REFERENCES37
The references span ABC methodology, summary-statistic construction, neural-network learning, regularization, and the Ising model.
- ABC and summary statistics: Several references address approximate Bayesian computation without likelihoods and the construction or selection of summary statistics.
- ABC and summary statistics: The bibliography includes work on semi-automatic ABC and nonlinear regression approaches for dimension reduction.
- Neural networks: Neural-network references cover gradient-based learning, dimensionality reduction, deep belief networks, representation learning, and universal approximation.
- Neural networks: The references include foundational work on overfitting, regularization, dropout, and neural-network consistency.
- Models and inference: Additional citations cover the Ising model, point estimation, and likelihood-free Monte Carlo methods.