Estimating the evidence -- a review

Nial Friel; Jason Wyse

Source-linked AI summary

Estimating the evidence -- a review

Nial Friel, Jason Wyse

arXiv:1111.1957v1Nov 8, 2011stat.ME

TL;DR

Bayesian model evidence is central to comparing models, but it is usually analytically intractable and sensitive to prior specification. This paper reviews computational estimators and practical guidance, finding important performance and sensitivity differences across methods and examples.

Problem
Model evidence is vital for Bayesian model comparison but generally requires difficult high-dimensional integration and is sensitive to the prior.
Method
The paper reviews computational methods for estimating evidence, including Laplace’s method, the harmonic mean estimator, Chib’s method, annealed importance sampling, and nested sampling.
Results
The examples show that estimator performance can differ substantially, while Bayes factors can be highly sensitive to prior settings.
Takeaways & Limitations
Evidence estimation should be assessed across methods and prior choices rather than treated as computationally routine.
Takeaways & Limitations
Nested sampling has computational overhead, and its situations of reliable performance require further experimentation.

Abstract

from arXiv · show

The model evidence is a vital quantity in the comparison of statistical models under the Bayesian paradigm. This paper presents a review of commonly used methods. We outline some guidelines and offer some practical advice. The reviewed methods are compared for two examples; non-nested Gaussian linear regression and covariate subset selection in logistic regression.

1 Introduction

Bayesian model analysis increasingly addresses uncertainty across competing models, using model evidence to support model choice. This review focuses on computational methods for estimating that evidence.

Markov chain Monte Carlo methods and software such as WinBUGS made Bayesian analysis practical for complex models and widely accessible.
Model evidence is the probability of the observed data given a statistical model, providing a basis for Bayesian model choice.
The evidence combines the likelihood with the prior through integration over the model parameters.
Evidence is generally analytically intractable because it requires high-dimensional integration over complicated, variable functions and is sensitive to the prior.
The article reviews computational methods for estimating model evidence and provides an accessible starting point for readers unfamiliar with Bayesian methods.

2 Why do we need the evidence?

Model evidence supports Bayesian comparison and prediction when several models could plausibly have generated the data. It yields posterior model probabilities, Bayes factors, and model-averaged predictions.

Posterior model probabilities combine the evidence terms for competing models through Bayes’ theorem.
The Bayes factor is the ratio of the evidence terms for two competing models.
Larger BFij values indicate greater evidence in favour of model mi compared with model mj.
Bayesian model averaging predicts future observations by weighting each model’s posterior predictive distribution by its posterior model probability.

3 Review of methods to compute the model evidence

The review introduces several approaches for estimating model evidence, beginning with methods that approximate or evaluate the required normalising constant within a fixed model. Laplace’s method illustrates the role of posterior approximation and numerical optimisation.

The methods are presented chronologically while conditioning on a given model and suppressing repeated model-index notation.
3.1 Laplace’s method: Laplace’s method assumes that the posterior is adequately approximated by a normal distribution, often when the sample size is large.
3.1 Laplace’s method: A quadratic expansion around the posterior mode approximates the likelihood-prior product by a Gaussian with covariance ˜Σ = (−D2l(˜θ))−1.
The running example uses n independent observations from N(µ, τ −1), with normal and gamma priors on µ and τ.
3.1 Laplace’s method: The posterior mode can be found iteratively with Newton’s method, and the final iterate can be substituted into the Laplace evidence estimate.

3.2 Harmonic mean estimator

The harmonic mean estimator estimates evidence from posterior MCMC draws, making it easy to implement but vulnerable to prior sensitivity. A tractable Gaussian example shows that its estimates can remain nearly unchanged while the true evidence varies.

3.2 Harmonic mean estimator: The harmonic mean estimator uses posterior MCMC draws and the corresponding likelihood values to estimate the evidence.
3.2 Harmonic mean estimator: The estimator follows from an identity relating the posterior expectation of the reciprocal likelihood to the reciprocal evidence.
3.2 Harmonic mean estimator: Its severe limitation is that posterior-only draws may barely change when the prior changes, although the evidence is highly prior-sensitive.
3.2.2 A tractable example: Table 1 reports the true log π(y) values alongside harmonic-mean estimates across τ0 settings.
3.2.2 A tractable example: In the normal-gamma example, the posterior remains similar as τ0 changes from 1 to 0.0001, producing similar harmonic mean estimates.
3.2.2 A tractable example: The evidence nevertheless changes with the diffuseness of the prior because it integrates over µ and τ.

3.3 Chib’s method

Chib’s method estimates model evidence from Gibbs-sampler output by rearranging Bayes’ formula and estimating a posterior ordinate at a selected parameter point. The approach extends to high-dimensional spaces but can encounter difficulties with label switching and nonidentifiability.

Chib’s estimator rearranges Bayes’ formula to express the evidence using the prior, likelihood, and posterior density at θ∗.The prior and likelihood can usually be evaluated in closed form, while the posterior density is estimated from Gibbs-sampler output.
The posterior ordinate at θ∗ is estimated by a Monte Carlo average based on Gibbs-sampler draws.The method can factor the posterior using full-conditional distributions when the parameter vector is partitioned into components.
Chib’s method can be extended to high-dimensional parameter spaces and applied broadly where Gibbs sampling is available.The implementation requires storing draws from various full-conditional distributions, which is rarely prohibitive.
Difficulties can arise for mixture models, hidden Markov models, and other models with label switching or parameter nonidentifiability.For finite mixtures, Gibbs sampling may take a long time to visit all k! unique modes.

3.4 Annealed importance sampling

Annealed importance sampling estimates evidence by constructing a tempered sequence from the prior to the posterior and using Markov transitions along that sequence. It produces independent target-distribution draws while averaging importance weights to estimate the evidence.

Importance sampling uses draws from an easily sampled importance function to estimate a target distribution or the ratio of normalizing constants.The importance function generally needs heavier tails than the target distribution.
AIS adaptively defines an importance function by tempering the posterior through distributions π_tj(θ|y) that transition from the prior to the posterior.The tempered posterior raises the un-normalized posterior to power t_j, with 1 = t_0 > · · · > t_m = 0.
AIS uses Markov transition kernels with invariant distributions π_tj to generate draws along the tempered sequence.Metropolis-Hastings and Gibbs kernels are given as examples, and transitions may need repeated application to obtain representative draws.
The average importance weights estimates the ratio of the target and prior normalizing constants, providing an estimator of the evidence.AIS also yields an independent sample from the target distribution, described as a major strength.
AIS is suited to complicated or multimodal posterior distributions that can challenge standard MCMC algorithms.Its tempered construction is intended to help in multimodal settings, while the transition-kernel choice remains vital.

3.5 Nested sampling

Nested sampling computes evidence by viewing it as an integral over prior mass and approximating that integral with likelihood-ordered prior draws. Its main computational burden is repeatedly sampling from the prior under likelihood constraints, and its performance boundaries remain incompletely characterized.

Nested sampling expresses evidence as a one-dimensional integral over prior mass and estimates it using a weighted sum of likelihood values.The construction uses the inverse likelihood–prior-mass relationship and quadrature weights.
The main computational burden is sampling from the prior subject to the likelihood constraint π(y|θ) > l.An alternative uses a surviving sample point and constrained Metropolis-Hastings moves.
The algorithm maintains N prior samples, removes the point with the smallest likelihood, and replaces it with a constrained prior draw.Each iteration adds the removed likelihood multiplied by a prior-mass weight to the evidence estimate.
The deterministic prior-mass step X_i = exp{−i/N} can be replaced stochastically, and termination may occur when new contributions become negligible.One cited criterion stops when a new contribution is less than 10^-8 times the current value.
Nested sampling has convergence rate O(N^1/2) under the cited deterministic-level setting, while comparable MCMC results are unknown.Further experimentation is needed to determine when the algorithm works well; cited examples include mixture and probit models.

3.6 Power posteriors

Power posteriors estimate model evidence by integrating expected log likelihood across a temperature path from prior to posterior. The method is generic and relatively easy to implement, but temperature scheduling and approximation errors require care.

Power posterior identity: Thermodynamic integration connects the log evidence to the expected posterior deviance across temperatures.The gradient of the log normalising constant is expressed as an expected posterior deviance, and integration over t yields the evidence identity.
Estimation procedure: The estimator discretises t from 0 to 1, samples π(θ|y, tj) at each temperature, and applies a trapezoidal approximation.At each tj, samples estimate Ej = Eθ|tj log π(y|θ), which are then combined numerically.
Error sources: Temperature discretisation and Monte Carlo approximation are the two main sources of estimation error.The discretisation error depends on the Kullback-Leibler distance between successive tempered distributions.
Error control: For a fixed number of temperatures, minimise discretisation error by positioning successive temperatures to minimise their Kullback-Leibler distance.This criterion concerns the distances between πtj and πtj+1 along the temperature path.
Practical considerations: The power posterior approach is generic and relatively easy to implement, but choosing the temperature schedule is an immediate difficulty.The method can also often be implemented in WinBUGS.

3.7 Other approaches

Evidence estimation can also use sequential Monte Carlo methods, which are close in spirit to tempering methods. In the cited mixture-model example, SMC did not outperform AIS, while optimal temperature paths remain an issue.

Sequential Monte Carlo: Sequential Monte Carlo provides an alternative evidence-estimation approach that is close in spirit to tempering methods.The paper cites an example estimating mixture-model evidence with SMC and AIS.
Comparison with AIS: In the cited mixture-model example, SMC did not give superior performance to AIS.The passage also identifies the optimal temperature schedule or path as an issue for these methods.

4 Some examples

The reviewed evidence-estimation methods are compared on analytically benchmarked Gaussian regressions and intractable logistic-regression model selection. Results indicate that method performance depends on the algorithm, prior specification, and model complexity.

4.1 Non-nested linear regression models: The first example compares two Gaussian linear regression models for predicting radiata-pine compression strength, with exact evidence available as a benchmark.The models use density and resin-adjusted density as alternative predictors, based on 42 specimens.
4.1 Non-nested linear regression models: The Gaussian comparison evaluates evidence estimates and Bayes factors across eighteen runs of the reviewed algorithms, including Laplace, harmonic-mean, Chib, annealed-importance-sampling, nested-sampling, and power-posterior methods.The exact Bayes factor and algorithmic estimates are compared using the reported tables and boxplots.
4.1 Non-nested linear regression models: The harmonic mean estimator and nested sampling show the poorest performance, particularly for the standard error of the Bayes factor.Nested-sampling performance might improve with better constrained-distribution simulation or a stricter termination criterion, though runtimes could remain time-consuming.
4 Some examples: The reported comparisons are rough accuracy guides because several algorithms were evaluated with largely vanilla implementations rather than fully optimized schemes.The authors identify tempering scales and constrained sampling as examples of choices that could be optimized.
4.2 Choosing between two logistic regression models: The logistic-regression example evaluates evidence for competing subset models using seven predictors of diabetes among 532 Pima Indian women.The analysis considers 129 potential models, with covariates standardized before analysis.
4.2 Choosing between two logistic regression models: The two highest-posterior-probability logistic models differ by inclusion of AGE, and a reduced reversible-jump run gives BF12 = 13.96 as the comparison benchmark.The benchmark uses τ = 0.01 and compares the reviewed methods using their best approximation over five runs and relative execution speed.
4.2 Choosing between two logistic regression models: Increasing τ to 1 reduces the Bayes factor to about 1.3 and is accompanied by much better performance from all algorithms in estimating BF12.The change in prior informativeness produces a much weaker model-comparison conclusion.

5 Concluding thoughts

The article surveys widely used approaches for estimating model evidence, emphasizing that evidence is fundamental but generally difficult to estimate. Practical reliability may require more implementation effort than the easiest available method.

5 Concluding thoughts: The paper surveys widely used approaches to estimating model evidence, a fundamental quantity in Bayesian statistics.It presents the review as a starting point for readers exploring these methodologies.
5 Concluding thoughts: Evidence estimation is generally difficult, especially when the prior distribution is diffuse, and may require substantial computational runtime and coding effort.The conclusion links estimation difficulty to implementation demands.
5 Concluding thoughts: The easiest method to implement may not provide the most reliable estimates.The conclusion presents implementation simplicity and estimation reliability as potentially distinct considerations.