PaperlayerView exact revision on arXiv

Source-linked AI summary

Marginal likelihood computation for model selection and hypothesis testing: an extensive review

Fernando Llorente, Luca Martino, David Delgado, Javier Lopez-Santiago

arXiv:2005.08334v4stat.COcs.LG

TL;DR

Marginal-likelihood computation is important for model selection, but the evidence is usually unknown and difficult to approximate. This review organizes and compares computational approaches and discusses prior-related issues; it also reports that a summary of one-dimensional evidence representations is obtained.

  • Problem

    The model-selection evidence Z = p(y|M) is usually unknown and difficult to approximate, although it is needed for model selection.

    Source: §2.1 · p. 4

  • Method

    The paper reviews and relates marginal-likelihood and normalizing-constant estimators, including importance-sampling and MCMC-based methods, and discusses prior choices.

    Source: §9 · pp. 77, 79

  • Results

    The authors derive two one-dimensional integrals for expressing Bayesian evidence Z.

    Source: §6.1.7 · p. 53

  • Takeaways & Limitations

    The review compares methods’ relationships, limitations, and strengths and discusses improper-prior issues and possible solutions.

    Source: §9 · pp. 77, 79

    The harmonic-mean estimator’s variance is often high and may be infinite.

    Source: §4.1 · p. 16

Abstract

from arXiv · show

This is an up-to-date introduction to, and overview of, marginal likelihood computation for model selection and hypothesis testing. Computing normalizing constants of probability models (or ratio of constants) is a fundamental issue in many applications in statistics, applied mathematics, signal processing and machine learning. This article provides a comprehensive study of the state-of-the-art of the topic. We highlight limitations, benefits, connections and differences among the different techniques. Problems and possible solutions with the use of improper priors are also described. Some of the most relevant methodologies are compared through theoretical comparisons and numerical experiments.

1 Introduction

Marginal likelihoods and Bayes factors underpin Bayesian hypothesis testing and model selection, while normalizing constants also matter in statistical physics, numerical analysis, and double-intractable posteriors. This review unifies computational methods, compares their properties, and discusses improper-prior issues and numerical experiments.

1 Introduction

  • Marginal likelihoods and Bayes factors are central to Bayesian hypothesis testing and model selection.Normalizing constants and their ratios also arise in statistical physics and numerical analysis, and in double-intractable posterior analysis.Source: §1 · p. 1
  • The review jointly describes marginal-likelihood computational schemes under common notation, emphasizing their relationships, differences, strengths, and limitations.It surveys methods from the 1990s through recent algorithms and includes some novel material.Source: §1 · p. 2
  • The review organizes computational techniques into four families and covers methods for estimating individual normalizing constants or ratios.Most methods rely on importance sampling, with several combining MCMC and importance sampling.Source: §1 · p. 2
  • The review discusses improper-prior problems and possible solutions, and compares selected methods through theory and numerical experiments.Source: §1 · p. 2

2 Problem statement and preliminary discussions

The section defines the evidence as a model’s normalizing constant and explains its role in model comparison, including Bayes factors and model averaging. It relates evidence to model fit and complexity, then outlines six computational-method families and criteria.

2.1 Framework and notation

  • Evidence Z = p(y|M) is the model’s marginal likelihood, used for model selection; ratios of evidences are Bayes factors, and model posteriors can weight model averaging.The review assumes proper priors in its main treatment and points to a later discussion of improper priors.Source: §2.1 · pp. 4–5

2.2.1 Bounds of the evidence Z

  • Evidence is a prior-weighted average of likelihood values, bounded by the likelihood’s minimum and maximum, with the prior choice determining where within that range it falls.The extreme prior choices correspond to the worst and best model fit, respectively.Source: §2.2.1 · p. 6

2.2.2 Occam factor and implicit/intrinsic complexity penalization in Z

  • The Occam factor W captures evidence’s intrinsic complexity penalty through the overlap between likelihood and prior, and depends on the prior and data amount.Model-prior probabilities can add another penalty to posterior model probabilities.Source: §2.2.2 · p. 6

2.2.3 Occam factor with uniform priors

  • With uniform priors, the Occam factor is the likelihood mass within the prior bounds divided by prior width, and it can shrink as prior width or parameter dimension grows.When fit gains are eventually dominated by complexity penalization, evidence penalizes additional parameters.Source: §2.2.3 · p. 7

2.2.4 Marginal likelihood and information criteria

  • Information criteria connect likelihood fit and complexity through penalty terms, but criteria such as AIC and DIC are not themselves marginal-likelihood approximations.BIC is asymptotically related to −2 log Z; the Bayesian approach can assign different penalties to parameters through their priors.Source: §2.2.4 · pp. 7–8

2.3 A general overview of the computational methods

  • The review classifies methods into deterministic approximations, density estimation, importance sampling, and vertical representations converted to one-dimensional integrals.Examples include Laplace/BIC, Chib’s method, numerous IS variants, and nested sampling; quadrature is applied to the one-dimensional representation.Source: §2.3 · pp. 8–9

3 Methods based on deterministic approximations and density estimation

This section reviews deterministic approximations and density-estimation approaches for marginal likelihoods, from Laplace and BIC to pointwise density estimators and interpolation. Their assumptions and limitations differ, including sensitivity to tuning choices and degradation with dimension.

3.1 Laplace’s method

  • Laplace’s method approximates the posterior near its MAP with a Gaussian whose covariance uses the negative inverse Hessian of the log unnormalized posterior.The approximation follows from a quadratic expansion, and variants include Laplace-Metropolis and INLA for Gaussian Markov random field models.Source: §3.1 · pp. 9–10

3.2 Bayesian-Schwarz information criterion (BIC)

  • BIC uses the maximum-likelihood estimate and a parameter-count penalty, with BIC ≈ −2 log Z asymptotically as the data count grows.Smaller BIC values indicate better models, while the penalty Dθ log Dy discourages overfitting.Source: §3.2 · p. 10

3.3 Kernel density estimation (KDE)

  • KDE estimates posterior density at selected points to approximate Z, but the resulting estimator is generally biased and depends on point selection, bandwidth, and sample count.An importance-sampling variant drawing points from the KDE mixture is described as unbiased.Source: §3.3 · p. 11

3.4 Chib’s method

  • Chib’s method estimates posterior density at a selected point using Metropolis-Hastings output and the sampler’s acceptance-probability relationship.The point is usually chosen in a high-probability region, and the estimator is related to bridge sampling.Source: §3.4 · p. 12

3.5 Interpolative approaches

  • Interpolative approaches fit an unnormalized posterior using basis functions at chosen nodes, then integrate the interpolant to approximate Z.The coefficients solve a linear system; suitable functions include rectangular, triangular, and Gaussian forms.Source: §3.5 · pp. 12–13
  • Interpolation quality deteriorates as parameter dimension grows.Adaptive schemes can add nodes sequentially to improve the approximation.Source: §3.5 · p. 13

4 Techniques based on IS

This section presents importance sampling as a central framework for estimating normalizing constants, with one-proposal, multiple-proposal, and adaptive variants. Estimators differ in bias, proposal requirements, and optimal proposal choices.

4 Techniques based on IS

  • Importance sampling is a central framework for marginal-likelihood computation, and many other methods are directly or indirectly related to it.The review summarizes estimators and special cases in a table.Source: §4 · pp. 13, 15
  • The standard IS estimator is unbiased when it averages weights from independent draws of a known proposal density.Source: §4 · p. 14
  • For standard IS, the proposal minimizing mean square error is the posterior density.Source: §4 · p. 14
  • An alternative IS version permits an unnormalized proposal and is biased, but is a convex combination of likelihood values that respects the evidence bounds.Its MSE-optimal proposal is proportional to the absolute difference between posterior density and prior density.Source: §4 · pp. 14–15
  • IS approaches include a single known proposal, multiple proposals or mixtures, and adaptive schemes that update proposals toward an optimal proposal.Some adaptive methods combine MCMC and importance sampling.Source: §4 · p. 15

4.1 Techniques using draws from one proposal density

The section organizes one-proposal estimators as importance-sampling schemes, then connects ratio-estimation identities to umbrella sampling and its special case for estimating Z.

4.1 Techniques using draws from one proposal density

  • One-proposal methods use a single proposal density, with the techniques differing in their choice of that density.The ideal posterior proposal is difficult to use because it requires posterior draws and the unknown Z.Source: §4.1 · p. 15
  • The naive Monte Carlo estimator averages likelihoods from prior draws, but can have large variance when prior and posterior overlap is small.It is a special case of importance sampling with the prior as proposal.Source: §4.1 · p. 16
  • The harmonic-mean estimator converges almost surely, but its variance is often high and can be infinite.It is a special case of reverse importance sampling; that scheme requires posterior draws, and its auxiliary function must have lighter tails than the unnormalized posterior.Source: §4.1 · pp. 16–17

4.1.1 The pre-umbrella estimators

  • The pre-umbrella identity estimates a ratio of normalizing constants using draws from one normalized density, and its relative MSE depends on the Pearson divergence between the densities.The framework covers almost all estimators discussed in the section, except IS vers-2.Source: §4.1.1 · p. 17

4.1.2 Umbrella Sampling (a.k.a. ratio importance sampling)

  • Umbrella sampling adds a middle density with overlap with both target densities; its optimal proposal is asymptotically more efficient than the pre-umbrella estimator.The optimal proposal depends on the unknown ratio, so a two-stage procedure is used in practice.Source: §4.1.2 · p. 18

4.1.3 Umbrella for Z: the self-normalized Importance Sampling (Self-IS)

  • For estimating Z, self-normalized IS is the more general form and includes the other one-proposal estimators as special cases.Its optimal proposal cannot be evaluated directly, so the section recommends a two-stage procedure.Source: §4.1.4 · p. 20, §4.1.3 · p. 20

4.2 Techniques using draws from two proposal densities

The section develops estimators using draws from two proposal densities, centering on bridge sampling and its connections to other methods. It also describes restricted and iterative variants.

4.2 Techniques using draws from two proposal densities

  • Two-proposal methods use samples from both normalized densities, equivalently a deterministic mixture of the two proposals.The section presents them as estimators of ratios of normalizing constants.Source: §4.2 · p. 21

4.2.1 Bridge sampling identity

  • The bridge sampling identity generalizes the pre-umbrella identity through an arbitrary function defined on the overlap of the two proposal supports.Setting that function to 1/q2 recovers the pre-umbrella identity.Source: §4.2.1 · p. 21

4.2.2 Relationship with Chib’s method

  • Chib’s estimator connects to bridge sampling: its proposal and acceptance function play the roles of the bridge proposal and function, subject to a balance condition.The more general expression also applies to non-symmetric proposals.Source: §4.2.2 · pp. 22–23

4.2.3 Locally-restricted IS and RIS

  • Locally restricted IS and RIS limit integration to a subset B, often a high-posterior or high-likelihood region, to address efficiency concerns.The restricted estimates use estimated posterior or proposal mass within B.Source: §4.2.3 · pp. 23–24

4.2.4 Optimal construction of bridge sampling

  • The optimal bridge density depends on the unknown ratio, and optimal bridge sampling has greater asymptotic error than optimal umbrella sampling.The optimal bridge procedure is iterative; all samples from one proposal reduce it to standard IS, while samples from the other reduce it to RIS.Source: §4.2.4 · pp. 25–26

4.2.5 Other estimators drawing from a generic proposal and the posterior

  • Iterative mixture-based estimators converge to optimal bridge sampling, and the bridge iteration seems to converge quickest in the simulation study.The mixture estimators cannot be used directly because they require evaluating the posterior density, which depends on Z.Source: §4.2.5 · pp. 26–27

4.2.6 Summary

  • The section’s methods are unified by a generic formula, while differing in which densities they sample from and which they only evaluate.The text summarizes these distinctions in Table 8.Source: §4.2.6 · p. 27

4.3 IS based on multiple proposal densities

The section surveys importance-sampling estimators built from multiple proposals, especially tempered distributions linking an easier density to the posterior. It compares their constructions, relationships, and practical choices.

4.3 IS based on multiple proposal densities

  • Tempered posteriors interpolate between the posterior and an easier density; their flatter distributions can improve MCMC mixing and exploration of high-posterior regions.The user selects the number of intermediate densities, which can define a temperature schedule linking prior and posterior.Source: §4.3 · pp. 27–28

4.3.1 Multiple Importance Sampling (MIS) estimators

  • MIS draws from multiple proposals and uses proper importance weights; the deterministic-mixture estimator is unbiased, while an alternative estimator is biased.The schemes vary in which densities are sampled and which are only evaluated.Source: §4.3.1 · pp. 28–29

4.3.2 Tempered posteriors as proposal densities

  • With a tempered posterior as proposal, IS-P uses likelihood-powered weights; mixing samples from several temperatures instead requires unknown normalizing constants, which RLR estimates iteratively.The method uses densities proportional to g(θ)ℓ(y|θ)^β, with β between 0 and 1.Source: §4.3.2 · pp. 29–30

4.3.3 Stepping-stone (SS) sampling

  • Stepping-stone sampling estimates successive normalizing-constant ratios across temperatures and multiplies them to estimate Z; the resulting estimator is unbiased.With K = 1, it reduces to the naive Monte Carlo estimator.Source: §4.3.3 · p. 31

4.3.4 Path sampling (a.k.a., thermodynamic integration)

  • Path sampling links prior and posterior along a density path and estimates log Z by integrating the path’s normalizing-constant derivative.The geometric path is generic and simple to implement, although the optimal path is impractical because it depends on the constants being estimated.Source: §4.3.4 · p. 32

4.3.5 Connections among path sampling, bridge sampling and stepping-stones

  • Path sampling arises as a continuous chain of bridge-sampling steps, whereas stepping-stone sampling uses a different identity to estimate each adjacent normalizing-constant ratio.The section presents these as connections among the methods’ identities and multi-stage extensions.Source: §4.3.5 · p. 33

4.3.6 Method of Power Posteriors

  • The power-posterior method estimates log Z by quadrature over temperatures and is a geometric-path special case of path sampling; deterministic quadrature makes its approximation biased.A small number of temperatures can yield a poor approximation, unlike in stepping-stone sampling.Source: §4.3.6 · p. 34

4.3.7 On the selection of βk

  • Temperature ladders for stepping-stone and power-posterior methods can concentrate values near β = 0, where the tempered posterior changes more rapidly.Path sampling also requires choosing a density p(β); for a given path, the generalized local Jeffreys prior is optimal.Source: §4.3.7 · p. 35

5 Advanced schemes combining MCMC and IS

The section develops methods that combine MCMC and importance sampling, including sequential schemes, multiple-try MCMC, and layered adaptive importance sampling. These methods differ in how they move, weight, and reuse samples.

5 Advanced schemes combining MCMC and IS

  • MCMC and importance sampling can be combined as MCMC-within-IS, IS-within-MCMC, or IS-after-MCMC; the section covers An-IS, SMC, MTM, and LAIS.An-IS and SMC use tempered targets; MTM embeds importance sampling in proposals, while LAIS separates chain adaptation from sampling.Source: §5 · p. 36

5.1.1 Weighting a sample after one MCMC iteration

  • After one MCMC update that preserves the target, the importance weight remains unchanged, yielding an unbiased estimator of Z.The construction begins with a draw from a normalized proposal and then applies a kernel whose invariant density is the target.Source: §5.1.1 · p. 37

5.1.2 Annealed Importance Sampling (An-IS)

  • An-IS moves through a sequence of increasingly tempered targets and weights the resulting samples using an extended-space importance-sampling construction.The final target is the posterior; earlier targets can be flatter to support exploration.Source: §5.1.2 · pp. 38–39

5.2.1 Generic Sequential Monte Carlo

  • SMC extends sequential importance sampling with MCMC transitions and resampling across a sequence of targets; An-IS is a special case under particular forward and backward choices.The sequence may represent a dynamic state-space model or tempered densities in a static problem.Source: §5.2.1 · pp. 40, 42, 43

5.2.2 Evidence computation in a sequential framework with resampling steps

  • With proper weighting of resampled particles, the two evidence estimators are equivalent; without it, only the second estimator is valid.The estimators are also equivalent in sequential importance sampling without resampling.Source: §5.2.2 · p. 44

5.3 IS-within-MCMC: Estimation based on Multiple Try MCMC schemes

  • Multiple Try Metropolis compares several weighted candidates at each iteration, aiming to explore more of the sample space and reduce correlation between chain states.The described adaptive independent scheme also uses weighted candidates to estimate the marginal likelihood.Source: §5.3 · pp. 44–45

5.4 IS-after-MCMC: Layered Adaptive Importance Sampling (LAIS)

  • LAIS uses MCMC samples as proposal locations, then applies a multiple-importance-sampling step to weight newly generated samples.Its denominator can mix all proposals or use temporal or spatial mixtures; the all-proposal choice is most costly but is reported to give the best estimator efficiency.Source: §5.4 · pp. 45–46
  • When LAIS uses many chains or long chains, clustering can reduce the cost of computing mixture denominators by using K << NT clusters.Compressed LAIS uses cluster centroids and weights proportional to cluster sizes.Source: §5.4 · p. 48

6 Vertical likelihood representations

This section introduces a different approach to marginal-likelihood computation: transform its integral into one-dimensional representations, then consider quadrature and nested sampling.

6 Vertical likelihood representations

  • The section develops one-dimensional Lebesgue representations of the marginal-likelihood integral, considers quadrature for these forms, and presents nested sampling as a possible solution when quadrature is not straightforward.The approach first derives two one-dimensional integral representations.Source: §6 · p. 48

6.1 Lebesgue representations of the marginal likelihood

The section derives one-dimensional representations of the evidence using likelihood levels and prior mass, and describes related sampling distributions. It notes that direct quadrature is difficult when the resulting functions are intractable and vary rapidly.

6.1.1 First one-dimensional representation

  • Assuming the likelihood is bounded, switching the integration order in an indicator-function representation yields a one-dimensional integral for Z.The upper integration limit is the supremum of the likelihood.Source: §6.1.1 · p. 49

6.1.2 The survival function Z(λ) and related sampling procedures

  • Z(λ) is the prior mass where the likelihood exceeds λ, so it is a non-increasing survival function; for prior draws, Z of their likelihood values is uniform on [0, 1].The likelihood under the prior has cumulative distribution F(λ) = 1 − Z(λ).Source: §6.1.2 · pp. 49–50

6.1.3 The truncated prior pdf g(θ|λ) and other sampling procedures

  • The prior truncated to likelihoods above λ has normalizing constant Z(λ); under this truncated prior, Z of sampled likelihood values is uniform on [0, Z(λ0)].A sample from the truncated prior can be generated by accepting prior draws whose likelihood exceeds λ0.Source: §6.1.3 · pp. 50–51

6.1.5 Distributions eamax

  • The maximum of N uniform random variables follows a Beta(N, 1) distribution, a result used to derive the standard nested-sampling method.The section summarizes relationships among the random variables used in this derivation.Source: §6.1.5 · p. 52, Table 15 · p. 52

6.1.6 Second one-dimensional representation

  • A second one-dimensional representation uses the inverse function Ψ(a), with Z(λ) > a exactly when λ < Ψ(a).The section then summarizes this representation alongside the first one.Source: §6.1.6 · pp. 52–53, §6.1.7 · p. 53

6.1.7 Summary of the one-dimensional representations

  • Direct quadrature of Z(λ) or Ψ(a) is difficult because these functions are usually intractable and change rapidly, making performance sensitive to careful grid selection.The section summarizes one-dimensional expressions for Z and log Z, whose integrands depend on θ explicitly or implicitly.Source: §6.1.7 · p. 53

6.2 Nested Sampling

Nested sampling estimates the marginal likelihood by quadrature over ordered likelihood ordinates and corresponding prior masses. The section discusses constrained-prior sampling challenges and connects related estimators through vertical representations.

6.2 Nested Sampling

  • Nested sampling estimates Z by quadrature over likelihood ordinates and corresponding prior masses, which are approximated because Ψ(a) is generally intractable.The method selects likelihood ordinates and approximates the associated prior masses rather than evaluating Ψ(a) on a chosen grid.Source: §6.2 · pp. 53–54, Table 16 · p. 54
  • At each iteration, nested sampling replaces the lowest-likelihood sample with one drawn from the prior truncated above the current likelihood threshold.The sequence of likelihood thresholds increases as the sample population is replenished.Source: §6.2 · pp. 54–55

6.2.2 Further considerations

  • Rejection sampling acceptance falls as the likelihood threshold rises, while MCMC can perform poorly with correlated samples, disjoint regions, or distant modes.The standard method also approximates prior-mass shrinkage; MCMC sampling can introduce further approximation through burn-in.Source: §6.2.2 · p. 57

6.2.3 Generalized Importance Sampling based on vertical representations

  • Vertical generalized importance sampling uses a user-defined W(λ); W(λ) = λ recovers the harmonic mean estimator, while W(λ) = λ^β gives a power-posterior proposal.The authors suggest nested sampling may also fit this framework.Source: §6.2.3 · p. 57

7 On the marginal likelihood approach and other strategies

The section reviews how prior choice and improper priors affect marginal-likelihood model comparison, alongside alternatives based on prior- and posterior-predictive approaches. It describes strategies for handling these issues and notes that posterior predictive approaches are less affected by prior choice with informative data.

7.1.1 Dependence on the prior

  • The marginal likelihood is highly sensitive to the prior, and more diffuse proper priors generally yield smaller Z values that can change model selection.The marginal likelihood averages likelihood values weighted by the prior.Source: §7.1.1 · p. 58

7.1.2 Safe scenarios for fair comparisons

  • Using the same, even improper, priors supports comparisons only when models share parameters and support; likelihood-based priors can instead use tempered or subset data.Shared nuisance-parameter constants cancel in Bayes factors, and likelihood-based priors connect to partial and intrinsic Bayes factors.Source: §7.1.2 · pp. 58–59

7.2 Bayes factors with improper priors

  • An improper prior leaves its normalizing constant unspecified; although the posterior can be well-defined, marginal likelihoods and Bayes factors depend on those constants.Different choices of constants can therefore yield different preferred models.Source: §7.2 · pp. 59–60
  • Partial Bayes factors use training data to construct proper priors, but their model comparison can depend on the chosen training set.Finding a minimal suitable training set is not straightforward.Source: §7.2 · pp. 60–61
  • Fractional Bayes factors remove dependence on the arbitrary prior constants, while intrinsic Bayes factors average partial Bayes factors over minimal training sets.Intrinsic Bayes factors may use either an arithmetic or geometric mean.Source: §7.2 · pp. 61–62

7.3 Marginal likelihood as a prior predictive approach

  • The prior-predictive approach compares simulated datasets with observed data or estimates each model’s marginal likelihood from its simulated-data set.The simulated data are generated by drawing parameters from the model prior and then drawing data from its likelihood.Source: §7.3 · p. 62

7.4 Other ways of model selection: the posterior predictive approach

  • Posterior predictive model checking compares simulated data with observations, and its posterior-based distribution is less affected by prior choice when data are informative.Checks can use test quantities, graphical checks, or posterior predictive p-values.Source: §7.4 · pp. 62–63
  • Using p(yi|y) overestimates ELPD, so LOO-CV instead uses p(yi|y−i); importance-sampling weights for the direct approach can have high or infinite variance.The true data-generating density used to define ELPD is unknown, requiring approximations.Source: §7.4 · pp. 63–64

8 Numerical comparisons

The numerical comparisons evaluate marginal-likelihood estimators in illustrative Gaussian, concentrated-likelihood, and multimodal settings and real-data analyses, including regression, biochemical oxygen demand, and COVID-19. They examine how prior–likelihood overlap and proposal tuning relate to estimator performance, and compare support for candidate bases in the COVID-19 analysis.

8.1.1 First setting: Gaussians with same mean and different variances

  • When the Gaussian proposal variance is smaller than the target variance, IS can have very high MSE; in the realistic setting, iterative mixture estimators converge to optimal bridge sampling.In the ideal setting, mixture IS and mixture self-IS perform best, but this advantage is unachievable in practice; iterative optimal bridge sampling converges fastest, and two-stage optimal self-IS does not beat it when h < 1.Source: §8.1.1 · p. 66, Figure 8 · p. 67, Figure 9 · p. 67

8.1.2 Second setting: Gaussians with same variance and different means

  • As the Gaussian proposal mean moves farther from the target, IS and RIS MSE diverge as e^µ2, while optimal bridge and self-IS estimators are less affected.Optimal self-IS performs best; optimal bridge and suboptimal self-IS perform similarly, and both are worse than optimal self-IS.Source: §8.1.2 · p. 67, Figure 10 · p. 68

8.2 Second experiment: Gaussian likelihood and uniform prior

  • With a broad prior and concentrated likelihood, naive Monte Carlo underestimates and harmonic mean and tempered-posterior IS overestimate; stepping stone and power posterior are accurate.For the less challenging scenario, naive Monte Carlo performs well because the prior is a good proposal; increasing K improves power posterior, while stepping-stone error slightly worsens as fewer samples come from the prior.Source: §8.2 · p. 69, Figure 11 · p. 70
  • In the concentrated-likelihood scenario, stepping stone and power posterior greatly outperform the other estimators; concentrating β values near zero improves both methods.The scenario uses 100 observations and prior width 1000; power-posterior error decreases with K, while stepping-stone error improves with K for α = 0.25 but worsens for α = 1.Source: §8.2 · p. 70, Figure 12 · p. 71

8.3 Third experiment: posterior as mixture of two components

  • RIS and CLAIS maintain small, nearly constant relative error as posterior mixture modes separate in dimensions 1 and 5, while Laplace degrades and harmonic mean performs poorly.Naive Monte Carlo has the lowest error at the smallest separation, where prior and posterior are similar, but is quickly outperformed by RIS and CLAIS.Source: Figure 13 · p. 72, §8.3 · p. 72

8.4 Experiment with biochemical oxygen demand data

  • Table 19 compares seven methods by relative MAE and its standard error for estimating the marginal likelihood, whose ground-truth log Z is −16.208 for the two-parameter target after integrating out σ.The target is θ = [θ1, θ2], with uniform priors θ1 ∼ U([0, 60]) and θ2 ∼ U([0, 6]); σ has improper prior g3(σ) ∝ 1/σ and is integrated out.Source: §8.4 · p. 73, Table 19 · p. 74
  • Naive Monte Carlo performs best when prior and likelihood overlap amply; IS and CLAIS also perform well, RIS-kde beats RIS, and HM performs worst.Source: §8.4 · p. 74
  • For tested C and h, CLAIS and RIS with clusterized KDE are robust and comparable to Chib’s estimator, whose error is lower when θ* is near the posterior mode than in the fair scenario.For CLAIS at C=10, h should not be too small; RIS performs best at h=0 and worsens as h increases.Source: §8.4 · pp. 74–75, Figure 14 · p. 75

8.5 Experiment with COVID-19 data

  • Across all four bases, both methods favor M ∈ {5,6,7,8}; interval probability is generally near 0.2 and exceeds 0.2 for ν=2,3, reaching 0.35 for NMC at ν=2.Source: §8.5 · p. 77
  • M=8 already fits well, and M=5–8 approximates RVM/GP models with M=140 [98]; ν=3 is discarded, while NMC slightly favors Laplacian and MTM nearly ties it with Gaussian.Source: §8.5 · p. 77

9 Final discussion

The discussion synthesizes marginal-likelihood methods for model selection and hypothesis testing, including ratio estimators and prior considerations. It offers practical guidance while emphasizing method-specific trade-offs and unresolved comparisons.

Final discussion

  • The review covers estimators of both marginal likelihoods and ratios of normalizing constants, discusses prior choice and improper priors, and summarizes practical options.It also describes posterior-predictive alternatives to model selection.Source: §9 · p. 79
  • The authors recommend considering Naive Monte Carlo as an initial attempt, while describing the harmonic-mean estimator as the worst estimator of Z and a potentially loose upper bound.The harmonic-mean estimator may be very imprecise as an upper bound.Source: §9 · p. 79
  • MTM is recommended within MCMC schemes because it supplies two estimators of Z and samples usable by Chib, RIS, and LAIS.Those samples can be reused in other estimators.Source: §9 · p. 79
  • The optimal umbrella estimator performs best theoretically for estimating ratios of normalizing constants, but costly implementation may prevent that performance in practice.Sampling from its optimal proposal is not straightforward.Source: §9 · p. 79
  • Chib’s method is a strong alternative, but its need for internal MCMC details rules it out as post-processing for a black-box chain.HM or LAIS can be used for that post-processing role.Source: §9 · p. 79
  • LAIS lies between Naive Monte Carlo and the harmonic-mean estimator, combining posterior-localized samples with standard importance-sampling properties.The review places bridge sampling, SS, path sampling, and other tempered-posterior methods in this middle ground as well.Source: §9 · p. 80
  • Tempered-posterior methods perform well, but depend on temperature choices; among SS, PS, PP, An-IS, and SMC, the authors regard SS as more robust and simpler.SS also requires fewer tempered posteriors than PS and PP; the authors call for more comparisons among SS, An-IS, and SMC.Source: §9 · p. 80
  • Nested sampling is widely used, but sampling from truncated priors is difficult; its implicit likelihood optimization is identified as an important feature.The authors note that high likelihood values matter to other computational schemes too.Source: §9 · p. 80

A Table of other reviews

The review compares related surveys by topic coverage, material, and schemes, while also considering completeness and depth of derivations. It presents the comparison as illustrative rather than exhaustive.

A Table of other reviews

Loading 2005.08334v4…