PaperlayerView exact revision on arXiv

Source-linked AI summary

Validating Bayesian Inference Algorithms with Simulation-Based Calibration

Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, Andrew Gelman

arXiv:1804.06788v2stat.ME

TL;DR

Bayesian computation is difficult to validate for complex models because both model implementations and inference algorithms can be wrong. The paper introduces simulation-based calibration, which validates posterior-sampling algorithms using simulations from the Bayesian joint distribution and graphical diagnostics. SBC identifies computational or implementation problems and helps indicate their effects, while remaining limited to computational validation and sensitive to diagnostic noise and unresolved statistic-selection questions.

  • Problem

    Complex Bayesian models require sophisticated implementations and algorithms, but validation is needed to determine whether computation and model specification are correct.

  • Method

    SBC simulates ground truths and data from the Bayesian joint distribution, compares computed posterior samples with those simulations, and uses graphical diagnostics.

  • Results

    SBC identifies biased computation and incorrect model specifications, while its visualizations can indicate how resulting inferences are affected.

  • Takeaways & Limitations

    SBC is presented as an important step in a robust Bayesian workflow and as a tool for computational algorithm and statistical software development.

  • Takeaways & Limitations

    SBC validates computational behavior only; it does not guarantee posterior coverage for an individual observation or that the model captures the truth.

Abstract

from arXiv · show

Verifying the correctness of Bayesian computation is challenging. This is especially true for complex models that are common in practice, as these require sophisticated model implementations and algorithms. In this paper we introduce \emph{simulation-based calibration} (SBC), a general procedure for validating inferences from Bayesian algorithms capable of generating posterior samples. This procedure not only identifies inaccurate computation and inconsistencies in model implementations but also provides graphical summaries that can indicate the nature of the problems that arise. We argue that SBC is a critical part of a robust Bayesian workflow, as well as being a useful tool for those developing computational algorithms and statistical software.

1. INTRODUCTION

Bayesian inference becomes difficult to validate as models and datasets grow more complex. The paper introduces SBC as a generic validation procedure for algorithms that produce posterior samples.

  • 1. INTRODUCTION: Complex models require inference algorithms to handle structured data and posterior distributions accurately.Algorithm performance depends on model and data details, so methods that work in one analysis may fail in another.
  • 1. INTRODUCTION: Bes​​poke models increase the opportunity for specification mistakes and inaccurate computations.Validation must check both whether code implements the intended model and whether the inference algorithm performs the necessary computations accurately.
  • 1. INTRODUCTION: SBC validates Bayesian computational methods capable of producing posterior samples or approximate posterior samples.The procedure applies to Monte Carlo methods and deterministic methods such as INLA and ADVI when their approximate posteriors can be sampled.
  • 1. INTRODUCTION: SBC provides qualitative diagnostics that can identify how a validated algorithm might be failing.The paper examines SBC output interpretation and its incorporation into a robust Bayesian workflow.

2. SELF-CONSISTENCY OF THE BAYESIAN JOINT DISTRIBUTION

Exact posterior expectations are usually unavailable for realistic models, and a single simulated ground truth cannot characterize an algorithm reliably. The paper instead evaluates inference across ground truths drawn from the Bayesian joint distribution, whose exact posterior averages reproduce the prior.

  • 2. SELF-CONSISTENCY OF THE BAYESIAN JOINT DISTRIBUTION: Exact posterior expectations are unavailable for all but the simplest models, motivating validation methods that do not require them.Those simple models may also differ structurally from the models used in applications.
  • 2. SELF-CONSISTENCY OF THE BAYESIAN JOINT DISTRIBUTION: A single simulated ground truth can make correct inference appear unsuccessful or incorrect inference appear successful.Individual simulations therefore do not meaningfully characterize an algorithm; multiple ground truths are needed.
  • 2. SELF-CONSISTENCY OF THE BAYESIAN JOINT DISTRIBUTION: The evaluation should cover the parameter space according to the prior, because the prior quantifies which parameter values are relevant.The procedure samples a ground truth from the prior, simulates corresponding data, and compares the recovered posterior with that ground truth.
  • 2. SELF-CONSISTENCY OF THE BAYESIAN JOINT DISTRIBUTION: Averaging exact posteriors over the Bayesian joint distribution returns the prior distribution.Equivalently, data-averaged exact posterior expectations equal the corresponding prior expectations.
  • 2. SELF-CONSISTENCY OF THE BAYESIAN JOINT DISTRIBUTION: Any discrepancy between the data-averaged posterior and the prior indicates inaccurate posterior computation or a model mis-implementation.This comparison provides a generic validation method within the scope of the modeling assumptions.

3. EXISTING VALIDATION METHODS EXPLOITING THE BAYESIAN JOINT DISTRIBUTION

Earlier validation methods exploit Bayesian self-consistency but face difficulties from convergence, finite MCMC samples, autocorrelation, discretization, and continuity corrections. A regression example shows that these issues can produce apparent non-uniformity even for accurate computation.

  • 3. EXISTING VALIDATION METHODS EXPLOITING THE BAYESIAN JOINT DISTRIBUTION: Geweke’s method uses an auxiliary Gibbs sampler, but its diagnostics are meaningful only after convergence.Strong data-parameter correlations in generative models can make convergence slow and difficult to assess.
  • 3. EXISTING VALIDATION METHODS EXPLOITING THE BAYESIAN JOINT DISTRIBUTION: Cook, Gelman and Rubin’s method evaluates posterior CDF values, which should be uniformly distributed under exact computation.It transforms these values into z-scores and can quantify deviations with a χ2 test.
  • 3. EXISTING VALIDATION METHODS EXPLOITING THE BAYESIAN JOINT DISTRIBUTION: Finite MCMC samples and autocorrelation can make estimated quantiles deviate from the assumed uniform distribution independently of computational bias.Without suitable asymptotic conditions, deviations may reflect pre-asymptotic behavior rather than inaccurate posterior computation.
  • 3. EXISTING VALIDATION METHODS EXPLOITING THE BAYESIAN JOINT DISTRIBUTION: Empirical CDF values are discrete, taking L + 1 evenly spaced values on [0,1].This discretization creates visualization artifacts and requires continuity corrections at finite-sample boundary values.
  • 3. EXISTING VALIDATION METHODS EXPLOITING THE BAYESIAN JOINT DISTRIBUTION: A Stan regression example produced strong histogram deviations from uniformity despite Stan’s known accuracy for the analysis.Posterior-sample autocorrelation also appeared in the histogram, illustrating how the framework can suggest algorithmic problems that are not present.

4. SIMULATION-BASED CALIBRATION

Simulation-based calibration (SBC) compares rank-statistic histograms with the discrete uniform distribution expected under correctly implemented Bayesian inference. Deviations can reveal autocorrelation, dispersion errors, or directional bias, while calibration remains limited to computational accuracy.

  • SBC procedure: SBC compares histograms of rank statistics with the discrete uniform distribution expected when the analysis is correctly implemented.The procedure uses repeated prior draws, simulated datasets, posterior samples, and rank statistics across one-dimensional quantities.
  • Interpreting deviations: Boundary spikes indicate non-negligible autocorrelation among posterior samples, which can bias ranks toward extremely small or large values.The paper discusses thinning correlated Markov chain Monte Carlo samples to reduce this deviation.
  • Expected behavior: Uniform rank statistics are consistent with independent samples from the exact posterior of a correctly specified model.The rank statistic is uniformly distributed over the integers [0,L] under the Bayesian joint distribution.
  • Interpreting deviations: A characteristic ∩-shaped histogram indicates overdispersed computed posteriors, whereas a ∪-shaped histogram indicates under-dispersed computed posteriors.Overdispersion makes rank statistics concentrate toward extremes; underdispersion produces the converse pattern.
  • Interpreting deviations: Asymmetry in the rank histogram indicates bias in the computed posterior, with ranks shifted in the opposite direction.Posterior samples biased smaller produce higher rank statistics, while samples biased larger produce lower rank statistics.
  • Scope and workflow: SBC adapts to the specific model design but cannot guarantee posterior coverage for a single observation or validate whether the model captures the truth.Sensitivity analysis and posterior predictive checks address posterior behavior for a given observation and model assumptions, respectively.

5. EXTENDING SIMULATION-BASED CALIBRATION

SBC can be extended to practical settings by mitigating autocorrelation in MCMC samples and improving sensitivity to small deviations from uniformity. These extensions remain subject to diagnostic and computational limitations.

  • SBC can validate simulation-based Bayesian algorithms, but its usefulness is limited in several practical circumstances.
  • Mitigating the Effect of Autocorrelation: MCMC posterior samples can produce nonuniform SBC histograms because dependence obscures bias in the samples.The proposed remedy is to thin chains before computing rank statistics.
  • Mitigating the Effect of Autocorrelation: Effective-sample-size-based thinning can remove enough autocorrelation for SBC, although more conservative thinning may yield much smaller samples.
  • Mitigating the Effect of Autocorrelation: Residual deviations after thinning indicate that MCMC estimators may lack a central limit theorem and that chains may not adequately explore the target space.
  • Simulation-Based Calibration for Small Deviations: SBC histograms may miss small deviations, motivating ECDFs and other statistics that improve sensitivity near extreme ranks.ECDFs reduce variation at small and large ranks, while specialized statistics can be harder to interpret and may be less sensitive to systematic deviations.

6. EXPERIMENTS

The experiments apply SBC across prior misspecification, hierarchical MCMC, ADVI, and INLA examples, showing that rank diagnostics reveal distinct computational and modeling problems. They also demonstrate how thinning, parameterization, and finer ECDF views affect interpretation.

  • With L = 100 posterior samples, correctly calibrated rank statistics should follow a U[0,100] discrete uniform distribution.
  • 6.1 Misspecified Prior: A wider data-generating prior than the fitting prior produces a characteristic ∪-shaped SBC histogram for β.
  • 6.2 Biased Markov chain Monte Carlo: Centered eight-schools geometry causes HMC posterior samples to be biased toward larger τ values and produces a funnel that impedes exploration.
  • 6.2 Biased Markov chain Monte Carlo: Thinning removes the apparent bias in the non-centered eight-schools parameterization, whereas unthinned samples show autocorrelation through spikes at L = 100.
  • 6.3 ADVI: ADVI drastically underestimates the posterior for the regression slope β, unlike HMC, whose rank histogram is consistent with uniformity.
  • 6.4 INLA: For the Kenya INLA analysis, N = 1000 replications yield slightly more low ranks than uniformity predicts, while wide histogram uncertainty motivates ECDF inspection.
  • 6.4 INLA: INLA’s bias is linked to low-information binomial observations and is viewed as acceptable near 5.4% prevalence but inappropriate at 0.1%.

7. CONCLUSION

The conclusion presents SBC as a readily implemented validation procedure for identifying and interpreting biased computation and model-specification errors. It also highlights visualization, autocorrelation, and multivariate calibration as important boundaries and future directions.

  • SBC identifies biased computational algorithms and incorrect model specifications, while its visualizations indicate how these problems affect inferences.
  • Visualization-based interpretation can be limiting for models with many parameters, motivating numerical summaries and automated diagnostics.
  • Global χ2 goodness-of-fit summaries performed poorly because deviations from uniformity often follow a few systematic failure modes.
  • Future work could calibrate SBC deviations under autocorrelation, remove thinning, and define effective sample size beyond MCMC.
  • SBC histograms assess only one-dimensional posterior summaries, leaving multivariate quantities as an important limitation.

APPENDIX A: CODE LISTINGS

The appendix provides code listings for the linear-regression and eight-schools examples, with an implementation note about Stan’s normal-distribution parameterization.

  • Stan parameterizes the normal distribution by mean and standard deviation, whereas the text uses mean and variance.
  • The code uses loops over N observations and J groups and includes a transformed-parameters block.
  • The listings include the linear-regression data-generating process and inference model.
  • The eight-schools examples include centered and non-centered parameterizations.

APPENDIX B: PROOF OF THEOREM 1

The proof establishes that, under prior-generated data and independent exact posterior samples, a one-dimensional rank statistic is discrete uniform over all possible ranks.

  • Theorem 2 states that the rank statistic of any one-dimensional variable is uniform over integers [0,L].
  • The proof evaluates the variable on the prior draw and each posterior draw, then orders the posterior evaluations.
  • The rank probability mass function is written using the combinatorial factor L! / (r!(L − r)!).
  • Conditioning on the observation makes posterior samples independent of the conditioning model configuration, enabling the uniform-rank argument.
  • The resulting rank distribution is consistent with a uniform distribution over the L + 1 possible ranks.
Loading 1804.06788v2…