Source-linked AI summary
Visualization in Bayesian workflow
Jonah Gabry, Daniel Simpson, Aki Vehtari, Michael Betancourt, Andrew Gelman
TL;DR
Bayesian data analysis requires an iterative workflow beyond posterior computation, and the paper examines how visualization can support that process. Using a global PM2.5 example, it applies visual checks and comparisons across model building, computation, evaluation, and expansion, concluding that visualization complements quantitative procedures throughout.
Problem
Bayesian analysis and visualization must address the full iterative workflow of model building, inference, checking, evaluation, and expansion rather than posterior computation or trace plots alone.
Method
The paper follows a global PM2.5 example using visualization for exploratory analysis, prior and posterior predictive checks, computational diagnostics, model comparison, and iterative model improvement.
Results
Visualization is demonstrated as a complement to quantitative procedures for comparing models, identifying fit failures, checking computation, setting priors, and improving models.
Takeaways & Limitations
Visualizations can be used throughout the Bayesian workflow to understand models, assess predictions and computation, and guide model changes.
Takeaways & Limitations
The paper uses a smaller three-model network than a comprehensive PM2.5 analysis would require, and prior predictive checks cannot select between models.
Abstract
from arXiv · showhide
Bayesian data analysis is about more than just computing a posterior distribution, and Bayesian visualization is about more than trace plots of Markov chains. Practical Bayesian data analysis, like all data analysis, is an iterative process of model building, inference, model checking and evaluation, and model expansion. Visualization is helpful in each of these stages of the Bayesian workflow and it is indispensable when drawing inferences from the types of modern, high-dimensional models that are used by applied researchers.
1. Introduction and running example
The paper presents visualization as a tool used throughout the Bayesian workflow, illustrated through global PM2.5 prediction. The running example combines sparse ground-monitor measurements with satellite estimates to build calibrated prediction intervals.
- Visualization supports exploratory analysis, prior predictive checks, computational diagnostics, posterior predictive checks, and model comparison.
- The running example estimates global ambient PM2.5 concentration from city-level data linked to health outcomes.
- Direct PM2.5 measurements come from 2980 ground monitors with heterogeneous spatial coverage, especially sparse across Africa, central Asia, and Russia.
- Satellite aerosol optical-depth measurements are calibrated against ground monitors to estimate PM2.5 at the spatial resolution required by population data.
- The analysis focuses on three simple, plausible models for constructing, checking, computing, and evaluating PM2.5 predictions with calibrated prediction intervals.
2. Exploratory data analysis goes beyond just plotting the data
Exploratory visualization is used to build a network of increasingly complex models that reflects heterogeneity and limitations in sparse, unbalanced PM2.5 data. Regional plots reveal that a single linear regression is inadequate, motivating multilevel alternatives.
- Exploratory analysis builds increasingly complex models intended to capture features and heterogeneities present in the data.
- The simplest model treats satellite estimates as predictors of ground-monitor data after a simple affine adjustment, with the log-log fit achieving R2 ≈0.6.
- A single linear regression may introduce ecological bias because regional trends differ across WHO and clustered super-regions.
- Sparse data in Sub-Saharan Africa and clustered regions 1 and 6 make their linear trends difficult to determine comprehensively.
- The resulting model network contains one simple linear regression and two linear multilevel models stratified by WHO or clustered super-regions.
- The three-model network is sufficient for demonstrating the workflow but is smaller than a comprehensive PM2.5 analysis, which could include finer regions, country variation, spatial coefficients, and further covariates.
3. Fake data can be almost as valuable as real data for building your model
The paper uses simulations from the prior predictive distribution to assess whether priors and their implied data-generating processes are consistent with domain knowledge before fitting models.
- The exploratory analysis produced one linear regression model and two linear multilevel models requiring priors for all parameters.
- Prior predictive simulations assess whether the chosen priors generate data consistent with domain knowledge.This approach evaluates the joint effect of parameter priors and the likelihood rather than inspecting parameters one at a time.
- Vague priors generated completely impossible PM2.5 data because they did not respect contextual knowledge.
- Priors centered on intercept 0 and slope 1 produced more plausible, though still miscalibrated, satellite-model data.
- Repeated simulations from the tighter priors still produced data points exceeding 22,000µgm−3, which remained very high in context.
- The prior predictive distribution also supplies the marginal likelihood at the observed data, but it is not useful for selecting between models.
4. Graphical Markov chain Monte Carlo diagnostics: moving beyond trace plots
The paper moves beyond trace plots by using visualization to diagnose Hamiltonian Monte Carlo trajectories and posterior geometry. Divergence patterns distinguish false positives from localized high-curvature regions that obstruct exploration.
- Traditional trace plots and autocorrelation functions can help investigate numerical problems but are not always needed when chains mix well.
- HMC diagnostics exploit the geometry of the typical set, whose smoothness can be checked numerically through trajectory behavior.
- The divergence heuristic can falsely flag stable trajectories, so visual comparisons of divergent and non-divergent samples help identify false positives.
- A concentration of divergences in small parameter-space neighborhoods indicates posterior high curvature that obstructs exploration.
- Bivariate scatterplots marking divergent transitions and parallel coordinate plots diagnose troublesome parameter-space regions and differentiate problematic models.
- Posterior predictive density overlays compare the observed dataset with 100 simulated datasets drawn from the posterior predictive distribution.
5. How did we do? Posterior predictive checks are vital for model evaluation
Posterior predictive checks compare replicated data with observed data to assess model fit, using both overall and group-specific statistics. In this example, hierarchical models generally outperform the single-level model, but their predictive distributions remain miscalibrated.
- Posterior predictive checking simulates replicated datasets from the data-informed posterior predictive distribution and compares their features with observed data.The checks are mostly qualitative and can reveal that the model needs extension or modification.
- Multilevel Models 2 and 3 simulate log(PM2.5) values more similar to the observed distribution than Model 1.
- Model 3 best captures observed skewness, Model 2 performs adequately, and Model 1 fails completely.Test statistics should be chosen to be orthogonal to model parameters because posterior predictive checks use the data twice.
- The two hierarchical models fit region-level medians better than the non-hierarchical model.
- LOO-PIT plots show miscalibration in all models, although the hierarchical models improve on the single-level model.The plots compare computed LOO-PIT densities with simulations from a standard uniform distribution.
- The frown shapes for Models 2 and 3 indicate predictive distributions that are too broad, suggesting further modeling to capture uncertainty accurately.One proposed direction is subdividing super-regions to represent within-region variability.
6. Pointwise plots for predictive model comparison
Pointwise LOO diagnostics evaluate predictive performance and observation influence across models. They identify where models differ and flag observations that disproportionately affect posterior predictions.
- Visual posterior predictive checks help identify outliers and high-leverage observations that may indicate needed changes to the model.Possible changes include nonlinear regression or heavier-tailed observation error.
- LOO predictive distributions assess observations individually, while influence diagnostics examine how leaving each observation out changes predictive quantities.
- PSIS-LOO compares models pointwise using differences in estimated expected log predictive densities for individual observations.Model 3 appears slightly better than Model 2 in the pointwise comparison.
- Positive PSIS-LOO differences indicate that Model 3 outperformed Model 2, with results colored by WHO cluster.
- The 2674th observation, the only measurement from Mongolia, is flagged as influential under Model 2 by the k-hat diagnostic.Its log satellite and log(PM2.5) values are (1.95, 4.32), appearing extreme in the scatterplot.
7. Discussion
Visualization complements quantitative procedures throughout the Bayesian workflow, supporting model comparison, computational checking, prior setting, fit assessment, and iterative model improvement. Using data to guide model building raises overfitting and generalization concerns, which the authors address with stringent predictive checks and cautious model extensions.
- Visualization supports model comparison, fit assessment, computational checking, prior setting, and iterative model improvement.
- Using measured data to guide model building can raise concerns that the resulting model will generalize poorly to new datasets.
- Prior predictive checks should target plausible datasets broadly rather than replicate the observed data indistinguishably.
- Posterior predictive checks should examine influential measurements and motivate weakly informative extensions centered on the previous model.
- The authors argue that statisticians should understand this common workflow’s consequences and provide concrete suggestions for more robust applied statistics.