Source-linked AI summary
Using stacking to average Bayesian predictive distributions
Yuling Yao, Aki Vehtari, Daniel Simpson, Andrew Gelman
TL;DR
In M-open settings, BMA can be unreliable because the true data-generating process is not among the fitted candidates. The paper extends stacking to predictive distributions, using proper scoring rules and PSIS-LOO for efficient weight estimation. Simulations and applications motivate recommending predictive-distribution stacking, with BB-pseudo-BMA as a computational alternative.
Problem
Bayesian model averaging is flawed in M-open settings where the true data-generating process is not one of the candidate models.
Method
The paper combines posterior predictive distributions with stacking under proper scoring rules and uses PSIS-LOO to estimate the required leave-one-out predictive densities.
Results
The paper recommends stacking of predictive distributions; BB-pseudo-BMA is presented as an approximate alternative when computation cost is an issue.
Takeaways & Limitations
Stacking is recommended for combining separately fitted Bayesian posterior predictive distributions, while BB-pseudo-BMA offers a simpler approximate option.
Takeaways & Limitations
Separate-fit combination does not pool information across models and is less suitable when the candidate models share substantial structure.
Abstract
from arXiv · showhide
The widely recommended procedure of Bayesian model averaging is flawed in the M-open setting in which the true data-generating process is not one of the candidate models being fit. We take the idea of stacking from the point estimation literature and generalize to the combination of predictive distributions, extending the utility function to any proper scoring rule, using Pareto smoothed importance sampling to efficiently compute the required leave-one-out posterior distributions and regularization to get more stability. We compare stacking of predictive distributions to several alternatives: stacking of means, Bayesian model averaging (BMA), pseudo-BMA using AIC-type weighting, and a variant of pseudo-BMA that is stabilized using the Bayesian bootstrap. Based on simulations and real-data applications, we recommend stacking of predictive distributions, with BB-pseudo-BMA as an approximate alternative when computation cost is an issue.
1 Introduction
The paper addresses Bayesian model combination when the true data-generating process is absent from the candidate list, arguing that predictive accuracy is a better target than selecting one model. It extends stacking from point predictions to predictive distributions and develops efficient cross-validation-based weighting.
- Motivation: Model selection can be unstable and wasteful, whereas model averaging seeks an optimal combination of candidate models for future prediction.In Bayesian prediction, the target is a predictive distribution close to the true data-generating distribution.
- Motivation: M-open refers to settings where the true model is not in the candidate list and its predictive form cannot be explicitly specified.The paper distinguishes this from M-closed settings, where the true model is among the candidates.
- Bayesian model averaging: BMA is appropriate for M-closed problems but asymptotically selects the single candidate closest in KL divergence in M-open and M-complete settings.Its model weights depend on marginal likelihoods, which can be sensitive to the specified priors.
- Stacking: The paper generalizes stacking from point estimates to Bayesian posterior predictive distributions by optimizing proper scoring rules using leave-one-out predictions.This extends stacking beyond squared-error averaging of posterior means.
- Computation: PSIS-LOO provides a practically efficient way to estimate the leave-one-out predictive densities needed for log-score stacking without restricting the method to time series.Exact leave-one-out validation can otherwise require repeated model refits.
- Alternative weighting: The paper proposes Bayesian-bootstrap Pseudo-BMA+ weighting as a simpler alternative and reports that it improves on regular BMA and Pseudo-BMA but remains inferior to log-score stacking.Pseudo-BMA+ also supplies an initial guess for the stacking optimization.
2 Theory and methods
The theory defines stacking as selecting a convex combination of predictive distributions by a proper scoring rule and estimating its weights with leave-one-out predictive densities. The methods use PSIS to reduce computation, while diagnosing and correcting unstable approximations when necessary.
- Scoring rules: Stacking of predictive distributions extends stacking of means by optimizing a proper scoring rule over the whole predictive distribution rather than only its posterior mean.The energy score with β = 2 corresponds to stacking of means but is not strictly proper.
- Scoring rules: Proper scoring rules compare predictive distributions through an associated divergence; logarithmic scoring corresponds to KL divergence.The logarithmic score is the only proper local score under the stated regularity conditions.
- Predictive-distribution stacking: Stacking chooses a convex combination of candidate predictive distributions whose divergence from the true distribution is minimized, equivalently maximizing the selected scoring rule.The weights are nonnegative and sum to one.
- Weight estimation: Leave-one-out stacking evaluates each candidate through predictive densities based on data with observation i removed, then optimizes the resulting scores for the combination weights.The combined predictive density is formed from the weighted candidate predictive densities.
- Pareto smoothed importance sampling: PSIS approximates exact leave-one-out calculations from full-data posterior draws by smoothing the largest importance ratios with a generalized Pareto distribution.This avoids refitting each model n times, while retaining a diagnostic based on the estimated Pareto shape parameter.
- Pareto smoothed importance sampling: When the estimated Pareto shape parameter exceeds 0.7 for left-out observations, the paper recommends exact LOO or k-fold validation instead of the PSIS approximation.Small samples can also produce high-variance, unstable weighting.
- Pseudo-BMA: Pseudo-BMA uses LOO estimated expected log predictive densities for AIC-type weights, while Bayesian-bootstrap regularization accounts for uncertainty in the future-data proxy.The estimated elpd is computed from pointwise components and their standard errors.
3 Simulation examples
Across simulation examples, stacking of predictive distributions generally performs best in M-open settings, remains robust to redundant candidate models, and can combine uncertain approximations. Its main practical weakness is instability when samples are extremely small or PSIS estimates are unreliable.
- Gaussian mixture model: In the Gaussian mixture M-open example, stacking combines models while BMA selects the single candidate closest in KL divergence, with stacking's benefits emerging for large samples.Stacking of means can have a reasonable point estimate but an unappealing predictive distribution because it does not match distributional shape.
- Gaussian mixture model: Stacking of predictive distributions achieves nearly the same optimal mean squared error as stacking of means, while both outperform BMA.The comparison supports matching the predictive distribution rather than only its moments.
- Gaussian mixture model: Adding repeated N(4, 1) candidates worsens BMA and Pseudo-BMA weighting, whereas stacking is theoretically unchanged apart from numerical fluctuation.Stacking depends on the space expanded by the candidate models, while BMA effectively assigns extra prior mass to duplicated models.
- Linear subset regressions: In linear subset regressions, stacking outperforms the other combination methods across sample sizes and is asymptotically better than them, while Pseudo-BMA+ improves on naive Pseudo-BMA.BMA resembles Pseudo-BMA and remains better than model selection in the reported M-open setting.
- Linear subset regressions: For the M-closed nested-regression example, almost all methods recover the full model at large n, except BMA and marginal-likelihood model selection.The reported BMA weakness is attributed to its parameter priors when conditioning on particular parameter values.
- Limitations and diagnostics: Stacking is unstable for extremely small n when PSIS approximations are crude, and LOO weighting can become unstable when relative sample size is small relative to effective parameter count.The paper recommends exact LOO for problematic observations diagnosed by high Pareto shape estimates and leaves broader small-sample adjustment for future work.
- Comparison with mixture models: Mixture models and single-model selection perform worse than the proposed averaging methods, while stacking and Pseudo-BMA+ outperform all alternatives at moderate sample sizes.The mixture model takes about 30 more times to run than stacking or Pseudo-BMA+ and is too complex to fit when samples are small.
- Variational inference with different initial values: In the variational-inference example, stacking or Pseudo-BMA+ combines approximations from different initializations to average posterior distributions for a multimodal target.The example addresses initialization-dependent local optima in non-convex variational optimization.
4 Discussion
The discussion favors stacking predictive distributions for combining separately fit Bayesian models, while emphasizing regularization and the limits of separate model fitting.
- Regularization: The simplex constraint gives combined predictive distributions meaningful weights that are positive and sum to one.It is natural for predictive-density combination and is automatically satisfied by BMA and Pseudo-BMA weights.
- Limitations: Convex linear combinations are an assumption that may be inadequate when the true model distribution is a convolution rather than a mixture.The subset regression example illustrates a setting where going beyond convex combinations could provide gains.
- Regularization: Stronger priors on combination weights can improve stability when the model space is high-dimensional.The authors motivate this by arguing that a larger convex span can better approximate the true model.
- Limitations: Separate model fitting does not pool information across models, making the approach less suitable when the models share substantial structure.In such cases, the discussion favors a larger super-model containing the separate models as special cases.
- General recommendations: Stacking predictive distributions directly targets performance of the combined predictive distribution, unlike weighting based on marginal likelihood.The authors recommend stacking for separately fit Bayesian posterior predictive distributions, with Pseudo-BMA+ as a cheaper alternative.
Appendix A. Implementation in Stan and R
The appendix describes how to compute model-combination inputs in Stan and obtain stacking, Pseudo-BMA, and Pseudo-BMA+ weights in R.
- Stan implementation: Cross-validated log likelihoods for each observation and model can be generated in Stan and used for model combination.The resulting n×K matrix contains p(y_i|y_-i, M_k) values across observations and candidate models.
- Stan implementation: The Stan model declares regression data, coefficients, and noise parameters, then samples observations from a normal likelihood.The shown program places normal and gamma priors on beta and sigma and records pointwise log likelihoods.
- Stan and R workflow: Each regression model is fit separately in Stan using its corresponding covariate column, and posterior log likelihoods are extracted.The R example loops over K models, fits each model, and stores its extracted log_lik draws.
- R implementation: The loo package provides stacking, Pseudo-BMA, and Pseudo-BMA+ model-combination weights from the supplied likelihood calculations.Bayesian bootstrap regularization can additionally be applied to Pseudo-BMA when computation time is not a concern.
- Recommendation: The appendix reiterates the general recommendation to use stacking for combining separate Bayesian predictive distributions.This recommendation is presented as the preferred approach among the discussed weighting methods.