PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian inference and the parametric bootstrap

Bradley Efron

arXiv:1301.2936v1stat.AP

TL;DR

Efficient posterior computation and frequentist assessment of Bayesian estimates are the motivating questions. The paper reweights parametric-bootstrap samples for posterior inference; in the student-score example, the BCa density nearly agrees with the Jeffreys posterior.

  • Problem

    The paper addresses efficient posterior computation and how Bayesian estimates vary across future data sets.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The method reweights raw parametric-bootstrap replications for posterior computation, with a simplified conversion in exponential families under Jeffreys prior.

    Source: §1 · p. 2, §3 · p. 9

  • Results

    In the student-score example, the BCa density nearly agrees with the Jeffreys posterior, with a 95% central interval θ0 ∈(0.074,0.748).

    Source: §2 · p. 6

  • Takeaways & Limitations

    Bootstrap reweighting provides a way to assess both computational accuracy and the frequentist variability of Bayesian estimates.

    Source: §1 · p. 2

    Jeffreys priors can give inaccurate inferences for individual parameters in multiparameter settings, where specifically tailored priors may be needed.

    Source: §4 · p. 14

Abstract

from arXiv · show

The parametric bootstrap can be used for the efficient computation of Bayes posterior distributions. Importance sampling formulas take on an easy form relating to the deviance in exponential families and are particularly simple starting from Jeffreys invariant prior. Because of the i.i.d. nature of bootstrap sampling, familiar formulas describe the computational accuracy of the Bayes estimates. Besides computational methods, the theory provides a connection between Bayesian and frequentist analysis. Efficient algorithms for the frequentist accuracy of Bayesian inferences are developed and demonstrated in a model selection example.

1. Introduction.

The introduction presents parametric-bootstrap reweighting as an efficient, straightforward route to Bayesian posterior computation in the cases where it applies, and as a link between Bayesian and frequentist analysis. It previews extensions to exponential families and accuracy assessment of Bayesian estimates.

1. Introduction.

  • The authors position parametric bootstrapping as a way to link, rather than separate, frequentist and Bayesian practices.The introduction presents Jeffreys priors as examples for convenient presentation, not as a requirement for bootstrap reweighting.Source: §1 · pp. 1–2
  • Parametric bootstrap reweighting computes posterior distributions and estimates efficiently and straightforwardly in the comparatively limited cases where bootstrap methods apply.The approach reweights raw bootstrap replications rather than directly generating correctly distributed posterior draws as in Markov chain methods.Source: §1 · p. 2
  • In exponential families, the Bayes/bootstrap conversion can be explicitly characterized, and multivariate normal and generalized linear models are investigated.The paper notes that Jeffreys-type priors can yield unsatisfactory results for multiparameter problems [Ghosh (2011)].Source: §1 · p. 2
  • The paper develops internal estimates of the bootstrap replications needed and external estimates of how Bayesian results vary across future data sets.The external accuracy analysis concerns the frequentist behavior of Bayesian estimates.Source: §1 · p. 2

2. Conversion and reweighting.

The section illustrates posterior computation by reweighting parametric-bootstrap replications, first for a student-score correlation and then in the exponential-family framework. In the correlation example, the BCa density nearly agrees with the Jeffreys posterior, though that agreement does not generalize reliably to multidimensional settings.

2. Conversion and reweighting.

  • For the student-score example, 10,000 parametric-bootstrap samples produce correlation replications compared with Fisher’s theoretical bootstrap density.Figure 1 marks exact 95% confidence limits θ ∈(0.093,0.741).Source: §2 · p. 3, Figure 1 · p. 4
  • Reweighting raw bootstrap replications by wi = πiRi yields an estimated posterior distribution, with posterior expectations estimated by sample averages.The replications are i.i.d. draws from the parametric bootstrap density, so the law of large numbers gives convergence under mild regularity conditions.Source: §2 · p. 5
  • In the student-score example, the Jeffreys prior is described as yielding accurate frequentist coverage probabilities.The prior belongs to the Welch–Peers family discussed later in the paper.Source: §2 · p. 6
  • The BCa density nearly agrees with the Jeffreys posterior for the student-score correlation, with a 95% central interval θ0 ∈(0.074,0.748).The BCa density is slightly heavier in the left tail; the authors caution that this agreement is misleadingly soothing in light of the multidimensional context.Source: §2 · p. 6
  • In exponential families, the Bayes/bootstrap conversion simplifies; with Jeffreys prior, r(β) = exp(∆(β)), and the cubic ∆(β) makes tail reweighting more extreme.The section presents this simplification as facilitating application to multiparameter problems.Source: §3 · p. 6

3. Exponential families.

In exponential families, bootstrap replications can be reweighted to approximate Bayesian posterior distributions, with Jeffreys prior giving weights tied to the deviance difference. The section relates the accuracy and behavior of these estimates to density approximations, skewness, and the chosen family.

3. Exponential families.

  • The density approximation typically errs by a factor of only 1+O(1/n), and can perform better for discrete families such as the Poisson.It applies the central limit theorem at the center, where it is most accurate.Source: §3 · p. 8
  • For a normal translation model, Δ(β)=0, so the Bayes estimate equals the unweighted bootstrap estimate.In other cases, the difference between the estimates relates to the variability of Δ(β).Source: §3 · p. 9
  • With Jeffreys prior, the reweighting factor is r(β)=exp(Δ(β)), yielding a deviance-based adjustment to bootstrap replications.The section describes the Jeffreys-prior form as a particularly simple conversion.Source: §3 · pp. 8–9
  • The paper's Jeffreys-prior examples show substantial agreement between Bayes and unweighted bootstrap results, with multiparameter examples yielding small RBD values.Source: §3 · pp. 9–10
  • In repeated sampling, skewness decreases as n^-1/2, and the Bayes and bootstrap distributions converge at that rate.The normal translation family has zero skewness, Δ(β)=0, and R(β)=1.Source: §3 · p. 10
  • In the illustrated case, cubic Δ(β) makes exp(Δ_i) reweighting more extreme in the distribution tails than near β̂.Source: §3 · p. 10
  • Weights proportional to π(β_i)R(β_i) make the bootstrap distribution approximate the posterior distribution π(β|β̂).The conversion is invariant to whether the bootstrap is performed on the canonical parameter or another sufficient vector.Source: §3 · pp. 7, 10
  • For posterior prediction, weighting simulated future-data samples by w_i approximates the posterior predictive distribution g(y).For each β_i, the procedure samples a future-data replicate from g_{β_i}(·).Source: §3 · p. 11

4. The multivariate normal family.

For multivariate normal models, bootstrap replications can be reweighted to estimate Bayes posterior distributions. The eigenratio example contrasts Jeffreys-prior posterior limits with BCa confidence limits, while the section discusses limitations of general-purpose objective priors in multiparameter settings.

Bayes/bootstrap conversion

  • For multivariate normal samples, the theorem characterizes Jeffreys prior and Bayes weights through the deviance difference, with the expression valid regardless of coordinates.The theorem gives the quantities for coordinates consisting of the mean and covariance parameters.Source: §4 · pp. 11–12
  • Parametric bootstrap datasets are generated from the fitted model, and reweighted replicates estimate the posterior distribution and its expectations, percentiles, and credible limits.The posterior distribution is supported on the bootstrap replicate points.Source: §4 · p. 12

Multivariate normal example

  • (0.650, 0.908) are the Bayes 95% credible limits, while the BCa 95% interval (0.598, 0.890) is shifted far leftward.Figure 3 also shows the unweighted bootstrap density.Source: Figure 3 · p. 13, §4 · p. 13
  • BCa weights increase sharply toward smaller eigenratio replications, whereas Jeffreys-prior Bayes weights are nearly flat, leaving the posterior close to the unweighted bootstrap density.The BCa weighting pulls its weighted percentile points downward.Source: §4 · p. 14

Limitations and prior choice

  • Jeffreys priors can give inaccurate inferences for individual parameters in multiparameter settings, and the eigenratio example motivates parameter-specific priors.A standard inverse Wishart prior with a flat prior on the mean gave essentially the same eigenratio results as Figure 3.Source: §4 · p. 14
  • The BCa weights can be viewed as prior weights that make posterior credible limits equal the BCa limits, providing an automatic construction of probability-matching priors.The paper describes this as a way of tailoring the prior.Source: §4 · p. 15

Computational accuracy

  • Occasional large importance weights can make computations excessively variable; internal accuracy estimates warn of problems, and broader bootstrap sampling can reduce weight instability.The paper also mentions truncating the largest weight and expanding the sampling covariance in directions where weights increase.Source: §4 · p. 15

5. Generalized linear models.

The section applies Jeffreys-prior bootstrap reweighting to Poisson polynomial models for prostate-cancer gene z-values, using Fdr(3) to illustrate Bayesian inference. The results show that Fdr(3) depends strongly on model choice, while data-based model selection is unstable.

5. Generalized linear models.

  • The M4 fit is selected by AIC, with deviance dropping dramatically from M3 to M4 and changing only slowly for higher-degree models.M2 corresponds to a normal location-scale model for the marginal z-value density; higher-degree models are more flexible.Source: §5 · pp. 17–18
  • 32% of M8 bootstrap samples selected M4 by AIC, versus 51% selecting M8; weighting changed these proportions to 36% versus 45%.The weighted results still favor M8.Source: Table 2 · p. 18, §5 · p. 20
  • For M4, B = 4000 bootstrap samples yield β_i; from a Jeffreys prior, the posterior expectation of any θ = t(β) is estimated by Pwiti/Pwi, with wi = e^∆i.Source: §5 · p. 18
  • The M4-based Jeffreys posterior density and BCa confidence density for Fdr(3) are nearly the same, and both lie slightly left of the unweighted bootstrap density.Figure 6 presents the posterior densities based on B = 4000 parametric bootstrap replications.Source: §5 · p. 19, Figure 6 · p. 19
  • M8's 95% credible limits for Fdr(3) are (0.141, 0.239), versus M4's (0.154, 0.241), and its estimate is sharply lower.Figure 7 compares the bootstrap histograms from M8 and M4.Source: Figure 7 · p. 20, §5 · p. 20
  • Data-based model selection is unstable in this example, as confirmed by the section's accuracy calculations.Source: §5 · p. 20

6. Accuracy.

The section develops internal and external accuracy calculations for bootstrap-reweighted Bayesian estimates, using the i.i.d. sampling structure to avoid repeated computation. Its example shows substantial frequentist uncertainty in data-based model-selection probabilities.

6. Accuracy.

  • The i.i.d. nature of bootstrap sampling makes both computational error and future-sample variation of Bayesian estimates straightforward to assess.The section distinguishes internal bootstrap sampling error from external statistical sampling error.Source: §6 · p. 21
  • For an eigenratio, the Jeffreys Bayes estimate is 0.799, nearly the MLE 0.793, and its bootstrap coefficient of variation is 0.002.The calculation indicates that the estimate nearly equals the exact Bayes estimate, so B = 10,000 was excessive.Source: §6 · p. 21
  • Bootstrap-after-bootstrap estimates frequentist standard error by resampling from the fitted distribution, then reweighting the original replications to evaluate each resample's estimate.The reweighting requires no new evaluations of t(β), π(β), or R(β), and its factor is easily calculated in exponential families.Source: §6 · pp. 21–22
  • A bootstrap-after-bootstrap standard error of 0.20 for the Bayes probability 0.36 of selecting M4 indicates highly uncertain data-based model selection.The section interprets this uncertainty from a frequentist viewpoint.Source: §6 · p. 22
  • Jackknife estimates lie closer to the original estimate, making their reweighting factors closer to 1 and reducing strain on importance sampling.Source: §6 · pp. 22–23

7. Summary.

The parametric bootstrap provides a straightforward importance-sampling route to Bayesian posterior computation, with especially simple weights under Jeffreys invariant prior. The paper also develops computational-accuracy checks and a second-level bootstrap for assessing the frequentist accuracy of Bayesian inferences.

Posterior computation

  • Bootstrap replications are reweighted for posterior computation; in exponential families, Jeffreys-prior weights depend only on the deviance difference ∆(β).The parametric bootstrap is the starting distribution, rather than direct posterior draws as in MCMC.Source: §7 · p. 23

Prior behavior

  • Jeffreys-prior posteriors were not much different from the unweighted bootstrap distribution, which may be unsatisfactory for a single parameter in multiparameter families.Welch–Peers and reference priors are closely related to the frequentist BCa reweighting formula.Source: §7 · p. 23

Deviance behavior

  • The deviance difference has an asymptotic cubic normal form that depends on the family’s skewness.Source: §7 · p. 23

Computational accuracy

  • I.i.d. bootstrap resampling yields formulas for posterior-computation accuracy as a function of B, and the examples’ computation time was measured in seconds.Because importance sampling can be unstable, the paper urges internal accuracy calculations.Source: §7 · p. 23

Frequentist accuracy

  • A second-level bootstrap algorithm estimates the frequentist accuracy of Bayesian inferences, supporting assessment of inferences based on formulaic priors such as Jeffreys.Source: §7 · p. 23

APPENDIX

The appendix characterizes the asymptotic deviance difference and gives computational details for BCa acceleration and examples. It also reports a particularly large bias correction for the eigenratio.

Transformation coordinates

  • Jeffreys prior’s transformation invariance makes the transformed-coordinate expression identical to the corresponding original-coordinate expression.Source: APPENDIX · p. 24

Proof details

  • Under repeated sampling with β̂ fixed and β ∼f β̂(·), Z → N(0,1); skewness γ̂ is O(n^-1/2), so Δ is O_p(n^-1/2), and its first missing Taylor term, δ̂Z^4/12, is O_p(n^-1).Source: APPENDIX · p. 24

BCa computation

  • The abc2 program computes the BCa acceleration constant quickly and accurately but requires family-specific programming; accel works directly from bootstrap replications.Source: APPENDIX · p. 26

BCa examples

Loading 1301.2936v1…