Source-linked AI summary
The prior can generally only be understood in the context of the likelihood
Andrew Gelman, Daniel Simpson, Michael Betancourt
TL;DR
The paper examines the tension between treating priors as pre-data information and choosing them with reference to the likelihood. It resolves this tension by evaluating priors across inference, prediction, and model evaluation, emphasizing generative and predictive checks. The paper argues that prior appropriateness depends on the likelihood, data, and posterior predictive behavior.
Problem
Prior modeling faces a tension because priors should encode pre-data information independently of the data model, while common methods are often motivated by a likelihood.
Method
The paper evaluates priors through generative reasoning, prior predictive distributions, posterior predictive validation, and robustness analysis within the full Bayesian workflow.
Results
The paper argues that a prior can generally only be interpreted in the context of its paired likelihood, with posterior predictive properties indicating its appropriateness for a problem.
Takeaways & Limitations
Default and subject-matter-specific priors should be developed and assessed in relation to the likelihood and the predictive behavior of the resulting analysis.
Takeaways & Limitations
The general shape of appropriate prior neighborhoods remains an open question because relevant analyses often require marginalizing parameters tractable only in simple cases.
Abstract
from arXiv · showhide
A key sticking point of Bayesian analysis is the choice of prior distribution, and there is a vast literature on potential defaults including uniform priors, Jeffreys' priors, reference priors, maximum entropy priors, and weakly informative priors. These methods, however, often manifest a key conceptual tension in prior modeling: a model encoding true prior information should be chosen without reference to the model of the measurement process, but almost all common prior modeling techniques are implicitly motivated by a reference likelihood. In this paper we resolve this apparent paradox by placing the choice of prior into the context of the entire Bayesian analysis, from inference to prediction to model evaluation.
1. The role of the prior distribution in a Bayesian analysis
Priors can encode substantive information, stabilize inference, or express uncertainty, but their significance depends on how they interact with the likelihood and observed data. The paper resolves the resulting conceptual tension by evaluating priors through inference, prediction, and model evaluation.
- A prior may encode problem-relevant information, stabilize high-dimensional inference, or express uncertainty through Bayes’ theorem.
- The prior’s influence can only be judged in the context of the likelihood and the observed data.
- The paper addresses the paradox that priors should precede the data model but are often chosen with reference to a likelihood.
- The proposed resolution combines robustness to approximate models with attention to the prior’s differing roles across Bayesian analyses.
- Common prior strategies range from likelihood-independent maximalist choices to likelihood-informed minimalist, reference, structural, regularizing, and weakly informative approaches.
- Prior and posterior predictive distributions assess potential measurements before data collection and validate models against collected data, supporting robust prior choice.
2. A simple motivating example
A sex-ratio example shows how a seemingly weakly supported comparison can become strongly favored under a uniform prior, while informative or weakly informative priors pull inference toward smaller differences. The example illustrates that prior effects depend on the likelihood and the scale of plausible effects.
- The motivating analysis compared girl-birth proportions for the most attractive parents with categories 1–4, finding an 8-percentage-point difference with t-statistic 2.44.
- With a uniform prior on δ, the posterior implied a 99.2% chance that beautiful parents had more girls and a 50% chance that the difference exceeded 8 percentage points.
- Changing the uniform prior’s parameterization produced essentially no posterior change because the estimated probabilities were far from 0 and 1.
- A uniform prior contradicted the observed stability of human sex ratios, which remain near 48.5% across time and populations.
- A weakly informative prior with standard deviation 0.005 yielded a posterior mean difference of 0.2 percentage points and standard deviation 0.5 percentage points.
- The example’s central lesson is that a uniform prior is not inherently bad; its interaction with the likelihood and data can produce poor performance when the true difference is small.
3. When exactly is the prior irrelevant in practice?
Prior irrelevance is not guaranteed by large samples: it depends on the likelihood, signal size, identifiability, and whether asymptotic assumptions apply. In complex or poorly identified models, prior choices can continue to affect posterior inference.
- When priors become irrelevant: Prior influence depends on likelihood structure and effect size, with more complex likelihoods and smaller effects requiring more data to make the prior irrelevant.The authors describe this as a general issue in contemporary statistical practice.
- Uniform priors and small signals: Uniform priors can produce nonsensical inferences even with seemingly large samples when the true effect is small.The likelihood’s random fluctuation can overwhelm a small signal, so the prior’s interaction with the data matters.
- Limits of asymptotic reasoning: Asymptotic arguments may fail in real applications, so reference, maximum entropy, and matching priors can perform poorly when their assumptions do not hold.The paper recommends using asymptotics to identify when priors may perform poorly rather than to design default prior families.
- Poor identifiability: In models whose parameter count grows with sample size, posterior concentration may remain along a submanifold, preserving the prior’s influence asymptotically.The prior’s projection along that submanifold can continue affecting the posterior even as more data arrive.
- Gaussian-process example: For Gaussian processes under infill asymptotics, σ²√κ is consistently estimable, whereas κ and σ individually are not.The data cannot distinguish a long-range, high-variance process from a short-range, low-variance process.
- Gaussian-process example: A prior on the nonidentified range parameter κ affects the posterior for σ, while ridge-aligned coordinates can yield more meaningful prior specification.A light right tail on κ can leave the posterior for σ with almost no support near small variances.
- Complex models: Complex models can assign unexpected weight to prior information, making weak subject-matter understanding useful along nonidentified parameter submanifolds.The paper notes that general techniques for locating such problem areas in complex hierarchical models are unavailable.
4. A prior is more than just a probability measure, so we need to start thinking generatively
A probability measure becomes a meaningful prior only through its interaction with a likelihood and the data-generating processes it induces. This generative perspective exposes pathologies in complex models, motivates weakly informative priors, and shows that prior sensitivity depends on the analysis goal.
- Complex models can contain hidden pathologies that are difficult to explore mathematically, limiting the practical usefulness of mathematically clean default priors.The paper specifically questions the practical value of reference and maximum entropy priors in such settings.
- Weakly informative priors disfavor parameter regions corresponding to data-generating processes that would look strange, explaining their practical usefulness.The paper gives the half-t prior on a logistic-regression standard deviation as an example.
- A probability measure is a prior only when paired with a likelihood and judged by the data-generating processes it favors or disfavors.The same probability distribution can be appropriate in one problem but not another because its meaning depends on the measurement context.
- In high dimensions, independent standard Gaussian priors can induce posterior concentration near a unit sphere around the least-squares estimate.With one observation and orthonormal predictors, concentration-of-measure results produce this behavior exponentially in the number of coefficients.
- Prior assessment must consider joint behavior, because priors can create unintended structure that is invisible when parameters are examined individually.The paper warns that subtle joint effects become more common and important as models grow more complex and high-dimensional.
- For predictive performance, posterior-predictive utility is more robust than prior-predictive utility because it averages over the data-informed posterior.This robustness is conditional: it depends on the likelihood washing out prior details.
- Posterior-predictive model selection is relatively stable under weakly informative priors, whereas marginal likelihoods can remain highly sensitive to prior details.Changing a unit-scale parameter’s prior standard deviation from 100 to 1000 can reduce the marginal likelihood by about a factor of 10 without appreciably changing posterior inferences.
5. Generative priors need to be prediction focused
Priors should be evaluated not only for generative plausibility but also for their posterior predictive performance. The paper connects this evaluation to pre-data overfitting checks, shrinkage toward simpler submodels, and practical tail choices.
- Prior predictive distributions assess whether draws generate data consistent with substantive understanding, whereas posterior predictive distributions assess prediction of new data from similar experiments.The paper treats generative adequacy and predictive performance as distinct aims that can compete.
- Overfitting: A priori overfitting can be assessed by measuring how often prior draws place parameters outside a small neighborhood of the simpler model Θ0.Posterior predictive checks can then assess actual overfitting after data are observed.
- Shrinkage: Priors concentrating mass near simpler submodels can improve estimation, as illustrated by Stein shrinkage for multivariate normal means when N ≥3.Stein’s estimator improves on the usual estimator by shrinking sufficiently near-zero observations toward zero.
- Open problems: A general analogue of Stein’s neighborhood is unresolved because the appropriate shape around Θ0 is problem-dependent and marginalization is tractable only in simple cases.Localized Kullback–Leibler approaches provide a heuristic for some parameterizations but require extension to more general cases.
- Practical guidance: Generative and predictive goals can conflict, so priors optimized for predictive properties must still be checked for generative adequacy.For scale parameters, the paper discusses competing recommendations ranging from finite density at zero to heavy tails.
- Practical guidance: A unit-scale half-Cauchy proposes a standard deviation above 50 about 1.2% of the time, which the paper argues is usually not generative for reasonably scaled models.The paper links this behavior to numerical pathologies and recommends lighter-tailed alternatives when scaling confidence permits.
6. Discussion
The discussion argues that priors are interpretable only together with their likelihood and should incorporate real subject-matter information. Prior predictive and posterior predictive checks provide the framework for guiding default and application-specific prior construction.
- The literature’s tension is that priors conceptually precede the data model, while common defaults such as Jeffreys’ prior explicitly depend on the likelihood.Other constructions also depend on individual data points or face difficulties with unbounded parameters.
- Improper priors can undermine coherent Bayesian inference, and natural symmetry principles may force improper choices that are inappropriate for particular parameter spaces.The paper notes that these concerns may matter less when data are strongly informative and prior information is sparse.
- Recent theoretical, computational, and applied work converges on including real, subject-matter-specific prior information to obtain more stable and accurate inferences.This creates a responsibility to develop useful default priors or procedures for building them.
- A prior can generally be interpreted only in the context of its paired likelihood, with prediction and posterior predictive properties providing the appropriate evaluation context.The conclusion frames this observation as guidance for methodologists and practitioners developing default and subject-matter-specific priors.