Overall Objective Priors

1.1 The problem

• A reference prior typically changes with the parameter of interest, producing different marginal reference posteriors for different parameters.For the multinomial model, each θ_i has a Beta reference posterior with parameters x_i + 1/2 and n − x_i + 1/2.Source: §1.1 · pp. 1–2
• One prior could be useful when analyses concern several parameters or functions, prediction, decisions, nonstandard quantities, computation, or communication with nonspecialists.Separate computations with a different reference prior for each parameter can be onerous.Source: §1.1 · p. 2
• The goal is a joint prior whose marginal posteriors for the quantities of interest are sensible from a reference-prior perspective, but the goal is not well-defined.The authors therefore explore several possible approaches.Source: §1.1 · p. 2

1.2 Background

• An informal study of 21 quantities of interest in the bivariate normal model judged πo(μ1, μ2, σ1, σ2, ρ) a good overall prior by frequentist coverage.The study assessed reasonable coverage for at least the most important quantities of interest.Source: §1.2 · p. 2

Invariance-based priors

• Right-Haar priors can yield sensible posteriors for selected parameters yet problematic posteriors for other means, standard deviations, or overall quantities.In the multi-normal means example, a constant right-Haar prior gives sensible individual-mean posteriors, but the posterior for an overall mean concentrates around an incorrect value.Source: Invariance-based priors · p. 3

Constant and vague proper priors

• Constant priors lack transformation invariance, often yield improper posteriors, and can perform terribly; vague proper priors inherit many flaws and may produce arbitrary posteriors.When a constant prior yields an improper posterior, a vague proper prior can give an essentially arbitrary posterior depending on its chosen vagueness; detecting this may require sensitivity analysis.Source: Constant and vague proper priors · pp. 3–4

Jeffreys-rule prior

• Jeffreys-rule priors have problems in multiparameter models, including wrong degrees of freedom for μ and σ and inferior posteriors for ρ and derived parameters.In a multinomial example with n = 3 and m = 1000, the prior effectively adds 1/2 to each of 998 empty cells, making them more important than cells with data; the authors know of no multivariable example where they would recommend it.Source: Jeffreys-rule prior · pp. 4–5

Prior averaging approach

• Geometric averaging is unaffected by normalizing constants and minimizes average KL divergence from the component priors, whereas arithmetic averaging of improper priors has arbitrary weights.In the bivariate normal example, πA is worse than either right-Haar prior, while πG is recommended as the overall objective prior.Source: Prior averaging approach · pp. 5–6
• Prior averaging is difficult to justify overall because reference priors depend on other parameters and their quality as overall priors is unclear.In the multinomial example, orderings produce m! full reference priors to average, often making computation prohibitive; the authors treat averaging as a way to generate candidates for further study.Source: Prior averaging approach · p. 6

Common reference prior

• When the reference prior is the same for every parameter of interest, the authors take that common prior as the natural overall choice.They catalogue settings where a common reference prior is known.Source: Common reference prior · p. 6

Reference distance approach

• The reference-distance approach seeks a prior whose marginal posteriors for each quantity of interest are close to their respective reference posteriors.For the multinomial model, the proposed choice a = 1/m adds only 1/m to each cell and yields sensible posterior results in the sparse-data example.Source: Reference distance approach · pp. 6–7

Hierarchical approach

• The hierarchical approach places a reference prior on a hyperparameter indexing a class of proper priors, using the induced prior as the overall prior.For the multinomial example, symmetric Dirichlet priors encode treating all cells similarly; the reference prior for a is proper because the marginal likelihood stays bounded away from zero as a → ∞. The authors prefer maximizing the reference posterior for a to maximizing the marginal likelihood, whose mode may not exist.Source: Hierarchical approach · pp. 7–8

1.4 Outline of the paper

• Section 2 catalogs common reference priors; Sections 3 and 4 apply the distance and hierarchical approaches to several models, and Section 5 presents conclusions.The applications include multinomial and normal models for the distance approach, and multinomial, hypergeometric, multinormal, and bivariate normal models for the hierarchical approach.Source: §1.4 · p. 8

• A common reference prior is unchanged across the specified parameters of interest and possible parameter orderings.Source: §2 · p. 8
• In general, a reference prior depends on the ordering used in its derivation.Source: §2 · p. 8
• When unique in this sense, the reference prior is typically an excellent overall prior.Source: §2 · p. 8

2.1 Structured diagonal Fisher information matrix

• Under the stated Fisher-information structure, the one-at-a-time reference prior is common across chosen parameters of interest and nuisance-parameter orderings.For a factorized density with decomposable observations, it is the product of the separate reference priors and is also the Jeffreys-rule prior.Source: §2.1 · pp. 8–9

Bivariate binomial distribution

• For the bivariate binomial model, the common prior is independent Beta(1/2, 1/2) distributions for θ1 and θ2, recommended as the overall prior.This result follows because the Fisher information has the stated structure (4).Source: Bivariate binomial distribution · p. 9

Multinomial distribution for directional data

• For directional multinomial data, independent Beta(1/2, 1/2) priors on the conditional probabilities ξj give the common one-at-a-time prior for any ordering of those parameters.This matches Berger and Bernardo’s reference prior for the specified cell ordering.Source: Multinomial distribution for… · pp. 9–10

A two-parameter exponential family

• In the two-parameter exponential family, the stated information structure gives a common one-at-a-time reference prior; the Inverse Gaussian case is an overall recommended prior.The family includes suitable forms of the normal, inverse normal, gamma, and inverse gamma distributions.Source: A two-parameter exponential family · pp. 10–11

A stress-strength model

• In the stress-strength model, the Jeffreys and one-at-a-time reference priors for θ and ψ coincide as 1/{θ(1 − θ)ψ}; it is also a second-order matching prior for θ when m/n → a > 0.Source: A stress-strength model · p. 12

2.2 Other scenarios with a common reference prior

• The section contrasts two scenarios with a common reference prior: one yields a fine overall prior and the other does not.Source: §2.2 · p. 12

The location-scale family

• For the location-scale family, the common one-at-a-time reference prior for either μ or σ is 1/σ and is recommended overall.It is also the right-Haar prior and is known to yield Bayes procedures with optimal frequentist properties.Source: The location-scale family · p. 12

Unnatural parameterizations

• In an unnatural bivariate-normal parameterization, the common reference prior is equivalent to a right-Haar prior argued to be poor overall, suggesting natural parameterizations matter.Source: Unnatural parameterizations · pp. 12–13

3 Reference distance approach

• The approach seeks one joint prior whose marginal posteriors are close, in an average sense, to the reference posteriors for the quantities of interest.The paper assumes a preferred reference prior has already been selected when reference priors are not unique.Source: §3 · p. 13

3.1 Exact solution

• When one prior makes every marginal posterior equal its corresponding reference posterior, the paper treats that exact match as a natural solution.The univariate normal model provides an exact solution: σ^-1 is the common reference prior for μ and σ.Source: §3.1 · p. 13

3.2 Reference distance solution

• When exact agreement is unavailable, the method selects from a candidate family the prior whose marginal posteriors are closest on average to the reference posteriors.The paper notes that the minimizing prior is defined assuming it exists and is unique.Source: §3 · p. 13, §3.2 · p. 14

Directed logarithmic divergence

• The discrepancy measure is directed Kullback-Leibler divergence, a nonnegative quantity invariant under one-to-one transformations and interpretable in nits.It measures the expected log-density ratio of a density over its approximation.Source: Directed logarithmic divergence · p. 14

Weighted logarithmic loss

• The best candidate minimizes weighted average expected logarithmic loss across the quantities of interest, with weights reflecting their relative importance.Uniform weights are a natural default, but the paper allows other choices.Source: Weighted logarithmic loss · p. 14

Multinomial model

• For m=10 and 1000, the minima are 0.08 and 0.0008; across sample sizes, the optimum is about 0.8/m, yielding approximate marginal reference posteriors.The paper recommends a global Dirichlet prior with parameters approximately 0.8 × {1/m, ..., 1/m}.Source: Figure 1 · p. 16, Multinomial model · p. 17

The normal model with coefficient of variation

• For n=10 and k=3, total loss is minimized at a*=1.016; φ contributes most near the optimum, and the usual normal-model objective prior is recommended pragmatically.The individual μ and σ losses each have a unique minimum at a=1.Source: The normal model with coefficient of… · pp. 18–19

4 Hierarchical approach with hyperpriors

• A natural family of proper priors indexed by a can be turned into an overall prior by finding the reference prior for the marginal likelihood p(x | a).This hierarchical specification is equivalent to using the resulting prior as the overall prior in the original problem.Source: §4 · p. 20

The hierarchical prior

• For large sparse contingency tables, πR(a | m, n) is proper and its approximation is reported to be good, with φ(a) = a/(a+(n/m)) having a Be(φ | 1/2, 1) distribution.The exact prior behaves as O(a^-1/2) near zero and O(a^-2) for large a; Figure 4 compares the exact prior and approximation for two (m, n) settings.Source: The hierarchical prior · pp. 21–22, Figure 4 · p. 22

Computation with the hierarchical reference prior

• Under the approximate reference prior, the marginal posterior of a is log-concave when at least three cells are nonempty, allowing adaptive rejection sampling.The paper also describes Metropolis-Hastings sampling for the exact hierarchical reference prior.Source: Computation with the hierarchical… · pp. 22–23

Posterior behavior as m →∞

• As m grows, the transformed reference posterior for v = ma depends on the data only through r0, the number of nonzero cell observations.The empirical-Bayes posterior mode of a equals 1/m only when r0 = log n; the reference-distance choice is pre-experimental.Source: Posterior behavior as m →∞ · pp. 25–26

4.2 Multivariate hypergeometric model

• For the multivariate hypergeometric model with a multinomial hierarchy, the problem reduces to the multinomial case and gives an approximate Dirichlet prior with all parameters 1/k.Source: §4.2 · p. 27

4.3 Multi-normal means

• For multi-normal means, the hierarchical prior is a shrinkage prior, and the paper finds it satisfactory for the individual means and |µ|².The paper also says the single reference prior for |µ| is satisfactory for all quantities of interest, so averaging among reference priors would not work in this example.Source: §4.3 · p. 28

4.4 Bivariate normal problem

• In the bivariate normal example, averaging the right-Haar priors hierarchically produces a prior reported inferior to either individual prior, while the empirical hierarchical approach performs much worse.The authors conclude that the hierarchical attempt based on this class of improper priors failed.Source: §4.4 · p. 29

5 Discussion

• The reference-distance approach depends on parameterization, accepted reference priors for each parameter, weights, and the chosen candidate-prior class.For improper reference priors, the approach may be implemented on expanding compact sets if the result is insensitive to their choice; minimizing over all priors would be formidable.Source: §5 · pp. 29–30
• Hierarchical priors are recommended when a natural hierarchy over proper priors exists, but the bivariate normal attempt using right-Haar priors was highly unsatisfactory.The discussion warns that hierarchical or empirical-Bayes approaches should start from proper priors.Source: §5 · p. 30
• Arithmetic averaging of right-Haar priors was inferior in the bivariate normal example, whereas their geometric average seemed reasonable in that example.Source: §5 · p. 30
• Common reference and reference-distance approaches coincide when a common reference prior exists, while reference-distance and hierarchical approaches will rarely agree.The latter differ because reference distance fixes the hyperparameter, whereas the hierarchical approach assigns it a reference prior.Source: §5 · p. 30
• Common reference and hierarchical approaches are recommended when applicable, while reference distance is useful when neither is available, as in the coefficient-of-variation example.Source: §5 · p. 30