Robust Bayesian inference via coarsening

Jeffrey W. Miller; David B. Dunson

Source-linked AI summary

Robust Bayesian inference via coarsening

Jeffrey W. Miller, David B. Dunson

arXiv:1506.06101v1Jun 19, 2015stat.ME

TL;DR

Bayesian inference can be highly sensitive to small model misspecifications, especially with large datasets and flexible models. The paper introduces a coarsened posterior that conditions on empirical-distribution neighborhoods; with relative entropy, it becomes a tempered likelihood and yields robust behavior in the paper’s examples.

Problem
Small departures from an assumed model can distort Bayesian inference, including uncertainty, component interpretation, and variable-selection conclusions.
Method
The coarsened posterior conditions on a neighborhood of the empirical distribution, with relative-entropy neighborhoods approximated by raising the likelihood to a fractional power.
Results
In a skew-normal mixture, the coarsened posterior stabilizes on the true two-component interpretation instead of favoring increasingly many components as n increases.
Takeaways & Limitations
The approach can provide computationally tractable Bayesian inference that is more tolerant of small departures while retaining interpretable model structure.
Takeaways & Limitations
Coarsened posteriors may be less concentrated than desired, and the paper identifies further work on small-sample corrections, alternative distances, and inferring α.

Abstract

from arXiv · show

The standard approach to Bayesian inference is based on the assumption that the distribution of the data belongs to the chosen model class. However, even a small violation of this assumption can have a large impact on the outcome of a Bayesian procedure. We introduce a simple, coherent approach to Bayesian inference that improves robustness to perturbations from the model: rather than condition on the data exactly, one conditions on a neighborhood of the empirical distribution. When using neighborhoods based on relative entropy estimates, the resulting "coarsened" posterior can be approximated by simply tempering the likelihood---that is, by raising it to a fractional power---thus, inference is often easily implemented with standard methods, and one can even obtain analytical solutions when using conjugate priors. Some theoretical properties are derived, and we illustrate the approach with real and simulated data, using mixture models, autoregressive models of unknown order, and variable selection in linear regression.

1. Introduction

The paper addresses Bayesian inference under model misspecification by conditioning on neighborhoods of empirical distributions rather than exact data. Its coarsened posterior is designed to preserve interpretability and computational tractability while improving robustness to small departures from the model.

Motivation: Small model violations can strongly affect Bayesian likelihoods and become more disruptive as sample sizes increase.Standard posteriors may underestimate uncertainty and concentrate on parameters or models that do not resemble the observed distribution.
Motivation: Under misspecification, model averaging can favor increasingly complex models even when a simpler model describes the data adequately for practical purposes.This can obscure interpretable components or variables as the sample size grows.
Illustration: In the skew-normal mixture example, the standard posterior favors more components as n increases, whereas the coarsened posterior stabilizes on a macroscopic two-group interpretation.The standard posterior produces many small components that obscure the two large groups corresponding to the true components.
Approach: The coarsened posterior conditions on the empirical distribution being close to model-generated data under a chosen statistical distance.This changes the conditioning event while retaining the same underlying model.
Approach: Coarsened posteriors inherit continuity from the chosen distance, so small changes in the data distribution produce small changes in inference.The approach is presented as valid Bayesian inference based on limited information.
Computation: Relative-entropy neighborhoods yield a fractional-power likelihood approximation that can use standard algorithms and sometimes analytical conjugate-prior calculations.Tempering the likelihood may also improve MCMC mixing time and permits robustified marginal-likelihood expressions in exponential families.
Limitation: A main disadvantage is that coarsened posteriors can be less concentrated when the actual misspecification is smaller than expected.The paper also notes a connection between Gibbs posteriors and approximate Bayesian computation.

2. Toy example: Bernoulli trials

The Bernoulli example replaces exact conditioning with conditioning on a relative-entropy neighborhood whose size reflects expected corruption. The resulting c-posterior remains robust to small perturbations while responding to larger departures and closely matches its approximation.

Setup: The toy problem tests H0: θ = 1/2 against H1: θ ≠ 1/2 when Bernoulli observations may not follow the assumed model exactly.The setup contrasts standard Bayesian conditioning with inference under possible corruption.
Construction: The c-posterior conditions on a relative-entropy neighborhood between the observed and hypothetical model-generated empirical distributions.A prior R ∼ Exp(α) represents rough uncertainty about the amount of corruption.
Calibration: Choosing α = 1/(2ε^2) corresponds approximately to tolerating a sample-mean shift of ε when θ = 1/2.This uses the chi-squared approximation to relative entropy near 1/2.
Approximation: The approximate c-posterior differs from the standard posterior by replacing n with α_n = 1/(1/n + 1/α).This acts as an effective sample-size adjustment in the Bernoulli expression.
Results: For Bernoulli(0.51) data, the standard posterior probability of H0 eventually goes to 0, while the c-posterior remains high as n increases.The approximate and exact c-posteriors are indistinguishable in the reported plots.
Results: For Bernoulli(0.56) data, the c-posterior behaves more like the standard posterior and favors H1 when n is sufficiently large.The method therefore does not treat departures beyond the chosen tolerance as compatible with H0 indefinitely.
Limitation: Unlike the standard posterior, the c-posterior does not concentrate as n →∞ because uncertainty about corruption remains.This nonconcentration is presented as appropriate when the observed data may be corrupted.

3. Method

The c-posterior replaces exact conditioning on potentially corrupted data with conditioning on a neighborhood of the empirical distribution. With relative entropy neighborhoods, it admits a likelihood-tempering approximation that supports efficient inference and robustness to small distributional changes.

Core construction: The approach is motivated by model misspecification, where exact conditioning can produce non-robust inference and underestimated uncertainty.As datasets grow, even slight model deficiencies can become disruptive.
Core construction: The c-posterior conditions on the event that idealized and observed empirical distributions are close under a chosen statistical distance.A prior can also be placed on the neighborhood radius when it is not known a priori.
Relative entropy approximation: Relative entropy is used in applications because it avoids explicitly computing the discrepancy between idealized and observed data and yields a tractable approximation.The approach also applies more generally to other statistical distances, including Wasserstein and maximum mean discrepancy.
Relative entropy approximation: For relative entropy, the c-posterior can be approximated for small n by tempering the likelihood with a fractional power.This small-sample correction improves on the asymptotic approximation, which is reliable only when n is sufficiently large.
Consequences for inference: The fractional-power posterior adjusts the effective sample size from n to nζ_n and can produce robustified Bayes factors and model posteriors.These model-inference procedures are robust to small perturbations in P_o measured by relative entropy.
Computational implementation: Standard MCMC can sample from the power posterior, and fractional likelihood powers often improve mixing by flattening multimodal targets.However, latent-variable moves may not work the same way for the power posterior, requiring care in MCMC design.

4. Connections with previous work

The c-posterior is related to approximate Bayesian computation, partial-information conditioning, Gibbs posteriors, and power likelihoods, but differs in motivation and theoretical form. Its robustness interpretation also distinguishes it from methods that use power likelihoods for other purposes.

Approximate Bayesian computation: Unlike approximate Bayesian computation, the c-posterior targets robustness to model misspecification rather than inference with intractable likelihoods.The likelihood is generally assumed computable, making inference more computationally efficient; the coarsened posterior is the desired object rather than an unwanted approximation.
Partial information: The c-posterior conditions on a neighborhood of the empirical distribution, extending partial-information conditioning beyond insufficient-statistic approaches.Earlier methods may condition on statistics such as the sample median, whereas this approach conditions on information about the empirical distribution.
Gibbs posteriors: For large samples, the c-posterior can be viewed as a Gibbs posterior whose risk is the distance between the model distribution and the empirical distribution.This interpretation may provide insight into calibrating the Gibbs learning-rate parameter.
Power likelihoods: Previous power-likelihood methods used fixed powers for purposes including MCMC mixing, historical-data discounting, and model selection, whereas this work uses a power tending to 0 as n →∞.The paper states that neither this form of power likelihood nor its theoretical motivation had appeared in prior work.
Related approaches: Goodness-of-fit neighborhoods and model credibility indices are conceptually related, but the paper uses different methods to construct and analyze the c-posterior.The connection is conceptual rather than an assertion of mathematical equivalence.

5. Theory

The theory establishes asymptotic forms for c-posteriors and shows that their robustness follows from continuity of the distance used to define them. Relative-entropy and weakly continuous distances yield specialized convergence results, while standard model inference can remain discontinuous under small distributional perturbations.

5.1. Asymptotic form: Under mild regularity conditions, c-posteriors have a simple asymptotic form, with general, relative-entropy, and weakly continuous-distance results established separately.The framework models i.i.d. data from an observed distribution, introduces a discrepancy distance and finite-sample estimator, and proves convergence through a general theorem and corollaries.
5.1. Asymptotic form: When R ∼Exp(α), the asymptotic c-posterior is proportional to exp(−αd(Pθ, Po))Π(dθ), and it is asymptotically equivalent to an empirical-distance form under consistency.The same result identifies this form with a Gibbs posterior; a fixed-radius R instead truncates the posterior outside the corresponding neighborhood.
5.2. Robustness: The standard posterior can change dramatically under slight distributional perturbations because its likelihood contribution scales with n, whereas the relative-entropy c-posterior uses a fixed α asymptotically.The standard posterior is approximately proportional to exp(−nD(po∥pθ))Π(dθ), while the c-posterior is proportional to exp(−αD(po∥pθ))Π(dθ).
5.2.1. Lack of robustness of model inference: Under misspecification, standard posterior model probabilities can vanish for every model or shift toward increasingly complex models as the sample size grows.For nested models, slight perturbations can increase the minimal adequate model index or prevent the infimum KL divergence from being attained, causing posterior divergence.
5.2.2. Robustness of the c-posterior: The asymptotic c-posterior inherits continuity properties of its defining distance, so appropriately chosen distances make inference robust to perturbations in Po.If the observed distribution is close to an ideal model distribution, the c-posterior is close to the posterior obtained under that ideal distribution.

6. Extensions

The c-posterior framework extends beyond i.i.d. data to stationary ergodic time series and conditional regression models. In both settings, relative-entropy-based discrepancies lead to power-posterior approximations that support robust model-order or variable-selection inference.

6.1. Time-series c-posterior based on relative entropy rate: For stationary ergodic processes and finite-order Markov models, relative entropy rate provides a natural discrepancy between the observed process Po and model process Pθ.The construction assumes finite-dimensional densities and integrability conditions, together with an almost-surely consistent estimator of the relative entropy rate.
6.1. Time-series c-posterior based on relative entropy rate: The time-series c-posterior is approximated by raising the likelihood to the power ζn, avoiding direct computation of the discrepancy estimator.The approximation combines the ergodic theorem with a small-sample correction and is applied to robust inference for autoregressive model order.
6.2. Regression c-posterior based on conditional relative entropy: In regression, conditional relative entropy measures discrepancy between po(y|x) and pθ(y|x) when observed targets are treated as slightly corrupted versions of model-generated targets.An almost-surely consistent estimator of conditional relative entropy supplies the discrepancy used to construct the c-posterior.
6.2. Regression c-posterior based on conditional relative entropy: The regression c-posterior likewise admits a power-posterior approximation, bypassing computation of the conditional discrepancy and enabling robust variable selection in linear regression.A small-sample correction replaces α by nζn before the empirical approximation is formed.

7. Applications

Applications show that coarsened posteriors can stabilize model selection and improve interpretability under misspecification across autoregression, variable selection, and mixture modeling. The approach trades some distributional fit or precision for representations that better reflect practically meaningful structure.

Autoregressive models: The c-posterior favors the correct AR order k = 4 across a fairly wide range of α values and stabilizes near k = 4 as n increases.For α between 100 and 1200, most posterior mass is on k = 4; standard and coarsened approaches diverge once n exceeds α.
Autoregressive models: Under misspecification, the standard AR posterior favors orders much greater than 4, whereas coarsening balances fit and complexity as though the sample size were capped at α.The standard marginal likelihood’s complexity penalty grows only logarithmically and is eventually overwhelmed by the order-n fit term.
Variable selection: In simulated variable selection, the c-posterior favors the true number of nonzero coefficients, k = 2, even as the sample size grows.The simulation uses a slightly nonlinear mean function, while the fitted regression model is subject to misspecification.
Variable selection: At n = 10000, all c-posterior 95% credible intervals contain the true coefficient values, while the standard posterior intervals miss several and can be considerably more concentrated.The c-posterior places most probability at zero for β3 through β6 but can be more diffuse than necessary for β2.
Variable selection: For birthweight, the standard posterior includes around 10 of 72 coefficients, while coarsening yields more parsimonious representations whose included variables increase with α.At α = 100, typically only body length is included; increasing α adds variables until the standard posterior is reached at α = ∞.
Mixture models: In mixture examples, coarsening prevents component counts from continually growing under non-Gaussian misspecification and produces more interpretable clusters, while sacrificing some fit to the data distribution.For the skew-normal simulation, the α = 100 c-posterior strongly supports two components; for Shapley galaxies, α around 100–500 yields representations corresponding to visible large groups.

8. Discussion

The discussion presents the c-posterior as a promising, computationally efficient approach to robust Bayesian inference while identifying several unresolved directions for future work.

Further work is needed to assess the accuracy of the small-sample correction and justify its use in non-discrete cases.
Future research could examine statistical distances beyond relative entropy, particularly where fast inference methods can be developed.
Inferring the coarsening parameter α from data could support robust inference, estimate misspecification, and potentially achieve efficiency when the model is correct.
Precise frequentist coverage guarantees for the c-posterior under misspecification remain a potentially difficult open goal.

Appendix A. Proofs

The appendix begins the toy-example derivation by introducing an indicator for the coarsening event and applying Bayes’ theorem to the hypotheses.

The event Z = 1(D(ˆp_x||ˆp_X) < R) records whether the empirical distributions fall within the random relative-entropy radius R.
Bayes’ theorem is applied to compute posterior probabilities for h ∈ {H0, H1} conditional on the coarsening event.
The derivation organizes the posterior through the likelihoods P(Z = 1|H0) and P(Z = 1|H1).

A.1. C-posterior for the toy example in Section 2.

The toy-example appendix evaluates the coarsening event under both hypotheses and derives an approximation for the posterior under H0 using a tempered beta-function expression.

Under H0, S|H0 follows Binomial(n, 1/2), while under H1, S|H1 follows a Uniform distribution on {0, 1, . . . , n}.
The approximation uses α_n = 1/(1/n + 1/α), the Bernoulli likelihood p_θ(x), and a constant c independent of θ and h.
For H0, the coarsening-event probability is approximated by c/2^α_n.
The resulting posterior approximation is Π(H0|Z = 1) ≈ 1/(1 + 2^α_n B(1 + α_n x̄, 1 + α_n(1 − x̄))).

A.2. Justification of the small-sample correction when |X| < ∞.

This appendix justifies the small-sample correction by approximating relative entropy with chi-squared distance and expressing the latter through the covariance structure of multinomial observations.

For empirical frequencies ˆs, the central limit theorem gives the approximation ˆs ≈ N(s, C(s)/n).
The derivation uses the approximation C(q′) ≈ C(s′) and the convolution formula for independent normal variables to obtain Equation A.1.
For p and q near each other, relative entropy satisfies D(p∥q) = 1/2χ2(p, q) + o(∥p − q∥2).
The chi-squared distance can be represented as a Mahalanobis distance using C, which equals Cov(Z′) for Z ∼ Multinomial(1, q).
For q ∈ ∆k, the covariance matrix has entries C_ij = q_i1(i = j) − q_iq_j, with its inverse obtained by a Sherman–Morrison rank-one update.

A.3. Proofs of results in Theory section.

The proofs establish convergence of conditional expectations by applying a general lemma to distances between distributions and empirical distances. Under stated convergence, continuity, positivity, and integrability conditions, the limiting conditional expectation yields the theoretical result.

Theoretical results: Theorem 5.3 applies the lemma with U = d(Pθ, Po), Un = dn(X1:n, x1:n), V = R, and W = h(θ).Its assumptions include Un converging almost surely to U, P(U = V) = 0, P(U < V) > 0, and E|W| < ∞.
Proof strategy: Lemma 5.1 shows that E(W | Un < V) converges to E(W | U < V) under almost-sure convergence and regularity conditions.The proof uses dominated convergence, with P(U = V) = 0, P(U < V) > 0, and E|W| < ∞.
Proof strategy: The lemma’s limit is expressed as E(W1(U < V)), using independence of V from U and W by construction.This identity establishes Equation 5.2 and supports Equation 5.1 for bounded continuous h.
Theoretical results: The proof of Corollary 5.5 uses the almost-sure behavior of sampled and observed sequences to obtain convergence of empirical distributions.The argument invokes Dudley’s theorem for ˆPX1:n and ˆPx1:n converging to Pθ and Po, respectively.
Theoretical results: Theorem 5.6 follows by applying Lemma 5.1 with U = d(Pθ, Po), Um = d(Pθ, Pm), V = R, and W = h(θ).This transfers the lemma’s conditional-expectation argument to the theorem’s specified distance quantities.