PaperlayerView exact revision on arXiv

Source-linked AI summary

On the asymptotic behaviour of $\alpha$-divergence generalized posteriors

Ioar Casado-Telletxea, Omar Rivasplata

arXiv:2610.35330v1math.STstat.MLstat.TH

TL;DR

BvM-type results beyond KL have been lacking for α-divergence generalized variational inference. This paper analyzes the unrestricted problem and shows that α determines the limiting shape and posterior scale, with corresponding coverage consequences.

  • Problem

    For α-divergence-regularized GVI, BvM-type results beyond KL had not characterized how the regularizer changes limiting shape and concentration scale.

    Source: §1.1 · p. 3

  • Method

    The authors analyze α-divergence-regularized GVI over distributions absolutely continuous with respect to the prior, under local regularity, identifiability, and prior assumptions.

    Source: §1 · p. 2, §5 · p. 12

  • Results

    For α ∈(0, 1) with d < 2/(1 −α), the limit is Student-t and width shrinks faster than n^-1/2; for α > 1, it is compactly supported Pearson type-II and width shrinks more slowly.

    Source: §5 · p. 12

  • Takeaways & Limitations

    For the studied credible-region classes, scale mismatch gives coverage tending to zero for α < 1 and one for α > 1, so posterior concentration alone does not ensure calibrated uncertainty quantification.

    Source: §5 · p. 12

    The regime α ∈(0, 1) with 2/(1 −α) ≤ d remains insufficiently characterized, and analysis over restricted distribution families is left for future work.

    Source: §5 · pp. 12–13

Abstract

from arXiv · show

We study the asymptotic behaviour of the solutions to the unconstrained generalized variational inference problem regularized by an $\alpha$-divergence. Under mild conditions, the minimizer is explicit up to a scalar normalizing constant, and we show that it admits a rescaling under which it converges in $L^1$ to a fixed non-Gaussian law, at a concentration scale around the empirical loss minimizer that departs from the typical $n^{-1/2}$ in the classical Bernstein--von Mises theorem. For $\alpha\in(0,1)$ and $d<2/(1-\alpha)$ the limit is a Student-$t$ distribution and the posterior width shrinks faster than the sampling error of the empirical minimizer. Conversely, for $\alpha>1$ the limiting distribution is a compactly supported Pearson type-II distribution, and the posterior width shrinks more slowly than the sampling error. The KL divergence ($\alpha=1$) is the unique member of the $\alpha$-divergence family whose posterior width matches the sampling scale and whose limit is Gaussian. We discuss the implications of these qualitatively different regimes for the frequentist coverage of credible regions, and a toy example on intercept-only logistic regression illustrates the theoretical claims.

1 Introduction

The paper studies α-divergence-regularized generalized variational inference over unrestricted distributions and establishes how the divergence changes posterior shape and concentration. Its results identify distinct coverage consequences for α below and above 1, with KL regularization retaining the classical Gaussian, sampling-scale behavior.

1 Introduction

  • The analysis treats α-divergence as the GVI regularizer over unrestricted distributions, isolating its effect from restrictions imposed by a variational family.The posteriors are absolutely continuous with respect to the prior.Source: §1 · p. 2

1.1 Contributions and related work

  • BvM-type results beyond KL were lacking for α-divergence GVI, leaving its limiting shape and concentration scale uncharacterized.The paper situates this gap alongside existing work on GVI consistency, finite-sample rates, and closed-form f-divergence solutions.Source: §1.1 · p. 3
  • For α ∈(0, 1), the rescaled limit is Student-t type under d < 2/(1 −α), while for α > 1 it is compactly supported Pearson type-II; their posterior widths shrink faster and slower than n^-1/2, respectively.The width constant in both non-KL regimes depends on the prior density at the population minimizer.Source: §1 · p. 2, §1.1 · p. 4
  • For the studied credible-region classes, scale mismatch yields frequentist coverage tending to zero for α ∈(0, 1) and to one for α > 1 when the empirical minimizer has nondegenerate √n fluctuations.The included classes cover equal-tailed intervals used in practice.Source: §1.1 · p. 4
  • As α →1, both limiting-density families converge in L1 to Gaussian, and KL is the unique α-divergence member with the usual posterior-width exponent.The limiting concentration exponents approach 1/2.Source: §1 · p. 2, §1.1 · p. 4

2 Preliminaries

The preliminaries define generalized variational inference and the local asymptotic description used in the paper. They distinguish posterior concentration from BvM behavior and define frequentist calibration separately from posterior credibility.

2.1 Generalized variational inference

  • GVI minimizes empirical loss plus a divergence penalty over a class of probability measures, with η controlling the relative weight of the divergence term.The paper's analysis considers unrestricted distributions absolutely continuous with respect to the prior.Source: §2.1 · p. 5
  • With KL divergence and a likelihood loss, the unrestricted formulation gives the ordinary Bayesian posterior; restricting the family gives variational inference, while an arbitrary loss gives a Gibbs posterior.These are examples of constructions encompassed by the GVI formulation.Source: §2.1 · p. 5

2.2 Asymptotic analysis

  • Consistency describes concentration around the population target but does not determine the concentration rate or local posterior shape; a BvM result supplies that finer local description.The local description centers at θ_n and rescales by s_n, with convergence typically in L1 or total variation.Source: §2.2 · p. 5
  • Asymptotic credibility is defined by posterior probability, whereas frequentist coverage is the repeated-sampling probability that a region contains the fixed population target.Calibration requires the frequentist coverage to match the nominal level.Source: §2.2 · p. 6

3 Main results

Under local regularity, identifiability, and prior assumptions, the unconstrained GVI solution is explicit up to normalization and has different limiting shapes and concentration rates across α-divergence regimes. These differences yield opposite frequentist coverage behavior away from α = 1, while the KL case recovers the classical Gaussian limit and sampling scale.

3 Main results

  • Under mild assumptions, the GVI problem has an explicit solution up to a scalar normalizing constant.The analysis assumes locally quadratic losses, identifiable minimizers, and a prior density continuous and positive at the population minimizer.Source: §3 · pp. 6–7

3.1 The α ∈(0, 1) case

  • For α ∈(0, 1) with 2κ > d, the rescaled posterior has a Student-t limit and concentrates around θn faster than n^-1/2.Its asymptotic width depends on the prior density at θ0; a larger prior density yields a wider posterior.Source: §3.1 · pp. 7–8
  • For α ∈(0, 1), the width mismatch yields coverage tending to zero for the specified credible regions, including equal-tailed intervals used in practice.The result applies when θn has nondegenerate √n sampling fluctuations.Source: §3.1 · p. 8

3.2 The α > 1 case

  • For α > 1, the rescaled posterior has a compactly supported Pearson type-II limit and concentrates around θn more slowly than the sampling fluctuations.Unlike the α ∈(0, 1) regime, a larger prior density at θ0 yields a narrower posterior.Source: §3.2 · p. 9
  • For α > 1, specified fixed-level credible regions containing a neighborhood of the origin in rescaled coordinates have frequentist coverage tending to one.The posterior concentrates more slowly than θn approaches θ0.Source: §3.2 · p. 9

3.3 The phase transition at α = 1

  • At α = 1, the limiting shape and concentration exponent approach their Gaussian and 1/2 counterparts, making KL the unique α-divergence case matching the sampling scale.The limiting densities in both non-KL regimes converge in L1 to the Gaussian law as α →1.Source: §3.3 · p. 10

4 Example: intercept-only logistic regression

The intercept-only logistic regression example illustrates the distinct rescaled posterior limits for α = 0.5 and α = 2, and examines their finite-sample convergence and credible-interval coverage.

4 Example: intercept-only logistic regression

  • The example uses independent Bernoulli observations with p0 = 0.7 and an intercept-only logistic model with logit link.The experiments set η = 1 and use a Student-t5 prior centred at zero with scale parameter 2.Source: §4 · p. 10
  • In the logistic regression example, the sn-rescaled α-divergence generalized posterior converges to a Student-t distribution for α = 0.5 and a Pearson type-II distribution for α = 2, with d = 1 and η = 1.Source: Figure 2 · p. 11, Figure 3 · p. 11
  • Coverage is computed exactly over the binomial distribution of the success count, with the ordinary Bayesian posterior as the Gibbs baseline.The comparison uses equal-tailed 95% credible intervals and includes the nominal-coverage line.Source: §4 · p. 11, Figure 4 · p. 12
  • For α = 0.5, posterior intervals become too narrow relative to centre sampling fluctuations, while for α = 2 they become too wide.The undercoverage for α = 0.5 develops slowly; for α = 2, coverage approaches one quickly.Source: §4 · pp. 11–12, Figure 4 · p. 12

5 Conclusions and future work

The conclusions characterize how α-divergence controls generalized posterior shape and concentration under fixed regularization. They connect the resulting scale mismatch to credible-region coverage and identify regimes and extensions left for future work.

5 Conclusions and future work

  • For α ∈ (0, 1) with 2/(1 − α) > d, rescaled posteriors converge in L1 to Student-t laws and shrink faster than n^-1/2; for α > 1, they converge to compactly supported Pearson type-II laws and shrink more slowly.In both regimes, asymptotic width depends on the prior density at θ0; as α → 1, both limit families converge in L1 to the Gaussian and their concentration exponents approach 1/2.Source: §5 · p. 12
  • For the specified credible regions, fixed α ≠ 1 yields frequentist coverage tending to zero when α < 1 and to one when α > 1, even in regular, correctly specified models.This follows when the empirical minimizer has nondegenerate √n sampling fluctuations and posterior width shrinks at a different rate.Source: §5 · p. 12
  • The analysis leaves the α ∈ (0, 1) regime 2κ ≤ d, with κ = (1 − α)^-1, for future work.Further directions include sample-size- or data-dependent η, other f-divergences, and variational problems over restricted distribution families.Source: §5 · pp. 12–13

A Proofs for the main results

The appendix lemma establishes uniform local quadratic control around the empirical minimizer, separation away from it, and local prior-density bounds under the stated assumptions.

A Proofs for the main results

  • Outside that neighbourhood, the loss gap is bounded below by a positive constant, so θn is the unique global minimizer for sufficiently large n.This uses the separation assumption together with the local quadratic lower bound.Source: A Proofs for the main results · p. 17
  • Under prior regularity, the prior density is bounded above and bounded away from zero throughout the neighbourhood of θn for sufficiently large n.The proof obtains these bounds from continuity and positivity at θ0 and convergence of θn to θ0.Source: A Proofs for the main results · p. 17
  • For sufficiently large n, the empirical loss gap is bounded above and below by constant multiples of the squared distance to θn within a fixed neighbourhood.The proof combines eigenvalue bounds for Hn with the local cubic remainder bound.Source: A Proofs for the main results · p. 17

Proof of Proposition 3.4

The proof establishes a unique normalizing constant and uses convexity to establish uniqueness of the variational minimizer.

Proof of Proposition 3.4

  • The proof begins by defining a normalizer δn and then applies convexity to the variational objective.The normalizer is chosen to satisfy Fn(δn) = 1.Source: Proof of Proposition 3.4 · p. 17
  • A unique δn > 0 normalizes Fn because Fn is finite, continuous, strictly decreasing, and ranges to zero at infinity while diverging at the other endpoint.The local quadratic loss bound and positive prior density yield the divergence when 2κ ≥ d.Source: Proof of Proposition 3.4 · p. 17
  • Convexity of fα gives the minimization inequality for probability densities with finite Jn, and strict convexity gives uniqueness.The objective is expressed using r = dQ/dΠ, up to constants independent of the minimization.Source: Proof of Proposition 3.4 · pp. 17–18

Proof of Theorem 3.5

The proof establishes that the normalizer-related sequence δ_n tends to zero, hence s_n tends to zero, and that the rescaled densities converge in L1(R^d).

Proof of Theorem 3.5

  • δ_n → 0, and consequently s_n → 0.The proof obtains a contradiction from any subsequence bounded away from zero, using the loss gap outside a neighborhood and the prior's density.Source: Proof of Theorem 3.5 · p. 18
  • Under θ = θ_n + s_n x, the proof uses the cubic remainder bound and Assumption 3.3 to establish pointwise convergence.The change of variables is introduced before applying dominated convergence to the rescaled densities.Source: Proof of Theorem 3.5 · pp. 18–19
  • Pointwise convergence of q_n to q, with both probability densities, yields L1(R^d) convergence by Scheffé’s lemma.The proof combines its preceding limits to obtain q_n → q pointwise.Source: Proof of Theorem 3.5 · p. 19

Proof of Corollary 3.6

The proof uses the concentration rate s_n ≍ n^−κ/(2κ−d) to show that √n s_n tends to zero, then completes a claim using Portmanteau’s theorem.

Proof of Corollary 3.6

  • On the event {A_n ⊆ B_M(0)}, θ_0 ∈ C_n implies ∥θ_n − θ_0∥ ≤ M s_n.This implication is used in the proof's bound on the event involving C_n.Source: Proof of Corollary 3.6 · p. 19
  • s_n ≍ n^−κ/(2κ−d), and √n s_n ≍ n^−d/(2(2κ−d)) → 0 when 2κ > d.The vanishing rate makes the last term in (54) vanish.Source: Proof of Corollary 3.6 · p. 19
  • The proof takes a lim sup and applies Portmanteau’s theorem after the first term vanishes by assumption and the last term vanishes by the rate bound.It then lets ε → 0 and uses P(Z = 0) = 0 to prove the second claim.Source: Proof of Corollary 3.6 · p. 19

Proof of Proposition 3.7

The proof constructs a unique positive normalizer δ_n by showing its defining function is finite, continuous, and strictly increasing.

Proof of Proposition 3.7

  • For sufficiently large n, Δ_n ≥ 0, which the proof uses to define the normalizer function F_n.The nonnegative integrand bounds the function's integrand above by δ^p, giving finiteness and continuity by dominated convergence.Source: Proof of Proposition 3.7 · p. 20
  • A local quadratic loss bound and a positive lower bound on the prior near θ_n support the proof's local integral bound.These conditions hold when ∥h∥ ≤ ε_1.Source: Proof of Proposition 3.7 · p. 20
  • F_n is strictly increasing because its integrand is nondecreasing everywhere and strictly increasing on a ball around θ_n with positive prior mass.The proof concludes that there is a unique δ_n > 0 satisfying F_n(δ_n) = 1.Source: Proof of Proposition 3.7 · p. 20
  • The objective is rewritten using r = dQ/dΠ, up to the constant nℓ_n(θ_n) and factor n.The candidate density r_n has finite objective, and convexity of f_α is used to compare it with any probability density of finite objective.Source: Proof of Proposition 3.7 · p. 20
  • Strict convexity of f_α gives uniqueness of the minimizer.The proof also notes that both densities integrate to one and the final integrand is nonnegative.Source: Proof of Proposition 3.7 · p. 20

Proof of Theorem 3.8

The proof of Theorem 3.8 establishes L1 convergence of rescaled densities; the following corollary and proposition proofs use that result and derive further limits.

Proof of Theorem 3.8

  • δ_n/n → 0, and consequently s_n → 0.The proof rules out a subsequence with δ_n bounded below proportionally to n, using a positive lower bound on prior mass near θ_n.Source: Proof of Theorem 3.8 · p. 21
  • Pointwise convergence of q_n to q, with both probability densities, yields L1(R^d) convergence by Scheffé’s lemma.The proof combines its preceding limits to obtain pointwise convergence.Source: Proof of Theorem 3.8 · p. 22

Proof of Corollary 3.9

  • The Corollary 3.9 proof relates membership of θ_0 in C_n to the distance between θ_n and θ_0, then applies Portmanteau’s theorem to a closed set.On {B_r(0) ⊆ A_n}, θ_0 ∉ C_n implies ∥θ_n − θ_0∥ > r s_n.Source: Proof of Corollary 3.9 · p. 22

Proof of Proposition 3.10

  • For Proposition 3.10, the kernel forms given for α < 1 and α > 1 are used in the normalization-limit argument.For α > 1, the proof bounds K_α(x) by e^−z(x) and invokes dominated convergence.Source: Proof of Proposition 3.10 · p. 23
  • The normalized densities converge pointwise to φ_H0, and Scheffé’s lemma gives L1(R^d) convergence.The proof also states concentration exponents κ/(2κ − d) for α < 1 and p/(2p + d) for α > 1, both equal to 1/2 + d(α − 1).Source: Proof of Proposition 3.10 · p. 23
Loading 2610.35330v1…