PaperlayerView exact revision on arXiv

Source-linked AI summary

The Bayesian Learning Rule Beyond KL Geometry

Sophia Sklaviadis, Wu Lin, Andre Martins, Mario Figueiredo, Thomas Moellenhoff, Emtiyaz Khan

arXiv:2610.31308v1cs.LG

TL;DR

The KL-based BLR provides additive updates for exponential-family posteriors, but its fixed-support geometry does not cover parameter-dependent support. This paper derives additive updates from mixed-space Bregman or Fenchel-Young discrepancies and applies them to Tsallis q-Gaussian posteriors. Experiments show controllable support and tail behavior with competitive performance, while the study stops short of demonstrating state-of-the-art performance or scale.

  • Problem

    The KL-based BLR is restricted to fixed-support exponential families, leaving open which non-KL geometries preserve its additive updates for broader posterior families.

    Source: §1 · p. 1, §2.3 · p. 5

  • Method

    The paper derives generalized BLR updates from mixed-space Bregman or Fenchel-Young discrepancies, inducing maximum-entropy families and dual coordinates while recovering the classical BLR.

    Source: §1 · p. 2, §3 · p. 5, §3.2 · p. 7

  • Results

    On MNIST, CIFAR-10, and CIFAR-100, q-VON generally improves over SGD and q-VSGD, while q-VON-MF is competitive with highly tuned IVON on CIFAR without additional tuning.

    Source: §4 · p. 10

  • Takeaways & Limitations

    With Tsallis negentropy and quadratic statistics, the generalized BLR yields Newton-like methods with bounded-support or heavy-tailed q-Gaussian posteriors.

    Source: §1 · p. 2, §3.2 · p. 7

    The paper does not demonstrate state-of-the-art performance or scale, and defers larger experiments to future work.

    Source: §1 · p. 3

Abstract

from arXiv · show

The Bayesian learning rule (BLR) exploits the relation between Bayesian updating and the Kullback-Leibler divergence (KLD) to derive efficient additive updates for exponential family variational posteriors. The KL geometry is powerful but restrictive: combined with posterior families with parameter-dependent support it can lead to undefined objectives. We ask which non-KL geometries preserve the additive BLR computation. Previous works that use $f$-divergences suggest that this is only possible with the KLD. Here, we show that this is not true. We show that additive information-geometric optimization algorithms can be derived for the class of mixed-space Bregman or Fenchel-Young discrepancies. These discrepancies induce generalized maximum-entropy families, dual coordinates, and generalized-BLR canonical gradient updates, recovering the classical BLR. The generalized BLR leads to new algorithms that \textit{cannot} be derived from the KL-based BLR. For the Tsallis negentropy with quadratic statistics, the generalized BLR yields bounded-support and heavy-tailed $q$-Gaussian posteriors and an easy Newton-like implementation. Experiments up to CIFAR-100 illustrate controllable support and tail behavior and competitive performance.

1 Introduction

The KL-based BLR gives efficient additive updates but is limited to fixed-support exponential families. This paper extends those updates through mixed-space Bregman or Fenchel-Young discrepancies, including Tsallis-based q-Gaussian methods with controllable support and tails.

1 Introduction

  • The KL-based BLR is tied to fixed-support exponential families, excluding posterior families with variable support or adaptive tail behavior.Such families can have parameter-dependent support or be outside the ordinary fixed-support exponential-family class.Source: §1 · p. 1
  • Mixed-space Bregman or Fenchel-Young discrepancies preserve BLR-style additive canonical updates, while recovering the classical BLR as a special case.The framework also yields generalized maximum-entropy families, dual coordinates, and canonical gradient updates.Source: §1 · p. 2
  • Tsallis negentropy with quadratic statistics yields Newton-like variational methods using bounded-support and heavy-tailed q-Gaussian posteriors.The methods adapt Improved Variational Online Newton (IVON) with a few lines of code.Source: §1 · p. 2
  • Experiments reach CIFAR-100 and show the proposed methods are competitive with Gaussian IVON without additional tuning.The paper defers demonstrations of state-of-the-art performance or scale, as well as larger experiments such as quantization-aware training, pruning, and language-model training with bounded variance.Source: §1 · p. 3

2 KLD Geometry and the BLR

The section derives the BLR’s additive updates from convex duality, moment-constrained maximum entropy, and exponential-family coordinates. It then identifies fixed-support restrictions of KL geometry and motivates a Fenchel-Young generalization that retains additive updating.

2 KLD Geometry and the BLR

  • The KLD is the Fenchel-Young slack generated by Shannon negentropy and a reference score, linking the Gibbs-Donsker-Varadhan variational formula to maximum-entropy inference.The density-space duality, restricted by finite moment constraints, characterizes exponential families as maximum-entropy distributions.Source: §2 · p. 3

2.1 Exponential Family Posteriors from Moment-Constrained Maxent

  • A finite-dimensional score and its log-normalizer define an exponential family whose expectation parameters are the moments and whose members maximize Shannon entropy under those moment constraints.For a non-constant base measure, the corresponding maximum is relative Shannon entropy with respect to that measure.Source: §2.1 · p. 4, §2.2 · p. 4

2.2 The BLR in Exponential Family Coordinates

  • In exponential-family coordinates, the KLD gradient is affine in the natural parameter, producing an additive BLR update in that coordinate.For conjugate models, a single BLR step coincides with classical Bayes’ rule; more generally, the BLR optimizes an objective whose minimizer over all densities is the exact posterior.Source: §2.2 · pp. 4–5

2.3 Limitations of the KLD and the BLR

  • KL geometry requires absolute continuity, so posterior families with bounded, parameter-dependent support fall outside the KL-based BLR’s scope.Fenchel-Young discrepancies instead support generalized maximum-entropy families and additive updates; the paper states that sequential coherence singles out KL only within updates using an f-divergence and linear expected loss.Source: §2.3 · p. 5

3 Fenchel-Young BLR

The section replaces KL geometry with Fenchel–Young discrepancies to preserve additive Bayesian learning updates beyond exponential families. It builds induced maximum-entropy families and dual coordinates, then derives generalized canonical-gradient updates that recover the classical BLR as a special case.

3.1 Generalizing the KLD to the Fenchel-Young Discrepancy

  • Fenchel–Young discrepancies generalize the KLD using a reference score and density as arguments, allowing generalized entropies with parameter-dependent support.The reference score need not be the log-density of a normalized prior.Source: §3 · p. 5, §3.1 · p. 6

3.2 Generalizing the Exponential Family

  • Finite-dimensional score families induce maximum-entropy distributions with canonical and expectation coordinates, extending the construction beyond exponential families.The generalized Fisher is derived from the convex potential; the induced density exists when the supremum is attained.Source: §3.2 · pp. 6–7
  • The mixed-space Fenchel–Young form enables additive canonical updates, with quadratic statistics and Tsallis negentropy inducing bounded-support and heavy-tailed q-Gaussians.For Shannon negentropy the construction recovers the classical exponential family.Source: §3.2 · p. 7

3.3 Generalizing the Variational Objective

  • The generalized variational objective recovers the BLR objective for Shannon negentropy, while other convex functions yield new objectives.Unlike f-divergences other than KLD, its loss and generalized negentropy separate; sequential losses compose associatively in score space.Source: §3.3 · p. 7

3.4 Generalizing the BLR

  • The generalized canonical-gradient update is intrinsic to the variational problem, recovers BLR when Ω = −H, and at unit step recovers a tilted-score conjugate posterior.At convergence, the canonical parameter matches the gradient of the loss.Source: §3.4 · p. 8

4 Example: Tsallis Negentropy and q-VON

The Tsallis-negentropy example uses quadratic statistics to construct q-Gaussian variational posteriors with tunable support and tails. Its bounded-support mean-field approximation can track posterior modes, and the section describes practical q-VON implementations and their scope.

4 Example: Tsallis Negentropy and q-VON

  • Quadratic statistics with Tsallis negentropy induce q-Gaussians: q < 1 gives parameter-dependent bounded support, while q > 1 gives heavy tails.The heavy-tailed case is parameterized as a Student-t distribution with ν = 2/(q − 1) − D.Source: §4 · pp. 8–9
  • Specializing the generalized BLR to quadratic families yields Newton-like canonical updates, with reparameterized gradients and practical diagonal implementations.The bounded-support and heavy-tailed algorithms differ in their scale parameterization and escort reweighting.Source: §4 · p. 9
  • The q-VON algorithms retain an IVON-like update form but change the sampling law, scale constants, and diagonal Stein estimator.They use minibatch stochastic gradients, diagonal precision, and momentum terms.Source: §4 · p. 9
  • Mean-field q-VON assigns each parameter a one-dimensional q-Gaussian, avoiding high-dimensional constraints on heavy tails and bounded-support radii.Each variance is updated separately using one-dimensional constants.Source: §4 · p. 9
  • Changing q lets bounded-support mean-field q-VON means approach posterior modes for skewed marginals that Gaussian approximations poorly match.Figure 2 illustrates the comparison on Bayesian logistic regression using the UCI Credit Approval dataset.Source: Figure 2 · p. 9

A The sharp Gibbs bound forces the KLD

  • The appendix proposition considers loss-independent functionals under a sharp Gibbs bound, with equality at the Gibbs density.The provided passage states the proposition’s setup and equality case.Source: A The sharp Gibbs bound forces the KLD · p. 14

B.1 Expfam as max Shannon entropy

  • The passage states that, when all of the integrals are finite, it proceeds to characterize a maximum of Shannon entropy.Source: §B.1 · p. 14

B.2 q-expfam as Max Tsallis (2 −q)-entropy

Under the stated feasibility and finiteness assumptions, the q-exponential candidate minimizes the Tsallis negentropy among densities with the same moment constraints. The argument establishes optimality conditionally, not existence of the candidate.

B.2 q-expfam as Max Tsallis (2 −q)

  • Given the stated assumptions, πλ minimizes Ω2−q, equivalently maximizes S2−q, over densities satisfying the same moment constraints.The argument certifies a feasible q-exponential candidate; it does not prove that λ or a feasible πλ exists.Source: §B.2 · pp. 16–17
  • For q < 1, q-Gaussians have bounded support; for the stated q > 1 range, they are Student-t q-Gaussians.The degrees of freedom are ν = 2/(q−1) − D > 2.Source: §B.2 · p. 17

B.3 Existence

  • A general existence theorem for arbitrary Ω would require additional infinite-dimensional assumptions on support, moment functions, and the regularizer.For the Tsallis families used in Sec. F, the optimizer is explicit, and normalization and moment conditions can be checked from closed forms.Source: §B.3 · p. 17

C Proof of Thm. 1

The proof derives a generalized canonical-gradient update from the mixed-space Fenchel-Young objective. The accompanying constructions express q-Gaussians in canonical and location-scale forms and provide identities used by q-VON.

C Proof of Thm. 1

  • When the generalized Fisher matrix exists and is invertible, the Fenchel-Young BLR has the same canonical-gradient update form as Khan and Rue [2023].The derivation uses dual coordinates and the chain rule; its fixed point satisfies the stationary condition of the generalized variational objective.Source: C Proof of Thm. 1 · p. 18

D.1 Canonical q-exponential form

  • The q-Gaussian construction writes bounded-support and heavy-tailed densities in q-exponential canonical form, with q < 1 and q > 1 respectively.The heavy-tailed form is a Student-t family with parameter ν; the bounded-support form has a closed-form maximum support radius.Source: §D.1 · pp. 18–19, §D.2 · pp. 19–20

D.2 Radial laws and location-scale form

  • The location-scale representation uses θ = m + Σ^1/2(ru), with direction u uniform on the unit sphere and independent of radius r.The squared radial fraction for q < 1 is Beta-distributed; the heavy-tailed radial law is a Beta-prime kernel and has an equivalent chi-square-ratio representation.Source: §D.2 · pp. 19–20

D.3 Expectation parameters

  • For q > 1, the covariance is ν/(ν−2)Σ when ν > 2; for q < 1 it is κ(q,D)Σ.The expectation parameters are the mean and second moment, with the second moment equal to mm⊤ plus the covariance.Source: §D.3 · p. 20

D.4 FY Mixed-space Bregman between two heavy-tailed q-Gaussians

  • For quadratic statistics and q > 1, the paper derives a Fenchel-Young mixed-space Bregman divergence between heavy-tailed q-Gaussians.The derivation uses their first and second moments and the deformed cumulant expression.Source: §D.4 · p. 21

E Bonnet-, Price-, and Stein-type identities for q-Gaussians

  • The q-VON algorithms retain VON’s form, differing by factors depending only on q and D and by closed-form escort reweighting; as q → 1 they recover standard VON.They use q-Gaussian samples and generalized Bonnet and Price identities.Source: E Bonnet-, Price-, and Stein-type… · pp. 21–22

E.1 Bonnet-type identities

  • For q < 1 and q > 1, the location-parameter gradient satisfies a Bonnet identity for differentiable functions with suitable integrability.The paper reviews these gradient identities as ingredients for deriving the q-VON updates.Source: §E.1 · p. 22, E Bonnet-, Price-, and Stein-type… · p. 22

E.1.1 Proof of the q > 1 or Student-t Bonnet-type identity

This material establishes gradient identities for q-Gaussians and uses a Stein identity to express the diagonal escort-Hessian estimate in a form used by the algorithms.

E.1.1 Proof of the q > 1 or Student-t Bonnet-type identity

  • For Student-t posteriors with ν > 2, the Bonnet identity proof differentiates under the integral and uses vanishing boundary terms.The assumptions include differentiability and integrability of f; polynomial decay of the posterior is given as sufficient for the boundary condition.Source: §E.1.1 · p. 22

E.2 Price-type identities

  • The scale-parameter gradient admits a generalized Price identity, with the q-dependent radial weighting and covariance factor specified for q < 1 and q > 1.For q > 1, the stated factor ν/(ν−2) requires ν > 2.Source: §E.2 · p. 23

E.2.1 Diagonal Stein estimator

  • The diagonal Stein estimator rewrites escort-averaged Hessian entries using first derivatives evaluated at samples from πλ.The q-VON algorithms use this form to avoid explicit Hessian evaluations; the identity requires vanishing boundary terms.Source: §E.2.1 · pp. 23–24

E.2.2 Proof of the q > 1 Student-t Price-type identity

The proof establishes the q > 1 Student-t Price identity by differentiating expectations and integrating by parts under regularity conditions. The Tsallis construction relates that identity framework to q-VON updates for q-Gaussian families.

E.2.2 Proof of the q > 1 Student-t Price-type identity

  • For Student-t posteriors with ν > 2, the Price-identity proof assumes f is twice continuously differentiable and integrable, with boundary terms vanishing.The paper states bounded f and gradient as sufficient for the boundary condition.Source: §E.2.2 · p. 24
  • The proof differentiates the expectation with respect to Σ and uses integration by parts twice to express it through the Hessian, yielding the q > 1 Price identity.It identifies the resulting expression with the stated identity to complete the proof.Source: §E.2.2 · pp. 24–25

F Tsallis Negentropy and q-VON

  • With quadratic statistics, Tsallis negentropy yields q-Gaussians: bounded-support for q < 1 and heavy-tailed for q > 1, corresponding to q-VON and Student-t-VON.Changing Ω changes support and tail behavior while keeping the quadratic statistic unchanged.Source: F Tsallis Negentropy and q-VON · p. 25
  • The Tsallis generalized BLR uses the dual-coordinate relation ∇µΩ2−q(πλ) = λ to derive Newton-like updates.The canonical-parameter Hessian F2−q recovers the Fisher matrix when q = 1.Source: F Tsallis Negentropy and q-VON · p. 26

F.1 q-Variational Online Newton

The section derives q-VON updates for bounded-support and heavy-tailed q-Gaussian posteriors from a shared canonical update, then extends them to mean-field variants.

F.1 q-Variational Online Newton

  • Both tail regimes start from the same canonical update; the derivation converts expectation-parameter gradients to location-scale gradients and applies Bonnet- and Price-type identities for Newton-like updates.For q > 1, the Student-t covariance exists under the stated condition ν > 2, equivalent to 1 < q < (D + 4)/(D + 2).Source: §F.1 · pp. 26–28
  • Tsallis negentropy with quadratic statistics yields q-Gaussian families whose q value determines bounded support or heavy tails, while the statistics determine their elliptical geometry.The q > 1 subfamily coincides with Student-t densities.Source: §F.1 · pp. 26–27
  • For q < 1, bounded-support q-Gaussians have a maximum support radius, and the scale matrix is expressed through a determinant reweighting to avoid dependence on S^-1.The resulting Newton-like update uses the (2 − q)-escort distribution.Source: §F.1 · p. 27
  • For q > 1, the q-Gaussians are Student-t distributions with unbounded support, and the derived Student-t canonical parameter updates are used in Alg. 2.The section assumes ν > 2 for finite covariance and optionally ν > 4 when finite kurtosis is needed.Source: §F.1 · p. 28
  • The (2 − q)-escort reweighting makes the updates’ sampling behavior differ by regime: q > 1 scale estimates are tail-aware, while q < 1 concentrates on the support interior and gives a stable scale estimate with bounded exploration.For q > 1, large-radius samples affect the Hessian estimate; for q < 1, the scale update has lower variance.Source: §F.1 · pp. 28–29
  • Deep-learning implementation uses minibatch stochastic gradients, a diagonal scale parameterization, momentum, and bounded noise; the momentum does not change fixed points and was found to improve results.The scale parameter is reparameterized using a positive L2 regularizer, and its entries are clipped to a positive minimum.Source: §F.1 · p. 29

F.2 Mean-Field Variants

  • Mean-field q-VON variants factorize the posterior into one-dimensional q-Gaussians and require dimension-one constants, per-parameter scale computation, and one-dimensional sampling.The paper refers to these variants as q-VON-MF.Source: §F.2 · p. 29

G Experiments

The experiment uses bounded-support q-VON-MF to approximate the posterior mode in Bayesian logistic regression on UCI Credit Approval. It compares mean-field Gaussian and bounded-support q-Gaussian posterior approximations using MCMC-computed marginals.

G.1 Bayesian logistic regression

  • The experiment evaluates bounded-support q-VON-MF as an approximation to the posterior mode in Bayesian logistic regression on UCI Credit Approval.Posterior marginals for each feature are computed with MCMC.Source: §G.1 · p. 29
  • The variational approximations compared are mean-field Gaussian with q = 1 and bounded-support q-Gaussians with q = −2 and q = −10.Source: §G.1 · p. 29
  • The comparison is based on posterior marginals for each feature, computed using MCMC.Source: §G.1 · p. 29

G.2 Image Classification with Neural Networks

The experiments compare q-VON variants with SGD, VSGD, IVON, and q-VSGD on MNIST, CIFAR-10, and CIFAR-100. q-VON improves over SGD and q-VSGD, is competitive with IVON, and shows different generalization and calibration trade-offs across posterior shapes.

G.2 Image Classification with Neural Networks

  • Using the posterior predictive distribution generally boosts performance for q-VON and IVON on the CIFAR datasets.Source: §G.2 · p. 30
  • q-VON improves over q-VSGD and SGD on MNIST and CIFAR-10, while q-VON-MF is competitive with IVON on CIFAR-10 and CIFAR-100 without hyperparameter tuning.On MNIST, heavy-tailed q-VON has better generalization than IVON, while compact-support q-VON has better calibration.Source: §G.2 · p. 30
  • The q-VON algorithms and mean-field variants scale to large neural networks, with experiments using the same hyperparameters as IVON.The reported comparisons cover test accuracy, NLL, ECE, Brier score, and AUROC.Source: §G.2 · pp. 29–30
  • Heavy-tailed posteriors can improve generalization, whereas truncated posteriors can improve calibration on MNIST.Source: §G.2 · p. 30, Table 2 · p. 31

G.3 Empirical Verification of Convergence

In the 2D logistic-regression diagnostic, implemented updates move toward low-objective regions and stabilize near minima, although convergence guarantees for this general setting remain open. The q-sweep sections describe optimizer behavior and evaluation setup, but the supplied passages do not report their detailed outcomes.

G.3 Empirical Verification of Convergence

  • All tested trajectories move toward low-objective regions and stabilize near minima, but convergence analysis for BLR/VON-type methods in this generality remains largely open.The diagnostic uses optimizer-specific objectives for bounded-support, Gaussian, and Student-t variants, with full and mean-field implementations.Source: §G.3 · p. 31, Figure 3 · p. 32

H Varying q

  • q-VON retains a VON-style Newton-like mean update preconditioned by learned diagonal precision, using minibatch stochastic gradients, momentum, and Hessian-diagonal estimates.Iteration counts across q values are not directly comparable because q changes the objective, scale-precision map, and stochastic Hessian estimator.Source: H Varying q · p. 32

H.1 1D Logistic Regression

  • Figure 4’s synthetic 1D q-sweep reports average posterior means or variances over five seeds, with 95% confidence intervals and grid-based exact-posterior references.Each q-VON run uses 3,000 steps on a dataset of 256 observations.Source: §H.1 · p. 32, Figure 4 · p. 33

H.2 MNIST

Loading 2610.31308v1…