Source-linked AI summary
Statistical exponential families: A digest with flash cards
Frank Nielsen, Vincent Garcia
TL;DR
The document presents a concise reference for exponential-family distributions, covering their definitions, sufficient statistics, common decompositions, and geometric dualities. It organizes these models through log-normalizers, Bregman divergences, Fisher geometry, and dual affine coordinates, while including software support for processing mixtures.
Problem
Statistical practice needs a concise, maintainable reference for the widespread exponential-family distributions, their decompositions, and their associated geometric properties.
Method
The document summarizes exponential-family definitions and properties, catalogs common distribution formulas, and relates natural and expectation parameters through Legendre and Bregman dualities.
Results
Exponential-family manifolds are flat in dual θ/η coordinates, with canonical divergence represented as a Bregman divergence and linked to its Legendre-conjugate form.
Takeaways & Limitations
The resulting framework connects statistical distributions, Fisher-Rao geometry, dual affine information geometry, and soft Bregman clustering for expectation-maximization.
Abstract
from arXiv · showhide
This document describes concisely the ubiquitous class of exponential family distributions met in statistics. The first part recalls definitions and summarizes main properties and duality with Bregman divergences (all proofs are skipped). The second part lists decompositions and related formula of common exponential family distributions. We recall the Fisher-Rao-Riemannian geometries and the dual affine connection information geometries of statistical manifolds. It is intended to maintain and update this document and catalog by adding new distribution items.
A digest of exponential families
Exponential families organize statistical models around sufficient statistics, a log-normalizer, and canonical parameterizations. Their Legendre duality connects natural and expectation parameters to Bregman divergences, flat geometry, and common distribution formulas.
- Sufficient statistics: Sufficient statistics concentrate the information needed to recover or estimate model parameters from finite samples.The Fisher-Neyman factorization theorem identifies such statistics through the density decomposition.
- Definition and properties: Exponential families are characterized by a canonical decomposition with sufficient statistic t(x), log-normalizer F, and carrier measure k(x).The carrier measure is typically Lebesgue for continuous distributions or counting measure for discrete ones.
- Definition and properties: The family includes common continuous, discrete, categorical, and matrix-valued distributions, while uniform, Cauchy, and Lévy skew α-stable distributions are excluded.Examples include Gaussian, Poisson, multinomial, Gamma, Beta, Wishart, Dirichlet, Rayleigh, Weibull, and Pareto families.
- Definition and properties: The natural parameter space is an open convex set, and products remain exponential families whereas mixtures generally do not.The document also distinguishes univariate from multivariate families by observation-space dimension and family order by natural-parameter dimension.
- Dual parameterizations: Legendre transformation maps natural parameters θ to expectation parameters η = ∇F(θ), with inverse gradients supplied by the conjugate log-normalizer.Figure 2 summarizes the conversion between source, natural, and expectation parameterizations.
- Geometry and duality: The statistical manifold is flat with dual θ/η coordinates, and its canonical divergence equals a Bregman divergence generated by F.The same divergence also admits a Legendre-conjugate form and supports soft Bregman clustering for expectation-maximization.
Software library
The section presents jMEF as a Java library for processing mixtures of exponential families.
- jMEF is a Java library for processing mixtures of exponential families.
- The library implements hard, soft, and hierarchical techniques.
- Its techniques use sided and symmetrized Bregman divergences.
2 Univariate Gaussian distribution
The section identifies Kullback-Leibler divergence.
- Kullback-Leibler divergence is identified in the section.
- The section names Kullback-Leibler divergence without further elaboration.
- No additional property or formula is stated for Kullback-Leibler divergence here.
3 Univariate Gaussian distribution, σ2 fixed
The section identifies Kullback-Leibler divergence.
- Kullback-Leibler divergence is identified in the section.
- The section names Kullback-Leibler divergence without further elaboration.
- No additional property or formula is stated for Kullback-Leibler divergence here.
4 Multivariate Gaussian distribution
The section identifies Kullback-Leibler divergence.
- Kullback-Leibler divergence is identified in the section.
- The section names Kullback-Leibler divergence without further elaboration.
- No additional property or formula is stated for Kullback-Leibler divergence here.
5 Multivariate isotropic Gaussian distribution
The section includes Kullback-Leibler divergence as a topic associated with the multivariate isotropic Gaussian distribution.
- Kullback-Leibler divergence is identified in this section.
- The passage names the divergence without supplying its formula or derivation.
- No additional distribution-specific property is stated in the supplied passage.
6 Poisson distribution
The supplied passages identify Kullback-Leibler divergence in the Poisson-distribution section, without providing further readable detail.
- Kullback-Leibler divergence is identified in this section.
- The passages do not state a Poisson-specific formula or result.
- The same divergence label appears repeatedly across the supplied passages.
9 Binomial distribution, n fixed ∈N+
The supplied passages associate the fixed-n binomial section with Kullback-Leibler divergence and display expressions beginning with factorial terms.
- One displayed expression begins with f(x; n, p) = n!.
- The section includes Kullback-Leibler divergence.
- Another displayed expression begins with f(x1, · · · , xk; p1, · · · , pk, n) = n!.
- Kullback-Leibler divergence is also listed in two additional supplied passages.
12 Gamma distribution
The supplied passages identify Kullback-Leibler divergence and include a parameterized expression together with a gradient formula involving the digamma function Ψ.
- Kullback-Leibler divergence is identified in the Gamma-distribution section.
- A displayed expression contains B(αP ,βP ) and several Ψ terms involving αP, βP, αQ, and βQ.
- Kullback-Leibler divergence appears in another supplied passage for this section.
- The gradient is given as ∇F(Θ) = (Ψ(θ1 + 1) −Ψ(θ1 + θ2 + 2), Ψ(θ2 + 1) −Ψ(θ1 + θ2 + 2)).