Source-linked AI summary
On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
Frank Nielsen
TL;DR
The paper addresses the lack of closed-form Jensen-Shannon divergence for some parametric families, especially Gaussian distributions. It introduces generalized JS divergences and JS-symmetrizations from abstract means, obtaining closed forms with geometric means for exponential families and harmonic means for scale Cauchy distributions.
Problem
Ordinary Jensen-Shannon divergence lacks a closed-form expression for generic exponential-family cases such as distinct Gaussian mixtures.
Method
The paper defines generalized (M, N)-Jensen-Shannon divergences and JS-symmetrizations of arbitrary distances using statistical mixtures derived from abstract means.
Results
The paper reports closed-form geometric Jensen-Shannon formulas for exponential families and a closed-form harmonic Jensen-Shannon formula for scale Cauchy distributions.
Takeaways & Limitations
Choosing an abstract mean according to the parametric family provides a route to closed-form Jensen-Shannon-type divergences and supports clustering applications.
Takeaways & Limitations
Matrix extensions face nonuniqueness because geometric matrix means are not uniquely defined, yielding different matrix Hellinger distances.
Abstract
from arXiv · showhide
The Jensen-Shannon divergence is a renown bounded symmetrization of the unbounded Kullback-Leibler divergence which measures the total Kullback-Leibler divergence to the average mixture distribution. However the Jensen-Shannon divergence between Gaussian distributions is not available in closed-form. To bypass this problem, we present a generalization of the Jensen-Shannon (JS) divergence using abstract means which yields closed-form expressions when the mean is chosen according to the parametric family of distributions. More generally, we define the JS-symmetrizations of any distance using generalized statistical mixtures derived from abstract means. In particular, we first show that the geometric mean is well-suited for exponential families, and report two closed-form formula for (i) the geometric Jensen-Shannon divergence between probability densities of the same exponential family, and (ii) the geometric JS-symmetrization of the reverse Kullback-Leibler divergence. As a second illustrating example, we show that the harmonic mean is well-suited for the scale Cauchy distributions, and report a closed-form formula for the harmonic Jensen-Shannon divergence between scale Cauchy distributions. We also define generalized Jensen-Shannon divergences between matrices (e.g., quantum Jensen-Shannon divergences) and consider clustering with respect to these novel Jensen-Shannon divergences.
1 Introduction and motivations
The paper motivates generalized Jensen-Shannon constructions by contrasting bounded Jensen-Shannon divergence with asymmetric, potentially unbounded Kullback-Leibler divergence and related symmetrizations. It then extends symmetrization to arbitrary distances using abstract means, targeting closed-form formulas for structured distribution families.
- Kullback-Leibler divergence is asymmetric, unbounded, and may be infinite, motivating bounded or symmetric alternatives.
- Jeffreys divergence symmetrizes Kullback-Leibler divergence but remains unbounded and can cause numerical sensitivity.
- Jensen-Shannon divergence measures divergence to an average distribution and is always bounded; its square root is a metric.
- The paper generalizes JS-symmetrization to arbitrary distances through abstract weighted means, recovering ordinary Jensen-Shannon divergence with arithmetic means.
- It selects means suited to distribution families, using geometric means for exponential families and harmonic means for scale Cauchy distributions.
2 Jensen-Shannon divergence in mixture and exponential families
For mixture families, Jensen-Shannon divergence admits a Jensen-Bregman representation, whereas generic exponential-family cases lack a practical closed form because ordinary mixtures generally leave the family. This limitation motivates generalized divergences based on abstract means.
- Mixture families use prescribed component distributions and include categorical distributions as a special case.
- 69 Theorem 1 gives a closed-form Jensen-Shannon expression for distributions in the same mixture family.
- For mixture families, the Jensen-Shannon divergence can be expressed as a Jensen-Bregman divergence for the negentropy generator.
- The mixture of two distinct Gaussian components is generally not Gaussian, so its ordinary Jensen-Shannon divergence lacks an obvious closed-form expression.
- The paper introduces generalized Jensen-Shannon divergences to obtain closed-form formulas by choosing abstract means matched to the relevant parametric family.
3 Generalized Jensen-Shannon divergences
This section generalizes Jensen-Shannon divergences by constructing statistical mixtures from abstract weighted means. It also establishes boundedness conditions and extends JS symmetrization to arbitrary distances.
- Abstract means and statistical mixtures: Abstract means generate statistical M-mixtures, which serve as the basis for generalized Jensen-Shannon divergences.The construction begins with weighted means and their normalized density interpolations.
- Abstract means and statistical mixtures: Quasi-arithmetic means unify common weighted means, including the geometric mean obtained with the logarithm generator.The arithmetic, geometric, and harmonic means are treated within this broader framework.
- Generalized Jensen-Shannon divergence: The M-Jensen-Shannon divergence replaces the usual arithmetic mixture with an M-mixture, recovering the ordinary JSD when M is arithmetic.The framework also extends from two-component to k-component mixtures.
- Generalized JS symmetrizations: The same construction defines M-JS symmetrizations for arbitrary distances, including generalized K-divergences and generalized Jeffreys divergences.These definitions combine distance evaluations through weighted means rather than restricting the construction to Kullback-Leibler divergence.
- Boundedness: M-JS divergences are bounded by log ZM, while the arithmetic case recovers the α-skew bound −log(1 −α), including log 2 at α = 1/2.The normalization factor is controlled using total variation, with ZM α(p, q) ∈[0, 2].
- Family-adapted means: Choosing the abstract mean according to the distribution family is proposed as a route to closed-form statistical distances.The paper motivates this choice because ordinary Gaussian mixtures are generally not Gaussian, obstructing closed-form JSD expressions.
4 Some closed-form formula for the M-Jensen-Shannon divergences
The paper chooses abstract means that preserve parametric families under generalized mixing, yielding closed-form Jensen-Shannon divergences and related symmetrizations. It develops geometric formulas for exponential families, including Gaussians, and a harmonic formula for scale Cauchy distributions, alongside clustering and matrix-distance applications.
- Abstract means are chosen to match parametric families, so generalized M-Jensen-Shannon divergences can admit closed-form formulas.The section focuses on geometric means for exponential families and harmonic means for the Cauchy scale family.
- 4.1 The geometric Jensen-Shannon divergence: G-JSD: The weighted geometric mean Gα(x, y) = x1−αyα is used for exponential families because normalized weighted products remain in the same family.Convexity of the natural parameter space ensures the geometrically mixed distribution remains an exponential-family member.
- 4.1.1 Case study: The multivariate Gaussian family: For multivariate Gaussians, the exponential-family representation converts KL into Bregman divergence and yields explicit G-JS formulas for KL and reverse KL.The construction uses composite vector-and-matrix parameters and the Gaussian log-normalizer.
- 4.1 The geometric Jensen-Shannon divergence: G-JSD: The geometric JS-symmetrization of reverse KL equals a Jensen/Burbea-Rao divergence between the corresponding natural parameters.The reverse divergence swaps the calling arguments, producing a distinct JS construction from the ordinary one.
- 4.1.2 Applications to k-means clustering: The generalized divergences are applied to generalized k-means clustering, while matrix extensions face non-unique geometric matrix means and multiple Hellinger-distance notions.The generalized k-means objective is NP-hard to optimize exactly.
- 4.3 The skewed Bhattacharyya distance interpreted as a geometric Jensen-Shannon symmetrization (G-JS): The skewed Bhattacharyya distance is identified as a geometric JS divergence for reverse KL, with a unique maximizing exponent α∗ called the Chernoff information.At α∗, KL(pα∗: p1) = KL(pα∗: p0).
- 4.4 The harmonic Jensen-Shannon divergence (H-JS): The harmonic mean is well-suited to the Cauchy scale family and gives a closed-form harmonic Jensen-Shannon divergence.KL divergence within the same scale family is scale-invariant, supporting the scale-family treatment.
5 Conclusion and perspectives
The paper generalizes Jensen-Shannon divergences and JS-symmetrizations through abstract means, deriving closed forms for exponential and Cauchy scale families and extending the framework to other distances and applications.
- The (M, N)-Jensen-Shannon divergences use abstract means for statistical mixtures and for symmetrizing asymmetric Kullback-Leibler divergence.
- Arithmetic and geometric weighted means yield closed-form M-Jensen-Shannon divergences for mixture and exponential families, respectively.
- The paper reports closed-form geometric Jensen-Shannon formulas for exponential-family densities and a harmonic Jensen-Shannon formula for scale Cauchy distributions.
- The framework extends JS-symmetrization to any base distance D, including skew N-Jeffreys and skew (M, N)-JS constructions.
- The geometric Jensen-Shannon divergence has applications in machine learning, while the paper also discusses clustering and extensions involving matrix divergences.
A Summary of distances and their notations
This summary defines the notation for generalized means, statistical distances, divergences, and their Jensen-Shannon, Jeffreys, and Bregman symmetrizations.
- The arithmetic and geometric means are Aα(x, y) = (1 − α)x + αy and Gα(x, y) = x^(1−α)y^α.
- A quasi-arithmetic mean is Mα(x, y) = f^−1((1 − α)f(x) + αf(y)) for strictly monotone f.
- The reverse distance is D*(p : q) = D(q : p), and the Jeffreys divergence combines forward and reverse Kullback-Leibler divergences.
- The skew Bhattacharyya divergence uses the logarithm of an α-weighted product integral, while the Mahalanobis distance is listed separately among statistical distances.
- The skew Jeffreys-Bregman divergence averages two directed Bregman divergences, and the skew Jensen divergence is defined alongside it.
- Generalized JS notation includes skew JS-symmetrization, skew M-Jensen-Shannon divergence, N-Jeffreys divergence, and skew (M, N)-JS divergence.
B Symbolic calculations in Maxima
The appendix uses Maxima code to calculate the normalizer for harmonic mixtures of Cauchy distributions.
- The Maxima program calculates the normalizer Z for harmonic H-mixtures of Cauchy distributions.
- The code defines the Cauchy density, harmonic mean, positive scale parameters, and positive α before integration.
- The normalizer is obtained by integrating the harmonic-mixture expression over x from −∞ to ∞.