A closed-form expression for the Sharma-Mittal entropy of exponential families

TL;DR

The paper addresses the need for closed-form Sharma–Mittal entropy and relative-entropy formulas for exponential families, alongside the challenge of estimating entropy when the underlying distribution is not explicit. It derives formulas for exponential families and specializes them to multivariate Gaussians; the same-family divergence has a closed form even when the carrier measure is nonzero.

• Problem

  Entropy estimation is difficult when the underlying distribution is not explicit, and k-nearest-neighbor methods suffer dimensionality costs and fall short at moderate dimensions.

  Source: p. 8

• Method

  The authors derive Sharma–Mittal entropy and same-family divergence formulas for exponential families, then specialize the entropy formula to multivariate Gaussians.

  Source: pp. 3, 6, 7

• Results

  Same-family Sharma–Mittal divergence is closed form even for k(x) ≠ 0; for k(x) = 0, limit cases yield Rényi, Tsallis, and Shannon entropies.

  Source: pp. 4, 7

• Takeaways & Limitations

  The formulas specialize explicitly to multivariate Gaussians, whose natural parameters can be estimated by maximum likelihood with a unique global optimum.

  Source: pp. 5, 8

  The differential Sharma–Mittal entropy of Gaussians may be negative.

  Source: p. 5