Source-linked AI summary
Extropy: Complementary Dual of Entropy
Frank Lad, Giuseppe Sanfilippo, Gianna Agrò
TL;DR
The paper addresses the unresolved conceptual status of Shannon’s refinement axiom and develops extropy as its complementary dual. It characterizes the duality across distributions and divergences, finding connections to Kullback–Leibler divergence, L2 distance, and sequential forecast scoring. The paper also states scope boundaries for continuous extropy and information measures.
Problem
Jaynes identified Shannon’s third axiom as mathematically important but conceptually unresolved in establishing entropy’s special status.
Method
The paper constructs and analyzes extropy as an alternative information measure linked to entropy through complementary mass functions and partitions.
Results
Extropy forms a complementary dual to entropy, with the analysis connecting relative extropy to Kullback–Leibler divergence and identifying half the L2 metric as an extropic dual.
Takeaways & Limitations
Entropy and extropy provide paired assessments whose relevance may extend to applications using entropic computations, including proper scoring rules and other information-theoretic settings.
Takeaways & Limitations
Continuous extropy is limited to L2 densities, alongside measurability concerns in continuous information measures.
Abstract
from arXiv · showhide
This article provides a completion to theories of information based on entropy, resolving a longstanding question in its axiomatization as proposed by Shannon and pursued by Jaynes. We show that Shannon's entropy function has a complementary dual function which we call "extropy." The entropy and the extropy of a binary distribution are identical. However, the measure bifurcates into a pair of distinct measures for any quantity that is not merely an event indicator. As with entropy, the maximum extropy distribution is also the uniform distribution, and both measures are invariant with respect to permutations of their mass functions. However, they behave quite differently in their assessments of the refinement of a distribution, the axiom which concerned Shannon and Jaynes. Their duality is specified via the relationship among the entropies and extropies of course and fine partitions. We also analyze the extropy function for densities, showing that relative extropy constitutes a dual to the Kullback-Leibler divergence, widely recognized as the continuous entropy measure. These results are unified within the general structure of Bregman divergences. In this context they identify half the $L_2$ metric as the extropic dual to the entropic directed distance. We describe a statistical application to the scoring of sequential forecast distributions which provoked the discovery.
1. SCOPE, MOTIVATION AND BACKGROUND
The paper addresses the unresolved conceptual status of Shannon’s refinement axiom by introducing extropy as a complementary dual to entropy. It develops the duality and connects it to information measures and proper scoring rules.
- Broader connections: The resulting duality connects extropy with the Kullback–Leibler divergence, Bregman divergences, and the standard L2 distance between densities.These connections extend the analysis from discrete distributions to continuous information measures.
- Contribution: For a discrete distribution pN, extropy is defined using the complementary probabilities 1 − pi and is interpreted as uncertainty in the distribution.Its duality with entropy arises through complementary event partitions and a transformed complementary mass function.
- Contribution: The entropy and extropy measures are linked to the complementary mass function qN = (N −1)−1(1N −pN), which normalizes probabilities of complementary events.This construction establishes extropy as the complementary dual of entropy.
- Motivation: Jaynes questioned whether Shannon’s third axiom provided a conceptually satisfactory foundation for entropy’s uniqueness.The paper presents this as a longstanding open issue in entropy’s axiomatization.
- Contribution: The paper introduces extropy as a distinct information measure produced by an alternative to Shannon’s third axiom.Extropy is denoted J(·) alongside entropy H(·).
- Statistical application: The paper motivates extropy partly through proper scoring rules, where completing the logarithmic score requires assessing negextropy in addition to negentropy.The expected logarithmic score equals −H(pN), while the proposed completion adds the corresponding extropic assessment.
2. THE CHARACTERIZATION OF EXTROPY
The paper characterizes extropy as entropy’s complementary dual, sharing continuity, permutation invariance, and uniform maximization while differing in refinement behavior.
- Definitions: Extropy is defined alongside entropy as a distributional uncertainty measure, using complementary event probabilities rather than the probabilities themselves.Its duality with entropy derives from symmetric relationships across crude event partitions.
- Basic characterization: For binary distributions, entropy and extropy coincide; for N ≥ 3 with at least three positive components, entropy exceeds extropy.The binary equality follows directly from the two complementary probabilities.
- Shared properties: The attainable (H,J) region expands with dimension but is nonconvex, with scalloped upper and lower boundaries and a shared binary flat edge.Each successive simplex adds a section to the previous range.
- Shared properties: Extropy satisfies Shannon’s continuity and dimension-monotonicity axioms, is permutation invariant, and is maximized by the uniform distribution.Its minimum occurs at simplex vertices, where J(e_i) = 0.
- Shared properties: Unlike entropy’s unbounded maximum log(N), extropy’s maximum is bounded by 1 and equals (N − 1)log[N/(N −1)] at the uniform distribution.The extropy maximum approaches 1 as N increases.
- Refinement: Under refinement [tp,(1−t)p,1−p], extropy gains a nonnegative component that varies nonlinearly with the refined probability, unlike entropy’s linear increase.Extropy rises more slowly for small p and more quickly for large p, equalizing with entropy at p = 1.
3. ISOENTROPY, ISOEXTROPY CONTOURS IN THE UNIT-SIMPLEX
For three possible outcomes, entropy and extropy form complementary contour patterns within the probability simplex. Their contours provide a geometric comparison of the two measures.
- For N = 3, Figure 3 compares constant-entropy contours with constant-extropy contours in the two-dimensional unit-simplex.
- The contours exhibit a geometric sense in which extropy and entropy are complementary.
- At H(p3) = 0.9028, Appendix B shows one isoentropy contour together with isoextropy contours that intersect it.
4. EXTROPY AS THE COMPLEMENTARY DUAL OF ENTROPY
Extropy is defined as entropy’s complementary dual through symmetric partition relationships and a complementary-distribution mapping. The mapping contracts the simplex toward the unique uniform fixed point, while the dual is not an involution.
- Symmetric equations relate the sums of a distribution’s entropy and extropy to the corresponding measures of its component probabilities.
- Extropy equals the difference between entropy summed over crude binary partitions and entropy on the finest partition, with entropy and extropy each representable through the other.
- The complementary mass function qN = (N −1)−1(1N −pN) represents a distribution of unlikeliness and makes extropy a rescaled entropy of qN rather than pN.
- The complementary mapping is a contraction into nested inscribed simplices, and the uniform distribution is its unique fixed point.
- The duality uses forward and backward images rather than a cycle: vertex triangles are not contraction images, so the dual is not an involution.
5. DIFFERENTIAL EXTROPY AND RELATIVE EXTROPY FOR CONTINUOUS DISTRIBUTIONS
The paper extends extropy from discrete distributions to relative and differential settings, treating it as a complement to entropy and Kullback–Leibler divergence. Bregman divergences unify the paired constructions and connect extropy with half the squared L2 distance.
- For continuous densities, the paper develops differential extropy through relative entropy and general Bregman functions, with a uniform dominating measure underlying the integral form.
- Relative extropy is defined as a function complementary to the Kullback–Leibler divergence.
- When sN is uniform, relative entropy and extropy reduce to rescaled discrete entropy and extropy measures.
- The relative extropy of pN relative to sN equals an extropy difference adjusted by expectations involving the component of tN complementary to sN.
- Relative entropy and relative extropy are complementary Bregman divergences associated with Φ(pN) = −H(pN) and Φc(pN) = −J(pN).
- Half the usual squared Euclidean distance between pN and sN is identified as the extropic dual to the entropic directed distance.
- For a density f on [x1,xN] with uniform density u, the relative entropy–extropy pair identifies differential entropy and extropy forms.
6. STATISTICAL APPLICATION TO PROPER SCORING RULES
The paper identifies a limitation of the logarithmic scoring rule: it evaluates only the probability of the observed outcome, despite complementary nonoccurrence probabilities also being observed. It completes the assessment with extropy-based scoring components and applies them to forecast distributions.
- The logarithmic score is a proper rule based only on the probability assigned to the observed outcome, leaving probabilities of unobserved possibilities unassessed.Its expected value equals the distribution’s negentropy.
- Because observing X = xo also entails observing X ≠ xi for every other outcome, the total logarithmic score incorporates these complementary probabilities.The added terms account for nonoccurrence probabilities inherent in the asserted distribution.
- The expected total logarithmic score equals the sum of negentropy and negextropy.
- Each component of the total logarithmic score, and every positive linear combination of the two components, is a proper scoring rule.
- In stock-price forecasting, the two score components assess expected prices and tail-area probabilities differently across distributions with differing attitudes toward extreme events.The application was motivated by the importance of evaluating probabilities for rarely observed extreme events.
7. CONCLUDING DISCUSSION
The conclusion frames extropy as a complementary measure to entropy and discusses its relevance for discrete information and possible applications. It also identifies a scope limitation for continuous extropy.
- Extropy is presented as an exterior measure of nonoccurrence probabilities, complementary to entropy’s measure of the occurring outcome.Together, entropy and extropy jointly assess information in a probability system.
- The paper argues that continuous extropy is limited to L2 densities.
- The authors state that statistical measurements are practically finite and discrete, while continuous mathematics remains useful for approximate computations.
- Extropic calculations could complement established entropic analyses in applications such as astronomical heat-distribution measurements.The paper presents this as an area where the duality would be worth investigating.
APPENDIX A: ENTROPY ≥EXTROPY
Appendix A proves that entropy and extropy coincide for binary distributions but entropy exceeds extropy when at least three probability components are positive. The proof uses properties of an auxiliary function and permutation invariance.
- The entropy–extropy difference is invariant under permutations of the probability components.
- For distributions containing a zero mass, deleting that component preserves both entropy and extropy.
- H(X) = J(X) when N = 2, while H(pN) > J(pN) for N ≥ 3 with three or more positive components.The binary case reduces to the common expression −p1 log(p1) − (1 − p1)log(1 − p1).
- For N = 3, strict concavity yields u(p1) + u(p2) > u(p1 + p2), which implies H(X) − J(X) > 0.
APPENDIX B: THE RANGE OF EXTROPY VALUES THAT SHARE AN ENTROPY
Appendix B examines distributions sharing a fixed entropy and characterizes the range of their extropy values geometrically. For H(p3) = 0.9028, extreme extropy contours intersect the entropy contour at three points, while intermediate contours intersect it at six.
- H(p3) = 0.9028 defines the entropy contour used to study the range of compatible extropy values.
- The maximum and minimum extropy contours each intersect this entropy contour at three points, and each corresponding mass function has two equal components.
- A companion extropy value places an empirical distribution’s disorder within the possible extremes associated with its entropy.
- Every intermediate iso-extropy contour intersects the fixed entropy contour at six points.
- The three angle-bisecting lines partition the unit simplex into six symmetric permutation kernels.
RESULT 5
Result 5 describes extropy contours as transformed counterparts of entropy contours, with complementary mass functions linking their values and geometry.
- RESULT 5: H(1/4) = 0.7781 identifies apex points shared by specific isoentropy and isoextropy contours tangent to the triangular sub-simplex Sc.These contours appear on both sides of Figure 3.
- RESULT 5: The complementary mass function to p3 = (1/8) is q3 = (3/8), whose entropy level is H(q3) = 1.0822.The complementary contour relation determines the corresponding extropy level for p3.
- RESULT 5: The H(q3) = 1.0822 contour transforms into an isoextropy contour with J(p3) = 0.7781 by flipping and expanding the entropy contour.The transformation follows the relationship prescribed by Result 5.
- RESULT 5: Starting from H(3/8) = 1.0822 instead yields the dual extropy contour J = 0.8033 containing J(3/8) = 0.8033.The paired contours are inscribed in the smaller sub-sub-simplex Scc.
- RESULT 5: The visualization completes the interpretation of extropy as the complementary dual of entropy.The contour construction connects the numerical values to the geometric duality.
D.1 Shannon’s Differential Entropy:
Shannon’s differential entropy is obtained by refining a discrete distribution over a fixed interval and taking a transformed limit as the grid spacing approaches zero.
- D.1 Shannon’s Differential Entropy:: For a refined grid, Δx = (xN − x1)/(N − 1) and f(xi) = pi/Δx relate discrete masses to density values.The interval endpoints remain fixed while additional points are inserted uniformly.
- D.1 Shannon’s Differential Entropy:: H(pN) is unbounded as N increases and Δx → 0, but subtracting the finite location shift log(Δx) yields a meaningful transformed expression.The relocated summand uses −Σf(xi)log(f(xi))Δx.
- D.1 Shannon’s Differential Entropy:: Differential entropy is defined as the limit of H(pN) + log Δx as Δx → 0.This definition is the continuous analogue of the discrete entropy measure.
- D.1 Shannon’s Differential Entropy:: Differential entropy lacks absolute meaning because its value depends on the coordinate system used to express the variable.Under transformation from X to Y, the measure changes according to the transformation’s Jacobian.
- D.1 Shannon’s Differential Entropy:: Relative entropy avoids the invariance problem that affects differential entropy and remains suitable for comparing uncertainties between densities.The passage contrasts coordinate-dependent absolute values with comparative use.
D.2 Motivating the Differential Extropy Measure as −1
Differential extropy is motivated by expanding the discrete extropy expression for small probability masses and taking its continuous-density limit.
- D.2 Motivating the Differential Extropy Measure as −1: The expression −Σ(1 − pi)log(1 − pi) appears problematic if discrete masses are directly replaced by density values, since densities may exceed 1.A Maclaurin expansion resolves this issue under refinement.
- D.2 Motivating the Differential Extropy Measure as −1: As Δx → 0 and max pi → 0, the expansion makes discrete extropy closely approximate a quadratic expression in the probability masses.The approximation is derived by summing the expansion terms over the refined distribution.
- D.2 Motivating the Differential Extropy Measure as −1: For large N, representing pi as f(xi)Δx converts the extropy approximation into a location and scale transformation of −1/2 Σf^2(xi)Δx.The continuous analogue is therefore quadratic in the density.
- D.2 Motivating the Differential Extropy Measure as −1: Differential extropy is defined through the limit of J(pN) under the same refinement framework used for differential entropy.Definition D.2 formalizes the continuous-density measure.
- D.2 Motivating the Differential Extropy Measure as −1: The sum of squared probability masses is known as the repeat rate and was previously treated as an alternative uncertainty measure.The paper relates a rescaled version of this index to extropy when the maximum mass is small.
- D.2 Motivating the Differential Extropy Measure as −1: Half the negative expected value of a density function value is identified as the continuous differential analogue of extropy.This connects the density formulation to the proposed extropy measure.