Source-linked AI summary
Pointwise Partial Information Decomposition using the Specificity and Ambiguity Lattices
Conor Finn, Joseph T Lizier
TL;DR
Multivariate information decomposition lacks a generally satisfactory way to evaluate its partial-information atoms. This paper applies the redundancy-lattice approach pointwise to specificity and ambiguity, and reports that the recombined decomposition satisfies the target chain rule.
Problem
Existing redundancy measures have been criticized, and no generally accepted replacement fully compatible with PID had emerged.
Source: §I · p. 1, B. Partial Information Decomposition · p. 3Method
The paper applies redundancy-lattice axioms separately to pointwise specificity and ambiguity, then recombines their partial-information atoms.
Source: §VI · p. 19Results
Upon recombination, the measure rmin satisfies the target chain rule.
Source: §VI · p. 19Takeaways & Limitations
The authors regard the result as supporting their operational account of redundancy, while acknowledging that it does not satisfy the identity property.
Source: §VI · p. 19
Abstract
from arXiv · showhide
What are the distinct ways in which a set of predictor variables can provide information about a target variable? When does a variable provide unique information, when do variables share redundant information, and when do variables combine synergistically to provide complementary information? The redundancy lattice from the partial information decomposition of Williams and Beer provided a promising glimpse at the answer to these questions. However, this structure was constructed using a much criticised measure of redundant information, and despite sustained research, no completely satisfactory replacement measure has been proposed. In this paper, we take a different approach, applying the axiomatic derivation of the redundancy lattice to a single realisation from a set of discrete variables. To overcome the difficulty associated with signed pointwise mutual information, we apply this decomposition separately to the unsigned entropic components of pointwise mutual information which we refer to as the specificity and ambiguity. This yields a separate redundancy lattice for each component. Then based upon an operational interpretation of redundancy, we define measures of redundant specificity and ambiguity enabling us to evaluate the partial information atoms in each lattice. These atoms can be recombined to yield the sought-after multivariate information decomposition. We apply this framework to canonical examples from the literature and discuss the results and the various properties of the decomposition. In particular, the pointwise decomposition using specificity and ambiguity satisfies a chain rule over target variables, which provides new insights into the so-called two-bit-copy example.
I. INTRODUCTION
Information decomposition seeks to assign target information to unique, redundant, and synergistic contributions, but multivariate atoms require both a structure and a way to evaluate them. The introduction presents PID’s redundancy lattice and the limits of existing redundancy measures, then motivates a pointwise perspective on information.
B. Partial Information Decomposition
- Williams and Beer’s axioms yield a redundancy lattice of partially ordered atoms, but Imin was criticized for conflating identical information with equal amounts and lacking the target chain rule [1].Other proposed replacements also lack the target chain rule and general multivariate compatibility, and no generally accepted fully compatible measure had emerged.Source: B. Partial Information Decomposition · p. 3
I. INTRODUCTION
- Multivariate decomposition must organize partial-information atoms and evaluate them, with redundancy capturing the same information rather than merely the same amount and satisfying the target chain rule.The bivariate decomposition separates unique, redundant, and complementary information; extending it to arbitrary predictor sets requires a structure consistent with that decomposition.Source: §I · pp. 1–2, B. Partial Information Decomposition · p. 3
II. POINTWISE INFORMATION THEORY
- Pointwise entropy and mutual information describe individual events, and taking expectations recovers the usual average measures.Unlike average mutual information, pointwise mutual information can be negative when a predictor event lowers the target event’s probability relative to its prior.Source: §II · p. 4
A. Pointwise Information Decomposition
Pointwise decomposition assigns information atoms to individual realizations, but remains underdetermined until an atom is defined. The paper addresses signed pointwise mutual information by separating probability-mass exclusions into unsigned components for separate redundancy treatment.
A. Pointwise Information Decomposition
- The pointwise bivariate equations decompose joint and individual information into redundant, unique, and complementary atoms, but remain underdetermined without defining one atom.Taking expectations of the pointwise atoms recovers the average decomposition.Source: A. Pointwise Information Decomposition · pp. 4–5
B. Pointwise Unique
- In PwUnq, pointwise reasoning yields zero redundant information and 1/2 bit of average unique information from each predictor, unlike Imin, Ired, f UI, and SVK.One predictor provides full information while the other provides zero information in each realization.Source: B. Pointwise Unique · p. 5, Table IV · p. 5
C. Pointwise Partial Information Decomposition
- PPID applies redundancy axioms to source events, but signed pointwise mutual information makes monotonicity problematic; splitting it into unsigned components preserves overlap-based redundancy intuition.The approach treats informative and misinformative components separately.Source: C. Pointwise Partial Information… · pp. 5–6
III. PROBABILITY MASS EXCLUSIONS AND THE DIRECTED COMPONENTS OF POINTWISE MUTUAL INFORMATION
- For an event s, pointwise information tracks the change from prior p(t) to posterior p(t|s), described as excluding the joint probability mass associated with the complementary event.The remaining mass is normalized to obtain P(T|s).Source: §III · p. 6
A. Two Distinct Types of Probability Mass Exclusions
- Informative exclusions remove mass from p(tc), whereas misinformative exclusions remove mass from p(t); both types can occur together for one event.Pointwise mutual information increases with informative-exclusion size and decreases with misinformative-exclusion size.Source: A. Two Distinct Types of Probability… · pp. 6–7
B. The Directed Components of Pointwise Information: Specificity and Ambiguity
The paper separates pointwise information into positive specificity and negative ambiguity, then defines redundancy for each by comparing probability-mass exclusions. This yields operational measures based on the smallest relevant exclusions.
B. The Directed Components of Pointwise Information: Specificity and Ambiguity
- Postulates 1–4 uniquely decompose the pointwise information provided by s about t into entropic components, with the resulting forms unique up to the logarithm base in mutual information in Postulates 1 and 3.The decomposition does not directly correspond to separate components associated with informative and misinformative exclusion.Source: B. The Directed Components of… · p. 8
- Specificity depends on the predictor event, while ambiguity depends on both predictor and target events.Specificity reflects total excluded probability mass; ambiguity reflects uncertainty in the predictor given the target.Source: B. The Directed Components of… · p. 8
C. Operational Interpretation of Redundant Information
- The paper treats information as redundant when predictor events induce matching exclusions with respect to the realized target’s two-event partition.Informative and misinformative exclusions are compared separately, avoiding the difficulty of signed pointwise mutual information.Source: C. Operational Interpretation of… · p. 9
- Redundant specificity uses the source event with the smallest total exclusions, whereas redundant ambiguity uses the one with the smallest misinformative exclusion.The comparison is based on exclusion size, not on which elementary events in the target complement are excluded.Source: C. Operational Interpretation of… · p. 9
D. Motivational Example
A motivating example shows that equal pointwise mutual information can conceal different exclusion patterns, which the separate specificity and ambiguity measures distinguish. The paper then applies these components to lattices defined for each discrete realization.
D. Motivational Example
- Equal pointwise information can arise from different predictor-event exclusions, so mutual information alone obscures how information is provided.In the example, both events exclude target event t2, while one also excludes mass associated with t1 and t3.Source: D. Motivational Example · pp. 9–10
- The example assigns log 4/3 bit to redundant specificity and 1 bit each to the additional event’s unique specificity and unique ambiguity.The equal unique specificity and ambiguity offset, leaving the events no more informative on net.Source: D. Motivational Example · p. 10
IV. POINTWISE PARTIAL INFORMATION DECOMPOSITION USING SPECIFICITY AND AMBIGUITY
- Axioms 1–3 induce separate specificity and ambiguity lattices for every discrete realization of the joint distribution.The lattices have the same structure as the PID redundancy lattice, and Möbius inversion yields unique contributions at each source-event node.Source: §IV · p. 11, Figure 3 · p. 11
- The lattices organize decomposition, but evaluating their atoms requires separate definitions of redundant specificity and redundant ambiguity.The pointwise decompositions can then be used as the basis for multivariate information decomposition.Source: §IV · p. 11
A. Bivariate PPID using the Specificity and Ambiguity
The bivariate framework decomposes pointwise specificity and ambiguity separately into redundant, two unique, and complementary atoms. These components can be recombined to recover pointwise and average information decompositions.
A. Bivariate PPID using the Specificity and Ambiguity
- Pointwise specificity and ambiguity each split into redundant, two unique, and complementary atoms.The two decompositions share the bivariate structure but apply it to different entropic components.Source: A. Bivariate PPID using the… · pp. 11–12
- Subtracting ambiguity atoms from specificity atoms recovers the pointwise unique, redundant, and complementary information atoms.Taking expectations of specificity or ambiguity atoms instead gives their respective average partial-information atoms.Source: A. Bivariate PPID using the… · p. 12
- Because averaging and recombination are linear, either order recovers the average unique, redundant, and complementary information atoms.This procedure recovers the atoms from PID.Source: A. Bivariate PPID using the… · p. 12
B. Redundancy Measures on the Specificity and Ambiguity Lattices
The redundancy measures are constructed to meet lattice axioms, and their partial specificity and ambiguity atoms are nonnegative. Recombined information can nevertheless be negative, while satisfying a target chain rule.
B. Redundancy Measures on the Specificity and Ambiguity Lattices
- The proposed redundant specificity and ambiguity measures satisfy Axioms 1–4 and increase monotonically on the lattice.The corresponding partial specificity and ambiguity atoms are nonnegative.Source: B. Redundancy Measures on the… · p. 13
- Recombined average information atoms are not guaranteed nonnegative, and Rmin can be negative when sources redundantly misinform about a target event.This means Rmin fails local positivity and the target monotonicity property.Source: B. Redundancy Measures on the… · p. 13
- The pointwise redundant information rmin satisfies a chain rule over target realizations, as does its average Rmin.The rule also holds for pointwise and average partial-information atoms, but not for specificity or ambiguity alone.Source: B. Redundancy Measures on the… · p. 13
V. DISCUSSION
The discussion compares specificity-and-ambiguity PPID with existing measures and examines its decomposition of XOR. In XOR, the only nonzero partial-information atom is 1 bit of complementary information.
A. Comparison to Existing Measures
- Unlike Iccs and Hcs, which restrict or discard co-information-based overlap in difficult cases, this approach retains redundancy as overlapping exclusions [18, 19].Iccs interprets co-information as overlap only when the relevant pointwise terms have the same sign; Hcs does so only when co-information is positive.Source: A. Comparison to Existing Measures · p. 14
- Its event-level redundancy resembles Imin [1] and Immi, but is fully pointwise because it evaluates predictor events rather than whole variables [1].Imin is semi-pointwise: it considers information from whole predictor variables about a target event.Source: A. Comparison to Existing Measures · p. 14
- The approach also shares similarities with f_UI and SVK; their bivariate decompositions are equivalent.Axiom 4 is a pointwise adaptation of an f_UI assumption, and the paper identifies corresponding similarities with SVK.Source: A. Comparison to Existing Measures · p. 14
B. Probability Distribution Xor
- In XOR, each predictor’s 1-bit specificity and 1-bit ambiguity balance to zero pointwise mutual information despite inducing exclusions.The two predictor events induce the same exclusions, yielding 1 bit each of redundant specificity and redundant ambiguity per realisation.Source: B. Probability Distribution Xor · p. 15
- Every XOR realisation has 1 bit of complementary information as its only nonzero partial-information atom.The figure also reports 1 bit of average complementary information.Source: B. Probability Distribution Xor · p. 15, Figure 4 · p. 15
C. Probability Distribution PwUnq
The paper applies separate specificity and ambiguity lattices to pointwise information, recombines their atoms for multivariate decomposition, and obtains a target chain rule without the PwUnq pointwise-unique problem. Its examples also expose trade-offs involving negative average atoms, identity, and local positivity.
C. Probability Distribution PwUnq
- In PwUnq, recombination avoids the pointwise-unique problem; each realization has 1 bit of redundant specificity and ambiguity, with another 1 bit of unique specificity at the fully informative event.The caption identifies the top as probability-mass diagrams, the middle as per-realisation decompositions, and the bottom as the recombined decomposition.Source: C. Probability Distribution PwUnq · p. 15, Figure 5 · p. 16
D. Probability Distribution RdnErr
- −0.811 bit is S2's average unique information when its error probability is ε = 1/4; its unique ambiguity, without unique specificity, explains the negative value.S2 still provides 0.189 bit of information, including 1 bit of average redundant information.Source: D. Probability Distribution RdnErr · p. 16
E. Probability Distribution Tbc
- Tbc yields 1 bit of average redundant and 1 bit of average complementary information; independent predictors can be redundant when their exclusions match the two-event target partition.For composite T1,3, rmin/Rmin give consistent decompositions, unlike f UI, Ired and SVK; the target chain rule, identity property and local positivity cannot all hold.Source: E. Probability Distribution Tbc · pp. 17–18, Table V · p. 18
F. Summary of Key Properties
- The decomposition satisfies the target chain rule but not identity or target monotonicity; partial specificity and ambiguity atoms are locally nonnegative, while recombined PI atoms need not be.At least one source event has zero unique specificity and at least one has zero unique ambiguity; they need not be the same event.Source: F. Summary of Key Properties · p. 19
VI. CONCLUSION
- Separate specificity and ambiguity lattices are recombined for arbitrary predictor sets, yielding a decomposition that satisfies the target chain rule and avoids the PwUnq pointwise-unique problem.The authors note that their redundancy measure does not satisfy the identity property.Source: §VI · p. 19
Appendix A: Kelly Gambling, Axiom 4, and Tbc
The appendix uses Kelly gambling to interpret pointwise information and redundancy, then uses accumulator bets to motivate a target chain rule. Its Tbc example treats independent singleton information as redundant for predicting the realized outcome, while their joint use adds complementary information.
1. Pointwise Side Information and the Kelly Criterion
- Kelly’s optimal strategy allocates bets in proportion to outcome probabilities, and the increase in doubling rate from side information equals the mutual information between the message and race outcome.With fair odds, proportional betting returns the stake regardless of the winner; independent side information does not increase the doubling rate.Source: §1 · pp. 20–21
2. Justification of Axiom 4 and Redundant Information in Tbc
- Two gamblers with independent wires each double their stake using one bit of information, so the paper regards their equal predictive power as redundant despite distinct exclusions.The distribution of exclusions across losing horses does not affect pointwise mutual information for the horse that wins.Source: §2 · pp. 21–22
- Using both wires lets a third gambler quadruple the stake, making the additional bit available only jointly complementary information.Each singleton wire provides one bit; together they identify the winning horse and exclude the other three.Source: §2 · p. 22
3. Accumulator Betting and the Target Chain Rule
- Accumulator bets on component variables reach the same final returns as direct bets on the joint target, but through different information paths, motivating a target chain rule.The section argues that a decomposition should reflect those distinct paths while preserving the same result.Source: §3 · p. 22
Appendix B: Supporting Proofs and Further Details
Appendix B derives specificity and ambiguity lattices from source events and axioms, then uses Möbius inversion to obtain their partial atoms. The lattice structure alone does not evaluate those atoms: separate pointwise redundancy measures are required.
1. Deriving the Specificity and Ambiguity Lattices from Axioms 1–4
- The appendix derives a source-event collection that captures distinct ways of providing redundant information and orders it into a lattice with the same structure as the PID redundancy lattice.The collection excludes source-event supersets of other source events; the ordering organizes which redundancies include others.Source: §1 · p. 23
- Applying the redundancy construction separately to specificity and ambiguity yields two lattices, whose node contributions can be recovered by Möbius inversion.The resulting partial specificity and partial ambiguity also have a closed-form expression based on inclusion-exclusion.Source: §1 · pp. 23–24
- The lattices provide a decomposition structure but require separate definitions of pointwise redundant specificity and pointwise redundant ambiguity to evaluate its atoms.The appendix notes that these redundancy functions integrate the unique specificity or ambiguity contributed at lattice nodes.Source: §1 · pp. 23–24
2. Redundancy Measures on the Lattices
This section establishes properties of the proposed redundancy measures and their lattice atoms. Pointwise specificity and ambiguity atoms are nonnegative, but recombined and averaged partial-information atoms need not be.
2. Redundancy Measures on the Lattices
- Theorem 1 states that the definitions of r+ min satisfy Axioms 1–4; Theorem 2 reports that the redundancy measures r+ min increase monotonically.Source: §2 · pp. 24–25
- The partial specificity and partial ambiguity atoms evaluated with the proposed measures are nonnegative.The theorem applies to the atoms on their respective lattices.Source: §2 · p. 25
- Recombining and averaging the pointwise specificity and ambiguity atoms can produce negative partial-information atoms.The section establishes this possibility by a counterexample using RdnErr.Source: §2 · p. 25
3. Target Chain Rule
The proposed pointwise redundant information obeys a chain rule over target variables. The section also shows a trade-off: target chain rule, identity, and local positivity cannot all hold simultaneously.
3. Target Chain Rule
- Pointwise redundant information satisfies a chain rule when the target is split into component variables.The section also relates conditional redundant specificity and ambiguity across conditioned and joint targets.Source: §3 · p. 26
- The target chain rule, identity property, and local positivity cannot all be satisfied simultaneously.The section’s argument uses isomorphic target distributions to derive a contradiction under those combined requirements.Source: §3 · pp. 26–27
- In the XOR example, applying the target chain rule alongside the identity property requires a −1-bit conditional synergy, despite there being no ambiguity terms in the split pointwise information.The authors say this example is not a formal proof of incompatibility between identity and the target chain rule.Source: §3 · p. 27
1. Probability Distribution Tbep
The Tbep example yields both three-way redundant and pairwise complementary specificity. Unq distinguishes useful prediction from shared specificity, while And and Or have the same decomposition.
1. Probability Distribution Tbep
- Tbep has been used to show that no redundant-average-information measure for more than two predictors can satisfy the Williams–Beer axioms, identity, and local positivity simultaneously.The cited discussion presents it as a counterexample to consistency with PID for arbitrary numbers of predictors.Source: §1 · p. 27
- Tbep averages 1 bit of three-way redundant information and 1 bit of three-way redundant, pairwise complementary information.Each realization has 1 bit of three-way redundant specificity; the three predictor pairs also share 1 bit of pairwise complementary specificity.Source: §1 · p. 27, Figure 8 · p. 28
2. Probability Distribution Unq
- In Unq, S1's 1 bit is redundant with S2, which has −1 bit of unique misinformative information; 1 bit of complementary information balances unique ambiguity.The authors avoid calling this synergy because S1 alone enables perfect predictions of T.Source: §2 · p. 28, Figure 9 · p. 29
3. Probability Distribution And
- And and Or have the same decomposition because their target distributions are isomorphic.Figure 10 shows the And decomposition.Source: §3 · p. 29, Figure 10 · p. 29