Source-linked AI summary
Nonnegative Decomposition of Multivariate Information
Paul L. Williams, Randall D. Beer
TL;DR
Existing multivariate information measures, especially interaction information, do not provide a consistently nonnegative and interpretable account of information structure. The paper defines outcome-sensitive redundancy, uses it to construct a redundancy lattice and partial information decomposition, and shows that the resulting atoms are nonnegative while interaction-information negativity reflects mixed redundancy and synergy.
Problem
Interaction information, a widely used multivariate measure, can be negative, complicating its interpretation as an informational quantity.
Method
The paper defines redundancy as the outcome-averaged minimum information supplied by sources, then derives a redundancy lattice and partial information atoms from it.
Results
The partial information decomposition exhaustively decomposes Shannon information into atoms that are never negative and have clear informational interpretations.
Takeaways & Limitations
The analysis explains interaction-information negativity as a consequence of confounding redundant and synergistic interactions.
Abstract
from arXiv · showhide
Of the various attempts to generalize information theory to multiple variables, the most widely utilized, interaction information, suffers from the problem that it is sometimes negative. Here we reconsider from first principles the general structure of the information that a set of sources provides about a given variable. We begin with a new definition of redundancy as the minimum information that any source provides about each possible outcome of the variable, averaged over all possible outcomes. We then show how this measure of redundancy induces a lattice over sets of sources that clarifies the general structure of multivariate information. Finally, we use this redundancy lattice to propose a definition of partial information atoms that exhaustively decompose the Shannon information in a multivariate system in terms of the redundancy between synergies of subsets of the sources. Unlike interaction information, the atoms of our partial information decomposition are never negative and always support a clear interpretation as informational quantities. Our analysis also demonstrates how the negativity of interaction information can be explained by its confounding of redundancy and synergy.
I. INTRODUCTION
The paper addresses the limitations of existing multivariate information measures by introducing a nonnegative partial information decomposition grounded in a new redundancy measure and lattice structure.
- Total correlation measures dependency as a single quantity and does not reveal how multivariate information is distributed among variables.
- Interaction information characterizes higher-order interactions but is hindered by sometimes taking negative values for three or more variables.
- The paper defines redundancy as the minimum information any source provides about each outcome, averaged across outcomes.
- This redundancy measure induces a lattice and supports an exhaustive decomposition of Shannon information into partial information atoms.
- The resulting atoms are nonnegative and interpretable, while interaction-information negativity reflects confounding between redundancy and synergy.
II. THE STRUCTURE OF MULTIVARIATE INFORMATION
The paper decomposes information that sources provide about a target into unique, redundant, and synergistic contributions, illustrating how these components differ in simple systems.
- The goal is to decompose information that a source vector provides about a target into contributions from individual and joint subsets of sources.The framework is motivated by distinguishing information carried by individual neural responses from information carried by their combinations.
- For two sources, total information is measured by I(S; R1, R2) and can consist of unique information, redundancy, or synergy.
- Unique information is supplied by one source but not the other, whereas redundancy is overlapping information supplied by both sources.
- Synergy occurs when the combination provides information unavailable from either source alone, as in exclusive-OR coding.For S = R1 ⊕ R2, each source individually provides no information while the pair provides complete information.
- These components form the basic atoms of multivariate information, with unique information later treated as a degenerate form of redundancy or synergy.
III. MEASURING REDUNDANCY
The paper defines redundancy using outcome-specific information, simplifies its source domain by removing supersets, and organizes the resulting possibilities in a redundancy lattice.
- Specific information measures the information a source provides about an individual outcome, while mutual information averages this quantity over outcomes.
- Redundancy is defined as the expected minimum information that any source provides about each outcome of S.This preserves outcome-specific differences that average mutual information can miss.
- Imin is nonnegative, bounded above by every source’s mutual information, and equals each source’s information exactly when all sources provide identical outcome-specific information.
- When one source is a subset of another, redundancy reduces to the smaller source’s self-redundancy, so supersets can be removed from source collections.
- The reduced domain contains collections of nonempty sources in which no source is a superset of another.
- An ordering based on source inclusion produces a redundancy lattice whose higher elements provide at least as much redundant information as lower elements.
IV. PARTIAL INFORMATION DECOMPOSITION
The partial information decomposition uses a redundancy lattice and its Möbius-inverse PI-function to partition mutual information into nonnegative atoms representing redundancies among source synergies. For three and four variables, the resulting diagrams organize unique, redundant, and synergistic contributions while showing that interaction information can mix redundancy and synergy.
- Redundancy lattice and PI-function: The PI-function is defined as the Möbius inverse of the cumulative redundancy function Imin on the redundancy lattice.It quantifies information redundantly supplied by a source collection but not by any simpler collection below it.
- Redundancy lattice and PI-function: The resulting PI-atoms are nonnegative and can therefore be interpreted as informational quantities.The paper derives nonnegativity recursively from the lattice formulation.
- Three-variable decomposition: For three variables, the decomposition separates each source’s unique information, their redundancy, and their joint synergy.The redundancy term is Imin for the two sources, while unique information subtracts that redundancy from each source’s total information.
- Illustrative distribution: In the example distribution, the sources provide distinct outcome-specific information, shared uncertainty reduction, and joint information about whether S = 0 occurs.The sources have equal mutual information, but each determines a different outcome individually; together they determine the S = 0 event.
- Higher-dimensional decomposition: For four variables, the PI-diagram retains the three-variable atoms and adds redundancy and synergy involving all three sources.It includes unique regions, pairwise redundancy and synergy, three-way redundancy, and three-way synergy.
- Higher-dimensional decomposition: The redundancy lattice and PI-diagram are complementary representations of the same structure, with the diagram collapsing regions according to lattice ordering.The lattice orders redundant information, whereas the diagram displays its corresponding overlapping regions.
V. WHY INTERACTION INFORMATION IS SOMETIMES NEGATIVE
Interaction information can be negative or ambiguous because its PI-decomposition combines redundant and synergistic information with signed terms. This confounding becomes especially problematic for four variables, where purely synergistic and purely redundant systems can receive the same value.
- Interaction information is defined recursively for more than three variables using conditional interaction information.The measure remains symmetric across permutations of its arguments.
- For three variables, interaction information equals synergistic information minus redundant information, so positive values indicate synergy and negative values indicate redundancy.When both interactions coexist, the signed difference makes the interpretation ambiguous.
- In a three-variable example, interaction information can be negative even when synergistic interactions are present because redundancy exceeds synergy.Another example yields zero interaction information when redundant and synergistic information are balanced.
- For four variables, pure 3-parity synergy and pure redundancy both produce +1 bit of interaction information.The redundant system has each source copied from the target, while the synergistic system requires all three sources to determine it.
- The four-variable PI-decomposition adds highest-order synergy and redundancy while subtracting several lower-order redundant-synergy terms, including one term twice.This structure explains why the measure fails to distinguish purely synergistic from purely redundant systems and becomes harder to interpret in larger systems.
VI. DISCUSSION
The paper presents partial information decomposition as a framework for illuminating multivariate interaction structure, while noting that its number of terms grows rapidly for larger systems.
- The paper defines Imin, derives a redundancy lattice, and uses it to decompose mutual information into interpretable partial information contributions.The partial terms capture combinations of redundant and synergistic interactions among subsets of variables.
- The number of partial information terms grows rapidly for larger systems, creating an application challenge.
Appendix A: Measures of Specific Information
The appendix distinguishes response-specific and stimulus-specific measures of information and explains the outcome-specific measure used in the paper.
- Response-specific information measures the change in uncertainty about S when a response r is observed.Its weighted average gives mutual information, although the measure can be negative.
- Stimulus-specific information measures how a particular stimulus s tends to evoke responses informative about the stimulus ensemble.Unlike the response-specific measure, it weights responses by their information about the entire ensemble.
- The paper uses I(S = s; R), which measures the reduction in surprise about a particular stimulus gained from responses, averaged over responses associated with that stimulus.
Appendix B: Lattice Theory Definitions
The appendix introduces posets and lattice-theoretic concepts used to represent ordered structures. It defines chains, antichains, bounds, meets, joins, covering relations, and down-sets, illustrated through the power-set lattice.
- A poset is a set equipped with a reflexive, transitive, and antisymmetric binary relation.
- A lattice is a poset in which every pair of elements has both a greatest lower bound and a least upper bound.
- The power set ordered by inclusion is a canonical lattice represented by a Hasse diagram, whose edges encode covering relations.
- A chain consists of mutually comparable elements, whereas an antichain contains no distinct comparable elements.
- The top and bottom elements bound every element from above and below, while a down-set contains an element and everything below it.
Appendix C: A(R) and the Redundancy Lattice
The collection A(R) of nonempty source collections forms the basis of a redundancy lattice. Its size follows Dedekind numbers, and its lattice structure organizes redundant information.
- A(R) is the set of antichains on the lattice of nonempty subsets of R, excluding the empty set.
- For |R| = n − 1, the cardinality of A(R) is the (n − 1)-th Dedekind number, with values 1, 4, 18, 166, 7579, … for n = 2, 3, 4, 5, 6, ….
- The ordered set of source collections forms a redundancy lattice with defined meet and join operations.
Appendix D: Supporting Proofs
The appendix proves monotonicity and nonnegativity properties underlying the partial information decomposition. These results establish that the constructed informational quantities are well behaved on the relevant lattices.
- Outcome-specific information I(S = s; A) is nonnegative and increases monotonically when the source set A expands.
- The proof of outcome-specific monotonicity uses a conditional-information decomposition for nested source sets.
- Minimum information Imin increases monotonically on the redundancy lattice ⟨A(R), ≼⟩.
- The partial information atom ΠR is nonnegative, including the bottom element where it equals Imin.
Appendix E: Supplementary Figures
The supplementary figures visualize PI-diagrams and the term-by-term decomposition of interaction information. They distinguish added and subtracted partial-information regions across three- and four-variable cases.
- A four-variable PI-diagram represents individual-source information and regions for redundancies among multiple source subsets.
- The three-variable interaction-information decomposition combines one joint-information term with two subtracted single-source terms.
- The four-variable interaction-information decomposition alternates joint and lower-order terms, with some PI-regions subtracted twice.