Source-linked AI summary
Measuring multivariate redundant information with pointwise common change in surprisal
Robin A. A. Ince
TL;DR
Quantifying redundant information remains difficult because predictors may share information about a target without carrying identical content. The paper introduces Iccs, derives it using game-theoretic constraints and a maximum-entropy distribution, and applies it within PID to separate redundant, unique, and synergistic information across example systems.
Problem
Redundancy is the information about a target S that is common to two or more predictors, but existing Imin can overestimate overlap when predictors carry different information.
Method
Iccs measures common changes in surprisal pointwise, using game-theoretic unique-information constraints to construct a maximum-entropy distribution.
Results
The paper uses Iccs within PID to decompose multivariate mutual information into redundant, unique, and synergistic contributions across example systems, including continuous Gaussian variables.
Takeaways & Limitations
Iccs provides an overlapping-information perspective in which predictor dependence can alter the balance between redundancy and unique information.
Takeaways & Limitations
Iccs does not generally preserve PID non-negativity: the authors argue that negative atoms are necessary when redundancy genuinely quantifies overlapping information content.
Abstract
from arXiv · showhide
The problem of how to properly quantify redundant information is an open question that has been the subject of much recent research. Redundant information refers to information about a target variable S that is common to two or more predictor variables Xi. It can be thought of as quantifying overlapping information content or similarities in the representation of S between the Xi. We present a new measure of redundancy which measures the common change in surprisal shared between variables at the local or pointwise level. We provide a game-theoretic operational definition of unique information, and use this to derive constraints which are used to obtain a maximum entropy distribution. Redundancy is then calculated from this maximum entropy distribution by counting only those local co-information terms which admit an unambiguous interpretation as redundant information. We show how this redundancy measure can be used within the framework of the Partial Information Decomposition (PID) to give an intuitive decomposition of the multivariate mutual information into redundant, unique and synergistic contributions. We compare our new measure to existing approaches over a range of example systems, including continuous Gaussian variables. Matlab code for the measure is provided, including all considered examples.
1 Introduction
The paper addresses how to quantify redundant information about a target when multiple predictors represent it, distinguishing shared, unique, and synergistic contributions. It introduces Iccs, a pointwise redundancy measure grounded in common changes in surprisal and develops it within PID.
- Motivation: Redundant information concerns content about a target S that is shared across predictor representations.The paper frames this as overlap in how predictors represent S.
- Motivation: The central challenge is distinguishing whether joint predictor observations add synergy or merely repeat information available from one predictor.This distinction determines whether predictors provide complementary or overlapping representations of S.
- Partial Information Decomposition: PID decomposes multivariate mutual information into redundant, unique, and synergistic contributions using a redundancy measure over predictor subsets.The target variable S is privileged, while the remaining variables form the predictor set.
- Limitations of Existing Measures: Existing Imin redundancy can overstate overlap because it measures the minimum information carried by variables without requiring that they carry the same information.This motivates alternative measures based on overlapping information content.
- Contribution: The paper presents Iccs, a redundancy measure based on common changes in surprisal at the local or pointwise level.It applies the measure within PID and demonstrates it across previously studied examples and continuous Gaussian variables.
2 Interaction Information (co-Information)
Interaction information expresses overlap among information quantities through entropy and mutual-information relations, with local terms describing changes in surprisal. However, its aggregate value conflates redundancy and synergy, making their separate interpretation difficult.
- Definitions: Entropy measures uncertainty in a probability distribution, while local information quantifies the change in surprisal associated with particular observed values.For a value x, surprisal is h(x) = −log2 p(x); lower-probability values have higher surprisal.
- Definitions: Mutual information can be represented as the overlap of two entropies by adding individual entropies and subtracting joint entropy.The corresponding local quantity is expressed as a difference in surprisal when one value is observed.
- Interaction Information: For predictors X and Y about target S, positive intersection represents net redundancy, whereas negative intersection represents net synergy.The sign convention is tied to the overlap of the two mutual-information terms.
- Interaction Information: Interaction information generalizes through alternating sums over subsets of predictors and has equivalent local formulations using pointwise information quantities.In three variables it is bounded by conditional mutual-information terms and is symmetric in its arguments.
- Interpretation: Decoding performance provides intuition for redundancy and synergy but is difficult to compare quantitatively across individual and joint predictor models.The relevant performance metrics may differ across X, Y, and the joint variable (X, Y).
- Interpretation: Interaction information contrasts joint-response information with individual-response information, but its interpretation is complicated by mixed local signs and ambiguous terms.Consequently, a single aggregate value can conflate synergistic and redundant contributions rather than separating them.
3 The Partial Information Decomposition
The Partial Information Decomposition represents multivariate mutual information as redundant, unique, and synergistic atoms on a redundancy lattice. The section reviews candidate redundancy measures, identifies conceptual problems with Imin and Ibroja, and introduces an independent identity property.
- The Partial Information Decomposition: PID decomposes multivariate mutual information about target S into non-negative redundant, unique, and synergistic information atoms organized on a redundancy lattice.Each lattice node corresponds to an antichain of predictor-variable sources and has redundancy and partial-information values.
- Axioms and alternative operational arguments: The authors propose an independent identity property and distinguish subset equality from monotonicity as conceptual contributions to PID axioms.They argue the game-theoretic perspective challenges the necessity of decision-theoretic constraints used in Ibroja.
- An example PID: RDNUNQXOR: In RDNUNQXOR, the intended PID contains one bit each of redundancy, unique information from each predictor, and synergy.The reported partial-information calculations include I∂(S;{1}) = 2 −1 = 1 and I∂(S;{12}) = 4 −1 −1 −1 = 1.
- Measuring redundancy with minimal specific information: Imin: Imin measures the minimum specific information about each target value conveyed by the considered sources, but does not require them to convey identical information.This can overstate redundancy relative to overlapping information content.
- Measuring redundancy with minimal specific information: Imin: The two-bit copy problem shows Imin assigns 1 bit of redundancy to independent predictors that provide different components of a copied target.X1 and X2 each provide one bit about S, but their information does not overlap.
- Measuring redundancy with maximised co-information: Ibroja: The section argues that co-information-based approaches conflate synergistic and redundant effects and can be incompatible with synergistic entropy.It also states that an associated redundancy measure may provide only a bound rather than the true value.
4 Measuring redundancy with pointwise common change in surprisal: Iccs
Iccs measures redundancy as common pointwise changes in surprisal, retaining only local co-information terms with an unambiguous redundant interpretation. It combines this local criterion with a maximum-entropy distribution and a game-theoretic account of unique information.
- Iccs derivation: Iccs defines redundancy as the expected pointwise change in surprisal common to the input sources.The common change may be positive or negative and is computed from local co-information terms.
- Iccs derivation: Iccs counts only terms where all source changes and their local co-information overlap share the same sign.This excludes synergistic, ambiguous, and mixed-sign interaction effects while retaining redundant information and redundant misinformation.
- Distribution construction: The measure uses a maximum-entropy joint distribution constrained by source-target marginals and the multivariate target marginal.The constraints are motivated through a game-theoretic operational definition of unique information.
- Example systems: For the redundant-information system, each predictor contributes 1 bit and the changes overlap completely, so Iccs equals co-information.All local information and co-information signs are positive in both configurations.
- Example systems: For the synergistic system, equal predictor values contribute independently while unequal values provide a joint 1-bit change with −1 bit of local co-information.The zero-overlap and negative-overlap cases distinguish independent and synergistic contributions.
- Game-theoretic unique information: The game-theoretic definition allows unique information when an agent gains an advantage in a fair simultaneous game with equal utility functions.The construction uses two agents, shared utility structure, and an asymmetric stake dependence while preventing information leakage through sequential play.
- Game-theoretic unique information: In REDUCEDOR, Ibroja reports zero unique information, whereas Iccs reports zero redundancy and unique information in both predictors.The difference arises because the actual system decouples predictor evidence, unlike the Ibroja-optimised distribution.
5 Two variable examples
Across two-variable examples, Iccs often assigns redundancy according to shared local surprisal changes, while distinguishing independent unique information and synergy that other measures can conflate.
- 0.195 bits from each predictor forms 0.39 bits of redundancy when both make the same local surprisal reduction.No other redundant local changes occur in this example.
- Ibroja can include synergistic effects when positive local predictor changes coexist with negative local co-information.The local changes are both 0.585, while local co-information is −0.415.
- For the ambiguous binary system, Iccs reports redundancy and unique information without synergy, whereas Imin reports both redundancy and synergy with no unique information from X1.
- For XOR, all three measures quantify the 1 bit of information as fully synergistic.
- For AND and OR, Iccs assigns less redundancy and unique information to both predictors, unlike Imin and Ibroja.The Iccs redundancy remains within the bounds proposed by Griffith and Koch.
- In SUM, Iccs treats each independently conveyed output distinction as unique, contrasting with Ibroja’s 0.5 bits of redundancy.The final full discrimination requires both inputs and contributes 0.5 bits of synergy.
- For RDNXOR, Iccs correctly identifies 1 bit of redundancy and 1 bit of synergy.
- Iccs also correctly separates redundant, unique, and synergistic components in the RDNUNQXOR construction.
6 Three variable examples
Three-variable examples show that Iccs reproduces intuitive decompositions for simple systems while permitting negative partial-information terms when mechanistic redundancy spans synergistic nodes.
- Three-variable PID comparisons exclude Ibroja because that measure is defined only for two input sources.
- In DBLXOR, the upper lattice nodes carry non-zero redundant information, while lower nodes are argued to have zero redundancy.
- DBLXOR produces a −1 partial-information term because three child nodes each contribute 1 while the top node contributes 2 bits.The negative term is interpreted as mechanistic redundancy among pairwise synergistic partial-information terms.
- For XORCOPY, Iccs matches the system’s intuitive redundancy structure but also yields a negative term caused by mechanistic redundancy between synergistic pairs.
- Iccs matches proposed intuitive PIDs for several additional XOR-based examples, including XORDUPLICATE, XORLOSES, XORMULTICOAL, and PARITYRDNRDN.
- The XORUNQ example combines pairwise synergy with an independently copied input component, and Iccs obtains the correct PID.
- In ANDDUPLICATE, partial-information values around 0.14 bits arise from local misinformation and ambiguous entropy terms.The corresponding entropy terms are H∂({1}{23}) = H∂({2}{13}) = 0.14.
7 Continuous Gaussian Variables
Iccs extends directly to continuous Gaussian variables and changes with predictor-predictor correlation, unlike Immi’s correlation-invariant redundancy in the examined families.
- Iccs applies directly to continuous variables by integrating local common surprisal changes, with support for univariate and multivariate Gaussian systems.
- The Gaussian systems are evaluated as functions of predictor-predictor correlation b under unit-variance assumptions.
- For equally target-correlated predictors, Immi reports constant redundancy and zero unique information regardless of predictor-predictor correlation.
- Iccs transitions from purely unique information under negative predictor correlation to purely redundant information under strong positive correlation.
- The Gaussian implementation enables PID estimation for experimental continuous data using Gaussian-copula mutual-information methods.Monte Carlo integration code for Gaussian variables is included.
8 Discussion
The discussion presents Iccs as a pointwise redundancy measure that yields intuitive PIDs across examples, while acknowledging negative terms, sampling challenges, and scalability limits for larger systems.
- Iccs defines redundancy as expected pointwise surprisal change shared by all input sources.
- The measure counts unambiguous pointwise overlap directly through surprisal additivity, avoiding ambiguities that can conflate redundancy with other interaction types.
- Iccs satisfies several core redundancy axioms, including symmetry, self-redundancy, modified identity, and subset equality, but is not monotonic on the redundancy lattice.
- Negative partial-information terms are treated as compatible with genuine overlapping information and appear across the reported example systems.
- Limited sampling bias and appropriate permutation-based statistical inference remain open practical issues for estimating PID quantities from experimental data.
- Four-variable PIDs pose a substantial interpretation challenge because their redundancy lattice contains 166 nodes.The discussion suggests collapsing terms with the same order structure as a more tractable representation.
- The framework also supports partial entropy decomposition, which can separately quantify source and mechanistic redundancy.
- A practical redundancy measure could provide insights into common mechanisms while addressing synergistic interactions beyond classical statistical approaches.