Source-linked AI summary
Quantifying synergistic mutual information
Virgil Griffith, Christof Koch
TL;DR
The paper addresses how to quantify cooperation among predictors in predicting a single target, where prior synergy measures lack consensus. It reviews three measures and introduces SVK as the difference between the whole and the union of its parts. Across binary-circuit examples, the paper argues that this measure captures intuitive synergy and shows that independent predictors can have positive redundant information.
Problem
Existing information-theoretic synergy measures lack consensus about which measure is most valid for quantifying predictor cooperation.
Method
The paper reviews three prior measures and introduces SVK, defined from the difference between the whole and the union of its parts, alongside the candidate union-information measure IVK.
Results
The paper argues that defining synergy as the whole minus the union avoids double-subtracting redundancies and captures intuitive synergy across its examples.
Takeaways & Limitations
For SVK, independent predictors can convey redundant information about a target, and a target state can contain redundant, unique, and synergistic information simultaneously.
Takeaways & Limitations
WholeMinusSum underestimates intuitive synergy for all n, with a potentially looser lower bound as n increases.
Abstract
from arXiv · showhide
Quantifying cooperation or synergy among random variables in predicting a single target random variable is an important problem in many complex systems. We review three prior information-theoretic measures of synergy and introduce a novel synergy measure defined as the difference between the whole and the union of its parts. We apply all four measures against a suite of binary circuits to demonstrate that our measure alone quantifies the intuitive concept of synergy across all examples. We show that for our measure of synergy that independent predictors can have positive redundant information.
1 Introduction
The paper frames synergistic information as cooperation among predictors to reduce uncertainty about one target, addressing the lack of consensus among existing synergy measures. It defines synergy as how much the whole exceeds the union of its atomic elements and motivates applications in genetics, neuroscience, and complex-system modules.
- Several information-theoretic measures have been proposed, but no consensus exists about which measure is most valid.
- Synergy measures cooperation among a set of predictors in reducing uncertainty about a single target variable.
- Applications include gene interactions underlying phenotypic traits, presynaptic influences on neuronal firing, and discovery of informationally synergistic modules.
- The paper defines synergy as how much the whole exceeds the union of its atomic elements.
- The paper reviews prior Partial Information Decomposition approaches and introduces a synergy measure that also yields a union-information measure.
2 Information can be redundant, unique, or synergistic
PI-diagrams partition mutual information into redundant, unique, and synergistic regions. The examples show how these categories differ: redundancy is shared information, uniqueness belongs to one predictor, and synergy is information specified only jointly.
- Redundant information is carried by a singleton predictor and available somewhere else, whereas unique information is available only from exactly one singleton predictor.
- Synergistic information is any mutual information not carried by a singleton predictor, including information specified only by multi-predictor coalitions.
- A single target state can simultaneously contain positive redundant, unique, and synergistic information.The paper identifies this coexistence as a surprising property demonstrated by Figure 9.
- In Rdn, predictors redundantly specify one bit; in Unq, each predictor uniquely specifies one bit; in Xor, the predictors jointly specify one bit that neither singleton specifies.For Xor, the absence of singleton information leaves the joint information as synergistic.
3 Two examples elucidating properties of synergy
The paper uses Xor-derived examples to test desired properties of synergy measures, especially invariance to duplicate predictors and the effect of adding a predictor. Duplicating a predictor changes the coalition structure while preserving the synergistic information.
- The examples are designed to develop intuition for desired synergy properties before comparing prior measures.
- Duplicating X1 as X3 in Xor makes both coalitions X1X2 and X3X2 specify the target.
- Mutual information remains invariant to duplicate predictors, motivating the analogous requirement that synergistic information also remain unchanged.
- XorLoses adds a singleton predictor that fully specifies Y, making the original X1X2 synergy intuitively redundant.
- In XorDuplicate, synergistic information remains 1 bit after duplication: I(X3X2 :Y ) = I(X1X2 :Y ) = H(Y ) = 1 bit.
4 Prior measures of synergy
The reviewed synergy measures differ in how they separate synergistic information from redundancy and unique information. Their examples expose distinct failure modes, especially for Smax, WholeMinusSum, and ∆I.
- Imax synergy: Smax (X : Y ): Smax defines synergy as the whole beyond the state-dependent maximum of its parts.It obeys 0 ≤ Smax(X1...n : Y ) ≤ I(X1...n :Y ) and is invariant to duplicate predictors.
- Imax synergy: Smax (X : Y ): Smax can miscategorize unique information as synergy, reporting 1 bit for two unique bits despite no synergy or redundancy.The overestimate occurs because the whole, 2 bits, exceeds the maximum individual contribution, 1 bit.
- WholeMinusSum synergy: WMS (X : Y ): WholeMinusSum is a signed measure that subtracts redundancy from synergy, reducing to conditional total correlation minus total correlation.For n > 2, it subtracts pairwise and higher-order redundancy multiple times.
- WholeMinusSum synergy: WMS (X : Y ): WholeMinusSum can equal zero when synergy and redundancy have equal magnitude, as in RdnXor, and generally underestimates synergy with a gap that can increase with n.RdnXor contains 1 bit of synergy and 1 bit of redundancy, yielding 1 − 1 = 0 bits under WMS.
- Correlational importance: ∆I (X; Y ): ∆I agrees with intuition in the reported examples but measures something different from synergistic information and does not upperbound it.For And, synergy lies in (0.189, 0.5] bits while ∆I = 0.104 bits; with a duplicate predictor, ∆I drops 63% to 0.038 bits.
5 Synergistic mutual information
The paper formalizes synergy as mutual information in the whole beyond the union of its parts, introducing a union-information construction to avoid redundancy’s double-counting. The resulting SVK measure is bounded between prior underestimating and overestimating measures, though its exact value is not analytically computable.
- Synergy occurs when the whole exceeds the union of its parts, because union avoids counting duplicated information multiple times.The paper replaces the informal “whole exceeds the sum” intuition with “whole exceeds the union.”
- SVK(X:Y) measures the mutual information in the whole beyond the union of elements {X1, . . . , Xn}.
- Union information is computed by noisifying Pr(X1...n|Y) while preserving correlations with singleton predictors.
- The minimization preserves each joint distribution Pr(Xi,Y), preventing singleton information from changing while defining union information.
- The exact union-information optimization lacks an analytic calculation, but an analytic upper bound tightens the And example’s synergy bounds.
- SVK is bounded by WholeMinusSum synergy, which underestimates intuitive synergy, and Smax, which overestimates it.
6 Properties of IVK
The proposed union-information measure IVK is characterized by axiomatic properties governing positivity, identity, self-redundancy, symmetry, monotonicity, and target monotonicity. The construction also preserves the same predictor-target distributions rather than merely equal mutual-information magnitudes.
- IVK preserves the same predictor-target distributions Pr*(Xi,Y) = Pr(Xi,Y), ensuring singleton predictors retain the same information content.The paper distinguishes preserving the distributions from preserving only the magnitudes I*(Xi:Y).
- IVK is invariant under reordering predictors under Weak Symmetry.
- IVK(X1,...,Xn:Y) ≤ IVK(X1,...,Xn,W:Y), with equality when W is informationally poorer than an existing predictor.
- IVK(X:Y) ≤ IVK(X:YZ) for all random variables Y and Z under Target Monotonicity.
- For n = 2, Weak Local Positivity requires the derived partial informations to be nonnegative.
- IVK(X1,...,Xn:X1...n) = H(X1...n) under Strong Identity.
7 Applying the measures to our examples
Across binary circuit examples, prior synergy measures each conflict with intuition in at least one case, whereas SVK agrees with the intended results in the reported examples. The comparisons expose distinct failures involving unique information, redundancy, and duplicate predictors.
- Smax reports 1 bit of synergy in Unq despite two bits of unique information and no synergy.
- All measures reach the expected 1 bit for XorDuplicate, while all account for the redundant coalition-singleton information in XorLoses and report 0 bits.
- WholeMinusSum reports 0 bits for RdnXor by calculating 1 − 1 = 0 bits, although the example contains one bit each of synergy and redundancy.
- 0.270 ≤ SVK ≤ 1/2 in And, while WMS = 0.189 bits and ∆I = 0.104 bits.
- The supplementary examples RdnUnqXor, AndDuplicate, and XorMultiCoal are presented as nonessential for understanding the paper.
- No prior measure consistently matches intuitive synergy even for n = 2.
- ∆I is unbounded by Shannon mutual information, underestimates the known lower bound in And, and often decreases when predictors are duplicated.
8 Conclusion
The paper defines synergy as how much the whole exceeds the union of its parts and introduces SVK to quantify this idea. Its examples show that redundant, unique, and synergistic information can coexist in one target state, while SVK has computational limitations and supports positive redundancy from independent predictors.
- Synergy is defined as how much the whole exceeds the union of its parts, avoiding the double-subtraction of redundancies.
- A single target state can simultaneously convey redundant, unique, and synergistic information.This is demonstrated with RdnXor and RdnUnqXor.
- SVK is introduced as a novel synergy measure, alongside IVK as a candidate union-information measure and its analytic upperbound.
- SVK is not easily computable because an analytic solution to the minimization in IVK is unavailable, so numerical optimization remains necessary.
- For SVK, independent predictors can convey redundant information about a target.
A.0.1 Example AndDuplicate
AndDuplicate adds a copy of X1 to And, preserving the original unique, redundant, and synergistic structures while duplicating some coalition information. The example shows that ΔI changes under duplicate predictors even though total mutual information and intuitive synergy do not.
- A.0.1 Example AndDuplicate: AndDuplicate adds a predictor identical to X1, testing how ΔI responds when unique, redundant, and synergistic information coexist.
- A.0.1 Example AndDuplicate: X2 remains unchanged, so its unique information is preserved from And to AndDuplicate.
- A.0.1 Example AndDuplicate: X1’s unique information becomes redundant between X1 and X3 because X3 is identical to X1.
- A.0.1 Example AndDuplicate: And’s synergy between X1 and X2 remains, while an identical synergy also appears between X3 and X2.
- A.0.1 Example AndDuplicate: 0.811 bits is the shared total mutual information for And and AndDuplicate, but ΔI changes from 0.104 bits to 0.038 bits.
C.0.2 Proof of (GP)
This proof establishes structural properties of the artificial distribution used to define IVK, including invariance under predictor reordering and an upper bound by total mutual information. It also connects these properties to duplicate-predictor invariance of SVK, conditional on the corresponding intersection-information property.
- IVK is invariant under reordering the predictors because the artificial distribution is symmetric under predictor ordering.
- IVK(X:Y) ≤ I(X1...n:Y), because the original joint distribution satisfies the minimizing distribution’s constraints.
- IVK is computed from a novel artificial joint distribution formed from the target marginal and conditional distributions while preserving singleton-predictor relationships.
- The construction preserves every singleton joint distribution, satisfying Pr∗(Xi,Y) = Pr(Xi,Y) for all i.
- If the intersection information satisfies property (M), duplicate predictors do not increase synergy, although the paper does not explicitly prove that property for IVK.
E.2 Proof of bounds of SVK(X:Y )
The proof establishes that SVK lies between WholeMinusSum synergy and Smax synergy. It derives these bounds by comparing IVK with constrained and maximized mutual-information quantities and using nonnegativity of conditional mutual information.
- WMS(X:Y) ≤ SVK(X:Y) ≤ Smax(X:Y).
- SVK is defined as total mutual information minus IVK, while Smax is total mutual information minus Imax.
- The upper bound SVK(X:Y) ≤ Smax(X:Y) follows by proving IVK(X:Y) ≥ Imax(X:Y).
- The proof uses the predictor with maximum state-dependent mutual information and the nonnegativity of the resulting Kullback–Leibler divergence.
- The lower bound WMS(X:Y) ≤ SVK(X:Y) is obtained by relaxing singleton constraints and adding their mutual-information contributions.