Shared Information -- New Insights and Problems in Decomposing Information in Complex Systems

1 Introduction

• Neural responses can be redundant or synergistic, with synergy recoverable only from the joint responses of both cells.The paper also illustrates unique and shared information through visual pathways and agents with different areas of expertise.Source: §1 · pp. 1–2
• For multiple explanatory variables, total mutual information should be decomposed into interpretable shared, unique, and synergistic terms, but the number of terms needed is unclear.Williams and Beer proposed a partial information decomposition derived from intuitive properties.Source: §1 · p. 2
• The paper examines the two-variable case, additional properties, Williams and Beer’s Imin measure, and the difficulty of assigning plausible values in the three-variable XOR example.It also proposes a geometric picture and relates information decomposition to shared and common knowledge in game theory.Source: §1 · p. 2

2 The Case of Two Variables

• The two-variable decomposition divides I(S : X1X2) into shared information, unique information from each variable, and complementary information.The individual mutual informations are expressed as shared information plus the corresponding variable’s unique information.Source: §2 · p. 3
• The decomposition equations do not uniquely determine the four contributions; defining one term fixes two, while nonnegative induced values remain difficult to ensure.A proposed unique-information definition must also satisfy a consistency condition relating the two variables’ unique information and mutual information.Source: §2 · p. 4
• The paper treats unique information as a version of conditional information that excludes complementary information.The co-information is expressed as complementary information minus shared information.Source: §2 · pp. 3–4
• The paper identifies generalizing the two-variable decomposition to more explanatory variables as a second problem addressed by Williams and Beer’s proposal.Source: §2 · p. 4

3 Natural Properties of Shared Information and the Partial Information Lattice

• Williams and Beer define shared information I∩ over sets of variables, using it to construct a nonnegative decomposition of total mutual information [4].Source: §3 · p. 4
• The proposed properties include global positivity, weak symmetry, self-redundancy, and monotonicity; Williams and Beer proposed weak symmetry, self-redundancy, and monotonicity as axioms [4].Global positivity follows from the other properties according to Williams and Beer.Source: §3 · pp. 4–5
• The intrinsic mutual information from cryptography does not satisfy the consistency condition required for unique information in this decomposition.Source: §3 · p. 4
• Weak symmetry and monotonicity imply that shared information need only be defined on anti-chains, which form the partial information lattice.On this lattice, shared information is monotone; the paper focuses on two or three random variables.Source: §3 · p. 5
• Möbius inversion uniquely determines local decomposition terms from shared-information values, but those local terms may be negative.The two-variable lattice’s lowest node represents shared information, and its terms correspond to unique and shared contributions.Source: §3 · pp. 5–6

4 Further Natural Properties of Shared Information

• Williams and Beer’s properties do not uniquely determine shared information, particularly when adding arguments on the right or varying the left-hand target.The authors therefore propose further properties describing the role of the left-hand side.Source: §4 · p. 6
• Strong symmetry makes shared information symmetric in the target and explanatory sets, but conflicts with local positivity for three variables under the other properties.When the target is the set of all variables, monotonicity and strong symmetry allow omission of the first argument.Source: §4 · pp. 6–7
• Left monotonicity requires shared information about S not to exceed shared information about the joint target SS′.The authors motivate this by noting that information about S also reduces uncertainty about the joint outcome of S and S′.Source: §4 · p. 7
• The proposed left chain rule decomposes shared information about SS′ into information about S plus conditional shared information about S′; none of the discussed measures satisfies it.The conditional term averages over S, generalizing the mutual-information chain rule; most discussed measures also fail left monotonicity.Source: §4 · p. 7
• The identity property makes shared information among independent variables vanish, although the authors note that game-theoretic agents may still have shared knowledge.The paper distinguishes this shared-knowledge possibility from common knowledge elsewhere in its discussion.Source: §4 · p. 7

5 The Functions Imin and II

• Imin compares each predictor’s posterior for S with the prior, then averages using the joint distribution after minimizing across predictors.This construction is designed to quantify shared information by combining the prediction comparisons with a minimum over predictors.Source: §5 · p. 8
• II results from reversing Imin’s minimization and averaging order and satisfies the same stated properties, including local positivity.The authors note that the proof of local positivity for Imin also applies to II.Source: §5 · p. 8
• II compares absolute mutual-information values without checking whether different variables contain the same information, so the authors say it misses the intuition of shared information.They argue that any measure satisfying the listed properties is bounded above by II, including Imin.Source: §5 · p. 8
• II satisfies left monotonicity, but Imin does not; the authors also report that the discussed measure IHSP violates left monotonicity in the same example.The example illustrates the failure of Imin when the target is expanded to include S′.Source: §5 · pp. 8–9

6 The Case of Three Variables

• Under strong symmetry, the three-variable PI lattice leaves only the information shared by all three variables undetermined.Figure 2 depicts the lattice and its node redundancies under the assumption S = {X1, X2, X3}.Source: §6 · p. 9, Figure 2 · p. 9
• In the XOR example, X1 and X2 are independent binary variables and X3 = X1 ⊕ X2, yet Imin assigns the three variables one shared bit.The example is symmetric in X1, X2, and X3, and each variable carries at most one bit.Source: §6 · pp. 9, 11
• Strong symmetry, monotonicity, self-redundancy, and local positivity cannot all hold for a shared-information measure.The contradiction is demonstrated for the three-variable XOR example.Source: §6 · p. 11
• The contradiction follows because monotonicity and strong symmetry force the shared information of all three variables to vanish, while local positivity fails.The proof uses the vanishing pairwise mutual informations and bounds on the remaining terms.Source: §6 · p. 11
• Keeping strong symmetry and local positivity requires replacing monotonicity, but dropping its equality condition leaves PI-lattice values on arbitrary subset collections undetermined.The equality condition is what makes the PI lattice a natural domain for shared information.Source: §6 · p. 11
• In the XOR example, the three third-layer mutual informations each equal one bit, but their pairwise dependencies prevent simply adding them as shared information.The total redundancy is bounded above by I(S : X1X2X3) = 2, and assigning nonnegative local terms is not straightforward.Source: §6 · p. 12

7 A Geometric Picture of Shared Information

• The geometric approach addresses the lack of a known experimental way to extract shared information by representing the information itself as a family of distributions.The paper proposes two ways to quantify it, SIKL and SIlr; SIKL is globally positive, whereas SIlr can be negative depending on the construction.Source: §7 · pp. 12, 14
• SIKL guarantees global positivity, but SIlr can be negative; in the same example, SIlr violates monotonicity and SIKL synergy can become negative.The paper also derives a lower bound on shared information from non-negativity of conditional information.Source: §7 · pp. 14–15
• The construction seeks a symmetric, pointwise posterior representation that returns the posterior for one variable and becomes less informative as variables are added.The authors require symmetry, recovery of p(S|x1) when k = 1, and non-increasing KL divergence from the prior as variables are added.Source: §7 · p. 14
• The construction selects the least informative distribution in the convex hull of the posteriors, defined as the point closest to the prior under KL divergence.Convexity of KL divergence makes the minimum unique; the resulting distribution uses only information present in those posteriors.Source: §7 · p. 14
• The paper’s binary example shows that conflicting posteriors can yield the prior, while matching posteriors select p(S|X1 = 0), violating the stated monotonicity property.In the conflicting case, the constructed distribution equals the prior; when both posteriors favor the same state, the closest point is p(S|X1 = 0).Source: §7 · p. 15, Figure 4 · p. 15
• Unlike the geometric construction, IHSP approximates one variable’s posteriors using the other’s posterior family, breaking symmetry and restricting the definition to two variables.The paper also reports that IHSP violates left monotonicity.Source: §7 · p. 16

8 Game Theoretic Intuitions

• The example has no common knowledge, because iterating the shared-knowledge operator yields the empty set.Shared knowledge means every agent knows the event; common knowledge additionally requires this to be known iteratively.Source: §8 · p. 17
• Independent binary variables can have non-trivial shared knowledge: the agents jointly rule out one state, although neither knows the other’s state.In the example, X1 and X2 are independent and S contains copies of both variables.Source: §8 · pp. 17–18
• With independent, uniformly distributed inputs, both Imin and II equal 1 bit, implying no unique information under that interpretation despite the inputs’ independence.The paper notes that a reasonable shared-information value might be positive but below 1 bit, and suggests log(4/3) as a possibility.Source: §8 · p. 18
• Shared knowledge is not represented as a partition, unlike common knowledge, which corresponds to a partition and thus to a random variable after a probability measure is introduced.The paper says it is unclear how to formulate the distinction between shared and common information in information-theoretic terms.Source: §8 · p. 18

• Some intuitive desiderata conflict: a positive Williams–Beer PI decomposition violates strong symmetry, and the authors leave unresolved whether symmetry or the PI lattice should change.They cite the difficulty of assigning plausible values to the PI decomposition for the XOR example.Source: §9 · p. 19
• Imin can assign unreasonably large shared-information values because it confuses variables carrying the same information with variables carrying the same amount of information.Source: §9 · p. 19
• Imin is not monotone in its left argument: random variables can share less information about more.Source: §9 · p. 19
• The general decomposition of multivariate information remains open, and further progress requires a more precise, operational idea of shared information.Source: §9 · p. 19