Source-linked AI summary
A unified framework for information integration based on information geometry
Masafumi Oizumi, Naotsugu Tsuchiya, Shun-ichi Amari
TL;DR
The paper addresses how to quantify and relate measures of spatio-temporal interaction in stochastic dynamical systems. It uses information-geometric KL divergence between full and disconnected models, deriving geometric integrated information whose value is bounded by mutual information.
Problem
The paper seeks a unified way to quantify spatio-temporal interactions and clarify relationships among existing information-theoretic measures.
Source: INTRODUCTION · p. 1Method
It measures interaction by the minimized KL divergence between a full probability model and a constrained model with selected transmission branches disconnected.
Source: INTRODUCTION · p. 1Results
Geometric integrated information ΦG quantifies causal interactions between parts and satisfies 0 ≤ ΦG ≤ I(X;Y), while stochastic interaction also includes equal-time interactions.
Source: INTRODUCTION · p. 2, Integrated information · p. 3, RELATION TO GRANGER CAUSALITY · p. 4Takeaways & Limitations
The framework gives geometric interpretations of interaction measures and a hierarchical view that clarifies their relationships.
Source: INTRODUCTION · pp. 1–2, HIERARCHICAL STRUCTURE · p. 5
Abstract
from arXiv · showhide
We propose a unified theoretical framework for quantifying spatio-temporal interactions in a stochastic dynamical system based on information geometry. In the proposed framework, the degree of interactions is quantified by the divergence between the actual probability distribution of the system and a constrained probability distribution where the interactions of interest are disconnected. This framework provides novel geometric interpretations of various information theoretic measures of interactions, such as mutual information, transfer entropy, and stochastic interaction in terms of how interactions are disconnected. The framework therefore provides an intuitive understanding of the relationships between the various quantities. By extending the concept of transfer entropy, we propose a novel measure of integrated information which measures causal interactions between parts of a system. Integrated information quantifies the extent to which the whole is more than the sum of the parts and can be potentially used as a biological measure of the levels of consciousness.
INTRODUCTION
The paper frames interaction measures as information-geometric distances between full and constrained disconnected models, then uses this framework to interpret existing measures and introduce geometric integrated information.
INTRODUCTION
- Integrated information is designed to measure causal interactions among system parts and the information integrated in a system; IIT postulates that it corresponds to consciousness levels.The passage notes experimental support from studies of different types of loss of consciousness.Source: INTRODUCTION · p. 1
- The framework quantifies interactions through the minimized KL divergence from a full model to a constrained disconnected model.The minimized divergence is interpreted as information loss caused by disconnecting transmission branches.Source: INTRODUCTION · p. 1
- The framework gives mutual information, transfer entropy, integrated information, and stochastic interaction geometric interpretations based on which interactions are disconnected.It also arranges measures hierarchically to clarify their relationships.Source: INTRODUCTION · pp. 1–2
- Geometric integrated information ΦG extends transfer entropy to quantify causal interactions between system parts and is non-negative and no greater than mutual information.Stochastic interaction can exceed this bound because it also quantifies equal-time interactions in the present states.Source: INTRODUCTION · p. 2
SPATIO-TEMPORAL INTERACTIONS
The paper distinguishes interactions within a time point from interactions across time, and uses “causal” for the latter in a statistical, not necessarily physical, sense.
SPATIO-TEMPORAL INTERACTIONS
- The past and present system states are represented by X and Y, respectively, with their joint distribution characterizing spatio-temporal interactions.The state vectors comprise the elements of the system at each time.Source: SPATIO-TEMPORAL INTERACTIONS · p. 2
- Equal-time interactions occur between elements at the same time and can be quantified from p(X) or p(Y).Source: SPATIO-TEMPORAL INTERACTIONS · p. 2
- Time-lagged interactions are quantified from p(Y|X) and are called causal in the statistical-inference sense, not necessarily as physical causation.The passage qualifies this as inference from a limited amount of observation.Source: SPATIO-TEMPORAL INTERACTIONS · p. 2
Mutual information: Total causal interactions
The section interprets mutual information as the total causal interaction between past and present states: it is the information loss when X and Y are disconnected.
Mutual information: Total causal interactions
- The full distribution p(X,Y) is approximated by a disconnected model in which X and Y are independent, eliminating causal interactions between them.The constrained distributions form a manifold within the space of full models.Source: Mutual information: Total causal… · p. 2
- The KL divergence is minimized by orthogonally projecting p(X,Y) onto the independent-model manifold, with q(X)=p(X) and q(Y)=p(Y).The minimizing model preserves both actual marginal distributions.Source: Mutual information: Total causal… · p. 2
- The minimized divergence equals mutual information I(X;Y), interpreted as the total causal interactions between X and Y.Source: Mutual information: Total causal… · p. 2
Transfer entropy: Partial causal interactions
The section defines a partial causal interaction by disconnecting one past-to-present transmission while conditioning on the other past states; the minimized KL divergence equals conditional transfer entropy.
Transfer entropy: Partial causal interactions
- A partial interaction from x_i to y_j is disconnected by requiring them to be conditionally independent given the other past states, ˜X_i.Source: Transfer entropy: Partial causal… · p. 3
- The minimizing disconnected model preserves p(X), substitutes p(y_j|˜X_i) for p(y_j|X), and preserves the specified conditional distribution of the other present states.Source: Transfer entropy: Partial causal… · p. 3
- The minimized KL divergence equals conditional transfer entropy from x_i to y_j given ˜X_i, measuring that partial causal interaction.Source: Transfer entropy: Partial causal… · p. 3
Integrated information
Geometric integrated information measures the KL divergence from the actual distribution to a model that disconnects causal interactions across system parts. Its construction bounds it by mutual information, while summed conditional transfer entropies can exceed that total.
Integrated information
- Geometric integrated information ΦG is the minimized KL divergence between the actual distribution and a disconnected model that removes causal interactions across parts.The model is constrained so each part's present states depend only on its own past states.Source: Integrated information · p. 3
- ΦG is always less than or equal to mutual information because the mutual-information constraint manifold is included in the larger ΦG manifold.Source: Integrated information · p. 3
- ΦG is non-negative and upper-bounded by the whole system's mutual information, satisfying a stated theoretical requirement for integrated information.Source: Integrated information · p. 3
- ΦG is not the sum of conditional transfer entropies, whose sum can exceed mutual information; for Gaussian variables, that sum equals causal density.The paper therefore says causal density does not satisfy the theoretical requirement for an appropriate integrated-information measure.Source: Integrated information · p. 3
Stochastic interactions
Stochastic interaction is the minimized KL divergence under constraints that decompose each part's present-state distribution given the system's past. Unlike geometric integrated information's time-lagged-only disconnection, it also breaks equal-time interactions and can exceed total causal interactions.
Stochastic interactions
- The stochastic-interaction constraints break equal-time interactions among present states given the whole past, whereas ΦG breaks only time-lagged interactions.Source: Stochastic interactions · p. 4
- Stochastic interaction SI(X; Y) equals the minimized KL divergence under constraints that factor each part's present-state distribution given the past.The minimizing distribution preserves the past marginal and each part's conditional distribution given its own past.Source: Stochastic interactions · pp. 3–4
- Because its constraint manifold does not include the mutual-information manifold, stochastic interaction can exceed total causal interactions and is unsuitable as an integrated-information measure.Source: Stochastic interactions · p. 4
RELATION TO GRANGER CAUSALITY
In the Gaussian case, the paper relates integrated information to multivariate Granger causality by comparing a full model with a disconnected model. The geometric constraints also show how integrated information relates to mutual information and stochastic interaction.
RELATION TO GRANGER CAUSALITY
- The constraints for ΦG set the off-diagonal elements of A′ to zero, while mutual information sets A′ to zero and stochastic interaction also constrains off-diagonal Σ(E)′.The figure depicts the mutual-information constraint as a line, the stochastic-interaction constraint as a line, and the integrated-information constraint as a plane.Source: RELATION TO GRANGER CAUSALITY · p. 4, Figure 3 · p. 4
- Because the mutual-information and stochastic-interaction manifolds are included in the integrated-information manifold, ΦG is smaller than I(X; Y) and SI(X; Y), while MI and MS have no inclusion relation.The Gaussian analysis compares these constraints as manifolds of probability distributions.Source: RELATION TO GRANGER CAUSALITY · p. 4
- In the Gaussian case, ΦG is the difference in prediction error between the full model and one with off-diagonal A′ set to zero.The prediction error is measured by generalized variance, which is used as a goodness-of-fit measure in multivariate Granger causality analysis.Source: RELATION TO GRANGER CAUSALITY · p. 4
HIERARCHICAL STRUCTURE
The disconnected models form a hierarchy for quantifying combinations of time-lagged interactions. Its ordering by KL divergence clarifies how interaction cuts relate to measures such as transfer entropy, integrated information, and mutual information.
HIERARCHICAL STRUCTURE
- For two units, the four time-lagged interactions are T11, T12, T21, and T22; transfer entropy and integrated information consider cross interactions T12 and T21, while self-interactions can also be quantified.Self-interactions can be represented by imposing constraints on the corresponding conditional distributions.Source: HIERARCHICAL STRUCTURE · p. 5
- Disconnected models form a partial order by KL divergence: models linked by adding or removing one interaction are connected, and breaking more interactions increases information loss.Models at the same hierarchy level have no ordering relation.Source: HIERARCHICAL STRUCTURE · p. 5
- At the top of the hierarchy, all four interactions are broken, maximizing information loss and yielding mutual information I(X; Y).The hierarchy generalizes the related measures and provides a systematic way to quantify combinations of spatio-temporal interactions.Source: HIERARCHICAL STRUCTURE · p. 5
CONCLUSION
The paper concludes that information geometry unifies interaction measures through KL divergence between full and disconnected models. It introduces integrated information ΦG and relates its Gaussian form to multivariate Granger causality.
CONCLUSION
- The framework interprets interaction measures, including transfer entropy, through KL divergence between full and disconnected models.It also provides a unified way to quantify spatio-temporal interactions in stochastic dynamical systems.Source: CONCLUSION · p. 5
- The proposed integrated-information measure satisfies 0 ≤ ΦG ≤ I(X; Y), the mutual information in the whole system.The paper presents this as an important theoretical requirement for a proper measure of integrated information [18].Source: CONCLUSION · p. 5
- In the Gaussian case, ΦG is closely related to multivariate Granger causality based on generalized variance.The paper expects the framework to be useful in diverse fields, including neuroscience, and ΦG to be useful to consciousness researchers.Source: CONCLUSION · p. 5