PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Measures of Integrated Information

Max Tegmark

arXiv:1601.02626v3q-bio.NC

TL;DR

Measuring integrated information in computational and cognitive systems is of growing interest, but practical evaluation is computationally demanding. The paper classifies Φ-measures and derives exact and approximate formulas. Its Gaussian formulas and graph-based approximation reduce computational costs, while Φ3.0 remains especially slow to evaluate.

  • Problem

    Evaluating integration measures in practice is computationally demanding, with Φ3.0 requiring costly numerical computation for each bipartition.

    Source: A. Which Φ-measures are best? · p. 19

  • Method

    The paper develops a taxonomy of Φ-measures and derives Gaussian formulas and a graph-theoretic approximation to reduce computation.

    Source: §VI · p. 19

  • Results

    For Gaussian systems, derived formulas are exponentially faster, and graph-theoretic approximation reduces the stated cost from doubly exponential to polynomial in system size.

    Source: §VI · p. 19

  • Takeaways & Limitations

    The formulas provide approaches intended for real-world laboratory data without unreasonable computational demands.

    Source: §VI · p. 19

    Φ3.0 is the slowest measure discussed; its per-bipartition computation grows faster than quadratically in the number of states and is infinite for continuous variables.

    Source: A. Which Φ-measures are best? · p. 19

Abstract

from arXiv · show

Although there is growing interest in measuring integrated information in computational and cognitive systems, current methods for doing so in practice are computationally unfeasible. Existing and novel integration measures are investigated and classified by various desirable properties. A simple taxonomy of Phi-measures is presented where they are each characterized by their choice of factorization method (5 options), choice of probability distributions to compare (3x4 options) and choice of measure for comparing probability distributions (7 options). When requiring the Phi-measures to satisfy a minimum of attractive properties, these hundreds of options reduce to a mere handful, some of which turn out to be identical. Useful exact and approximate formulas are derived that can be applied to real-world data from laboratory experiments without posing unreasonable computational demands.

I. INTRODUCTION

The paper addresses the computational difficulty of testing integration measures by classifying existing and novel Φ-measures and deriving formulas for practical data. It frames integration as the failure of a system’s transition dynamics to factor into independent subsystems, then organizes measures by their modeling choices.

I. INTRODUCTION

  • Testing IIT’s integration measure is hampered by computational costs that grow super-exponentially with system information content.The paper responds with a comprehensive classification of existing and novel measures and formulas for laboratory time-series data.Source: §I · p. 1

II. MEASURES OF INTEGRATION

  • Φ is defined by minimizing a partition-specific interaction measure φ over cuts, whose number grows super-exponentially with system size.The minimizing cut is called the “cruelest cut.”Source: §II · p. 2

A. Interpreting evolution as a Markov process

B. A taxonomy of integration measures

  • A φ-measure is specified by choices of factorization, distributions to compare, knowledge of the current state, and a distance measure.The paper organizes these choices in Tables III–V.Source: B. A taxonomy of integration measures · p. 4

C. Options for approximately factoring M

The paper presents five options for approximating the transition matrix by a product of subsystem matrices. One option uses noising, treating the other subsystem’s input as random noise.

C. Options for approximately factoring M

1. Approximately factoring M using noising

  • IIT’s noising choice uses maximum-entropy, uniform noise over the other subsystem’s possible states.The resulting subsystem transition matrices are listed in Table III.Source: §1 · pp. 4–5

2. Approximately factoring M using mild noising

Mild noising uses the observed marginal distribution rather than uniform noise. The paper also considers optimizing the factorization and compares which distributions different φ-measures evaluate.

2. Approximately factoring M using mild noising

  • Uniform-noise factorization is undefined for continuous variables such as measured voltages because a uniform distribution cannot be normalized.Mild noising instead uses the other subsystem’s marginal distribution.Source: §2 · p. 5

3. Optimally factoring M

  • Optimizing over factorizations addresses possible overestimation by fixed noising choices; two of the five options are identical.When M is factorizable, all five options coincide; otherwise, noising yields higher φ than an optimized factorization.Source: §3 · pp. 5–6

D. Options for which probability distributions to compare

  • Comparing only subsystem A’s predicted future makes φ vanish for afferent pathways, whereas comparing the full future prediction gives φ > 0 in the described one-way system.Afferent pathways carry information inward toward the rest of the system.Source: D. Options for which probability… · p. 6

E. Options for what to treat as known about the current state

The paper distinguishes assumptions about what is known of the current state from choices of distance for comparing probability distributions. It discusses alternatives and their practical boundaries.

E. Options for what to treat as known about the current state

  • The current state may be treated as unknown, separable across subsystems, or known, with the known-state option advocated by IIT.A uniform distribution for a completely unknown state is excluded from Table IV as inappropriate for most physical systems and undefined for continuous variables.Source: Table IV · p. 7, E. Options for what to treat as known… · p. 7

F. Options for comparing probability distributions

  • The seven candidate distances include KL divergence, L1 and L2 norms, Hilbert-space and Shannon-Jensen distances, Earth-Movers distance, and mismatched decoding.All have positivity and monotonicity; all except the first are also symmetric and true metrics.Source: F. Options for comparing probability… · p. 7, Table V · p. 8
  • Earth-Movers distance requires numerical linear programming and grows faster than quadratically with the number of system states, making it formally infinite for continuous variables.The number of states itself grows exponentially with the number of bits.Source: F. Options for comparing probability… · p. 8

III. TAXONOMY RESULTS

The dKL optimal factorization is analytically tractable and yields a simple integration measure. Its solution preserves three marginals and is also optimal for several other probability-distribution choices.

A. Optimal factorization with dKL

  • Only dKL permits an analytic solution to the factorization optimization; d2 is tractable except for a cumbersome final column-normalization step.The authors note dKL lacks symmetry but has positivity, monotonicity, and interpretability properties.Source: A. Optimal factorization with dKL · p. 9
  • The optimized distributions preserve three marginals of the original distribution, and substituting the solution yields a simple integration measure.The factorization is obtained by applying the Markov matrix estimator separately to each subsystem after marginalizing over the other.Source: A. Optimal factorization with dKL · pp. 9–10
  • The same optimal factorization applies to several probability-distribution choices, while two options remain undefined or analytically unsolved.The stated equivalences include a+u, s+f, and a+s; t+s is undefined and f+u gives equations the authors could not solve analytically.Source: A. Optimal factorization with dKL · p. 10
  • Under stationarity, the forward- and reverse-time integration measures become identical.The paper attributes their difference otherwise to time asymmetry in the forward Markov-process interpretation.Source: A. Optimal factorization with dKL · p. 10

B. Comparison with the Ay/Barrett/Seth integration measures

The paper relates the Barrett–Seth measure to a KL-divergence measure by restoring final mutual information. It also identifies the time-reversed Markov measure with the stochastic interaction measure.

B. Comparison with the Ay/Barrett/Seth integration measures

  • The alternative measure ˜φE is identical to the time-reversed Markov measure φo˜tu.It is interpretable as the average KL-divergence between the past-state distribution given the present and the product of subsystem conditional distributions.Source: B. Comparison with the Ay/Barrett/Seth… · p. 10
  • The Barrett–Seth measure can be negative, including for two perfectly correlated bits that never change, where its value is −1.This example illustrates the measure’s stated positivity and interpretability problem.Source: B. Comparison with the Ay/Barrett/Seth… · p. 10
  • Adding back final mutual information makes the Barrett–Seth measure non-negative and yields the KL-divergence between the actual distribution and its best separable approximation.The resulting measure is guaranteed to be non-negative.Source: B. Comparison with the Ay/Barrett/Seth… · p. 11

C. Comparison with the mismatched decoding integration measure

The mismatched decoding measure shares the Markov measure’s first three taxonomy choices but uses a different distance. For state-dependent factorization, the optimal KL approximation uses conditional marginals, yet its integration measure vanishes for deterministic systems.

C. Comparison with the mismatched decoding integration measure

  • The mismatched decoding measure shares φM’s optimal factorization, two-time distributions, and unknown-past choices, but uses mismatched decoding distance instead of KL-divergence.Whether the φM factorization also minimizes mismatched decoding distance is not obvious.Source: C. Comparison with the mismatched… · p. 11
  • φM can count shared correlated random noise as integration even when no information flows between the subsystems; whether that should count is disputed.The example gives φM = 1 bit while I(x0, x1) = 0.Source: C. Comparison with the mismatched… · p. 11

D. Optimal state-dependent factorization

  • For a known present state, the optimal factorization uses the marginal conditional distributions of each subsystem.For an arbitrary bivariate distribution, the best separable KL approximation consists of its marginals, and the resulting divergence is mutual information.Source: D. Optimal state-dependent factorization · p. 11
  • State-dependent factorization yields zero integration for every deterministic system, leaving correlated system-generated noise as its only source of integration.With the present state known, a deterministic system’s future is known, so the entropies in the measure vanish.Source: D. Optimal state-dependent factorization · p. 11

E. Minimizing integration on average

The paper finds that average-minimizing and optimal factorization coincide, reducing a nominal 420-measure taxonomy to 21 options. It favors a small set of candidates with useful properties, while noting important computational and interpretive qualifications.

E. Minimizing integration on average

  • Minimizing average KL-divergence across present states gives the same optimal factors and minimum as the earlier optimization, making factorization options “o” and “a” identical.The comparison option “t” gives the same result as well.Source: E. Minimizing integration on average · p. 12

F. The full taxonomy

  • The taxonomy’s 5 × 4 × 3 × 7 choices nominally give 420 measures, but zero, undefined, or identical options leave 21 distinct options to consider.The seven distance choices can be selected independently without changing whether a measure vanishes or is well-defined.Source: F. The full taxonomy · p. 12

G. Which integration measures are best?

  • The Earth-Mover’s distance remains attractive for small discrete systems but is otherwise computationally unfeasible; dKL gives the simplest, most intuitive formulas.The preferred factorization depends on the desired properties: “o” is not analytically available for other distance measures, while “n” is undefined for continuous systems.Source: G. Which integration measures are best? · p. 12
  • The measures φnas, φnps, φmas, and φnps tend to be extremely small in practice because they differ little from optimal-factorization measures that always vanish.The paper characterizes these as measures of factorization suboptimality rather than integration.Source: G. Which integration measures are best? · p. 13
  • φofu tends to be much smaller than some other measures and is expected to be at most on the order of the future subsystems’ mutual information.For the stated random-distribution setting, that mutual information approaches approximately 0.28 bits as n grows.Source: G. Which integration measures are best? · p. 13
  • Discarding problematic measures leaves nine top candidates, and complementary minima can produce measures that vanish for both afferent and efferent systems.The three proposed combinations include φ2.5, which combines IIT2.0’s KL-divergence with a cause/effect minimum, and φ2.5′, which remains defined for continuous variables.Source: G. Which integration measures are best? · p. 13

H. How large can φ get?

The taxonomy narrows attractive measures to a handful, while some measures can grow with subsystem size even though a particular swap example has no true integration. The paper then turns to continuous variables, where Gaussianity simplifies the formulas.

H. How large can φ get?

  • The taxonomy yields only a handful of truly attractive φ-measures; for randomly generated probability distributions, none of the measures shown exceeds 1 − 1/2 ln 2 ≈ 0.28 bits on average.Source: H. How large can φ get? · p. 13
  • Some measures reach φ values growing as log2 n, but the swap example’s large value is specific to one cut and another cut gives Φ = 0.A random-coded construction instead has φM ∼ log2 n even after minimization over all bipartitions.Source: H. How large can φ get? · pp. 13–14

IV. THE n →∞LIMIT OF CONTINUOUS VARIABLES

  • The measures apply to discrete and continuous variables; the continuous-variable section uses Gaussianity to simplify their formulas.The paper frames continuous variables as the n → ∞ limit of discrete variables.Source: §IV · p. 14

H. How large can φ get?

  • For 3,000 random trials, φofum and φotuk are similar and satisfy φofum ≤ φotuk.The comparison is shown in Figure 3.Source: Figure 3 · p. 14

A. How Gaussianity gives linearity

Under a Gaussian assumption, conditional dynamics reduce to an affine function of the current state plus independent noise. For stationary autoregressive processes, covariance matrices can be estimated from time series, and forward and backward conditional entropies agree.

A. How Gaussianity gives linearity

  • Gaussian conditional dynamics have an affine mean function and covariance independent of the current state, so the next state is an affine function plus independent noise.The paper notes this removes nonlinear conditional means and state-dependent scatter under Gaussianity.Source: A. How Gaussianity gives linearity · p. 15

B. Autoregressive processes

  • For a stationary process, covariance is time-independent, and the dynamics’ matrices satisfy the Lyapunov equation C = ACAt + Σ.When eigenvalues of A have magnitude below unity, a nonstationary process eventually converges to a stationary state.Source: B. Autoregressive processes · p. 15
  • Time-series covariance and cross-time covariance determine the matrices needed to characterize the process.The paper obtains the required matrices by equating measured covariance with the autoregressive covariance relation.Source: B. Autoregressive processes · p. 15
  • Forward and backward conditional entropies are equal because the determinants of Σ and ˜Σ are identical.The backward-time description uses time-reversed dynamics and covariance matrices.Source: B. Autoregressive processes · p. 15

C. Optimal factorization

For Gaussian systems, integration calculations can use covariance matrices instead of full continuous-state transition distributions. The optimal and noising factorizations have distinct matrix interpretations, and one noising option is undefined for continuous variables.

C. Optimal factorization

  • Gaussian descriptions replace the full transition distribution with a 2n × 2n covariance matrix, from which deterministic dynamics A and stochastic dynamics Σ are computed.The optimal factorization corresponds to A and Σ being block-diagonal.Source: C. Optimal factorization · pp. 15–16
  • The “o”-factorization approximates the dynamics with subsystem-specific means and covariances, while the “m”-factorization reinterprets cross-subsystem signal terms as zero-mean noise.The “m” option uses the actual distributions of the noised variables; unlike “o,” it does not use tilded A-matrices.Source: C. Optimal factorization · p. 16, D. Noising factorization · p. 16

D. Noising factorization

  • The uniform “n” noising option is undefined for continuous variables because it yields infinite variance, whereas mild “m” noising remains well-defined.Mild noising uses the actual distributions for the variables being noised.Source: D. Noising factorization · p. 16

E. Results

The Gaussian results express several integration measures through matrix formulas, including formulas derived from mutual information and KL divergence. The time-reversed versions of three measures reuse their forward-time counterparts with reversed dynamics and covariance matrices.

E. Results

  • The Gaussian mutual-information formula yields matrix formulas for φM, φB, φots, φofs, and φxfk.The passage also notes that the second listed version of φB is given in.Source: E. Results · p. 16
  • The KL-divergence between Gaussian distributions separates mean mismatch from covariance mismatch, with the covariance term also nonnegative.The Gaussian formulas are obtained by substituting the relevant means and covariances into the divergence formula.Source: E. Results · p. 16
  • The time-reversed measures φopk, φmps, and φmpk match φoak, φmas, and φmak after replacing A and Σ with ˜A and ˜Σ.The Gaussian expression for φMD also gives a simplified case at β = 1.Source: E. Results · p. 17

V. GRAPH-THEORY APPROXIMATION TO MAKE COMPUTATIONS FEASIBLE

The graph-based approximation makes Φ computation more tractable by replacing weak matrix links with zeros and using connected components to identify a candidate bipartition. It works best when one bipartition is clearly better, while less separable systems can produce overestimates.

A. The problem

  • For discrete variables, evaluating φ requires an n × n matrix with n = 2^b, making computation prohibitive even for modest system sizes.The passage cites 100 bits as computationally prohibitive and gives b ∼ 10^11 for the number of neurons in the human brain.Source: A. The problem · p. 17
  • Continuous-valued variables pose an additional challenge because representing even one variable requires infinitely many bits.The paper describes Gaussian approximations as removing this infinity problem for most measures and enabling φ computation in O(b^3) time.Source: A. The problem · p. 17
  • Even after speeding up φ computation, evaluating Φ remains exponentially slow because it minimizes φ across exponentially many system bipartitions.Restricting the search to symmetric bipartitions remains exponentially costly.Source: A. The problem · p. 17

B. An approximate solution

  • Graph connectivity can identify a zero-φ cut in polynomial time, avoiding the inefficient search over all element permutations or bipartitions.For a disconnected graph with two equal parts, the matrix can be permuted into block-diagonal form.Source: B. An approximate solution · pp. 17–18
  • Zeroing A-matrix elements below a threshold chosen to make the largest connected component near n/2 yields a candidate cut whose φ-value approximates Φ.The method uses interval halving to choose the threshold and can use weak or strong graph connectivity depending on the Φ-measure.Source: B. An approximate solution · p. 18, Figure 4 · p. 18
  • The approximation extends to asymmetric bipartitions, but it does not resolve conceptual questions about how to weight or handle them.The authors recommend computing candidate cuts from both A and C and selecting the one with the smaller φ-value.Source: B. An approximate solution · pp. 18–19
  • The approximation finds the exact answer 95% of the time when Φmax/Φ > 2 and 99.96% when Φmax/Φ > 3; for Φmax/Φ ∼< 2, median overestimation reaches 15%.The reported evaluation used 7,000 simulations of an n=16 ΦM example.Source: B. An approximate solution · pp. 18–19, Figure 5 · p. 19

VI. CONCLUSIONS

The paper classifies Φ-measures and identifies a small set with attractive properties, then derives faster exact and approximate formulas. It also outlines unresolved partition, physical-foundation, and practical-data questions.

VI. CONCLUSIONS

  • The taxonomy spans five factorization options, 3×4 probability-distribution choices, and five comparison measures; attractive-property requirements narrow hundreds of options to a handful.Some of the remaining measures are identical, and the leading contenders are summarized in Table II.Source: §VI · p. 19
  • A graph-theory approximation reduces the computational cost of general measures from doubly exponential to polynomial in system size.The exact Gaussian formulas are exponentially faster, manipulating a matrix whose size grows as 2^b rather than 2^(2^b).Source: §VI · p. 19

A. Which Φ-measures are best?

  • Six Φ-measures stand out from a taxonomy of hundreds as particularly attractive: ΦM, ΦMkk′, Φ3.0, Φ2.5, Φ2.5′ and Φ2.5′′; ΦM is guaranteed to vanish for separable systems and never be negative.Source: A. Which Φ-measures are best? · p. 19
  • Φ3.0 has attractive features described in, but is the slowest measure to evaluate numerically because each bipartition requires a linear-programming problem.Its cost grows faster than quadratically with the number of system states and is infinite for continuous variables.Source: A. Which Φ-measures are best? · p. 19
  • Φ2.5, Φ2.5′, and Φ2.5′′ are much quicker than Φ3.0 and share its feature of vanishing for afferent and efferent systems.Φ2.5 depends only on system dynamics and present state, while Φ2.5′ avoids its cousin’s infinite value for continuous variables.Source: A. Which Φ-measures are best? · p. 19

B. Outlook

  • Asymmetric partitions remain an open question: minimizing φ across partitions of any size can yield small values when either subsystem is very small.The paper also asks whether the taxonomy’s measures can be derived from a small set of axioms.Source: B. Outlook · p. 20
  • The information-bit starting point leaves open how to ground integration maximization in physical properties such as particle motions or quantum-state evolution.IIT’s proposed link to the space-time coarse-graining level that maximizes Φ is a prediction to test.Source: B. Outlook · p. 20
  • Further work could improve the graph approximation by using Σ’s structure and state x; real-brain computation also faces non-stationarity, estimation, bias, and numerical-stability issues.The paper notes that estimating many parameters from short data windows can lead to overfitting.Source: B. Outlook · p. 20
Loading 1601.02626v3…