PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Causal Coupling Strength: A Lag-specific Measure For Multivariate Time Series Related To Transfer Entropy

Jakob Runge, Jobst Heitzig, Norbert Marwan, Jürgen Kurths

arXiv:1210.2748v2physics.data-ancs.ITmath.ITstat.ML

TL;DR

Meaningful coupling strength is difficult to infer from mutual information and transfer entropy, which can be unintuitive and, for transfer entropy, not lag-specific. The paper generalizes MIT as a lag-specific conditional mutual information and shows coupling-strength autonomy under specified conditions.

  • Problem

    Mutual information and transfer entropy can be unintuitive measures of coupling strength, and transfer entropy is not lag-specific.

    Source: §I · p. 2, §X · p. 10

  • Method

    The authors first estimate a graphical model to identify lagged causal links, then weight each existing link with multivariate MIT, a source-entropy-based conditional mutual information.

    Source: §I · p. 2, §IV · p. 4

  • Results

    Analyses show that mutual information and transfer entropy can be unintuitive coupling-strength measures, while MIT has coupling-strength autonomy under conditions (1)–(3).

    Source: §VIII · p. 9, §X · p. 10

  • Takeaways & Limitations

    MIT can provide an interpretable coupling strength reminiscent of an experimentally manipulable setting.

    Source: §X · p. 10

    MIT’s autonomy applies only under conditions (1)–(3); for nonlinear dependencies or sidepaths, the authors suggest modifications such as IMITS.

    Source: §VIII · p. 9

Abstract

from arXiv · show

While it is an important problem to identify the existence of causal associations between two components of a multivariate time series, a topic addressed in Runge et al. (2012), it is even more important to assess the strength of their association in a meaningful way. In the present article we focus on the problem of defining a meaningful coupling strength using information theoretic measures and demonstrate the short-comings of the well-known mutual information and transfer entropy. Instead, we propose a certain time-delayed conditional mutual information, the momentary information transfer (MIT), as a measure of association that is general, causal and lag-specific, reflects a well interpretable notion of coupling strength and is practically computable. MIT is based on the fundamental concept of source entropy, which we utilize to yield a notion of coupling strength that is, compared to mutual information and transfer entropy, well interpretable, in that for many cases it solely depends on the interaction of the two components at a certain lag. In particular, MIT is thus in many cases able to exclude the misleading influence of autodependency within a process in an information-theoretic way. We formalize and prove this idea analytically and numerically for a general class of nonlinear stochastic processes and illustrate the potential of MIT on climatological data.

I. INTRODUCTION

The paper seeks multivariate measures that identify causal dependencies and provide interpretable coupling strengths autonomous of other processes. It proposes source-entropy-based MIT to weight graph links, while noting that equitability is only partially addressed.

I. INTRODUCTION

  • A useful multivariate association measure should be general, causal, autonomous of interactions with other processes, and practically computable.Autonomy is motivated by the possibility that a third process drives both components, confounding their measured coupling strength.Source: §I · p. 1
  • Transfer entropy is not lag-specific and can depend on autodependency and interactions with other processes, undermining its interpretation as coupling strength.The paper also notes that arbitrary truncation can complicate TE estimation.Source: §I · p. 2
  • MIT generalizes a source-entropy-based measure to multivariate data using graphical models, with generality, causality, and autonomy as its addressed properties.The paper says equitability is addressed only partially.Source: §I · p. 2
  • The approach first estimates a graph to identify causal links, then estimates MIT as a meaningful weight for each existing link.The paper presents this as a way to obtain links that are causally identified and meaningfully weighted, with coupling delay.Source: §I · pp. 1–2

II. TRANSFER ENTROPY AND THE CURSE OF DIMENSIONALITY

Transfer entropy aggregates influence across past lags, while its estimation requires handling infinite histories, often through truncation. Truncation can raise dimensionality and affect the estimated value and reliability of causal inference; decomposed TE does not make its lag terms causal contributions.

II. TRANSFER ENTROPY AND THE CURSE OF DIMENSIONALITY

  • TE aggregates the influence of X across past lags rather than measuring a lag-specific association.Its definition involves infinite-dimensional densities, commonly handled by truncating the histories.Source: §II · p. 2
  • The truncation lag τmax affects estimation dimension and can strongly influence TE values and causal-inference reliability.The dimension is described as D = N · τmax + 1, and τmax must be at least the maximum coupling delay.Source: §II · pp. 2–3
  • Graphical-model-based DTE permits estimation with finite vectors, but a truncation approach can still leave high estimation dimensions.A cited example had a maximum dimension of 24.Source: §II · p. 3
  • DTE terms represent contributions from different lags but are not lag-specific causal contributions, since they can be nonzero without a graph link at that lag.The paper also says TE and DTE do not fulfill coupling-strength autonomy.Source: §II · p. 3

III. GRAPHICAL MODELS AND CAUSALITY

Time-series graphs represent lag-specific conditional dependencies as links, providing the first step in identifying causal couplings. The second step assigns meaningful weights to existing links, for which MIT is intended.

III. GRAPHICAL MODELS AND CAUSALITY

  • In a time-series graph, a directed link from a lagged component to a later one represents conditional dependence given the rest of the process, and thus lag-specific causality.A finite set of parents can be used to separate a node from the remaining past under the Markov property.Source: §III · p. 3
  • Conditional mutual informations over infinite histories cannot be computed directly, whereas Markov properties allow finite parent sets to be used.The caption illustrates MIT conditioned on the parents of the source and the remaining parents of the target.Source: §III · p. 3, Figure 2 · p. 3
  • The procedure first estimates the time-series graph to identify links and then assigns a meaningful weight to each existing link.MIT is intended to provide an interpretable coupling strength for inferred links.Source: §III · pp. 3–4

IV. MOMENTARY INFORMATION TRANSFER AND SOURCE ENTROPY

MIT measures the part of Y’s source entropy shared with the source entropy of X at a selected earlier time, conditioning on their parents. This construction is intended to provide a lag-specific, causal association and a low-dimensional estimation problem without arbitrary truncation parameters.

IV. MOMENTARY INFORMATION TRANSFER AND SOURCE ENTROPY

  • Source entropy is the expected uncertainty remaining about a process value after conditioning on its parents.MIT is the conditional mutual information measuring the part of Y’s source entropy shared with X’s source entropy at time t−τ.Source: §IV · p. 4
  • MIT’s “momentary” attribute refers to information from X at time t−τ transferred to Y at time t.The paper relates this momentariness to coupling-strength autonomy.Source: §IV · p. 4
  • Because MIT conditions on Y’s parents, it is nonzero only for lagged processes that remain dependent conditional on the rest of the past.Like other conditional mutual information measures, MIT is sensitive to statistical association and fulfills generality.Source: §IV · p. 4
  • MIT and related measures yield a low-dimensional estimation problem without arbitrary truncation parameters.ITY isolates Y’s source entropy and ITX measures source entropy in X that reaches Y along any path, so ITX is not a causal measure.Source: §IV · p. 4

V. ANALYTICAL COMPARISON

The analytical examples show that mutual information, transfer entropy, and link-based measures can depend on process dynamics beyond the coupling being measured. MIT isolates coupling strength under specified conditions, while sidepaths and nonlinear source mixing limit that autonomy.

V. ANALYTICAL COMPARISON

  • Transfer entropy depends on X’s autodependency in one case and on the coupling with Z in another, making it an unintuitive measure of X–Y coupling strength.The authors identify both dependencies in their analytical model.Source: §V · p. 5
  • MI depends on input-process coefficients and variances, while conditioned measures can retain dependence on the unconditioned process; none of these three depends on interaction with Z in this model.The simplified formulas rely on independent inputs to X and Y.Source: §V · p. 5
  • With no sidepaths, MIT depends only on the coupling coefficient and source variances for the stated class of additive models.The authors state that the result holds for arbitrary multivariate linear autoregressive processes under the no-sidepath constraint.Source: §V · p. 5
  • With a sidepath, MIT can depend on the additional path’s coefficient and the variance of W, so the coupling cannot be attributed to cXY alone.The covariance along the direct and W-mediated paths can vanish when their contributions cancel.Source: §V · p. 6
  • For nonlinear coupling, source entropies mix, so the coupling is treated as emanating from Xt−1 together with its parents; the authors report no analytical MIT expression for this model.They study a more general case numerically in the appendix.Source: §V · p. 6
  • The sidepath and nonlinear examples motivate conditions that the next section formalizes as a coupling-strength autonomy theorem.Source: §V · p. 6

VI. COUPLING STRENGTH AUTONOMY THEOREM AND MODIFICATIONS OF MIT

The theorem specifies when MIT reduces to an information measure of the source processes and how its dependencies change when key conditions fail. The section also describes modified measures for sidepaths and nonlinear dependencies, alongside scope boundaries for autonomy.

VI. COUPLING STRENGTH AUTONOMY THEOREM AND MODIFICATIONS OF MIT

  • When additivity, linearity in f, and the no-sidepath condition hold, MIT can be expressed as mutual information of the source processes.The theorem concerns a link Xt−τ → Yt in a multivariate stationary discrete-time process.Source: §VI · p. 6
  • If additivity and linearity hold but a sidepath exists, MIT also depends on sidepath-parent distributions and their functional dependence on Yt.Source: §VI · p. 6
  • If only additivity holds and the coupling is nonlinear, MIT additionally depends on f, parent distributions, and their functional dependencies on Yt.Source: §VI · p. 7
  • For a contemporaneous link, MIT has no sidepaths; when X = Y, it measures autodependency strength.Source: §VI · p. 7
  • Unlike channel capacity, MIT cannot isolate capacity alone because the input sources cannot be manipulated; other conditional mutual informations can be written in signal-to-noise form.Source: §VI · p. 7
  • For sidepaths, modified MITS leaves open all paths from Xt−τ to Yt by excluding Yt parents that depend on Xt−τ.The S denotes “sidepath”; a nonlinear modification, MITN, conditions on the joint vector (Xt−τ, PXt−τ).Source: §VI · p. 7
  • The authors defer study of the MITS and MITN modifications to a separate paper.Source: §VI · p. 7
  • Under the theorem’s conditions, MIT is independent of coefficients on other links, but links from the same X still depend on ηX’s source variance.The authors describe this qualified disentangling as coupling-strength autonomy.Source: §VI · p. 7

VII. NUMERICAL COMPARISON

The numerical comparisons show that MIT best isolates lag-specific coupling strength from other inputs and distinguishes coupling coefficients, while MI, TE, and related measures retain dependencies or estimation biases.

VII. NUMERICAL COMPARISON

  • MI can be larger or smaller than MIT as inputs change; ITX and ITY can exclude inputs to X or Y, respectively, but their input dependence may vary across models.On average, ITX is no greater than MIT, while ITY is no less than MIT.Source: §VII · p. 8
  • Only MIT numerically fulfills coupling-strength autonomy, remaining largely invariant to other inputs except for slight dependence under strong inputs and short-sample estimation problems.It approximately reaches the analytical value when input coefficients are zero; all strengths from a source process still depend on that source’s variance.Source: §VII · p. 8
  • TE and DTE scale similarly with input coefficients, whereas LINK depends on aX and MIT shows only slight dependence for strong inputs.The MIT dependence is attributed to estimation problems for short samples; the numerical results support autonomy only for MIT.Source: §VII · p. 8
  • Only MIT clearly distinguishes the different coupling coefficients; MI spans many values at fixed coupling, while TE and LINK estimates suffer dimension-dependent negative bias.TE and LINK use estimation dimension 17, versus 6 for MIT; TE’s negative relative bias is about 50% at the analytically known zero-input value I ≈0.15.Source: §VII · pp. 8–9, Figure 4 · p. 8
  • DTE tends toward higher values with stronger X autodependency and has greater estimate variance because the summand variances accumulate.The distributions also show TE and LINK’s negative bias depends on the chosen τmax, so estimates using different τmax values cannot be compared.Source: §VII · p. 9
  • MIT acts as an information-theoretic filter excluding effects of autodependency and misleading inputs, isolating interactions between two lagged subprocesses and making coupling strengths more interpretable than MI or TE.The reported results use linear f(x)=x, for which the multivariate process satisfies all three conditions; each input-coefficient and functional-form combination has 120 trials, with CMI estimated by kNN at k=1.Source: §VII · pp. 8–9

VIII. DISCUSSION AND LIMITATIONS

MIT’s autonomy and causal interpretation have defined scope: they depend on observed processes, discrete-time graph assumptions, and conditions on the coupling and dependencies; estimation and equitability also remain limited.

VIII. DISCUSSION AND LIMITATIONS

  • Causality is defined relative to the observable processes included as parents, rather than excluding influences from the whole universe.The graphical model describes interactions discretely; continuous-time sampling leaves unobserved nodes between successive observations.Source: §VIII · p. 9
  • For continuous-time processes sampled at intervals, source entropy is accessible only as information aggregated over the interval, not solely at time t.For discrete processes, graphical models apply to the large class satisfying condition (S) in.Source: §VIII · p. 9
  • Even after graphical-model dimension reduction, dimensions can bias estimates on shorter samples, and highly skewed distributions challenge entropy estimation.The authors identify improved CMI estimators and further study of high-dimensional estimation as needed.Source: §VIII · p. 9
  • The two-step approach first requires estimating a time series graph, which carries risks of false positives from multiple testing and missed causal links.Those graph-detection problems are analyzed in Supplementary Material in.Source: §VIII · p. 9
  • MIT filters all parent influences only when coupling strength is attributable to one coefficient and conditions (1)–(3) hold; nonlinear dependencies or sidepaths may call for IMITS.For full autonomy, the link itself must be linear, though remaining dependencies may be nonlinear and source processes may have arbitrary distributions.Source: §VIII · p. 9
  • For Gaussian dependencies, MIT scales ∝ln(1+cXY · · · ); the transformation I → 1 −e−2I can achieve linear scaling in that case.For more complex dependencies, the authors note that estimators better adapted to the distributions may help.Source: §VIII · p. 9

IX. APPLICATION TO CLIMATOLOGICAL TIME SERIES

The climatological analysis compares contemporaneous surface–troposphere coupling across tropical grid points. MIT isolates a significant sub-month coupling peak and spatially distinguishes coupling strength from the influence of surface persistence.

IX. APPLICATION TO CLIMATOLOGICAL TIME SERIES

  • The analysis uses monthly tropical air-temperature anomalies from 1927–2011 at the surface and 200 hPa, across grid points between 30°S and 30°N.The dataset contains 1008 months; Figure 5 maps contemporaneous coupling and lag-1 surface autodependency for MI and MIT.Source: §IX · p. 9, Figure 5 · p. 10
  • MIT shows the contemporaneous surface–troposphere link as a significant peak, indicating a coupling timescale below one month; MI is significant across a wide range of lags.The time-series graph has lag-1 autodependencies on both series and a contemporaneous link between them.Source: §IX · pp. 9–10, Figure 5 · p. 10
  • MIT coupling strengths are more spatially confined, with their largest values between 90°E and 170°W, a region corresponding to strong Walker-circulation upwelling.The difference between MI and MIT is largest in the Eastern Pacific, where surface-temperature autodependency is also apparent.Source: §IX · p. 10, Figure 5 · p. 10
  • In the Eastern Pacific, surface persistence increases MI, whereas MIT distinguishes that autodependency from contemporaneous coupling.The authors describe the resulting MIT-based picture of Walker-circulation interactions as more reasonable.Source: §IX · p. 10

X. CONCLUSIONS

The conclusion presents MIT as a lag-specific coupling-strength measure with coupling-strength autonomy, while noting that its interpretability depends on the process model. The appendix supplies proofs and further discussion of the autonomy theorem and an inequality relation.

X. CONCLUSIONS

  • The proposed two-step approach first identifies lag-specific and contemporaneous links, then weights existing links with generalized MIT, which is practically computable without arbitrary truncations.Coupling-strength autonomy supports an interpretable measure for models whose coupling strength can be attributed to one coefficient; for other cases, the authors suggest MIT modifications.Source: §X · p. 10

Appendix

  • The appendix proves the inequality relation in Eq. (12) and the coupling-strength autonomy theorem, and discusses autonomy when the linearity condition is violated.Source: Appendix · p. 10

I. PROOF OF INEQUALITY RELATION EQ. (12)

The appendix bounds MIT between conditional mutual information measures, with the lower bound requiring an additional no-sidepath constraint. It also derives covariance quantities for the analytical model to compute conditional entropies.

I. PROOF OF INEQUALITY RELATION EQ. (12)

  • The right inequality holds under the general condition (S), while the left additionally requires the no-sidepath constraint for the link Xt−τ → Yt.That constraint requires Xt−τ to be separated from PXt−τ \ PYt by its parents in the time-series graph.Source: §I · p. 11
  • The right bound follows by adding parents of Xt−τ not already in PYt and applying the chain rule, using graph separation to establish conditional independence.The proof uses nonnegativity of conditional mutual information.Source: §I · p. 11
  • For the left bound, the no-sidepath constraint makes the extra parents of Yt conditionally independent of Xt−τ given its parents; the chain rule then yields the bound.Source: §I · p. 11

II. DERIVATIONS FOR ANALYTICAL MODEL EQ. (13)

  • For analytical model Eq. (13), the derivation defines variances, auto-covariances, and cross-covariances; these populate covariance matrices used to compute conditional entropies.The listed cross-covariances include terms involving X, Y, Z, and W.Source: §II · p. 11

A. Derivations of TE

The appendix derives TE and MIT-related quantities for selected cases, then proves conditions for coupling-strength autonomy. In nonlinear experiments, MIT is less affected than MI by other inputs, though the autonomy and inequality results have stated limits.

A. Derivations of TE

  • For the TE derivation, the general quotient of infinite-dimensional covariance matrices is difficult to evaluate, so the appendix treats two simpler cases.The derivation uses a Toeplitz-matrix limit under an assumption of absolutely summable entries.Source: A. Derivations of TE · p. 12, §1 · p. 12

1. cXZ = cW Y = 0

  • With cXZ = cWY = 0, the reported TE expression includes a zero term for aX ≤ 1 and depends on X’s autodependency strength.Source: §1 · p. 12

2. aX = 0

B. MIT and related measures

  • If Yt additively depends on its parents, MIT depends only on the distribution of the source process ηYt.Source: §III · p. 13

III. PROOF OF COUPLING STRENGTH AUTONOMY THEOREM

  • When sidepath parents are additively separated, MIT can be further simplified; with only condition (1), the corresponding relation cannot be simplified much further.Source: §III · p. 14

IV. FURTHER NUMERICAL EXPERIMENTS

  • For squared dependency, MIT rises with aX because the source mixes with its parents, but it is less affected than MI; the left inequality appears violated for large aX and small aY.The authors suggest highly skewed distributions under nonlinear f(x) as a possible explanation for the apparent violation.Source: §IV · pp. 14–15
  • In these nonlinear experiments, the right side of the inequality is unaffected by nonlinear dependency, while TE and LINK have higher relative variance despite appearing less affected.Source: §IV · p. 14
Loading 1210.2748v2…