PaperlayerView exact revision on arXiv

Source-linked AI summary

The relation between Granger causality and directed information theory: a review

Pierre-Olivier Amblard, Olivier J. J. Michel

arXiv:1211.3169v1cs.ITmath.IT

TL;DR

Assessing directional dependence among signals requires estimating which signals are linked and in what direction. This review connects Granger causality with directed information theory, using causal conditioning to frame the relationship. It presents decompositions involving transfer entropy and instantaneous coupling, while noting a no-feedback condition under which directed information and mutual information are equal.

Abstract

from arXiv · show

This report reviews the conceptual and theoretical links between Granger causality and directed information theory. We begin with a short historical tour of Granger causality, concentrating on its closeness to information theory. The definitions of Granger causality based on prediction are recalled, and the importance of the observation set is discussed. We present the definitions based on conditional independence. The notion of instantaneous coupling is included in the definitions. The concept of Granger causality graphs is discussed. We present directed information theory from the perspective of studies of causal influences between stochastic processes. Causal conditioning appears to be the cornerstone for the relation between information theory and Granger causality. In the bivariate case, the fundamental measure is the directed information, which decomposes as the sum of the transfer entropies and a term quantifying instantaneous coupling. We show the decomposition of the mutual information into the sums of the transfer entropies and the instantaneous coupling measure, a relation known for the linear Gaussian case. We study the multivariate case, showing that the useful decomposition is blurred by instantaneous coupling. The links are further developed by studying how measures based on directed information theory naturally emerge from Granger causality inference frameworks as hypothesis testing.

1 GIPSAlab/CNRS UMR 5216/ BP46,

The report examines how to assess directional dependence among measured signals, linking Granger causality with directed information theory. It frames Granger causality as statistical dependence over time, distinct from intervention-based causal analysis.

I. INTRODUCTION

  • The inference problem is to assess directional dependence among signals and estimate the corresponding edges in a graph.The signals may come from settings such as neuroscience, solar physics, or turbulence.Source: §I · p. 2

A. What is, and what is not, Granger causality

  • Granger causality assesses directional dependence between time series through statistical dependence between one process's past and another's present.The report contrasts this probability-conditioning framework with approaches based on causal calculus and intervention.Source: §I · p. 3, A. What is, and what is not, Granger… · p. 4

I. INTRODUCTION

  • Information theory's usual mutual information is symmetric, so directional dependence requires measures that break that symmetry.The report identifies directed information theory as the relevant framework for this review.Source: §I · p. 3
  • The report reviews conceptual and theoretical links between Granger causality and directed information theory, alongside a literature survey focused on their relationship.It excludes practical aspects and detailed applications.Source: §I · p. 3

A. What is, and what is not, Granger causality

  • Granger causality permits feedback: two signals can each Granger-cause the other, and the framework does not itself establish intervention-based causation.The authors state that its conclusions are relative to the measurements available.Source: A. What is, and what is not, Granger… · p. 4

B. A historical viewpoint

The historical account traces connections between prediction-based Granger causality, information-theoretic measures, and later directed-information formulations. It highlights how instantaneous coupling enters decompositions of dependence.

B. A historical viewpoint

  • The accepted Granger-causality definitions extended prediction-based ideas and relied on conditional independence; later work defined Granger-causality graphs for multivariable processes.These graphs provide a counterpart to graphical models for multivariate random variables.Source: B. A historical viewpoint · p. 5
  • Geweke's Gaussian-case result decomposed mutual information rate into causality indices in both directions and an instantaneous-coupling index.The account says the decomposition also holds generally when causality indices are replaced by transfer entropy rates and instantaneous coupling by an instantaneous information exchange rate.Source: B. A historical viewpoint · p. 5
  • Gouriéroux and colleagues used nested hypotheses and Kullback divergence in a decision-theoretic framework, generalizing the Gaussian decomposition.Their treatment was limited to the bivariate case.Source: B. A historical viewpoint · p. 5
  • Rissanen and Wax used system identification and model complexity in an information measure, extending Geweke's result to nonlinear and non-Gaussian modeling.Their approach also gave the tests an information-theoretic interpretation.Source: B. A historical viewpoint · p. 5
  • Transfer entropy was introduced as a measure of influence between signals and was noted as closely related to Granger causality.The account describes it in terms of conditional mutual information and notes that many studies considered only bivariate analysis.Source: B. A historical viewpoint · p. 6
  • The review notes that earlier links between Granger causality and directed information often addressed Gaussian or bivariate settings and omitted instantaneous coupling.The authors identify this as part of the context for their review of information flow in networks.Source: B. A historical viewpoint · p. 7

C. Outline

The outline presents directed-information tools as natural measures for studying Granger-causal influences, with causal conditioning central to the connection. The notation section sets the review's process assumptions and conventions.

C. Outline

  • The review presents directed-information tools as natural measures for assessing Granger causality, with causal conditioning at the root of the relationship.It identifies transfer entropy measures with the dynamical part and instantaneous information exchange measures with instantaneous dependence.Source: C. Outline · p. 7
  • The review moves from prediction-based definitions of Granger causality to conditional-independence definitions, then treats instantaneous coupling and causality graphs.It also considers how inference approaches lead to measures from directed information theory.Source: C. Outline · p. 7

D. Notations

  • The report assumes discrete-time stationary processes, sufficiently finite moments, and existing limits where needed.Continuously valued variables are assumed to have probability densities.Source: D. Notations · p. 7
  • The notation represents multivariate signals by indexed component sets and uses filtrations to denote information observed up to a given time.The text also defines partitions of the index set and conditional-independence notation.Source: D. Notations · p. 8

II. GRANGER’S CAUSALITY

Prediction-based Granger causality asks whether including one process’s past improves prediction of another, relative to a specified observation set. The review also presents a broader conditional-independence definition, whose assessment lacks the prediction definition’s inherent operational character.

Prediction-based definitions

  • Granger causality from xA to xB is measured by the loss in prediction quality when xA’s past is excluded from the available past of the full observation set.The comparison is between optimal risks for predicting xB with all observed pasts and with xA’s past omitted.Source: A. From prediction-based definitions · pp. 8–9
  • The prediction-based verdict depends on the chosen cost and function class; with quadratic loss and linear predictors, a nonlinear influence can be missed.In the example, y does not Granger-cause x under linear prediction even though the passage describes an underlying nonlinear cause.Source: A. From prediction-based definitions · p. 9

The observation set

  • The conclusion is relative to observation set V: adding measurements can change whether one process’s past improves prediction of another.The chain example reports that x causes z relative to {x,z}, but not relative to {x,y,z}.Source: A. From prediction-based definitions · p. 9

Operational assessment

  • Quadratic loss supports operational tests, but the general optimal predictors are conditional expectations that may be too difficult or impossible to compute.The review notes that parameterized function classes can make prediction-based tests efficiently evaluable from data.Source: A. From prediction-based definitions · pp. 9–10

Conditional-independence definition

  • The conditional-independence definition says xA does not Granger-cause xB when, given the past of the remaining processes, xA’s past adds no information about xB’s next value.It does not require a specified function class or cost, but conditional-independence assessment lacks an inherent operational procedure.Source: B. ...to a probabilistic definition · p. 10

C. Instantaneous coupling

Instantaneous coupling captures same-time statistical dependence, which the past-to-present Granger-causality definitions do not capture. The review distinguishes two conditional-independence definitions and explains how their conditioning choices affect interpretation in multivariate settings.

Meaning of instantaneous coupling

  • Instantaneous coupling measures common information between two processes’ next values that is not shared with their past, and is a statistical dependence rather than a causal link per se.The report uses “instantaneous coupling” to distinguish this relation from “instantaneous causality.”Source: C. Instantaneous coupling · p. 10

Two definitions

  • The two definitions differ in conditioning: one tests the next values of A and B given C’s next value, while the other conditions only on information through time n.Both definitions are symmetric in A and B.Source: C. Instantaneous coupling · p. 11

Multivariate interpretation

  • Conditioning only on information through time n leaves instantaneous dependence unconditioned on C, so apparent coupling between A and B may reflect links to C.The review characterizes this as bivariate coupling that does not account for possible coupling between A and C or B and C.Source: C. Instantaneous coupling · p. 11
  • Conditioning on all processes through time n + 1 tests dependence between A’s and B’s next values conditional on C’s next value.Source: C. Instantaneous coupling · p. 11

Graphical interpretation

  • Conditional-independence graphs can help eliminate apparent coupling due to third parties, but conditioning can also make independent variables dependent, as with z = x + y.The review notes that independent x and y are conditionally dependent given their sum z.Source: C. Instantaneous coupling · p. 11

D. More on graphs

This section introduces Granger causality graphs and reviews directed information for studying directional dependence, emphasizing causal conditioning and its relation to mutual information. It also describes how feedback and symmetric instantaneous dependence shape that relationship and its decomposition.

D. More on graphs

  • A Granger causality graph is a mixed graph with directed edges denoting Granger causation and undirected edges denoting instantaneous coupling between process components.Source: D. More on graphs · p. 11

III. DIRECTED INFORMATION THEORY AND DIRECTIONAL DEPENDENCE

  • The review presents directed information theory as a recent extension rooted in 1960s–1970s studies by Marko, using a presentation close to Granger causality and treating directional dependence first bivariately, then in multivariate networks.Source: §III · p. 12

A. Notation and basics

  • Mutual information measures dependence but cannot determine its direction, since the conditional-independence relation holds under either ordering.The passages introduce conditional mutual information to account for a third process or side information.Source: A. Notation and basics · pp. 12–13

B. Directional dependence between stochastic processes; causal conditioning

C. Directed information rates

Information rates normalize information measures by time-series length, and causal conditioning extends this rate perspective to directed information. The continuous-valued case requires a more involved proof than the discrete-valued case.

C. Directed information rates

  • Entropy and mutual information generally grow linearly with series length, motivating rates that compensate for this growth.The stated normalization is by the length n of the sample.Source: C. Directed information rates · p. 16
  • Dividing an information measure by series length and taking its limit defines its information rate, when the limit exists.For stationary processes, entropy rate is the limit of the normalized entropy.Source: C. Directed information rates · p. 16
  • Kramer generalized the entropy-rate result to causal conditional entropies, defining the directed information rate.The same result holds for the instantaneous information exchange rate.Source: C. Directed information rates · p. 16
  • The rate proof relies on positive entropy for discrete-valued processes; continuous-valued processes can have negative entropy and require more involved methods.The passage points to methods developed in,,.Source: C. Directed information rates · p. 16

D. Transfer entropy and instantaneous information exchange

Transfer entropy measures information flow from the past of one process to the present of another, while directed information also accounts for instantaneous exchange. Their decompositions separate past-directed influence from instantaneous coupling.

D. Transfer entropy and instantaneous information exchange

  • Introduced by Schreiber,, transfer entropy is expressed as the difference of conditional entropies: T (x n−1 A n−l+1 →x n B n−k+1) = H(xB(n)∣x n−1 B n−k+1) −H(xB(n)∣x n−1 B n−k+1,x n−1 A n−l+1).Source: D. Transfer entropy and instantaneous… · pp. 16–17
  • Transfer entropy is zero exactly when the stated conditional-independence relation holds.The relation conditions on the target’s past and the source’s past, under the joint Markov property.Source: D. Transfer entropy and instantaneous… · pp. 16–17
  • The transfer-entropy definition uses histories ending at the same time for practical comparison with directed information when delays are unknown.The passage notes that no a priori information about possible delays is available.Source: D. Transfer entropy and instantaneous… · p. 17
  • Transfer entropy measures the influence of one process’s past on another’s present, but does not account for instantaneous dependence.Directed information handles that instantaneous dependence.Source: D. Transfer entropy and instantaneous… · p. 17
  • Schreiber’s transfer entropy considers a single present-time term rather than directed information’s sum over time, implicitly assuming stationarity when interpreted as a rate.Palus considered summing over delays to reduce estimation errors.Source: D. Transfer entropy and instantaneous… · p. 17
  • Directed information decomposes into a past contribution and an instantaneous contribution, while mutual information decomposes into directional dependencies and instantaneous coupling.The instantaneous exchange is counted twice across the directional terms but only once in mutual information.Source: D. Transfer entropy and instantaneous… · p. 17

E. Accounting for side information

With side information, ordinary conditioning can include paths through another process at future times, so causal conditioning is used to define conditional information flow. In multivariate settings, instantaneous coupling complicates the decomposition into transfer entropy and exchange.

E. Accounting for side information

  • Conditioning on the whole observation can include a route from A to B via C in the future, motivating causal conditional directed information.The passage contrasts usual conditioning on the whole observation with causal conditioning.Source: E. Accounting for side information · p. 18
  • The multivariate case differs from the bivariate case because instantaneous coupling blurs the useful decomposition of conditional directed information.The section frames whether the bivariate decomposition extends as a question and identifies the extra coupling term.Source: E. Accounting for side information · pp. 17–18
  • Causal conditional transfer entropy tests flow from A to B while accounting for a possible route via C; it is zero when that flow is entirely relayed by C.In that situation, usual transfer entropy can remain nonzero, and conditioning on C examines whether the route goes through C.Source: E. Accounting for side information · p. 18
  • Defining conditional transfer entropy requires conditioning on C’s past; otherwise an extra term captures instantaneous exchanges involving C and B.The additional term distinguishes intrinsic coupling between C and B from extrinsic coupling involving other variables.Source: E. Accounting for side information · pp. 18–19
  • When C does not modify the bivariate results, the section retains the bivariate decomposition and its generalization to causally conditioned mutual information.The passages state that the bivariate results remain unchanged in this case.Source: E. Accounting for side information · p. 19
  • The section emphasizes Kramer’s causal conditioning for directed information and side information, and identifies transfer entropy as strict causal flow excluding simultaneous coupling.The distinction concerns the part of directed information transfer entropy measures.Source: E. Accounting for side information · p. 19

IV. INFERRING GRANGER CAUSALITY AND INSTANTANEOUS COUPLING

The section connects Granger-causality inference to information-theoretic measures of conditional independence. In particular, causal conditional transfer entropy characterizes Granger causality, while directed information in the bivariate case also captures instantaneous coupling.

IV. INFERRING GRANGER CAUSALITY AND INSTANTANEOUS COUPLING

  • Granger causality is not directly operational, so practical assessment requires tools for testing conditional independence.The section introduces directed-information measures as such tools within an inference framework.Source: §IV · pp. 19–20

A. Information-theoretic measures and Granger causality

  • The section recasts Granger-causality definitions based on conditional independence in information-theoretic terms.It relies on mutual information for independence and conditional mutual information for conditional independence.Source: A. Information-theoretic measures and… · p. 20
  • Causal conditional transfer entropy is zero if and only if A does not Granger-cause B relative to V.This equivalence makes the measure usable for assessing Granger causality.Source: A. Information-theoretic measures and… · p. 20
  • In the bivariate case, directed information decomposes into transfer entropy and instantaneous information exchange, representing Granger causality and coupling.The passage identifies directed information as summarizing both effects in this case.Source: A. Information-theoretic measures and… · p. 20

B. Granger causality inference

The section reviews inference of pairwise Granger causality and instantaneous coupling in a multivariate process, showing how directed-information measures arise in hypothesis-testing frameworks. It also identifies a restriction on the proposed multivariate testing extension: instantaneous uncoupling is assumed.

B. Granger causality inference

  • The inference task is to assess Granger causality and instantaneous coupling between component pairs of an observed multivariate process.The section considers evaluating directed-information measures from data; zero measures indicate no causality or coupling, respectively.Source: B. Granger causality inference · pp. 20–21

1) Directed information emerges from a hypotheses-testing framework:

2) Prediction based approach in the Gaussian case:

The section relates prediction-based Granger-causality indices to directed-information measures in Gaussian settings, then discusses model-based inference beyond that case. Its general likelihood-ratio connection to causal conditional transfer entropy is presented as a conjecture requiring suitable conditions.

2) Prediction based approach in the Gaussian case:

  • For Gaussian stationary processes, the prediction-based decomposition analogous to the mutual-information decomposition is given by the same relation.The section links the prediction-risk formulation to that decomposition.Source: 2) Prediction based approach in the… · p. 23
  • In the bivariate Gaussian case, Geweke’s indices for Granger causality and instantaneous coupling equal the corresponding directed-information measures.The related decomposition equates mutual information rate with the two directional measures and instantaneous coupling; the conditioned versions are also identified.Source: 2) Prediction based approach in the… · p. 23

3) The model-based approach :

  • Model-based inference compares models for B that include or exclude A; it can be parametric or search over a functional space.In the parametric case, inference estimates parameters, while a nonparametric example searches an appropriate function space.Source: 3) The model-based approach · p. 24
  • For nonlinear non-Gaussian prediction models, the mutual-information decomposition has no reason to remain valid.Source: 3) The model-based approach · p. 24
  • In the linear Gaussian case, model-based testing recovers Geweke’s measures, while convergence of the generalized log-likelihood ratio to causal conditional transfer entropy is conjectured under suitable conditions.The convergence statement depends on efficient maximum-likelihood estimates and sufficiently relevant model conditions.Source: 3) The model-based approach · p. 25

V. CONCLUSIONS

The conclusions identify conditional independence and causal conditioning as the key ingredients linking Granger causality with directed information theory. They also note practical and scope limits, including sparse consideration of nonstationary processes.

V. CONCLUSIONS

  • Conditional independence and causal conditioning are identified as the key ingredients connecting Granger causality and directed information theory.Source: §V · p. 25
  • The report leaves open how to use its definitions and measures practically, though some measures are readily implemented in linear Gaussian cases.For more general cases, the conclusions mention estimating information-theoretic measures or using prediction residuals to assess causality.Source: §V · p. 25
  • The report uses Kullback-divergence-based measures to illustrate the link, while noting that other divergences and probability-space metrics could assess conditional independence.Source: §V · p. 25
  • Causality between nonstationary processes has rarely been considered.The conclusions mention an ad-hoc neuroscience approach as an exception.Source: §V · p. 26
  • Sparse-network assumptions and Markov properties can support efficient estimation procedures in a graphical-modeling approach.Source: §V · p. 26
Loading 1211.3169v1…