Source-linked AI summary
On directed information theory and Granger causality graphs
P. O. Amblard, O. J. J. Michel
TL;DR
The paper addresses how to infer Granger causality graphs from multivariate stochastic processes using an information-theoretic framework that handles directional dependence. It establishes directed-information characterizations of graph edges and discusses estimation and testing issues for finite samples. The framework is presented as useful for connectivity inference, including neuroscience applications.
Problem
The paper addresses the need for an information-theoretic framework based on causal conditioning to assess directional dependencies and Granger causality graphs in networks.
Method
The paper uses directed information measures to characterize directed and instantaneous edges in Granger causality graphs.
Results
Directed information measures provide criteria for identifying absent directed and undirected edges in Granger causality graphs.
Takeaways & Limitations
Directed information is presented as an effective tool for connectivity inference across neuroscience measurements and scales.
Takeaways & Limitations
The paper notes that directed-information rates are generally defined as limits and cannot in general be directly evaluated, requiring finite-sample estimators.
Abstract
from arXiv · showhide
Directed information theory deals with communication channels with feedback. When applied to networks, a natural extension based on causal conditioning is needed. We show here that measures built from directed information theory in networks can be used to assess Granger causality graphs of stochastic processes. We show that directed information theory includes measures such as the transfer entropy, and that it is the adequate information theoretic framework needed for neuroscience applications, such as connectivity inference problems.
I. INTRODUCTION
The paper frames brain-connectivity inference as difficult because measurements vary across scales and signal types, while existing dependence measures differ in directionality and modeling assumptions. It proposes a general framework combining information theory and causality, centered on directed information for multivariate stochastic processes.
- Motivation: Brain connectivity inference lacks a universal graph-estimation method because observations differ in scale, signal properties, memory, and nonlinearity.The paper therefore advocates principles that can be adapted to each experimental situation.
- Contribution: The paper proposes a general connectivity framework based on information theory and causality principles.Dependence analysis supplies the main tools for inferring connectivity.
- Existing approaches: Connectivity measures range from correlation and mutual information to directional measures such as transfer entropy and Granger causality.Transfer entropy is highlighted as a popular information-theoretic measure applied to information flow in sensorimotor networks.
- Directed information: Directed information theory addresses feedback that symmetric quantities such as mutual information do not capture.The framework is recast from communication channels with feedback toward dependence analysis between stochastic processes.
- Paper scope: The paper develops causal conditioning for directional dependence, relates directed information to transfer entropy, and links the resulting measures to Granger causality graphs.The authors describe this link as one of the paper’s main points while keeping the treatment primarily conceptual.
II. GRANGER CAUSALITY GRAPHS
The section introduces graphical models for dependence structure and describes causality graphs as mixed graphs containing directed and undirected connections. Connections are defined using Granger causality, first in linear models and later under an unrestricted probability-based definition.
- Graph structure: A causality graph is a mixed graph whose nodes represent processes and whose directed or undirected edges encode different connectivity relations.The graph framework was introduced for graphical modeling of random processes and applied early to neuroscience.
- Edge definitions: Directed edges represent Granger-causal connections, whereas undirected edges represent instantaneous connections under the linear graph formulation.The later formulation generalizes the connection definition from linear models to probability measures.
A. Granger causality
Granger causality defines whether a process improves prediction of another beyond its past, conditional on other observed processes. The resulting mixed graph distinguishes directed temporal influence from symmetric instantaneous coupling, while emphasizing that causality depends on the information set.
- Granger causality: A process x_t does not Granger-cause y_t when y_t is conditionally independent of x’s past given y’s past and the additional process history.Equivalently, conditioning on x’s past does not change the conditional distribution of y_t.
- Conditioning and information sets: Adding an observed process can remove an apparent Granger-causal relation, so Granger causality is a property relative to the available information set.In the constructed three-process example, including z eliminates the direct relation from y to x.
- Instantaneous causality: Instantaneous causality captures contemporaneous coupling and is symmetric because it concerns present-time dependence conditional on joint past information.The paper distinguishes this relation from influence transmitted through a process’s past.
- Graph construction: A Granger causality graph assigns each time series to a node, directed edges to conditional temporal causation, and undirected edges to conditional instantaneous causation.The remaining observed processes are stacked into Z when defining each connection.
- Formal graph definition: For an M-dimensional process, directed and undirected edge sets are defined by whether conditional causal and instantaneous-causal relations are absent.The resulting mixed graph is denoted (V, E_d, E_u).
III. DIRECTED INFORMATION THEORY
The paper recasts directed information theory as a framework for dependence analysis between stochastic processes rather than communication theory alone. It emphasizes that the link between directed information measures and Granger causality graphs is developed through causal conditioning.
- Reframing directed information: This section reviews directed information theory to reformulate its definitions for dependence analysis between stochastic processes.The paper explicitly sets aside communication theory in its full generality.
- Causal conditioning: Causal conditioning is the central tool for assessing directional dependence among multiple time series.The paper develops the connection between directed information measures and Granger causality graphs in the following section.
A. Directional dependence between two stochastic processes
Mutual information measures dependence but is symmetric, so it cannot measure directional influence. Directed information replaces the ordinary input factor with a feedback-aware causal factorization, recovering mutual information when feedback is absent and decomposing shared information by direction.
- The framework uses Kullback-Leibler divergence to obtain mutual information by comparing a joint density with the product of its marginals.Mutual information is zero exactly when the two processes are independent.
- Mutual information is symmetric in x and y and therefore cannot measure directionality in their dependence structure.
- Directed information compares the joint process with a factorization that preserves feedback from y to x while removing influence from x to y.The feedforward and feedback factors describe the channel directions explicitly.
- Directed information is positive, no greater than mutual information, and equals mutual information if and only if there is no feedback.
- Its decomposition expresses shared information between two stochastic processes as information flowing in opposite directions.The paper identifies this directional decomposition as important for communication channels with feedback.
B. Causal conditioning, causal conditional directed information
Causal conditioning adapts entropy and directed information to time-ordered observations and additional processes. This yields causal conditional directed information, whose decomposition includes instantaneous information exchange and supports multivariate time-series analysis.
- Directed information can be decomposed into a transfer component and instantaneous information exchange, with the latter symmetric in x and y.
- Causal conditioning applies conditioning progressively through time, unlike mutual information, which conditions on the whole observation of x at each time.
- Causal conditioning and usual conditioning do not commute, and their definitions differ in whether the conditioning variable is global or causal.
- Causal conditional directed information extends conditional mutual information by conditioning directed information on an additional process z.The paper introduces it as the causal analogue of I(x; y|z).
- Causal conditional directed information is crucial for multivariate time series and appears in sums of directed information flowing in opposite directions.
C. Rates for stationary processes
For stationary processes, directed-information quantities are expressed as rates because information can grow with observation time. The resulting transfer entropy rate places directed information within a unified framework for stochastic-process dependence.
- Under stationarity, directed-information rates admit simpler expressions and extend classical entropy-rate results to discrete and continuous processes.
- Information rates are introduced because information quantities can diverge linearly when the process phase space grows with time.
- The framework separates transfer entropy from instantaneous information exchange, which is absent from Schreiber’s work according to the paper.
- The transfer entropy rate is identified with I∞(Dx1:t−1 →y), the limiting directed-information component based on past x.
- This identification recasts results and approaches from the literature within a single framework while making stationarity explicit in Schreiber’s intuition.
IV. CAUSAL INFORMATION MEASURES TO INFER GRANGER CAUSALITY GRAPHS
The paper links causal information measures to Granger-causality graph edges by conditioning on the remaining observed processes. Conditional transfer entropy detects directed edges, while conditional instantaneous exchange detects instantaneous causality.
- Inferring a graph from multivariate observations is formulated as assessing Granger causality between ordered node pairs relative to the remaining nodes.
- Conditional transfer entropy rate is zero if and only if a directed Granger-causality edge is absent between two nodes, conditioned on the remaining processes.
- Conditional instantaneous information exchange rate quantifies instantaneous causality relative to the other observed time series.
- A pair of nodes has no directed or undirected edge if and only if the causal conditional directed information rate conditioned on the remaining process is zero.
- The proof uses conditional independence and a Markov-chain dependence model to show that non-Granger-causally related processes have zero conditional transfer entropy rate.
V. DISCUSSION
The paper frames directed information theory as a framework for obtaining Granger causality graphs and studying directional dependencies in stochastic processes, particularly for neuroscience connectivity. It also emphasizes that practical inference remains constrained by estimation, finite-sample, stationarity, ergodicity, testing, and computational issues.
- V. DISCUSSION: Directed information theory provides a framework for obtaining Granger causality graphs and assessing directional dependencies in multiple time series.The framework includes causal relationships such as conditional transfer entropy and conditional instantaneous information exchange rate.
- V. DISCUSSION: The framework is presented as especially relevant to neuroscience because feedback is fundamental to modeling brain structures and directed information can support connectivity analysis.The discussion connects these tools to understanding information processing and coding in the brain.
- V. DISCUSSION: Practical estimation requires ergodicity and stationarity, but neural data may satisfy stationarity only over context-dependent time scales and may exhibit point-process or long-range-dependent behavior.Without ergodicity, time averages cannot replace ensemble averages, creating difficulties for estimating statistical quantities.
- V. DISCUSSION: Finite observation windows replace information-rate limits with finite-sample estimates that may retain initial-condition effects and introduce systematic bias.The discussion cites information flows in two-dimensional AR(1) processes as an illustration of this limitation.
- V. DISCUSSION: Estimating the required conditional mutual information is difficult because neural-data properties make information-measure estimation computationally demanding and theoretically incomplete.k-nearest-neighbor estimators reduce dependence on bin sizes or kernel widths and apply across continuous and point-process data, but their computational burden and convergence theory remain concerns.
- V. DISCUSSION: Graph inference additionally requires thresholding information-rate estimates and handling multiple testing, while null distributions for nonlinear statistics are difficult to obtain.Bootstrapping, surrogate data, and random permutations can provide thresholds, but increase computational load.