PaperlayerView exact revision on arXiv

Source-linked AI summary

JIDT: An information-theoretic toolkit for studying the dynamics of complex systems

Joseph T. Lizier

arXiv:1408.3270v2cs.ITcs.MScs.SInlin.AOphysics.data-an

TL;DR

Complex systems are increasingly analyzed as distributed information-processing systems, creating a need for a general-purpose toolkit for empirical information-dynamics measurements. The paper introduces JIDT, an open-source Java toolkit with broad estimators, local and multivariate measures, cross-environment access, and demonstrations of its use.

  • Problem

    Researchers need a general-purpose information-theoretic toolkit that supports multiple environments, measures, estimators, local values, and statistical significance.

  • Method

    JIDT implements information-theoretic measures of dynamics with discrete and continuous estimators, object-oriented interchangeable implementations, and interfaces for MATLAB, Octave, Python, and other environments.

  • Results

    JIDT provides a wider array of measures and estimators than related toolkits, including local measures and statistical significance, as standalone platform-agnostic software.

  • Takeaways & Limitations

    JIDT offers portable empirical tools for studying information storage, transfer, and modification in complex-system time-series data.

Abstract

from arXiv · show

Complex systems are increasingly being viewed as distributed information processing systems, particularly in the domains of computational neuroscience, bioinformatics and Artificial Life. This trend has resulted in a strong uptake in the use of (Shannon) information-theoretic measures to analyse the dynamics of complex systems in these fields. We introduce the Java Information Dynamics Toolkit (JIDT): a Google code project which provides a standalone, (GNU GPL v3 licensed) open-source code implementation for empirical estimation of information-theoretic measures from time-series data. While the toolkit provides classic information-theoretic measures (e.g. entropy, mutual information, conditional mutual information), it ultimately focusses on implementing higher-level measures for information dynamics. That is, JIDT focusses on quantifying information storage, transfer and modification, and the dynamics of these operations in space and time. For this purpose, it includes implementations of the transfer entropy and active information storage, their multivariate extensions and local or pointwise variants. JIDT provides implementations for both discrete and continuous-valued data for each measure, including various types of estimator for continuous data (e.g. Gaussian, box-kernel and Kraskov-Stoegbauer-Grassberger) which can be swapped at run-time due to Java's object-oriented polymorphism. Furthermore, while written in Java, the toolkit can be used directly in MATLAB, GNU Octave, Python and other environments. We present the principles behind the code design, and provide several examples to guide users.

I. INTRODUCTION

Information theory is increasingly used to study complex systems as distributed information-processing systems, motivating measures of information storage, transfer, and modification. JIDT addresses this need with a general-purpose toolkit implementing these measures and providing code examples, estimators, and multi-environment access.

  • Motivation: Complex-system behavior is increasingly studied as distributed information processing among interacting elements, including information storage, transfer, and modification.This perspective has been applied to computational neuroscience, canonical complex systems, complex networks, social media, Artificial Life, and modular robotics.
  • Motivation: Information theory offers model-free, stochastic, nonlinear, general, and mathematically sound tools for quantifying information processing in dynamics.Existing measures include transfer entropy, active information storage, and predictive information.
  • JIDT contribution: JIDT is a standalone, GNU GPL v3 open-source toolkit for empirical estimation of information-theoretic measures from time-series data.It provides portable implementations of information transfer, storage, shared information, and entropy.
  • JIDT contribution: The toolkit implements entropy, mutual information, conditional mutual information, active information storage, transfer entropy, conditional and multivariate forms, and local values.Local measures assign information-theoretic values to specific observations, allowing their dynamics to be examined over time.
  • Toolkit design: JIDT includes multiple estimator types for discrete and continuous data, organized through an object-oriented architecture with interchangeable estimator-specific implementations.The architecture uses common parent classes and specialized child classes, including KSG implementations, while estimator choices depend on data type and properties.
  • Information-theoretic measures: Table I summarizes basic information-theoretic quantities and information-dynamics measures in both average and local forms.The table provides equations and references to their presentations in the appendices.

Y1 B X A Y2 Variables

JIDT frames information dynamics as the decomposition of a destination observation into stored information, transfers from source variables, and related statistical relationships. It packages these measures with multiple estimators, data-type support, local analysis, and significance testing in a general-purpose toolkit.

  • Information-dynamics framework: The framework decomposes information in the next observation X_n+1 into information stored in X's past and information transferred from other processes.The toolkit also treats information processing as involving non-trivial combinations of information sources.
  • Information-dynamics framework: Local active information storage and local transfer entropy quantify how information processing unfolds over time in destination X.Figure 1 illustrates storage in X and transfers from Y1 and Y2 for a measurement at time n + 1.
  • Toolkit scope: The paper identifies a gap for a general-purpose toolkit combining broad measures, multiple estimators, local values, and significance measures across code environments.It contrasts this scope with TRENTOOL's specialization in effective connectivity analysis of continuous neural data.
  • Toolkit coverage: JIDT implements broad measure coverage, including conditional and multivariate transfer entropy, active information storage, and fully specified embedding parameters.These features are presented as part of JIDT's general-purpose scope.
  • Estimators and data types: JIDT supports discrete and continuous data through plug-in, Gaussian, kernel, permutation, and KSG estimator approaches.Continuous estimators differ in assumptions and computational behavior: Gaussian estimation assumes linear interactions, while KSG adapts neighborhood size to local sample density.
  • Toolkit coverage: JIDT provides local measurements for all estimators and statistical significance calculations for MI, TE, and their conditional variants.Permutation testing is used to construct surrogate populations under a null hypothesis of no directed relationship.

A. Installation and dependencies

JIDT is distributed as standalone Java software with minimal installation requirements, while optional build, testing, and language integrations add dependencies. Its distribution includes libraries, source, documentation, demos, tests, and licensing materials.

  • Installation: JIDT requires little installation beyond downloading the software and runs on platforms supporting a standard Java Runtime Environment.The supported platforms listed are Windows, Mac, Linux, and Solaris.
  • Building and testing: Building from source is optional and can use the supplied Ant script or an IDE; running unit tests requires JUnit.The build script supports compilation, packaging, documentation generation, and test execution.
  • Dependencies: Users need a JRE to run JIDT, while modifying, building, or writing Java access code requires the Java SE JDK.MATLAB already includes a JVM for using JIDT.
  • Language integrations: GNU Octave and Python use require additional Java integration packages and a JVM.Octave requires octave-java; Python requires a relevant Python-Java extension.
  • Distribution contents: The full distribution contains the JAR library, license, build script, source, unit tests, Javadocs, demos, sample data, and notices.A full distribution is recommended for access to the examples, while a jar-only distribution contains only the library archive.
  • Code organization: JIDT includes discrete, continuous, mixed, utility, and network-inference packages, but experimental discrete-to-continuous MI calculators are not discussed in detail.The continuous package organizes estimator implementations into subpackages such as Gaussian, kernel, Kozachenko, Kraskov, and symbolic.

D. JIDT architecture

JIDT uses an object-oriented architecture that separates measure interfaces, shared abstract functionality, and estimator-specific child classes. This design supports code reuse, maintenance, runtime estimator selection, and validation through unit tests compared with existing tools and published results.

  • Architecture: Continuous-data calculators are organized as hierarchies of interfaces, abstract classes, and estimator-specific child classes.The example UML structure covers conditional mutual information and transfer entropy using KSG estimators.
  • Interfaces: Interfaces define the methods for each measure, allowing estimator classes to share a common calling contract.The ConditionalMutualInfoCalculatorMultiVariate and TransferEntropyCalculator interfaces exemplify this top layer.
  • Inheritance and specialization: Abstract classes provide shared measure functionality, while child classes inherit it and add estimator-specific implementations such as KSG algorithm 1 or 2.TransferEntropyCalculatorViaCondMutualInfo uses a conditional-MI interface without fixing the concrete estimator or all parameters.
  • Design benefits: Decoupling common code from estimator types enables code reuse and simpler maintenance.Common parent classes hold reusable functionality that specialized child classes extend.
  • Design benefits: Subtype polymorphism enables code to compute a measure through its interface while selecting the estimator type at runtime.The paper identifies this as dynamic dispatch and demonstrates it in the usage examples.
  • Validation: Unit tests validate calculators against other toolkits, published transfer-entropy plots, and a modified Granger-causality script.Examples include MILCA, TRENTOOL, Schreiber’s transfer-entropy paper, and ChaLearn sample code.
  • Validation: Further unit-test code coverage is planned for future work.This is stated as an explicit validation limitation.

IV. JIDT CODE DEMONSTRATIONS

JIDT’s demonstrations teach common usage patterns across discrete and continuous data, estimator types, multivariate settings, ensembles, local measures, significance testing, and runtime dispatch. The examples are distributed as standalone Java programs with scripts and sample data.

  • Demonstration set: The Simple Java Demos provide eight standalone programs illustrating core toolkit usage.They are located in demos/java and cover several discrete and continuous information-theoretic calculations.
  • Running examples: The demonstration source files are accompanied by shell scripts showing how to compile and run programs against infodynamics.jar.The scripts include javac and java classpath commands.
  • Demonstration coverage: The examples cover transfer entropy for binary, kernel-estimated continuous, KSG-estimated continuous, multivariate, ensemble, binned, and selected multidimensional data.They also include mutual information with dynamic estimator dispatch.
  • Running examples: Release v1.0 does not include the batch files, which were available in the SVN repository and planned for future releases.This limitation concerns Windows batch-file support for the demonstrations.
  • Demonstration coverage: The demonstrations extend basic estimator use to local measures, statistical significance, ensembles of samples, multivariate measures, and dynamic dispatch.The paper presents discrete and continuous examples as typical calling patterns.
  • Discrete data pattern: A discrete transfer-entropy example constructs a calculator, initializes it, adds source and destination observations, and computes the average local result.For a destination copying the source’s previous randomized binary value, the result should approach 1 bit.
  • Discrete data pattern: Discrete calculators use int[] time-series arrays, with separate arrays for univariate source and destination data.Multidimensional time series are handled in a separate example.

2. Typical calling pattern for an information-theoretic measure on continuous data

For continuous data, JIDT calculators are configured, initialized, supplied with observations, and then used to compute average, local, or significance measures. The standard example uses a KSG estimator for transfer entropy.

  • Calling pattern: JIDT’s standard continuous-data workflow constructs a calculator, sets estimator properties, initializes it, supplies observations, and computes the measure.The example uses a KSG transfer-entropy calculator with k set to 4 and history length initialized to 1.
  • Derived quantities: After observations are supplied, calculators can return average measurements, local values at each time step, and empirical or analytical significance distributions.Local TE is returned as a double[] array, while computeSignificance() supports resampled null distributions and some calculators provide analytical alternatives.
  • Calling pattern: Continuous estimators use double[] time-series arrays, and their parameters can be supplied through constructors, setProperty(), or overloaded initialise() methods.Initialization finalizes property settings and may accept the embedded history length.
  • Derived quantities: For transfer entropy, the first k local values are zero because the required history is unavailable.This boundary applies when computing local TE from the supplied observations.

5. Ensemble approach: using multiple trials or realisations to construct PDFs

JIDT supports multiple time-series realisations for constructing probability distributions and provides distinct handling for multivariate discrete and continuous data. Multivariate inputs are represented as time-by-variable arrays, with joint-variable preparation differing by data type.

  • Ensemble observations: Multiple continuous time-series realisations can be accumulated with startAddObservations(), repeated addObservations(), and finaliseAddObservations().This provides an alternative to supplying one realisation with setObservations().
  • Ensemble observations: After ensemble observations are finalized, computations proceed as usual, although local-value return formats may differ after multiple addObservations() calls.The documentation describes the more complicated return format for local quantities in this case.
  • Multivariate data: For multivariate discrete data, JIDT combines each time-step’s joint vector into one discrete value before applying the standard discrete calculator.A v-variable vector with M possible values per variable is treated as a v-digit base-M number.
  • Multivariate data: Multivariate continuous data uses separate calculators that receive two-dimensional double[][] arrays indexed by time and variable.The dimensions of each variable must be specified during initialization.

8. Coding to interfaces; or dynamic dispatch

JIDT encourages coding against calculator interfaces so implementations can be selected dynamically at runtime. The toolkit also mirrors Java demonstrations in MATLAB, Octave, and Python through language-specific Java integration.

  • Dynamic dispatch: Coding against a calculator interface allows JIDT to instantiate different implementing classes dynamically through Java’s polymorphism and dynamic dispatch.The implementation class can be loaded from a variable rather than hard-coded directly.
  • MATLAB and Octave: MATLAB and Octave users add the JIDT classpath, construct Java objects, and convert native arrays where necessary before calling toolkit methods.Java arrays remain zero-indexed even when methods are called from MATLAB or Octave.
  • Cross-language use: The MATLAB, Octave, and Python scripts reproduce capabilities shown in the Java demonstrations, subject to their respective integration requirements.The Python examples specifically use JPype, while Octave examples provide array-conversion helpers.
  • Python: Python demonstrations use JPype to import packages, start the JVM with the JIDT classpath, construct Java classes, and shut down the JVM afterward.Native-to-Java array conversion can require special handling, and the examples would change for another Python-Java interface.

D. Schreiber’s Transfer Entropy Demos

JIDT includes demonstrations covering transfer entropy, active information storage, cellular automata, interaction lags, and null distributions. These examples show how the toolkit supports estimator selection, local information profiles, and multivariate analyses across environments.

  • D. Schreiber’s Transfer Entropy Demos: The Schreiber transfer-entropy demos reproduce examples using discrete and box-kernel estimators on map simulations and heart-breath rate data.They also clarify parameter settings such as bias correction that were not explicit in the original paper.
  • D. Schreiber’s Transfer Entropy Demos: The heart-breath rate demonstration uses KSG AIS estimates with K = 4 nearest neighbours while varying embedded history length k.The example is implemented through a MATLAB function for k values 1:15.
  • E. Cellular Automata Demos: Cellular-automata demos plot local information measures across space-time, identifying gliders as transfer entities, blinkers and background domains as storage components, and collisions as modification events.The demonstrations reproduce local information-dynamics results for cellular automata.
  • F. Other Demos: The interaction-lag demos compare TE and MIT, reporting that TE is more suitable for source-destination lags because MIT can be misled by source memory.The same demos report that symbolic TE can miss important information components.
  • V. CONCLUSION: The toolkit’s conclusion highlights broad measure and estimator coverage, local measures, significance testing, standalone use, and access from MATLAB, Octave, Python, and other environments.The authors also describe future work on multithreading, GPU computing, automated embedding selection, additional estimators, and expanded unit-test coverage.

Appendix A: Information-theoretic measures

JIDT implements basic information-theoretic measures and extends them to quantify information dynamics in time-series processes. Its framework focuses on how information is stored, transferred, and related across past and future observations.

  • Basic measures: JIDT covers entropy, mutual information, conditional mutual information, and related multivariate information-theoretic measures.The toolkit first presents basic measures before developing measures focused on information dynamics.
  • Information dynamics: Information dynamics asks where the information in a future observation X_n+1 comes from, including contributions from a process’s past and other processes.The framework applies these questions to time-series processes and their realizations across countable time indices.
  • Information dynamics: Entropy rate measures the limiting growth of uncertainty, while conditional entropy rate describes how X_n+1 relates to preceding observations.Finite-k estimates quantify conditional entropy rates at specific time indices.
  • Information storage: Excess entropy measures structure or memory, and its predictive-information formulation captures past information that is also present in the future.This formulation provides an interpretation of excess entropy as stored information potentially distributed across external variables.
  • Information storage: Active information storage measures information from a process’s past that is observed to be used in computing its next observation.It directly addresses the sources of information in X_n+1.

3. Local information-theoretic measures

Local information-theoretic measures assign information values to specific observations rather than only expected averages. In information dynamics, they reveal how storage and transfer fluctuate across time and space.

  • Local measures: Local measures characterize information associated with specific measurements, complementing traditional expected or average information measures.They can be formed from local versions of basic information-theoretic quantities.
  • Local information dynamics: Local transfer entropy produces time-resolved values describing information transfer between processes, while local active information storage tracks changing use of memory.The toolkit also defines local conditional transfer entropy and other local information-dynamics measures.
  • Local measures: Local mutual information measures the information provided by a particular y about a particular x and is symmetric in x and y.It represents the coding cost of encoding x without knowledge of y.
  • Local measures: A local mutual-information value may be positive or negative: knowing y can increase expectation of x or change belief in the opposite direction.Positive values occur when p(x | y) > p(x), while negative values occur when p(x | y) < p(x).
  • Estimation: Local probability distributions may be estimated across repeated realizations or across time from stationary time series.The latter approach uses multiple observations from one or several full stationary realizations.
  • Local information dynamics: Local active information storage can be positive or negative, indicating that a process’s past may inform or misinform its next value.These values expose fluctuations in memory use at different time points.

4. Differential entropy

For continuous-valued variables, JIDT uses differential entropy and related continuous-variable information measures. These retain key mutual-information properties while differing from discretized Shannon entropy in important ways.

  • Differential entropy: Continuous-valued variables are analyzed with differential entropy rather than strictly discrete Shannon entropy.The toolkit’s continuous estimators evaluate differential-entropy quantities for continuous data.
  • Differential entropy: Differential entropy changes when the variable is rescaled and may be negative, unlike the corresponding intuition for discretized Shannon entropy.It is strongly related to Shannon entropy but has important differences.
  • Derived measures: Joint and conditional differential entropies follow the same chain-rule structure used for Shannon measures.Differential mutual information and derived quantities such as transfer entropy are defined analogously.
  • Derived measures: Differential mutual information has the same properties as discrete mutual information and equals the mutual information of discretizations as bin size approaches zero.The equivalence is stated for discretizations X^Δ and Y^Δ in the limit Δ→0.

5. Statistical significance testing

JIDT evaluates statistical significance by comparing observed information measures with null distributions representing absent relationships. Analytic distributions apply in some settings, while resampling is required when no analytic form is known.

  • Testing rationale: Finite-sample estimates of MI, conditional MI, and transfer entropy can be nonzero even when no relationship exists, motivating significance testing.The test evaluates evidence against a null hypothesis of no relevant relationship.
  • Null construction: Under the null, surrogate variables retain appropriate marginal or conditional properties while destroying the relationship being tested.For MI, surrogates retain p(y) and the marginal properties while replacing p(x | y) with p(x); conditional MI uses p(x | z).
  • Null distributions: Analytic chi-square null distributions are available for linearly coupled Gaussian variables and, asymptotically, for discrete variables.The discrete result depends on alphabet sizes and sample count, and convergence is slower for skewed distributions.
  • Null distributions: When no analytic null distribution is known, JIDT estimates it empirically using permutation or bootstrap resampling.Surrogate measurements are generated repeatedly to form a population for comparison.
  • Transfer entropy: Transfer-entropy surrogates must preserve the destination’s conditional dependence on its own past; directly shuffling the source series is invalid.Valid alternatives include shuffling source-history vectors, rotating a stationary source series, or swapping trials.
  • Multiple comparisons: Multiple hypothesis tests require correction using family-wise error rates or false discovery rates.This issue arises in effective network inference and other multivariate time-series applications.

Appendix B: Estimation techniques

JIDT includes multiple estimators for empirical information-theoretic measures, whose suitability depends on data type and finite-sample properties. For continuous data, Gaussian estimation provides a fast, parameter-free option but assumes linear interactions.

  • Finite-sample estimation can introduce bias and variance, and estimator choice depends on the data type and its properties.The toolkit therefore distinguishes estimator approaches rather than treating the mathematical definitions as sufficient for empirical analysis.
  • Discrete estimators use plug-in probabilities from counts and can compute local and average measures in O(N) time.Embedded past vectors required for measures such as transfer entropy can be cached and updated at each step.
  • Continuous data can be binned with fast O(N) discrete estimators, but continuous estimators are more complicated and can improve accuracy.The latter operate through differential entropy and probability-density estimation rather than discretisation.
  • Gaussian estimation models multivariate variables and obtains information-theoretic measures from sums and differences of joint entropies.For a d-dimensional variable, the entropy uses the determinant of its d × d covariance matrix; the estimator is fast and parameter-free but assumes linear interactions.
  • Gaussian estimation also supports local measures by reconstructing the probability of each observation and using those values as plug-in estimates.The same local-entropy construction extends to other local quantities through sums and differences.

b. Kernel estimation

JIDT implements kernel and KSG estimators for continuous information-theoretic measures, alongside permutation-based approaches. These methods trade off nonlinear, model-free estimation, computational cost, parameter sensitivity, and assumptions about ordinal structure.

  • Kernel estimation: Box-kernel estimation uses sample similarity within resolution r to estimate joint PDFs and then evaluates local measures for each sample before averaging.JIDT uses a step kernel with maximum-distance norm by default, and conditional probabilities are formed from component joint probabilities.
  • Kernel estimation: Kernel estimation can capture nonlinear relationships without a model, but resolution choice, bias, and computational cost remain limitations.JIDT uses box-assisted methods for efficiency; naive algorithms require O(·) time, while neighbour-search methods can reduce complexity.
  • KSG technique: KSG improves kernel estimation through nearest-neighbour log-probabilities, bias correction, and an adaptive neighbourhood width based on a fixed K.It is designed to improve data efficiency and accuracy for sparse observations while remaining relatively stable to K.
  • KSG technique: KSG mutual-information estimation counts neighbours in marginal spaces using widths determined by the Kth neighbour in the joint space.The first algorithm uses the maximum of the x and y distances, whereas the second sets marginal widths separately.
  • KSG technique: KSG is bias corrected and robust to K from K = 4 upwards, while algorithm 1 is favoured for smaller samples and algorithm 2 for very large samples.Variance decreases with K; the two algorithms differ in their accuracy–bias trade-off across sample sizes.
  • KSG technique: KSG extends to conditional mutual information and transfer entropy through dedicated single estimators, including local variants based on nearest-neighbour counts.The toolkit also describes local KSG mutual-information and conditional-mutual-information calculations.
  • KSG technique: KSG offers stronger continuous-data capabilities than kernel estimation but can be computationally expensive in JIDT v1.0.The release used a naive algorithm, while faster nearest-neighbour search was available through project SVN for future releases.
  • Permutation entropy: Permutation entropy symbolises joint vectors by ordinal structure and can support symbolic transfer entropy, including local versions.These approaches are computationally fast but assume ordinal relationships contain all relevant information, which can produce misleading results.
Loading 1408.3270v2…