Source-linked AI summary
Information Theoretical Estimators Toolbox
Zoltan Szabo
TL;DR
Information-theoretic estimation packages, the paper notes, often focus on discrete variables or specialized methods, while many applications use these measures. ITE provides a modular Matlab/Octave toolbox for estimating a broad range of quantities and composing estimators for optimization, with an ISA prototype application.
Problem
Available packages are described as focusing on discrete variables or specialized estimation methods, despite the broad applications of information-theoretic measures.
Source: §1 · pp. 1–2Method
Results
ITE provides estimates for multiple entropy, mutual-information, association, divergence, and distribution-kernel variants, along with a prototype ISA application.
Source: §1 · p. 2Takeaways & Limitations
The toolbox supports applying and combining information-theoretic estimators in optimization problems, including an ISA prototype and extensions.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We present ITE (information theoretical estimators) a free and open source, multi-platform, Matlab/Octave toolbox that is capable of estimating many different variants of entropy, mutual information, divergence, association measures, cross quantities, and kernels on distributions. Thanks to its highly modular design, ITE supports additionally (i) the combinations of the estimation techniques, (ii) the easy construction and embedding of novel information theoretical estimators, and (iii) their immediate application in information theoretical optimization problems. ITE also includes a prototype application in a central problem class of signal processing, independent subspace analysis and its extensions.
1. Introduction
Information-theoretic measures support applications across machine learning, including signal processing problems such as independent subspace analysis. ITE addresses the limited availability of packages for continuous-variable estimation with a modular toolbox that also supports estimator construction, optimization, and an ISA prototype.
1. Introduction
- ITE targets the gap left by packages focused on discrete variables or specialized estimation methods, offering a modular, free, open-source, multi-platform toolbox for continuous variables.The introduction presents ITE as a response to the limited package options available to the authors.Source: §1 · p. 2
- ITE estimates many kinds of entropy, mutual information, association and divergence measures, and distribution kernels using nonparametric methods.Source: §1 · p. 2
- ITE supports constructing estimators from existing methods or from scratch and using them in information-theoretic optimization problems.Source: §1 · p. 2
- The toolbox includes a prototype application to independent subspace analysis and its extensions.Source: §1 · p. 2
2. Library Overview
ITE combines a broad collection of information-theoretic estimators with a modular framework for composing estimators and solving optimization problems. Its overview illustrates these capabilities with estimator examples, ISA solvers, and tests.
Information Theoretical Measures
- ITE includes estimators for entropy, mutual information, association, divergence, cross quantities, and kernels on distributions.The overview lists variants including Shannon, Rényi, and Tsallis entropy; Kullback–Leibler divergence; and distribution kernels.Source: §2 · pp. 2–3
Quick Tests
- The overview describes quick tests for estimator accuracy, positive semi-definiteness of distribution-kernel Gram matrices, and image registration problems.Source: §2 · p. 3
Modularity
- ITE uses a modular design that reduces many estimators to reusable computational techniques, relates quantities through information-theoretic identities, and decomposes ISA into subtasks.The listed techniques include nearest-neighbor and kernel methods; ISA subtasks include ICA, clustering, and regression.Source: §2 · p. 3
Dedicated Subtask Solvers, Extension
- ITE provides dedicated solvers for ISA subproblems, and extensions can be added by introducing a switch entry in a subtask solver.Source: §2 · p. 3
Base and Meta Estimators
- Base and meta estimators share a unified calling syntax, and new meta estimators can be derived from existing base or meta estimators.The J-distance example symmetrizes KL divergence estimated with an existing k-nearest-neighbor method.Source: §2 · pp. 3–4
ISA Objectives and Optimization
- ITE supports information-theoretic optimization, including ISA objectives that can use any base or meta estimator; spectral-clustering approximations handle unknown subspace dimensions.The overview reports that a general desktop computer can handle about a million ICA elements within several minutes.Source: §2 · p. 4
3. Availability and Requirements
ITE runs in Matlab or Octave with standard toolboxes and is released as free, open-source software. The package has been extensively tested on Windows and Linux, with alternative-platform compatibility expected.
3. Availability and Requirements
- ITE requires a Matlab or Octave environment with standard toolboxes and is released under the GNU GPLv3 or later license.Source: §3 · p. 4
- ITE has been extensively tested on Windows and Linux; compatibility with other platforms is expected because it uses standard Matlab/Octave and C++ files.The paper suggests Visual C++ for Windows and GCC for Linux.Source: §3 · p. 4
- The toolbox documentation includes comments, examples, extension instructions, and pointers to further mathematical details.Source: §3 · p. 4