Source-linked AI summary
Mutual information is copula entropy
Jian Ma, Zengqi Sun
TL;DR
The paper addresses how mutual information relates to entropy and copulas. It proves that mutual information is negative copula entropy, then proposes a two-step estimation method based on this result. The method provides a competitive way to estimate mutual information in the reported Gaussian-variable experiment.
Problem
The paper examines the relationship between mutual information, entropy, and copulas.
Method
The paper proves the mutual-information/copula-entropy relation and estimates mutual information from samples in two steps, including empirical copula-density derivation.
Results
The proposed method provides a competitive way of mutual information estimation in experiments with correlated standard Gaussian variables.
Takeaways & Limitations
The theorem connects information theory and copula theory while supplying a basis for mutual information estimation from data.
Abstract
from arXiv · showhide
We prove that mutual information is actually negative copula entropy, based on which a method for mutual information estimation is proposed.
I. INTRODUCTION
The paper reframes mutual information as negative copula entropy, using copulas to isolate dependence information from marginal distributions.
- Mutual information is presented as an entropy-like quantity called copula entropy, with MI equal to its negative.
- Copulas represent a joint distribution through a copula and its marginal distributions.
- Separating margins from the joint distribution leaves the dependence information of the random variables in the copula.
- The notation uses C and c for copula function and density, D and F for joint and marginal distributions, and H, I, Hc for entropy, mutual information, and copula entropy.
II. THEOREM AND PROOF
The paper proves that mutual information equals the negative entropy of the corresponding copula and connects this result to information theory and copula theory.
- Theorem 1 states that mutual information equals the negative entropy of the corresponding copula function.
- Copula entropy measures information contained in the copula density, while entropy concerns the joint density and marginal densities.
- The corollary follows directly from the definition of mutual information and Theorem 1.
- The result connects mutual information with copula theory and links information theory to copula theory.
III. ESTIMATING MUTUAL INFORMATION VIA COPULA
The proposed mutual-information estimator uses copula-based estimation in two steps: first estimate the empirical copula density, then estimate copula entropy.
- The estimator first estimates the empirical copula density.
- It then estimates copula entropy.
- Empirical copula density is derived from i.i.d. samples using empirical functions.
- Figure 1 compares analytic values with kNN-based and copula-entropy-based mutual-information estimates.
IV. EXPERIMENTS
The method is evaluated on correlated standard Gaussian variables across covariance values from 0 to 0.9 and is reported to provide a competitive mutual-information estimation approach.
- The experiment uses two correlated standard Gaussian variables with covariance ρ from 0 to 0.9 in increments of 0.1.For each covariance value, the authors generate 1,000 samples.
- 1,000 samples are generated for each covariance value, and copula entropy is estimated using the method in.
- The proposed method provides a competitive way of estimating mutual information against the contrasting estimator in.Both methods are run on the same sample sets.