Source-linked AI summary
MINE: Mutual Information Neural Estimation
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, R Devon Hjelm
TL;DR
Estimating mutual information for high-dimensional continuous variables is difficult with traditional approaches. The paper introduces MINE, a neural estimator based on dual KL-divergence representations, and applies it across generative modeling and Information Bottleneck settings. The applications demonstrate scalable estimation and utility for minimizing or maximizing mutual information, while the method has a mini-batch gradient-bias issue that requires correction.
Problem
Mutual information is difficult to compute for general high-dimensional continuous variables because exact computation is limited and common estimators rely on non-parametric methods or approximate Gaussianity.
Method
MINE estimates mutual information with a parametric neural estimator based on dual KL-divergence representations, trained by back-propagation and analyzed theoretically.
Results
MINE is applied to mutual-information estimation, maximization, and minimization, including reducing GAN mode-dropping, improving ALI inference and reconstructions, and enabling continuous Information Bottleneck.
Takeaways & Limitations
MINE provides a tractable approach for applying mutual-information objectives to generative models and continuous Information Bottleneck methods.
Takeaways & Limitations
In mini-batch training, MINE's stochastic-gradient estimate is biased, requiring a correction procedure.
Abstract
from arXiv · showhide
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, and strongly consistent. We present a handful of applications on which MINE can be used to minimize or maximize mutual information. We apply MINE to improve adversarially trained generative models. We also use MINE to implement Information Bottleneck, applying it to supervised classification; our results demonstrate substantial improvement in flexibility and performance in these settings.
1. Introduction
Mutual information measures dependence but is difficult to compute for general high-dimensional continuous variables. The paper introduces MINE, a scalable neural estimator, and applies it to generative modeling and Information Bottleneck methods.
- Mutual information measures relationships between random variables and captures nonlinear statistical dependencies beyond correlation.It has applications including biomedical science, blind source separation, Information Bottleneck, feature selection, and causality.
- Exact mutual-information computation is generally tractable only for discrete variables or limited settings with known distributions.Existing general approaches use non-parametric estimators or approximations such as approximate Gaussianity.
- MINE estimates mutual information through dual representations of KL divergence using a scalable, flexible neural estimator trained by back-propagation.The approach extends dual optimization beyond the minimax objective used in GANs and includes theoretical analysis.
- MINE supports mutual-information estimation, maximization, and minimization rather than only the GAN minimax objective.
- The paper applies MINE to reduce mode-dropping in GANs and improve ALI reconstructions and inference on large-scale datasets.
- The paper applies MINE to continuous Information Bottleneck and reports that it outperforms variational bottleneck methods.
2. Background
Mutual information can be expressed as KL divergence between a joint distribution and the product of its marginals. The paper develops dual representations for estimating this divergence, emphasizing the stronger Donsker–Varadhan bound for fixed function classes.
- 2.1. Mutual Information: Mutual information measures dependence as the reduction in uncertainty and equals the KL divergence between the joint distribution and product of marginals.The divergence vanishes for fully independent variables.
- 2.1. Mutual Information: The KL-divergence formulation assumes absolute continuity of P with respect to Q.Under the stated compact-domain setting, the divergence is defined through densities relative to the Lebesgue measure.
- 2.2. Dual representations of the KL-divergence.: MINE relies primarily on the Donsker–Varadhan dual representation and also considers the dual f-divergence representation.These representations convert divergence estimation into optimization over functions T.
- 2.2. Dual representations of the KL-divergence.: For any fixed T, the Donsker–Varadhan bound is stronger than the f-divergence bound, although both are tight for sufficiently large function families.
3. The Mutual Information Neural Estimator
MINE estimates mutual information by optimizing a neural statistics network using the dual representation of KL divergence. The paper analyzes its approximation, consistency, sample complexity, and minibatch-gradient bias.
- 3.1. Method: MINE parameterizes a neural statistics network and maximizes a KL-divergence-based lower bound using empirical joint and marginal samples.Marginal samples can be obtained by shuffling joint samples along the batch axis, and the objective is optimized by gradient ascent.
- 3. The Mutual Information Neural Estimator: MINE defines a new class of neural information measures whose expressive neural networks can approximate mutual information, rather than equating every finite-network estimate with the exact quantity.The estimator relies on a chosen statistics network and samples from the data distribution.
- 3.2. Correcting the bias from the stochastic gradients: Naive minibatch stochastic gradients are biased because the denominator contains expectations estimated from the same minibatch.Replacing the denominator with an exponential moving average can make the bias arbitrarily small for small learning rates and improved performance in experiments.
- 3.3. Consistency: The neural-network family can approximate mutual information with arbitrary accuracy, while empirical extremum-estimation results provide convergence under mild parameter-space conditions.The consistency argument separates approximation error from estimation error.
- 3.3. Consistency: MINE is strongly consistent: for sufficiently many samples and an appropriate statistics network, its estimate converges almost surely to mutual information within any positive error.The paper states this as Theorem 2 and defines the guarantee using ϵ, N, and almost-sure convergence.
- 3.3.2. SAMPLE COMPLEXITY: The sample-complexity analysis bounds the number of samples needed for a target accuracy and confidence under Lipschitz, boundedness, and bounded-parameter-domain assumptions.The theorem explicitly treats ϵ and δ as desired accuracy and confidence parameters, with parameter dimension d entering the bound.
4. Empirical comparisons
MINE is evaluated against a nonparametric estimator on correlated multivariate Gaussians and tested for invariance under deterministic nonlinear transformations. It matches ground truth in low dimensions, improves over the comparison in twenty dimensions, and captures equitability.
- 4.1. Comparing MINE to non-parametric estimation: Both MINE and Kraskov’s estimator are virtually indistinguishable from ground truth for mutual information between bivariate Gaussians.The comparison uses multivariate Gaussians with componentwise correlation ρ ∈(−1, 1).
- 4.1. Comparing MINE to non-parametric estimation: MINE shows marked improvement over Kraskov’s estimator when estimating mutual information between twenty-dimensional random variables.The paper also compares the Donsker–Varadhan-based MINE with the f-divergence-based MINE-f.
- 4.1. Comparing MINE to non-parametric estimation: MINE provides a tighter mutual-information estimate than MINE-f in the reported comparison.MINE-f uses the f-divergence representation, whereas MINE uses the Donsker–Varadhan representation.
- 4.2. Capturing non-linear dependencies: MINE captures equitability, the property that dependence quantification remains invariant across the deterministic nonlinear transformations in this relationship.The authors report this result in Figure 2.
- 4.2. Capturing non-linear dependencies: For Y = f(X)+σ⊙ϵ, mutual information should remain invariant to the deterministic nonlinear transformation f and depend on the noise magnitude.The experiment uses 2-dimensional X ∼U(−1, 1), transformations x, x3, and sin(x), and Gaussian noise.
5. Applications
MINE is applied to maximize mutual information in generative models, improving mode coverage and reconstruction quality, and to estimate the Information Bottleneck regularizer.
- MINE applications maximize mutual information to improve generative-model mode representation and reconstruction, or minimize it to implement the Information Bottleneck.
- 5.1. Maximizing mutual information to improve GANs: Mode collapse occurs when GAN generators produce insufficiently diverse samples and poorly represent some modes of the target distribution.
- 5.1. Maximizing mutual information to improve GANs: The proposed GAN objective maximizes mutual information between generated samples and code variables to palliate mode collapse.
- 5.1. Maximizing mutual information to improve GANs: Both spiral and 25-Gaussians experiments show improved mode coverage over the baseline without a mutual-information objective.
- 5.2. Maximizing mutual information to improve inference in bi-directional adversarial models: MINE improves reconstruction performance in bi-directional adversarial models, outperforming ALI on MNIST reconstruction error and showing stronger effects on CelebA.
- 5.3. Information Bottleneck: MINE estimates the Information Bottleneck regularizer, extending the cross-entropy objective with a term promoting minimal representations.
6. Conclusion
MINE is presented as a scalable mutual-information estimator applied to generative modeling and continuous Information Bottleneck settings. These applications address mode dropping, reconstruction and inference quality, and tractability.
- MINE is scalable in dimension and sample size.
- ALI+MINE balances reconstruction with capturing all modes of the underlying data distribution, unlike ALICE’s perfect reconstruction with significant mode dropping.
- MINE improves mode representation and reconstruction in generative models, including ALI.
- MINE enables tractable application of Information Bottleneck methods in a continuous setting.
8. Appendix
The appendix provides additional experiment details and proofs omitted from the main text.
- The appendix supplies additional experiment details and spells out omitted proofs.
8.1. Experimental Details
The appendix documents implementation choices for MINE applications, including adaptive gradient clipping, experimental datasets, network architectures, optimizers, and training schedules.
- Adaptive Clipping: Adaptive clipping limits the mutual-information gradient’s Frobenius norm to the discriminator-gradient norm when MINE is maximized.
- Adaptive Clipping: Adaptive clipping is applicable whenever MINE is maximized.
- Experimental Details: Mode-dropping experiments use 100,000 examples from spiral and 25-Gaussians datasets.
- Experimental Details: Stacked-MNIST experiments compare GAN+MINE with PacGAN and VEEGAN using the same experimental setup and 26,000 test samples.
- Experimental Details: Bidirectional-model experiments use DCGAN-like architectures, Adam with learning rate 0.0001, and mini-batches of 100.
- Experimental Details: Information Bottleneck experiments match Alemi et al.’s hyperparameters and architectures, using a two-layer MLP statistics network with additive noise and 512 ELU activations.
8.2. Proofs
The appendix proves the dual KL-divergence representation underlying MINE, establishes strong consistency, and derives a reconstruction-error bound under stated assumptions.
- Dual Representation: The KL divergence admits a dual representation optimized over functions with finite expectations.
- Dual Representation: The dual bound is tight when the Gibbs distribution equals the target distribution, with optimal T* = log dP/dQ + C.
- Consistency: The consistency analysis assumes compact input and parameter domains, absolutely continuous measures, and continuous feedforward-network activations.
- Consistency: The empirical neural information measure is optimized as I_Θ(X, Z) = sup_θ∈Θ Î(T_θ).
- Consistency: MINE is strongly consistent.
- Reconstruction Error: The reconstruction-error bound is tight when the induced marginal q(z) matches the prior p(z).
8.3. Embeddings for bi-direction 25 Gaussians experiments
Figure 7 presents embeddings for the bidirection 25 Gaussians experiments corresponding to Figure 6.
- Figure 7 presents the embeddings for the experiments in this section.
- The embeddings correspond to the experiments shown in Figure 6.
- The section concerns experiments involving bidirection 25 Gaussians.