Source-linked AI summary
CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, Lawrence Carin
TL;DR
Estimating and minimizing MI from samples is difficult, particularly in high-dimensional settings where lower-bound methods do not directly support minimization. The paper introduces CLUB and its variational form, then uses them for MI minimization with negative sampling; experiments report reliable estimation and effective minimization, while existing upper bounds retain practical limitations.
Problem
Sample-based MI estimation and minimization remain challenging in high-dimensional settings, while prior methods mainly approximate lower bounds unsuitable for MI minimization.
Method
The paper introduces CLUB, extends it to variational CLUB for unknown conditional distributions, and develops a negative-sampling-accelerated MI minimization algorithm.
Results
CLUB shows the lowest bias among compared methods in the reported MI estimation study, while CLUB and CLUBSample have an insignificant bias difference.
Takeaways & Limitations
CLUB provides a trainable MI upper bound for estimation and MI minimization, with reported effectiveness in simulation studies and real-world applications.
Takeaways & Limitations
Existing upper-bound approaches can require difficult marginal approximation, sufficient sample size, or known conditional information, and L1Out remains numerically unstable in real-world minimization.
Abstract
from arXiv · showhide
Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.
1. Introduction
MI is useful for controlling dependence in machine learning, but sample-based minimization is difficult because prior estimators mainly provide lower bounds. CLUB addresses this gap with a contrastive upper bound, a variational approximation, and an accelerated minimization scheme.
- MI is used as a criterion or regularizer to encourage or limit dependence between variables in machine learning.
- Exact MI calculation is usually unavailable because it requires tractable density functions and log-density ratios, while most tasks provide only joint-distribution samples.
- Previous sample-based MI methods mainly estimate lower bounds, whereas MI upper-bound estimation has received less exploration despite its relevance to minimization.
- CLUB estimates MI through differences between conditional probabilities for positive and negative sample pairs, connecting MI estimation with contrastive learning.
- The variational CLUB approximation uses a neural network for unknown p(y|x), while the proposed minimization algorithm is accelerated through negative sampling.
- CLUB is reported to improve the bias-variance estimation trade-off and effectiveness in MI minimization across simulation studies and real-world applications.
2. Background
MI estimation becomes unreliable in high-dimensional or intractable settings, and most differentiable estimators target lower bounds rather than minimization-compatible upper bounds. Existing upper bounds face approximation, sample-size, or numerical-stability limitations that motivate variational comparisons.
- Classical estimators such as binning, kernel density, likelihood-ratio, and nearest-neighbor methods become unreliable as data dimension increases.
- Deep neural MI estimators mainly construct lower bounds, including BA, MINE, NWJ, and InfoNCE, for differentiable MI estimation.
- Most existing MI upper bounds require a known conditional distribution, limiting their use when p(y|x) is unavailable.
- VUB replaces the marginal p(y) with a variational approximation r(y), but high-dimensional density approximation is challenging and a fixed standard normal can produce high-bias estimates.
- L1Out uses a Monte Carlo approximation that depends strongly on sufficient sample size and can become numerically unstable in real-world MI minimization.
- Variational L1Out performs better than previous lower bounds in simulation, but its numerical instability persists in real-world applications.
3. Proposed Method
CLUB provides a sample-based mutual-information upper bound using conditional log-likelihood differences between positive and negative pairs. Its variational form extends the approach when p(y|x) is unknown, supports MI minimization, and can be accelerated with unbiased negative sampling.
- CLUB with p(y|x) Known: CLUB estimates mutual information from positive and negative sample pairs when a conditional distribution such as p(y|x) is available.The estimator contrasts log p(y_i|x_i) with log p(y_j|x_i) for negative pairs.
- CLUB with p(y|x) Known: The CLUB quantity is an MI upper bound because Jensen’s inequality makes the gap between CLUB and mutual information non-negative.The bound is tight when x and y are independent.
- CLUB with p(y|x) Known: The sample estimator is unbiased and uses a linear combination of positive-negative conditional log-ratios, improving numerical stability for CLUB and its gradient.The linear form distinguishes CLUB from neural MI estimators with less stable nonlinear probability transformations.
- CLUB with Conditional Distributions Unknown: vCLUB replaces the unknown conditional p(y|x) with a neural-network variational approximation q_θ(y|x).The approximation is trained by maximizing conditional log-likelihood on sample pairs.
- CLUB with Conditional Distributions Unknown: With a good variational approximation, vCLUB can remain an MI upper bound or become an estimator whose absolute error is bounded by approximation error.The relevant approximation error is KL(p(y|x)∥q_θ(y|x)).
- CLUB in MI Minimization: The proposed training scheme alternates updating q_θ(y|x) and minimizing the vCLUB estimator with respect to the joint distribution.The reparameterization trick enables gradients to back-propagate through sampled embeddings.
- CLUB in MI Minimization: Negative sampling preserves unbiasedness while reducing per-iteration computation, and sampled vCLUB retains comparable estimation performance while improving generalization as a learning critic.This property follows from CLUB’s linear log-ratio summation rather than applying a nonlinear logarithm after probability aggregation.
4. Experiments
Experiments evaluate CLUB and its sampled or variational forms for MI estimation, computational efficiency, Information Bottleneck, and unsupervised domain adaptation. CLUB provides favorable estimation trade-offs and supports effective MI minimization across the tested applications.
- 4.1. MI Estimation Quality: Toy studies estimate MI on 20-dimensional Gaussian and Cubic data, with true MI values from 2.0 to 10.0.The Cubic task applies a smooth bijective transformation that preserves mutual information.
- 4.1. MI Estimation Quality: CLUB and CLUBSample achieve the lowest bias, while CLUB has the best bias–variance trade-off by MSE.CLUBSample has higher variance because of sampling, but its bias is insignificantly different from CLUB's, supporting unbiased stochastic approximation.
- 4.2. Time Efficiency of MI Estimators: MINE and CLUBSample are fastest with O(N) complexity, while CLUB is fastest among other O(N^2) estimators.CLUB's mean-of-log-ratios form is efficiently accelerated by matrix multiplication; L1Out is slowest because it repeatedly leaves out positive pairs.
- 4.3. MI Minimization in Information Bottleneck: In Information Bottleneck experiments, CLUB and vCLUB outperform previous methods with lower misclassification rates.Sampled variants improve accuracy over their original estimators, while L1Out fails during MI optimization because of numerical instability.
- 4.4. MI Minimization in Domain Adaptation: In UDA, Sampled CLUB uniformly outperforms other competitive MI estimators across four domain-adaptation tasks.The paper attributes this result to stochastic sampling improving generalization and reducing overfitting; VUB and L1Out fail to train satisfying models.
5. Conclusions
The paper introduces CLUB as an MI upper-bound estimator, extends it variationally for sample-only settings, and proposes accelerated MI minimization. Experiments support its performance for MI estimation and minimization, with potential applications in reducing correlations between model components.
- CLUB is introduced as a mutual information upper-bound estimator that can be extended to a variational version when only joint samples are available.
- The variational CLUB supports a new MI minimization algorithm, accelerated through negative sampling.
- Simulation and real-world experiments show attractive performance for both MI estimation and MI minimization tasks.
- The proposed estimator is intended for reducing correlations among model parts in interpretable machine learning, controllable generation, and fairness.
A. Proofs of Theorems
The proof analyzes the gap between the variational CLUB estimator and true MI. It characterizes when the estimator remains an upper bound and shows equality under independence.
- The proof evaluates the gap between IvCLUB(x; y) and I(x; y) using KL-divergence terms.
- The variational CLUB estimator is an MI upper bound exactly when KL(p(x)p(y)∥qθ(x, y)) ≥ KL(p(x, y)∥qθ(x, y)).
- When x and y are independent, the joint and product-of-marginals distributions coincide, making the variational CLUB estimator equal true MI.
B. Network Expressiveness in Variational Inference
The analysis studies variational approximation when the conditional distribution is Gaussian and the neural network approximates its mean. Under bounded approximation error and sufficient samples, the conditional KL divergence can be made small.
- The conditional distribution is modeled as N(µ∗(x), I), while qθ(y|x) uses the neural network mean µθ(x) with the same covariance.
- Training pairs are represented as (xi, µ∗(xi) + ξi), where ξi follows N(0, I).
- The log-density difference is expressed through the mean error and the residual between yi and the approximating mean.
- If ∥µ∗(x) −µθ(x)∥ is bounded by A, the absolute log-density difference is bounded by A∥yi −µθ(xi) + ξi∥.
- With sufficiently large sample size, KL(p(y|x)∥qθ(y|x)) can be guaranteed smaller than any given ε > 0.
C. Properties of Variational Upper Bounds
The variational upper bounds IvVUB and IvL1Out depend on the quality of the neural conditional approximation. Under stated KL conditions, they can remain valid MI upper bounds.
- Neural approximation can remove the guaranteed upper-bound property of IvVUB and IvL1Out.
- IvVUB remains an MI upper bound when KL(p(y|x)∥qθ(y|x)) ≤ KL(p(y)∥r(y)).
- Given N −1 samples from the marginal p(x) and the condition in Theorem C.2, IvL1Out is an MI upper bound.
- Applying the theorem condition across combinations of samples establishes I(x; y) ≤IvL1Out(x; y).
- With sufficiently accurate neural approximation, IvVUB and IvL1Out can preserve their MI upper-bound properties.
D. Implementation Details
This section describes the Gaussian parameterization used for vCLUB and the notation for computing its estimator from sampled pairs.
- vCLUB parameterizes qθ(y|x) as an isotropic Gaussian with mean µ(x) and variance σ2(x).
- The vCLUB estimator is then calculated from these Gaussian parameters and sampled pairs.
- For each sample, the method denotes µi = µ(xi) and σi = σ(xi).
E. Detailed Experimental Setups
The experiments use specified neural architectures for information bottleneck and domain adaptation, with normalized domain images and fixed optimization settings for the bottleneck model.
- Information Bottleneck: The information bottleneck model uses an MLP with layers 784 →1024 →1024 →2K and sets the bottleneck size to K = 256.
- Information Bottleneck: Its variational classifier is a one-layer MLP implementing the Barber-Agakov MI lower bound.
- Information Bottleneck: Training uses batch size 100, learning rate 10^-4, exponential decay 0.97, and decay step 1200.
- Domain Adaptation: For domain adaptation, both content and domain feature extractors are nine-layer convolutional networks with leaky ReLU nonlinearities.
- Domain Adaptation: The content classifier and domain discriminator are one-layer and two-layer MLPs, respectively, and domain images use Gaussian normalization.
F. Numerical Results of MI Estimation
The paper evaluates MI estimation using mean squared error and reports that CLUB achieves the lowest estimation error as ground-truth MI increases.
- CLUB has the lowest MI estimation error when the ground-truth MI value increases.The reported metric is MSE of MI estimation.
- Table 3 reports MSE of MI estimation for the numerical comparison.