PaperlayerView exact revision on arXiv

Source-linked AI summary

CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, Lawrence Carin

arXiv:2006.12013v6cs.LGstat.ML

TL;DR

Estimating and minimizing MI from samples is difficult, particularly in high-dimensional settings where lower-bound methods do not directly support minimization. The paper introduces CLUB and its variational form, then uses them for MI minimization with negative sampling; experiments report reliable estimation and effective minimization, while existing upper bounds retain practical limitations.

  • Problem

    Sample-based MI estimation and minimization remain challenging in high-dimensional settings, while prior methods mainly approximate lower bounds unsuitable for MI minimization.

    Source: §1 · p. 1

  • Method

    The paper introduces CLUB, extends it to variational CLUB for unknown conditional distributions, and develops a negative-sampling-accelerated MI minimization algorithm.

    Source: §1 · pp. 1–2

  • Results

    CLUB shows the lowest bias among compared methods in the reported MI estimation study, while CLUB and CLUBSample have an insignificant bias difference.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    CLUB provides a trainable MI upper bound for estimation and MI minimization, with reported effectiveness in simulation studies and real-world applications.

    Source: §1 · p. 2, §5 · p. 8

    Existing upper-bound approaches can require difficult marginal approximation, sufficient sample size, or known conditional information, and L1Out remains numerically unstable in real-world minimization.

    Source: §2 · p. 2

Abstract

from arXiv · show

Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.

1. Introduction

MI is useful for controlling dependence in machine learning, but sample-based minimization is difficult because prior estimators mainly provide lower bounds. CLUB addresses this gap with a contrastive upper bound, a variational approximation, and an accelerated minimization scheme.

  • MI is used as a criterion or regularizer to encourage or limit dependence between variables in machine learning.Source: §1 · p. 1
  • Exact MI calculation is usually unavailable because it requires tractable density functions and log-density ratios, while most tasks provide only joint-distribution samples.Source: §1 · p. 1
  • Previous sample-based MI methods mainly estimate lower bounds, whereas MI upper-bound estimation has received less exploration despite its relevance to minimization.Source: §1 · p. 1
  • CLUB estimates MI through differences between conditional probabilities for positive and negative sample pairs, connecting MI estimation with contrastive learning.Source: §1 · p. 1
  • The variational CLUB approximation uses a neural network for unknown p(y|x), while the proposed minimization algorithm is accelerated through negative sampling.Source: §1 · pp. 1–2
  • CLUB is reported to improve the bias-variance estimation trade-off and effectiveness in MI minimization across simulation studies and real-world applications.Source: §1 · p. 2

2. Background

MI estimation becomes unreliable in high-dimensional or intractable settings, and most differentiable estimators target lower bounds rather than minimization-compatible upper bounds. Existing upper bounds face approximation, sample-size, or numerical-stability limitations that motivate variational comparisons.

  • Classical estimators such as binning, kernel density, likelihood-ratio, and nearest-neighbor methods become unreliable as data dimension increases.Source: §2 · p. 2
  • Deep neural MI estimators mainly construct lower bounds, including BA, MINE, NWJ, and InfoNCE, for differentiable MI estimation.Source: §2 · p. 2
  • Most existing MI upper bounds require a known conditional distribution, limiting their use when p(y|x) is unavailable.Source: §2 · p. 2
  • VUB replaces the marginal p(y) with a variational approximation r(y), but high-dimensional density approximation is challenging and a fixed standard normal can produce high-bias estimates.Source: §2 · p. 2
  • L1Out uses a Monte Carlo approximation that depends strongly on sufficient sample size and can become numerically unstable in real-world MI minimization.Source: §2 · p. 2
  • Variational L1Out performs better than previous lower bounds in simulation, but its numerical instability persists in real-world applications.Source: §2 · p. 3

3. Proposed Method

CLUB provides a sample-based mutual-information upper bound using conditional log-likelihood differences between positive and negative pairs. Its variational form extends the approach when p(y|x) is unknown, supports MI minimization, and can be accelerated with unbiased negative sampling.

CLUB with p(y|x) Known

  • CLUB estimates mutual information from positive and negative sample pairs when a conditional distribution such as p(y|x) is available.The estimator contrasts log p(y_i|x_i) with log p(y_j|x_i) for negative pairs.Source: §3 · p. 3, §3.1 · p. 3
  • The CLUB quantity is an MI upper bound because Jensen’s inequality makes the gap between CLUB and mutual information non-negative.The bound is tight when x and y are independent.Source: §3.1 · p. 3
  • The sample estimator is unbiased and uses a linear combination of positive-negative conditional log-ratios, improving numerical stability for CLUB and its gradient.The linear form distinguishes CLUB from neural MI estimators with less stable nonlinear probability transformations.Source: §3.1 · p. 3

CLUB with Conditional Distributions Unknown

  • vCLUB replaces the unknown conditional p(y|x) with a neural-network variational approximation q_θ(y|x).The approximation is trained by maximizing conditional log-likelihood on sample pairs.Source: §3.2 · pp. 3–4
  • With a good variational approximation, vCLUB can remain an MI upper bound or become an estimator whose absolute error is bounded by approximation error.The relevant approximation error is KL(p(y|x)∥q_θ(y|x)).Source: §3.2 · p. 4

CLUB in MI Minimization

  • The proposed training scheme alternates updating q_θ(y|x) and minimizing the vCLUB estimator with respect to the joint distribution.The reparameterization trick enables gradients to back-propagate through sampled embeddings.Source: §3.3 · p. 5
  • Negative sampling preserves unbiasedness while reducing per-iteration computation, and sampled vCLUB retains comparable estimation performance while improving generalization as a learning critic.This property follows from CLUB’s linear log-ratio summation rather than applying a nonlinear logarithm after probability aggregation.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate CLUB and its sampled or variational forms for MI estimation, computational efficiency, Information Bottleneck, and unsupervised domain adaptation. CLUB provides favorable estimation trade-offs and supports effective MI minimization across the tested applications.

4.1. MI Estimation Quality

  • Toy studies estimate MI on 20-dimensional Gaussian and Cubic data, with true MI values from 2.0 to 10.0.The Cubic task applies a smooth bijective transformation that preserves mutual information.Source: §4.1 · p. 6
  • CLUB and CLUBSample achieve the lowest bias, while CLUB has the best bias–variance trade-off by MSE.CLUBSample has higher variance because of sampling, but its bias is insignificantly different from CLUB's, supporting unbiased stochastic approximation.Source: §4.1 · p. 6

4.2. Time Efficiency of MI Estimators

  • MINE and CLUBSample are fastest with O(N) complexity, while CLUB is fastest among other O(N^2) estimators.CLUB's mean-of-log-ratios form is efficiently accelerated by matrix multiplication; L1Out is slowest because it repeatedly leaves out positive pairs.Source: §4.2 · pp. 6–7

4.3. MI Minimization in Information Bottleneck

  • In Information Bottleneck experiments, CLUB and vCLUB outperform previous methods with lower misclassification rates.Sampled variants improve accuracy over their original estimators, while L1Out fails during MI optimization because of numerical instability.Source: §4.3 · p. 7

4.4. MI Minimization in Domain Adaptation

  • In UDA, Sampled CLUB uniformly outperforms other competitive MI estimators across four domain-adaptation tasks.The paper attributes this result to stochastic sampling improving generalization and reducing overfitting; VUB and L1Out fail to train satisfying models.Source: §4.4 · pp. 7–8

5. Conclusions

The paper introduces CLUB as an MI upper-bound estimator, extends it variationally for sample-only settings, and proposes accelerated MI minimization. Experiments support its performance for MI estimation and minimization, with potential applications in reducing correlations between model components.

  • CLUB is introduced as a mutual information upper-bound estimator that can be extended to a variational version when only joint samples are available.Source: §5 · p. 8
  • The variational CLUB supports a new MI minimization algorithm, accelerated through negative sampling.Source: §5 · p. 8
  • Simulation and real-world experiments show attractive performance for both MI estimation and MI minimization tasks.Source: §5 · p. 8
  • The proposed estimator is intended for reducing correlations among model parts in interpretable machine learning, controllable generation, and fairness.Source: §5 · p. 9

A. Proofs of Theorems

The proof analyzes the gap between the variational CLUB estimator and true MI. It characterizes when the estimator remains an upper bound and shows equality under independence.

  • The proof evaluates the gap between IvCLUB(x; y) and I(x; y) using KL-divergence terms.Source: A. Proofs of Theorems · p. 11
  • The variational CLUB estimator is an MI upper bound exactly when KL(p(x)p(y)∥qθ(x, y)) ≥ KL(p(x, y)∥qθ(x, y)).Source: A. Proofs of Theorems · p. 11
  • When x and y are independent, the joint and product-of-marginals distributions coincide, making the variational CLUB estimator equal true MI.Source: A. Proofs of Theorems · p. 11

B. Network Expressiveness in Variational Inference

The analysis studies variational approximation when the conditional distribution is Gaussian and the neural network approximates its mean. Under bounded approximation error and sufficient samples, the conditional KL divergence can be made small.

C. Properties of Variational Upper Bounds

The variational upper bounds IvVUB and IvL1Out depend on the quality of the neural conditional approximation. Under stated KL conditions, they can remain valid MI upper bounds.

D. Implementation Details

This section describes the Gaussian parameterization used for vCLUB and the notation for computing its estimator from sampled pairs.

E. Detailed Experimental Setups

The experiments use specified neural architectures for information bottleneck and domain adaptation, with normalized domain images and fixed optimization settings for the bottleneck model.

Information Bottleneck

Domain Adaptation

F. Numerical Results of MI Estimation

The paper evaluates MI estimation using mean squared error and reports that CLUB achieves the lowest estimation error as ground-truth MI increases.

Related papers

Loading 2006.12013v6…