PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Limitations of Variational Mutual Information Estimators

Jiaming Song, Stefano Ermon

arXiv:1910.06222v2cs.LGcs.ITstat.ML

TL;DR

Variational MI estimators offer a scalable neural approach, but their reliability is limited by bias-variance trade-offs and failures of basic MI consistency properties. The paper unifies these methods as density-ratio optimization, identifies partition-function estimation as a variance source, and develops variance-reduction methods that improve benchmark bias-variance trade-offs.

  • Problem

    Neural variational MI estimators scale to high-dimensional settings, but their estimation effectiveness, bias-variance behavior, and consistency with basic MI properties remain unclear.

  • Method

    The paper treats variational MI estimation as optimization over valid density ratios and reduces variance through improved partition-function estimation and clipped density ratios.

  • Results

    The proposed ISMILE estimator improves performance and bias-variance trade-offs on benchmark tasks, while existing estimators fail self-consistency tests in different MI regimes.

  • Takeaways & Limitations

    Estimator quality requires evaluating both statistical trade-offs and whether estimates preserve data processing, independence additivity, and related MI properties.

  • Takeaways & Limitations

    Under the paper's conditions, optimal-ratio NWJ and MINE estimates can have variance that grows exponentially with ground-truth MI, while lower-variance alternatives introduce looser bounds and higher bias.

Abstract

from arXiv · show

Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.

1 INTRODUCTION

MI estimation matters for modern machine learning but remains difficult to scale and interpret reliably. This paper analyzes variational estimators' limitations and proposes variance-reduction techniques with improved bias-variance trade-offs.

  • MI estimation supports representation learning and reinforcement learning, but traditional methods struggle to scale to high-dimensional variables.The paper highlights estimating MI between images and learned representations as a modern scaling challenge.
  • Variational estimators combine naturally with deep learning, yet their effectiveness for MI estimation remains unclear.Higher estimated MI does not necessarily indicate improved predictive performance in downstream supervised learning.
  • MINE and related estimators can have variance that grows exponentially with ground-truth MI, producing poor bias-variance trade-offs.The paper also tests whether variational estimators satisfy fundamental MI properties such as data processing and additivity under independence.
  • A unified variational perspective motivates variance reduction for partition-function estimation, yielding better bias-variance trade-offs on standard benchmarks.The proposed approach treats MI estimation as optimization over valid density ratios and targets the source of high variance in MINE.

2 BACKGROUND AND RELATED WORK

The paper places neural variational MI estimators within a broader landscape of lower-bound, density-ratio, and generative approaches. These methods differ in bias, variance, computational cost, and modeling requirements.

  • Variational MI estimation is attractive because it can combine with deep learning, unlike traditional approaches that struggle with modern high-dimensional problems.
  • Barber-Agakov estimates MI using a valid conditional distribution qφ(x|y) together with the marginal density p(x).Its density-ratio form is qφ(x|y)/p(x) when p(x) is known.
  • MINE applies the Donsker-Varadhan KL lower bound with a neural statistic, but mini-batch expectations make its estimator biased and require partition-function corrections.Exponential moving averages are proposed to reduce bias in estimating the partition function.
  • NWJ uses an invertible variational function to estimate density ratios, and its mini-batch estimator is unbiased.This avoids the special gradient-bias correction required by MINE.
  • CPC generally reduces variance but introduces bias because its estimate cannot exceed log n, while larger batches require n^2 evaluations.Interpolation between CPC and NWJ provides finer bias-variance trade-offs.

3 VARIATIONAL MUTUAL INFORMATION ESTIMATION AS OPTIMIZATION

The paper unifies variational MI estimators as constrained optimization over valid density ratios. This framework separates density-ratio estimation, projection, and MI estimation while clarifying differences between generative and discriminative methods.

  • Variational MI estimation can be formulated as constrained optimization over Δ(Q), the set of valid density ratios with respect to Q.The optimum is achieved when the ratio equals dP/dQ.
  • The general procedure estimates a density ratio, projects it toward Δ(Q) using empirical samples, and averages its logarithm under P.
  • Barber-Agakov directly constructs a valid density ratio, whereas MINE estimates its logarithm and requires normalization because the ratio may not be normalized.For MINE, the normalization is valid as a density ratio only as batch size tends to infinity.
  • Discriminative methods estimate dP/dQ directly, while generative methods separately model the relevant distributions.The two categories therefore make different modeling and estimation choices.
  • Generative approaches may require tractable likelihoods or evidence lower bounds, whereas discriminative approaches require product-of-marginals samples.For batch size n, generative evaluation costs are Ω(n), while discriminative costs can reach Ω(n^2).

4 LIMITATIONS OF EXISTING VARIATIONAL ESTIMATORS

Existing variational MI estimators face exponential variance costs at high MI and fail basic self-consistency tests, limiting their reliability for high-dimensional applications.

  • 4.1 GOOD DISCRIMINATIVE ESTIMATORS REQUIRE EXPONENTIALLY LARGE BATCHES: The sample-average partition functions used by NWJ and MINE can have variance that scales exponentially with ground-truth MI.This holds even when the optimal density ratio is already obtained.
  • 4.1 GOOD DISCRIMINATIVE ESTIMATORS REQUIRE EXPONENTIALLY LARGE BATCHES: Reducing variance with a non-optimal density ratio produces looser bounds and higher bias.The variance-bias trade-off therefore remains even when the optimal ratio is not used.
  • 4.1 GOOD DISCRIMINATIVE ESTIMATORS REQUIRE EXPONENTIALLY LARGE BATCHES: n = Θ(e^DKL(P ∥Q)) is required to keep NWJ and MINE variance relatively constant as MI grows.CPC has small variance but requires n ≥e^DKL(P ∥Q) for small bias because its estimate is bounded by log n.
  • 4.2 SELF-CONSISTENCY ISSUES FOR MUTUAL INFORMATION ESTIMATORS: The proposed self-consistency tests require zero MI under independence, data processing, and approximate additivity for independent concatenated variables.These properties are necessary but not sufficient for accurate MI estimation.
  • 4.2 SELF-CONSISTENCY ISSUES FOR MUTUAL INFORMATION ESTIMATORS: None of the considered estimators pass all self-consistency tests on images; generative methods fail at low MI, while discriminative methods fail at high MI.The failures involve independence and data processing for generative approaches and additivity for discriminative approaches.

5 IMPROVED MI ESTIMATION VIA CLIPPED DENSITY RATIOS

The paper reduces partition-function variance by clipping estimated density ratios, introducing a tunable bias-variance trade-off and yielding the SMILE estimator.

  • 5 IMPROVED MI ESTIMATION VIA CLIPPED DENSITY RATIOS: Clipping density ratios during partition-function estimation is proposed to address high variance in INWJ and IMINE.The method replaces the unbounded ratio contribution with a clipped version.
  • 5 IMPROVED MI ESTIMATION VIA CLIPPED DENSITY RATIOS: For empirical Qn, the partition function uses EQn[clip(r, e−τ, eτ)] instead of EQn[r], with τ ≥0 controlling the clipping range.This is equivalent to clipping the log density-ratio estimator between −τ and τ.
  • 5 IMPROVED MI ESTIMATION VIA CLIPPED DENSITY RATIOS: SMILE uses a neural network Tθ to estimate the log-density ratio and converges to IMINE as τ →∞.The experiments learn the density ratio with logistic regression, as in Deep InfoMax.
  • 5 IMPROVED MI ESTIMATION VIA CLIPPED DENSITY RATIOS: Smaller τ reduces variance at the cost of potentially increasing bias, while small τ can make both variance and bias small when the partition function S is close to 1.Theorems 3 and 4 analyze this worst-case trade-off for bounded density ratios.

6 EXPERIMENTS

Experiments compare variational MI estimators on tractable Gaussian and Cubic tasks, then test self-consistency on high-dimensional images. ISMILE offers favorable bias-variance behavior, while estimators differ substantially in data-processing and additivity tests.

  • Toy benchmarks: The experiments evaluate ICPC, INWJ, ISMILE, and IGM on Gaussian and Cubic tasks with tractable ground-truth MI.The ground-truth MI increases by 2 every 4k training iterations across 20k iterations.
  • Toy benchmarks: ISMILE has relatively low bias and variance across architectures and tasks, whereas ICPC is biased and INWJ becomes highly variable at high MI.Reducing τ consistently lowers SMILE variance, but its effect on bias depends on the task and critic.
  • Bias-variance analysis: Nearly two orders of magnitude lower variance than INWJ is achieved by ISMILE with similar bias, producing lower MSE than INWJ and ICPC.The variance of INWJ increases exponentially with mutual information, while small-τ SMILE reduces variance.
  • Self-consistency tests: On images, the first setting tests whether estimated MI is zero for independent variables and non-decreasing as more rows are retained.Most methods except IGM pass the independence test, and estimated MI increases with the number of retained rows, though slopes differ.
  • Self-consistency tests: For data processing, all methods except IMINE and IGM perform well on both datasets; IGM performs poorly on CIFAR10 and IMINE on MNIST.The tested ratio should be approximately 1 because additional processing should not increase information.
  • Self-consistency tests: For additivity, discriminative estimators generally approach a ratio of 1 at large t, while IGM performs near perfectly across all t values.The ideal ratio is approximately 2 for two independent image pairs; the discriminative behavior may reflect initialization and objective saturation.

7 DISCUSSION

The discussion identifies high variance and failed self-consistency tests as limitations of variational MI estimators, while motivating variance-reduced alternatives and caution in interpreting optimization results.

  • INWJ- and IMINE-based estimators are prone to high variance with mini-batches, motivating the ISMILE estimator.
  • None of the considered approaches pass all self-consistency tests for MI.
  • Generative approaches perform poorly when MI is small, failing independence and data-processing tests.
  • Discriminative approaches perform poorly when MI is large, failing additivity tests.
  • Optimization over variational estimators is not necessarily related to optimizing MI, so alternative information measurements may be useful.

A.2 PROOFS IN SECTION 4

These proofs establish variance results for empirical density-ratio estimates, including asymptotic behavior and consequences for estimators using independent empirical distributions.

  • Theorem 2 analyzes empirical averages of the ground-truth density ratio under i.i.d. sampling from Q.
  • The first theorem result follows from the variance of the mean of n independent random variables.
  • Applying f = log and E[X] = 1 yields the theorem’s asymptotic variance statement.
  • Corollary 1 extends the result to empirical distributions Pm and Qn formed from independent samples.

A.3 PROOFS IN SECTION 5

These proofs characterize clipping effects on density-ratio partition-function estimates, including extrema, variance, and the resulting bias-variance trade-off.

  • Theorem 3 analyzes a non-negative measurable density-ratio function constrained by an integral condition.
  • The proof parameterizes the expected clipped and unclipped ratios through K1 and K2 before optimizing their difference.
  • The clipping analysis finds that g(K1, K2) is largest when K1 →∞, K2 = τ and smallest when K1 = τ, K2 →∞.
  • Theorem 4 gives the variance of EQn[rτ] when estimating a clipped partition function from n samples.
  • Combining the clipping bounds and variance result yields a bias-variance trade-off for partition-function estimation.

B.1 BENCHMARK TASKS

The benchmark tasks use correlated Gaussian variables and neural estimators trained under specified architectures and procedures, then compare bias, variance, and MSE across methods.

  • The Gaussian benchmark samples each dimension independently from correlated Gaussian variables X = Y = R20 with correlation ρ.The true mutual information is computed analytically in the benchmark setup.
  • The initial mutual information is 2 and increases by 2 every 4k iterations over 20k total iterations.
  • Discriminative estimators use joint or separable two-layer MLP architectures, while generative estimators use invertible flow architectures.
  • The benchmark reports bias, variance, and MSE for estimators under the joint critic.
  • Smaller τ visibly decreases the variance of estimating the clipped partition function in the SMILE estimator.This agrees with the variance estimates shown for the benchmark estimators.

B.2 SELF-CONSISTENCY EXPERIMENTS

The experiments test mutual information estimators on image-row prediction, data-processing consistency, additivity under independent images, and additional geometric transformations. They use shared discriminative architectures, VAE-based generative architectures, and specified training schedules.

  • Tasks: The first task estimates ˆI(X; Y), where Y consists of the top t rows of an MNIST or CIFAR10 image X.The bottom rows are masked to zero to simplify architecture designs.
  • Tasks: The second task estimates ˆI([X, X]; [Y; h(Y)]), ideally matching ˆI(X; Y) despite concatenating transformed inputs.Here h(Y) contains the top (t −3) rows of Y.
  • Tasks: The third task estimates mutual information for two independent image pairs, ideally producing 2 · ˆI(X; Y).X1 and X2 are independent MNIST or CIFAR10 images, with Y1 and Y2 formed from their top t rows.
  • Architecture and training procedure: All discriminative approaches use the same convolutional architecture followed by a 1024-neuron fully connected layer and a scalar output layer.The convolutional layers have 64 and 128 output channels, kernel size 5, stride 2, and padding 2.
  • Architecture and training procedure: The generative approach uses a VAE whose encoder predicts the means and standard deviations of 10 Gaussian distributions, with dataset-specific decoders.The MNIST decoder is a two-layer 400-neuron MLP, while CIFAR10 uses a corresponding transposed convolution network.
  • Additional experiments on scaling, rotation and translation: Additional MNIST experiments replace row removal with random scaling from 1x to 1.2x, rotations between ±20 degrees, or translations of at most 3 pixels.These experiments evaluate self-consistency under alternative image transformations.
Loading 1910.06222v2…