PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning deep representations by mutual information estimation and maximization

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio

arXiv:1808.06670v5stat.MLcs.LG

TL;DR

Learning useful unsupervised representations requires objectives that retain relevant information while accounting for input structure and desired statistical characteristics. DIM estimates and maximizes mutual information, uses global or local information, and applies adversarial prior matching. It provides flexible representations that perform well across downstream classification and reconstruction-style evaluations.

  • Problem

    The central challenge is learning representations that retain useful information while reflecting characteristics such as independence and task-relevant input structure.

  • Method

    DIM estimates and maximizes mutual information between inputs and encoder representations, using global or local information and adversarial prior matching.

  • Results

    DIM provides representations that perform well across classification and reconstruction-style tasks, with local consistency across structural locations.

  • Takeaways & Limitations

    The method offers a flexible way to tailor representation objectives to downstream goals by selecting global or local mutual-information targets and statistical priors.

  • Takeaways & Limitations

    With limited-capacity encoders, global input-representation mutual information may favor information specific to local patches or pixels rather than information shared across the input.

Abstract

from arXiv · show

In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.

1 INTRODUCTION

Deep InfoMax learns unsupervised representations by estimating and maximizing mutual information between inputs and encoder outputs. It uses input structure, adversarial prior matching, and new quality measures to tailor representations to downstream tasks.

  • Global mutual information between the complete input and encoder output can be insufficient, so incorporating locality helps shape useful representations.
  • Deep InfoMax (DIM) estimates and maximizes mutual information between input data and learned high-level representations.
  • Global or local mutual information objectives can tune representations for classification or reconstruction-style tasks.
  • Adversarial prior matching constrains representations to have desired statistical characteristics.
  • DIM introduces MINE-based and neural-dependency measures to compare representation quality across unsupervised methods.

2 RELATED WORK

The paper situates DIM among representation-learning, generative, mutual-information, and structure-aware methods. It emphasizes that DIM uses local input structure while differing from CPC in how global and local features are predicted.

  • Existing representation-learning methods include classic ICA and self-organizing maps, deep volume-preserving maps, clustering, noise-as-targets, and self-supervised approaches.
  • Generative models use reconstruction objectives whose error can be related to mutual information between inputs and representations.
  • DIM follows mutual-information estimation for continuous variables but finds the generator unnecessary and JSD more stable than the exact KL-based formulation.
  • Prior work uses input structure, transformations, or spatial proximity in mutual-information objectives to avoid degenerate solutions and support unsupervised organization.
  • CPC and DIM both maximize mutual information between global and local representations, but CPC predicts future local features sequentially while DIM predicts all local features simultaneously.

3 DEEP INFOMAX

Deep InfoMax trains encoders by estimating and maximizing mutual information, while using global or locality-aware objectives and prior matching to shape representations for different downstream goals.

  • 3 DEEP INFOMAX: Global DIM summarizes an M × M feature map into a vector and scores that vector with the corresponding lower-level map through a discriminator.Fake samples pair the same high-level vector with a feature map from another image.
  • 3 DEEP INFOMAX: DIM can combine global and local mutual-information maximization with adversarial prior matching to encourage desired representation characteristics such as independence.The relative objective weights are controlled by hyperparameters, and the paper states that these choices affect learned representations.
  • 3 DEEP INFOMAX: Deep InfoMax estimates and maximizes mutual information between input data and learned high-level representations.Its estimator follows a discriminator-based approach that distinguishes joint samples from samples drawn from the product of marginals.
  • 3.1 MUTUAL INFORMATION ESTIMATION AND MAXIMIZATION: DIM’s local objective scores global features against local feature-map vectors at every location, incorporating structure such as spatial locality.The global feature is formed by summarizing the local feature map before being paired with each local feature.
  • 3.1 MUTUAL INFORMATION ESTIMATION AND MAXIMIZATION: Jensen-Shannon estimation provides a binary-classification objective that the authors find more stable in practice than the Donsker-Varadhan objective.The JSD estimator compares the true joint distribution with the product of marginals.
  • 3.1 MUTUAL INFORMATION ESTIMATION AND MAXIMIZATION: Global input mutual information can preserve task-irrelevant pixel noise, whereas local mutual information can make representations more suitable for classification.The paper motivates locality-aware objectives because limited-capacity encoders selectively retain input information.

4 EXPERIMENTS

DIM was evaluated on four imaging datasets using classification, information, dependence, reconstruction, and semi-supervised metrics. Local DIM generally outperformed the compared unsupervised methods, while structural augmentations further improved classification.

  • Evaluation setup: DIM was evaluated on CIFAR10, CIFAR100, Tiny ImageNet, and STL-10 against several unsupervised representation-learning methods.The comparison included VAE, β-VAE, AAE, BiGAN, NAT, and CPC.
  • Evaluation setup: The evaluation measured classification, semi-supervised learning, input-output mutual information, factor dependence, and reconstruction quality.Classification used linear and nonlinear probes, while MINE, NDM, and MS-SSIM supplied additional measures.
  • Classification results: DIM(L) outperformed the presented unsupervised models on all datasets except CPC and matched or exceeded similarly configured fully supervised classifiers without fine-tuning.The authors note that specialized supervised architectures and augmentations can perform substantially better.
  • Objective and architecture comparisons: infoNCE generally performed best, while its advantage over JSD diminished on larger datasets and DV performed much worse there.CPC and DIM both improved considerably with a strided-crop architecture, and DIM only slightly outperformed CPC in that setting.
  • Representation properties: On CIFAR10, mixed local/global DIM achieved high mutual-information scores and was close to the dependence profile of models with lower factor dependence.MS-SSIM correlated well with the MINE estimate, indicating that the models encoded pixel-wise information.
  • Structural augmentations: Adding input occlusion or coordinate-prediction tasks to DIM improved classification accuracy, particularly for highly compressed global features.These additions were interpreted as extensions of inpainting and context-prediction tasks.

5 CONCLUSION

DIM learns unsupervised representations by maximizing mutual information while incorporating local consistency across structural locations. The authors present this as a flexible approach that performs well across varied tasks.

  • Conclusion: DIM maximizes mutual information to learn representations containing locally consistent information across structural locations such as image patches.The method is presented as a new unsupervised representation-learning approach.
  • Conclusion: The authors report that DIM provides a straightforward and flexible way to learn representations that perform well on a variety of tasks.

A.1 ON THE JENSEN-SHANNON DIVERGENCE AND MUTUAL INFORMATION

The appendix examines the relationship between Jensen-Shannon divergence and mutual information for the coupled joint and product-of-marginals distributions used by DIM. Discrete experiments show an approximately monotonic relationship.

  • Theoretical relationship: The analysis relates JSD between the joint distribution and product of marginals to mutual information through the coupled distributions p(x,y) and p(x)p(y).The relationship does not generally hold for arbitrary distributions because these two distributions are coupled.
  • Theoretical relationship: The relevant expectation is a concave, monotonically increasing function of the ratio p(y|x)/p(y), identified as ePMI(x,y).
  • Experiments: Experiments sampled 1000 sparse discrete joint distributions at matched dimensions 8, 16, 32, 64, and 128, then computed KL and JSD directly.Uniform inputs, dropout-induced sparsity, and softmax were used to construct the distributions.
  • Experiments: The KL and JSD divergences showed an approximately monotonic relationship, with distributions having the highest mutual information also having the highest JSD.

A.2 EXPERIMENT AND ARCHITECTURE DETAILS

The appendix describes DIM’s encoder, global and local mutual-information discriminators, prior matching, and training setup. Two local architectures score global–local feature pairs using concatenation or embedded dot products.

  • Encoders: DIM experiments used DCGAN-like encoders for CIFAR10/CIFAR100 and AlexNet-like encoders for the larger datasets, with ReLU activations and batch normalization.
  • Global objective: The global objective concatenates flattened local features with the global representation and scores them using a fully connected discriminator.
  • Local objectives: The concat-and-convolve local architecture pairs the global vector with each local feature-map location and scores the pair using a 1 × 1 convolutional discriminator.Fake pairs combine global vectors and local maps from different images.
  • Local objectives: The encode-and-dot-product architecture embeds global and local features into equal-dimensional spaces and scores their pairwise compatibility with dot products.Its high-dimensional embeddings allow efficient evaluation of many positive and negative pairs.
  • Prior matching: Prior matching trains a discriminator to distinguish prior samples from encoder outputs while the encoder learns to fool it.The prior-matching discriminator is a fully connected network.
  • Objective placement: The local objective was most effective at different encoder layers depending on dataset size, reflecting sensitivity to receptive-field size.The next-to-last layer worked best on CIFAR10/CIFAR100, while the previous layer worked best on larger datasets.
  • Training: Training used Adam with learning rate 1 × 10^-4 for 1000 epochs on CIFAR10/CIFAR100 and 200 epochs on the other datasets.

A.3 SAMPLING STRATEGIES

DIM’s estimators differ in their sensitivity to negative-sample strategies. JSD is relatively insensitive, whereas infoNCE and DV depend strongly on sample count and sampling choices.

  • JSD is insensitive to the number of negative samples, while infoNCE is highly sensitive to that number.With many negatives, infoNCE can outperform JSD, but its performance drops quickly as the estimation sample size decreases.
  • 1–2% accuracy improvement was obtained by excluding positive samples from the product of marginals for JSD.The authors attribute this likely improvement to batchwise sampling overestimating positive examples relative to the full dataset.
  • DV was outperformed by JSD even with the maximum negative-sample count and became highly unstable as that count decreased.
  • The local DIM objective produced more interpretable Tiny ImageNet nearest neighbors than the purely global objective.The nearest neighbors were selected by L1 distance in the encoded representation space.

A.4 NEAREST-NEIGHBOR ANALYSIS

Nearest-neighbor analysis shows that local DIM representations organize images more interpretably than global representations, while local consistency can preserve non-class-specific visual patterns.

  • DIM(L) yields nearest neighbors with a much more interpretable structure across the image than DIM(G).Queries were randomly selected reference images, and neighbors were ranked by L1 distance in representation space.
  • CIFAR10 ablations in Figure 10 vary DIM settings across a grid, with heatmaps formed by bilinear interpolation and thresholded for visual clarity.The highest or lowest grid value is marked, and NDM is measured without the sigmoid function.
  • CelebA ablations report average, minimum, and maximum class accuracies across attributes in a multinomial classification task.The caption states that the local objective is crucial, while the global objective is stronger here than on other datasets.

A.5 ABLATION STUDIES

Ablations show that global and local mutual-information objectives serve different downstream goals, while the prior controls dependence and can also improve classification.

  • Local versus global mutual information maximization: Classification depends strongly on the local coefficient β, whereas reconstruction depends strongly on the global coefficient α.Small amounts of α can help classification, and small amounts of β can improve reconstruction.
  • Local versus global mutual information maximization: Combining α and β produces higher MINE estimates than relying on either objective alone.
  • Local versus global mutual information maximization: CelebA’s fine-grained attributes make the global objective more influential than in classification on datasets such as CIFAR10.
  • Neural dependency measure: The NDM discriminator distinguishes original representations from versions whose features were independently shuffled across the batch.An easier discrimination task indicates greater dependence among representation components.
  • The effect of the prior: Including the prior term γ is absolutely necessary for low dependence between high-level representation components as measured by NDM.A small prior term also improves classification when combined with the local term β.

A.6 EMPIRICAL CONSISTENCY OF NEURAL DEPENDENCY MEASURE (NDM)

NDM decreases as β increases across β-VAE models and shows relatively low variance, supporting its empirical consistency. The appendix also examines DIM-based generation and reports competitive qualitative but not superior quantitative results.

  • Empirical consistency of NDM: NDM shows a strong downward trend as β increases across β-VAE models.The β = 0.5 and β = 1.0 models produce similar values.
  • Empirical consistency of NDM: NDM estimates have relatively low variance across models and runs, indicating empirical consistency in this setting.Error bars cover five VAE runs and estimates from ten different networks.
  • Experimental details: The occlusion experiments randomly hide image regions while requiring at least one visible and one fully occluded 10 × 10 pixel block.The block size was chosen using the receptive fields of local encoder features.
  • Experimental details: Absolute coordinate prediction models i and j as independent categorical variables using categorical cross-entropies from global and local features.
  • Experimental details: Relative coordinate prediction samples source and target spatial locations and uses their local features with the global feature.
  • Training a generator by matching to a prior implicitly: The prior objective trains a generator whose encoded output is mapped implicitly toward a one-dimensional mixture of two Gaussians.The mixture components target the encoder push-forwards of the data and generator distributions.
  • Training a generator by matching to a prior implicitly: Training the generator requires encoded data and generated distributions to share support so that generator gradients remain stable and non-zero.Overtraining the discriminator produced non-overlapping modes associated with poor training.
  • Images generation: The proposed generator produced highly realistic images qualitatively, but did not surpass NS-GAN-CP or WGAN-GP quantitatively.The comparison used identical architectures and training procedures.
Loading 1808.06670v5…