Source-linked AI summary
Learning Disentangled Representations via Mutual Information Estimation
Eduardo Hugo Sanchez, Mathieu Serrurier, Mathias Ortner
TL;DR
Learning representations that separate shared from image-specific attributes in paired images is the paper's target. It uses mutual-information estimation and a two-stage training approach without image reconstruction or generation. The authors report useful classification and retrieval representations and better disentanglement classification results than a VAE/GAN-based model.
Problem
The paper targets useful, low-dimensional representations that separate common attributes across paired images from image-specific attributes.
Source: §1 · p. 1Method
The method learns shared and exclusive representations using cross mutual information and an adversarial objective to minimize mutual information between the two components, without reconstruction or generation models.
Source: §1 · p. 1, §4.1 · p. 3Results
Takeaways & Limitations
The learned shared or exclusive components are reported as useful for downstream image classification and retrieval.
Source: §1 · p. 2
Abstract
from arXiv · showhide
In this paper, we investigate the problem of learning disentangled representations. Given a pair of images sharing some attributes, we aim to create a low-dimensional representation which is split into two parts: a shared representation that captures the common information between the images and an exclusive representation that contains the specific information of each image. To address this issue, we propose a model based on mutual information estimation without relying on image reconstruction or image generation. Mutual information maximization is performed to capture the attributes of data in the shared and exclusive representations while we minimize the mutual information between the shared and exclusive representation to enforce representation disentanglement. We show that these representations are useful to perform downstream tasks such as image classification and image retrieval based on the shared or exclusive component. Moreover, classification results show that our model outperforms the state-of-the-art model based on VAE/GAN approaches in representation disentanglement.
1. Introduction
The paper targets disentangled representations for paired images, separating their shared attributes from image-specific information. It proposes mutual-information-based learning without image reconstruction or generation, and evaluates the representations on downstream tasks.
1. Introduction
- Unsupervised representation learning seeks useful, transferable low-dimensional representations, while autoencoders can struggle with multimodal distributions and generative models can be computationally demanding to train.Source: §1 · p. 1
- Paired shared attributes have potential use in settings such as pose-invariant face recognition and satellite-image classification and segmentation.Source: §1 · p. 1
- The model learns shared and exclusive representations for image pairs, using cross mutual information to capture common information and an adversarial objective to minimize mutual information between the two parts.The shared part captures what the images have in common; the exclusive part captures what is specific to each image.Source: §1 · pp. 1–2
- The approach avoids costly image reconstruction or image generation models.Source: §1 · p. 1
- The training procedure first learns shared representations through cross mutual information estimation and maximization, then learns exclusive representations while minimizing their mutual information with the shared representations.Source: §1 · pp. 1–2
- Experiments on two synthetic and two real datasets report usefulness for image classification and retrieval, outperforming a state-of-the-art VAE/GAN-based model in representation disentanglement.Source: §1 · p. 2
2. Related work
Related work uses autoencoding, generative, and mutual-information-based approaches to learn representations and disentangle attributes. This paper adopts mutual-information estimation for paired shared and exclusive representations and compares its approach with a model targeting the same split.
2. Related work
- GANs are described as an adversarial game in which a generator learns samples resembling the data distribution while a discriminator distinguishes generated from dataset samples.Source: §2 · p. 2
- Mutual-information-maximization methods learn representations without decoder functions that map back to the image domain.Source: §2 · p. 2
- Deep InfoMax [14] uses a Jensen-Shannon-divergence objective, while another estimator uses the Wasserstein divergence [23].Source: §2 · p. 2
- Prior disentanglement methods include VAE variants [16], a GAN with mutual-information regularization [5], and a VAE-GAN image-translator model using gradient reversal [8] for paired shared and exclusive representations [9].Source: §2 · p. 2
- The paper uses mutual-information estimation rather than generative or autoencoding models, and introduces an adversarial objective that it reports as more effective than gradient reversal [8] for disentangling paired representations.Source: §2 · p. 2
3. Mutual information
The section frames mutual information as a quantity relating joint and marginal distributions, then describes a neural estimator used for maximization. It also reviews global and local mutual-information objectives for image representations.
3. Mutual information
- Mutual information between X and Z is expressed through the joint density p(x,z) and the corresponding marginal densities p(x) and p(z).Source: §3 · p. 2
- Mutual information can be written as the Kullback-Leibler divergence between the joint distribution PXZ and the product of marginals PXPZ.Source: §3 · p. 2
- The paper uses Deep InfoMax [14], a Jensen-Shannon-based estimator, because it is stable and the goal is maximizing mutual information rather than obtaining its precise value.Source: §3 · p. 3
- The estimator uses a deep neural network Tθ, called the statistics network, to estimate mutual information.Source: §3 · p. 3
- Global mutual information maximizes dependence between an image and its feature representation, while local mutual information maximizes dependence between image patches and that representation.Source: §3 · p. 3
4. Method
The method splits paired-image representations into shared information common to both images and exclusive information specific to each. It learns these components in two stages: first the shared representation, then the exclusive representation using the learned shared component.
4. Method
- Shared and exclusive codes represent, respectively, information common to paired images and information specific to each image.For image X, the full representation is RX = (SX, EX), with an analogous decomposition for Y.Source: §4 · p. 3
4.1. Shared representation learning
- Cross mutual information maximization learns shared codes by pairing each image with the other image’s shared representation, encouraging capture of common information.The global and local mutual-information terms are weighted by αsh and βsh.Source: §4.1 · p. 3
- An L1-distance loss encourages the two images’ shared representations to be identical, and the shared objective combines the preceding loss terms.The combination uses the constant coefficient γ.Source: §4.1 · p. 3
4.2. Exclusive representation learning
- With the shared representation held constant, the second stage maximizes mutual information between each image and its full representation while minimizing dependence between shared and exclusive codes.The adversarial objective uses a discriminator and shuffled exclusive representations to approximate the product of the marginal distributions.Source: §4.2 · pp. 3–4
4.3. Implementation details
- The adversarial objective targets dependence between shared and exclusive parts, rather than independence among all representation dimensions.The encoders are DCGAN-like; networks are trained from scratch in batches of 64 with Adam at learning rate 0.0001.Source: §4.2 · p. 4, §4.3 · p. 4
5. Experiments
Experiments test whether shared and exclusive representations separate common from image-specific attributes across synthetic and real paired-image datasets. The authors report classification, retrieval, ablation, sensitivity, and satellite-application results.
5.1. Datasets
- Experiments use paired images from colored-MNIST, 3D Shapes, IAM handwriting, and Sentinel-2, with shared factors defined by each dataset’s pair construction.The IAM subset contains 6,711 images from the top 50 writers; Sentinel-2 pairs are from the same region at different times.Source: §5.1 · p. 5
5.2. Representation disentanglement evaluation
- On colored-MNIST and 3D Shapes, the learned representations show the expected separation of shared and exclusive attributes in classification.The shared code is evaluated on shared attributes and the exclusive code on exclusive attributes, with opposite attributes expected near chance.Source: §5.2 · p. 5
- For IAM, the shared representation outperforms a VAE representation on writer recognition, while the exclusive representation achieves similar word-classification performance.Word classification is less satisfactory than writer recognition; the [17] VAE latent representation has size 128.Source: §5.2 · p. 6
- Retrieval with shared codes groups colored-MNIST digits independently of color and IAM words by writer or style; exclusive codes group by color or word.The authors also report IAM nearest-neighbor writer and word recognition results similar to those from a neural-network classifier.Source: §5.2 · p. 6, Figure 4 · p. 7
5.3. Analysis of the objective function
- Ablations indicate that switching shared representations prevents exclusive-information leakage, while the local mutual-information term supports capturing shared information.Removing the L1 distance slightly reduces shared-attribute accuracy, and removing the global mutual-information term slightly increases exclusive information in the shared code.Source: Table 6 · p. 7, Table 7 · p. 7, §5.3 · p. 8
- As λadv increases, exclusive representations approach the expected disentangled behavior, but values above 0.025 reduce classification performance on exclusive attributes.At λadv = 0.0, the exclusive code retains shared information and classifies shared attributes above chance.Source: Figure 5 · p. 7, §5.3 · p. 8
5.4. Satellite applications
- 93.11% accuracy on EuroSAT using Sentinel-2 shared representations, outperforming VAE (87.64%) and VAE-GAN (92.38%) representations.The classifier was trained on 27,000 labeled EuroSAT images using representations learned from 100GB of unlabeled Sentinel-2 data.Source: §5.4 · p. 8
- Mutual-information distance between satellite-image pixels distinguishes river, urban, and agricultural areas, suggesting potential for unsupervised segmentation and object detection.Figure 6 illustrates pixel distances computed between a selected pixel and the remaining image pixels.Source: §5.4 · p. 8, Figure 6 · p. 8
6. Conclusions
The proposed two-stage mutual-information method disentangles paired-image representations and outperforms VAE-GAN and reconstruction- or generation-based models on image classification. The conclusions also report useful results for Sentinel-2 data, segmentation, and object detection.
6. Conclusions
- The model outperforms VAE-GAN in classification experiments on three datasets and models relying on reconstruction or generation at image classification.Source: §6 · p. 8
- The method performs paired-image representation disentanglement using mutual-information estimation and a two-stage training procedure.Source: §6 · p. 8
- An ablation study examines switched shared representations and local and global statistics networks, reporting their usefulness for representation disentanglement.Source: §6 · p. 8
- Analysis of λadv empirically demonstrates the model’s disentangling capability during training.Source: §6 · p. 8
- In a Sentinel-2 setting with large amounts of unlabeled paired data, the model demonstrates benefits over image reconstruction or generation models for image classification.Source: §6 · p. 8
- The cross-mutual-information objective is reported as potentially useful for unsupervised image segmentation and object detection.Source: §6 · p. 8