PaperlayerView exact revision on arXiv

Source-linked AI summary

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel

arXiv:1606.03657v1cs.LGstat.ML

TL;DR

Unsupervised learning lacks known downstream tasks, motivating representations that separate salient, meaningful factors without labels. InfoGAN maximizes mutual information between selected latent codes and generated observations, learning interpretable disentangled representations whose quality matches prior supervised approaches across image datasets.

  • Problem

    Unsupervised learning needs representations that separate salient, meaningful data attributes despite unknown downstream tasks and absent labels.

  • Method

    InfoGAN augments GAN training by maximizing mutual information between a fixed subset of latent codes and observations using an efficiently optimized variational lower bound.

  • Results

    InfoGAN learns interpretable, disentangled representations across MNIST, rendered images, SVHN, chairs, and CelebA, matching the quality of prior supervised approaches.

  • Takeaways & Limitations

    InfoGAN provides a completely unsupervised, easy-to-train approach that discovers discrete and continuous semantic factors on challenging image datasets.

Abstract

from arXiv · show

This paper describes InfoGAN, an information-theoretic extension to the Generative Adversarial Network that is able to learn disentangled representations in a completely unsupervised manner. InfoGAN is a generative adversarial network that also maximizes the mutual information between a small subset of the latent variables and the observation. We derive a lower bound to the mutual information objective that can be optimized efficiently, and show that our training procedure can be interpreted as a variation of the Wake-Sleep algorithm. Specifically, InfoGAN successfully disentangles writing styles from digit shapes on the MNIST dataset, pose from lighting of 3D rendered images, and background digits from the central digit on the SVHN dataset. It also discovers visual concepts that include hair styles, presence/absence of eyeglasses, and emotions on the CelebA face dataset. Experiments show that InfoGAN learns interpretable representations that are competitive with representations learned by existing fully supervised methods.

1 Introduction

The introduction motivates disentangled representation learning for unsupervised data and presents InfoGAN as a simple mutual-information modification to GANs that learns interpretable representations. It then outlines the paper’s related-work, theoretical, algorithmic, and experimental progression.

  • Representation learning uses unlabelled data to expose important semantic features as easily decodable factors for downstream tasks.
  • Disentangled representations explicitly encode salient attributes, potentially helping with relevant downstream tasks whose identities are unknown during training.
  • Generative modeling motivates unsupervised learning, but perfect generative models can still have arbitrarily bad representations.
  • InfoGAN modifies the GAN objective by maximizing mutual information between a fixed small subset of noise variables and observations to learn interpretable, meaningful representations.The paper describes this mutual-information objective as relatively straightforward and reports that the method was surprisingly effective.
  • The paper proceeds from related work and GANs to mutual-information-based interpretability, an efficient algorithm, and experiments comparing InfoGAN with prior approaches.

2 Related Work

Prior work spans unsupervised representation learning with autoencoders, restricted Boltzmann machines, Skip-gram-inspired methods, VAEs, GANs, and probabilistic inference. Disentangled representations have also been pursued through supervised and weakly supervised methods, whereas InfoGAN requires no supervision and differs from hossRBM in its handling of latent factors and computational cost.

  • Unsupervised representation learning: Unsupervised representation learning has used stacked autoencoders, restricted Boltzmann machines, Skip-gram-inspired models, ladder networks, VAEs, GANs, and probabilistic inference.Skip-thought vectors and image feature-learning techniques were inspired by Skip-gram; GANs learned representations supporting basic linear algebra on code space.
  • Supervised disentanglement: Supervised disentanglement methods have separated style and content, face identity and viewpoint, or learned latent-factor transformations from labeled data.Examples include bilinear models, multi-view perceptrons, and a recurrent variant that generates sequences of latent factor transformations.
  • Weakly supervised disentanglement: Weakly supervised approaches reduce explicit labeling by using matched data pairs or related constraints to learn disentangled representations.disBM clamps hidden units for pairs differing in one variation factor, while DC-IGN extends this idea to VAEs and learns pose and lighting codes in 3D rendered images.
  • Unsupervised disentanglement: InfoGAN requires no supervision, unlike cited prior disentanglement methods, while hossRBM is the only other identified unsupervised method in this comparison.hossRBM disentangles emotion from identity but only handles discrete latent factors, and its computational cost grows exponentially.

3 Background: Generative Adversarial Networks

Generative Adversarial Networks train deep generative models through a minimax game between a generator and an adversarial discriminator. The generator transforms noise into samples intended to match the real data distribution, while the discriminator distinguishes generated from real samples.

  • GAN framework: GANs provide a framework for training deep generative models using a minimax game.Goodfellow et al. introduced this framework.
  • Generator and discriminator: The generator network G transforms noise z ∼ Pnoise(z) into samples G(z) from the generator distribution PG(x).GANs learn a generator distribution without explicitly assigning probability to every data point x.
  • Generator and discriminator: Training aims for PG(x) to match the real data distribution Pdata(x), while discriminator D distinguishes samples from Pdata and PG.The generator and discriminator train adversarially against each other.

4 Mutual Information for Inducing Latent Codes

This section identifies the entanglement problem in standard GAN noise and introduces structured latent codes whose dependence on generated observations is encouraged through mutual information. The proposed decomposition separates incompressible noise from semantic factors and uses low conditional entropy to make codes informative about data.

  • Problem: Standard GAN noise can be used in highly entangled ways, so individual dimensions need not correspond to semantic data features.The generator is unrestricted in how it uses the factored continuous noise vector z.
  • Motivation: Meaningful domains may decompose into independent salient factors, such as MNIST digit identity, angle, and stroke thickness.The intended MNIST allocation uses one discrete variable for digit identity and two continuous variables for angle and stroke thickness.
  • Method: InfoGAN decomposes the input into incompressible noise z and latent code c, targeting salient structured semantic features with c.The generator receives both inputs, becoming G(z, c).
  • Method: To prevent the generator from ignoring c, the method regularizes it by maximizing mutual information between the latent code and generated observations.Without this constraint, a trivial solution can satisfy P_G(x|c) = P_G(x).
  • Mutual information: Mutual information measures uncertainty reduction: it is zero for independent variables and maximal when variables are related by a deterministic, invertible function.This motivates requiring P_G(c|x) to have small entropy for generated x.

5 Variational Mutual Information Maximization

InfoGAN replaces the intractable mutual-information objective with a variational lower bound using an auxiliary posterior approximation, then optimizes this bound within the GAN framework. The bound is efficiently estimable and becomes tight as the approximation approaches the true posterior.

  • Variational lower bound: Directly maximizing I(c; G(z, c)) is difficult because it requires access to the posterior P(c|x).InfoGAN addresses this difficulty by introducing an auxiliary distribution Q(c|x) to approximate P(c|x).
  • Variational lower bound: Variational Information Maximization lower-bounds mutual information through Q(c|x), while InfoGAN fixes the latent-code entropy H(c) as a constant for simplicity.The entropy H(c) could also be optimized for common distributions because it has a simple analytical form.
  • Optimization: The lower bound LI(G, Q) is Monte Carlo-approximable, optimized directly with respect to Q and through reparametrization with respect to G, and added to GAN objectives without changing training.The resulting algorithm is called Information Maximizing Generative Adversarial Networks, or InfoGAN.
  • Tightness and interpretation: The bound becomes tight as Q approaches P, and for discrete codes reaches maximal mutual information when LI(G, Q) = H(c).InfoGAN can also be interpreted as connected to the Wake-Sleep algorithm.
  • Optimization: InfoGAN is formulated as a minimax game with a variational mutual-information regularization weighted by the hyperparameter λ.This incorporates the lower-bound objective into the GAN game.

6 Implementation

InfoGAN implements its auxiliary distribution with a neural network that shares convolutional layers with the discriminator, adding negligible computation cost. The method uses softmax or factored Gaussian posterior approximations and simple λ tuning, while DC-GAN techniques stabilize training without new tricks.

  • 6 Implementation: InfoGAN’s auxiliary distribution Q shares convolutional layers with D and adds only a final fully connected output layer, resulting in negligible computation cost.The authors also observe that the mutual-information lower bound converges faster than normal GAN objectives.
  • 6 Implementation: Categorical latent codes use a softmax Q(c_i|x), while continuous codes use a factored Gaussian approximation.The factored Gaussian was sufficient in the reported experiments for continuous latent codes.
  • 6 Implementation: Setting λ=1 is sufficient for discrete latent codes, whereas continuous codes typically require a smaller λ to match GAN objective scales.For continuous codes, λL_I(G, Q) involves differential entropy and is adjusted to remain on the same scale as GAN objectives.
  • 6 Implementation: Existing DC-GAN techniques stabilize InfoGAN training, so the experiments require no new training tricks.The detailed experimental setup is provided in the Appendix.

7 Experiments

Experiments show that InfoGAN efficiently maximizes mutual information and learns disentangled, interpretable latent factors without supervision across MNIST, 3D faces and chairs, SVHN, and CelebA. Latent codes capture discrete and continuous visual concepts, including digit identity, style, pose, lighting, shape, and face variation.

  • Mutual-information maximization: The mutual-information lower bound quickly reaches H(c) ≈ 2.30 on MNIST, indicating a tight bound and maximal mutual information.InfoGAN uses c ∼ Cat(K = 10, p = 0.1) in this experiment.
  • MNIST disentanglement: On MNIST, the categorical code c1 switches digit shapes and classifies digits with a 5% error rate, while continuous codes c2 and c3 control rotation and width.The continuous codes also adjust thickness and stroke style to keep generated digits natural-looking.
  • 3D faces and chairs: On 3D faces, InfoGAN recovers azimuth, elevation, and lighting as disentangled continuous factors and discovers an unlabeled code that smoothly changes faces from wide to narrow.The experiment uses five continuous codes ci ∼ Unif(−1, 1) with 1 ≤ i ≤ 5.
  • 3D faces and chairs: On 3D chairs, InfoGAN learns continuous rotation and smoothly interpolates between similar chair types with different widths using a single continuous code.The latent model uses four categorical codes with K = 20 and one continuous code c5 ∼ Unif(−1, 1).
  • SVHN and CelebA: On SVHN and CelebA, InfoGAN learns interpretable factors despite noise, clutter, pose variation, and limited repeated views, including azimuth and other visual concepts.CelebA contains 200, 000 celebrity images, and the model uses 10 uniform categorical variables of dimension 10.

8 Conclusion

InfoGAN is an unsupervised method that learns interpretable, disentangled representations with negligible added computation and easy training. The paper also identifies extensions including hierarchical latent representations, improved semi-supervised learning, and high-dimensional data discovery.

  • Conclusion: InfoGAN learns interpretable and disentangled representations completely unsupervised, unlike approaches requiring supervision.The method adds negligible computation cost on top of GAN and is easy to train.
  • Conclusion: On SVHN, latent codes separate lighting variation and the central digit from its surrounding context.A continuous code captures lighting, while a categorical code controls contextual digits independently of the central digit.
  • Conclusion: On CelebA, categorical codes capture face azimuth, glasses, hair style, and emotion.The discovered variations include discretized azimuth, presence of glasses, hair amount, and a stern-to-happy emotional range.
  • Conclusion: Future directions include hierarchical latent representations, better codes for semi-supervised learning, and using InfoGAN for high-dimensional data discovery.These extensions are listed as areas for further work.

A Proof of Lemma 5.1

The section states a conditional-expectation identity for random variables X and Y, allowing f(x, y) to be evaluated using an independent draw x′ conditioned on y under suitable regularity conditions.

  • A Proof of Lemma 5.1: Lemma A.1 equates E_x∼X,y∼Y|x[f(x, y)] with E_x∼X,y∼Y|x,x′∼X|y[f(x′, y)].The equality replaces x inside f with x′, sampled from X conditioned on y.
  • A Proof of Lemma 5.1: The identity applies to random variables X and Y together with a function f(x, y).Its validity is stated under suitable regularity conditions.
  • A Proof of Lemma 5.1: The right-hand expectation introduces x′ as a draw from X conditioned on y while retaining the outer conditioning on x.This conditional resampling produces the same expectation as the original expression.

Proof · B Interpretation as “Sleep-Sleep” Algorithm

InfoGAN can be interpreted through the Helmholtz-machine framework and Wake-Sleep training. Its auxiliary-distribution update matches the sleep phase, while its generator update also occurs in that phase, motivating the name “Sleep-Sleep” algorithm.

  • Proof: InfoGAN’s variational formulation includes an integral over x′, y, and x involving P(x′|y)f(x′, y).The supplied proof passage presents this expression without further explanatory text.
  • B Interpretation as “Sleep-Sleep” Algorithm: InfoGAN can be viewed as a Helmholtz machine, with PG(x|c) as the generative distribution and Q(c|x) as the recognition distribution.Wake-Sleep was proposed to train Helmholtz machines through separate wake and sleep updates.
  • B Interpretation as “Sleep-Sleep” Algorithm: The Wake-Sleep wake phase updates the generator by optimizing a variational lower bound of log PG(x).The supplied passage specifies that this optimization is with respect to the generator.
  • B Interpretation as “Sleep-Sleep” Algorithm: The Wake-Sleep sleep phase updates Q using samples dreamed from the current generator distribution rather than samples drawn from real data.This contrasts generator-produced samples with the real data distribution used elsewhere in training.
  • B Interpretation as “Sleep-Sleep” Algorithm: Optimizing the surrogate loss LI with respect to Q produces exactly the Wake-Sleep sleep-phase update.The equivalence concerns the update of the auxiliary distribution Q.
  • B Interpretation as “Sleep-Sleep” Algorithm: InfoGAN differs from Wake-Sleep by optimizing LI with respect to G to make the generator use latent codes c across the full prior P(c).The distinction concerns the generator’s use of latent codes over the whole latent-code prior distribution.
  • B Interpretation as “Sleep-Sleep” Algorithm: Because InfoGAN also updates the generator during the sleep phase, the method is interpreted as a “Sleep-Sleep” algorithm.This interpretation highlights InfoGAN’s difference from Wake-Sleep, although the supplied passage ends before stating the full comparison.

C Experiment Setup … C.5 Chairs

The experiments use a shared optimization and architectural framework, with latent-code-specific recognition-network parameterizations. Dataset-specific configurations vary in latent-code composition, generator input dimension, and regularization.

  • C Experiment Setup: Discrete latent codes use softmax outputs, whereas continuous codes use diagonal-Gaussian posteriors with exponentially parameterized standard deviations.The exponential transformation ensures positive standard deviations.
  • C.1 MNIST: MNIST uses 1 ten-dimensional categorical code, 2 continuous codes, and 62 noise variables, yielding a concatenated dimension of 74.The discriminator and recognition network share most of their architecture.
  • C.2 SVHN: SVHN uses 4 ten-dimensional categorical codes, 4 continuous codes, and 124 noise variables, yielding a concatenated dimension of 168.The discriminator and recognition network share most of their architecture.
  • C.3 CelebA: CelebA uses 10 ten-dimensional categorical codes and 128 noise variables, yielding a concatenated dimension of 228.The discriminator and recognition network share most of their architecture.
  • C.4 Faces: Faces uses 5 continuous latent codes and 128 noise variables, so the generator input has dimension 133.The discriminator and recognition network share the same network and differ only in their final output units; separate configurations are used for each learned variation.
  • C.5 Chairs: Chairs uses 1 continuous code, 3 discrete codes of dimension 20 each, and 128 noise variables, giving the generator input dimension 189.The discriminator and recognition network share the same network and have separate output units only at the last layer.
  • C.5 Chairs: Chairs uses separate configurations for each learned variation and different regularization coefficients for continuous and discrete latent codes.The separate configurations are presented in Table 7.
Loading 1606.03657v1…