PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Feature Learning

Jeff Donahue, Philipp Krähenbühl, Trevor Darrell

arXiv:1605.09782v7cs.LGcs.AIcs.CVcs.NEstat.ML

TL;DR

GANs lack a learned inverse mapping from data to latent representations, limiting their direct use for unsupervised feature learning. BiGANs jointly learn this inverse with generation, yielding semantic features that are competitive with contemporary self-supervised and weakly supervised approaches on natural images.

  • Problem

    GANs provide generative latent representations but lack a direct way to project data back into the latent space for unsupervised feature learning.

  • Method

    BiGANs introduce an encoder alongside the generator and discriminator so the encoder learns the inverse mapping from data x to latent features z.

  • Results

    BiGAN representations are competitive with contemporary self-supervised and weakly supervised visual feature-learning approaches, and outperform discriminator and latent-regressor baselines on natural images.

  • Takeaways & Limitations

    BiGANs provide a generic unsupervised feature-learning framework that remains a purely generative model capable of both sampling data x and predicting latent representation z.

  • Takeaways & Limitations

    The reported results are a preliminary exploration of possible BiGAN architectures, and the encoder and generator are experimentally approximate rather than exact inverses because optimization may not reach the optimum.

Abstract

from arXiv · show

The ability of the Generative Adversarial Networks (GANs) framework to learn generative models mapping from simple latent distributions to arbitrarily complex data distributions has been demonstrated empirically, with compelling results showing that the latent space of such generators captures semantic variation in the data distribution. Intuitively, models trained to predict these semantic latent representations given data may serve as useful feature representations for auxiliary problems where semantics are relevant. However, in their existing form, GANs have no means of learning the inverse mapping -- projecting data back into the latent space. We propose Bidirectional Generative Adversarial Networks (BiGANs) as a means of learning this inverse mapping, and demonstrate that the resulting learned feature representation is useful for auxiliary supervised discrimination tasks, competitive with contemporary approaches to unsupervised and self-supervised feature learning.

1 INTRODUCTION

GANs learn generators whose latent spaces capture semantic variation, but standard GANs do not provide an inverse mapping from data to latent representations. BiGAN adds an encoder and jointly discriminates data–latent pairs, enabling unsupervised feature learning through learned latent representations.

  • Motivation: GANs map latent samples to complex data distributions, and their latent spaces can encode smooth semantic variations and attributes.Examples include latent directions associated with gender and eyeglasses in GANs trained on human faces.
  • Motivation: Standard GANs lack an inverse mapping from data back to latent representations, limiting their direct use for unsupervised feature learning.A latent regressor can instead predict z from generated data G(z), but this may be insufficient when the generator does not perfectly model complex data distributions.
  • BiGAN: BiGAN extends GANs with an encoder E that maps data x to latent representations z.The framework is introduced as an unsupervised feature learning method.
  • BiGAN: Its discriminator distinguishes real pairs (x, E(x)) from generated pairs (G(z), z) in joint data and latent space.The discriminator therefore evaluates both data-space samples and their associated latent components.
  • Feature learning: The encoder’s latent representations may serve as useful features for semantic tasks because they act like labels obtained without supervision.This motivation follows prior evidence that GAN latent features capture semantic attributes.
  • Results: BiGANs are reported as generic unsupervised feature learners competitive with contemporary self-supervised and weakly supervised approaches on natural images.The paper also reports theoretical results supporting their generality across data types.

2 PRELIMINARIES

GANs generate data by transforming samples from a fixed latent distribution, but using them as feature learners requires recovering latent representations from data. Direct regression from generated samples can be inadequate when the generator poorly models complex real-world distributions.

  • GAN formulation: GANs model the data distribution by transforming samples from a fixed latent distribution through a deterministic generator G.The training goal is for the generated distribution pG(x) to approximate the data distribution pX(x).
  • GAN formulation: The GAN framework trains G adversarially against a discriminator D that distinguishes real data samples from generated samples.The generator seeks to produce samples that the discriminator cannot distinguish from data.
  • GAN formulation: For an ideal discriminator, the GAN objective corresponds to the Jensen-Shannon divergence between pG and pX.This connects adversarial optimization to distribution matching.
  • Feature learning gap: Applying GANs to unsupervised feature learning is difficult because standard GANs provide no direct inverse mapping from data to latent representations.A latent regressor predicting z from G(z) may fail when the generator does not adequately model complex distributions such as high-resolution natural images.

3 BIDIRECTIONAL GENERATIVE ADVERSARIAL NETWORKS

BiGAN augments GANs with an encoder and a joint data–latent discriminator, learning representations by matching encoder and generator joint distributions. At the optimum, the encoder and generator invert one another almost everywhere, although practical optimization may yield only approximate inverses.

  • 3 BIDIRECTIONAL GENERATIVE ADVERSARIAL NETWORKS: BiGAN adds an encoder E mapping data x to latent representations z, while its discriminator distinguishes real and generated joint pairs in data–latent space.The discriminator compares (x, E(x)) with (G(z), z).
  • 3.1 OPTIMAL DISCRIMINATOR, GENERATOR, & ENCODER: The BiGAN objective minimizes the Jensen-Shannon divergence between the encoder and generator joint distributions P_EX and P_GZ.With an optimal discriminator, C(E, G) = 2 DJS(P_EX || P_GZ) − log 4.
  • 3.1 OPTIMAL DISCRIMINATOR, GENERATOR, & ENCODER: At the global minimum, P_EX = P_GZ, the objective equals −log 4, and the optimal discriminator outputs one-half.Equality of the joint distributions characterizes the global optimum.
  • 3.2 OPTIMAL GENERATOR & ENCODER ARE INVERSES: Optimal deterministic BiGAN encoders and generators satisfy G(E(x)) = x and E(G(z)) = z almost everywhere on the supports of the data and latent distributions.The inversion property follows because fooling the discriminator requires encoder and generator pairs to satisfy both consistency conditions.
  • 3.3 RELATIONSHIP TO AUTOENCODERS: The objective is related to an ℓ0 autoencoder loss, while imposing no assumptions about the data’s structure or distribution beyond what the discriminator learns.Its reconstruction indicators enforce consistency in both data-to-latent and latent-to-data directions.
  • 3.4 LEARNING: In practice, BiGAN trains D, G, and E with alternating stochastic gradient steps using mini-batch estimates of the objective.The discriminator is updated in the positive gradient direction, while encoder and generator parameters are updated in the negative direction.
  • 3.5 GENERALIZED BIGAN: For generalized BiGANs, transformed data and latent variables define joint distributions whose equality and inversion properties extend the original theoretical results.The data transformation g_X can downsample high-resolution encoder inputs, while g_Z may remain the identity.

4 EVALUATION

BiGAN representations are evaluated on permutation-invariant MNIST and ImageNet through nearest-neighbor classification, qualitative analysis, and transfer to supervised vision tasks. On ImageNet, the learned encoder features are competitive with contemporary unsupervised and self-supervised approaches, while the framework also supports generation and reconstruction.

  • Evaluation setup: BiGAN features are evaluated on permutation-invariant MNIST and ImageNet, including transfer to classification, detection, and segmentation tasks.The evaluation compares BiGAN with GAN-based, unsupervised, self-supervised, weakly supervised, and autoencoder baselines.
  • MNIST results: On permutation-invariant MNIST, BiGAN, latent-regressor variants, and autoencoders perform at roughly the same level in 1NN classification.The comparison uses the same architecture and optimization algorithm across methods.
  • ImageNet qualitative results: BiGAN encoder filters on ImageNet show Gabor-like structure and color- versus edge-sensitive grouping similar to filters from supervised AlexNet.The comparison includes BiGAN’s D, G, and E modules, a discriminator baseline, self-supervised filters, and fully supervised ImageNet filters.
  • ImageNet qualitative results: ImageNet reconstructions G(E(x)) are imperfect but empirically demonstrate that the BiGAN encoder and generator learn approximate inverse mappings.The figure also presents generator samples G(z) and real data samples x.
  • Transfer results: BiGAN is competitive with contemporary visual feature-learning methods and outperforms the discriminator and latent-regressor baselines on complex natural-image data.Using a higher-resolution encoder input, BiGAN 112 × 112 E performs better still; this strategy is unavailable to the discriminator and latent-regressor baselines.
  • Discussion: The reported results are a preliminary exploration of BiGAN architectures, with expected improvements from advances in generative image models and discriminative convolutional networks.This scope boundary applies to the architectures explored in the paper.

A.1 PROOF OF PROPOSITION 1 (OPTIMAL DISCRIMINATOR)

The optimal BiGAN discriminator is the Radon–Nikodym derivative of the encoder joint measure with respect to the sum of encoder and generator joint measures. The objective can therefore be rewritten as a single expectation under their averaged measure.

  • The optimal discriminator D* equals f_EG, the Radon–Nikodym derivative dP_EX/d(P_EX + P_GZ).This follows by maximizing a log-likelihood expression pointwise, where the maximizing value is a.
  • The encoder and generator joint measures are each absolutely continuous with respect to P_EG, so their derivatives with respect to P_EG exist and sum to one.
  • The adversarial objective combines expectations over P_EX and P_GZ into one expectation under P_EG.The two terms become weighted by f_EG and f_GE, respectively.

A.2 PROOF OF PROPOSITION 2 (ENCODER AND GENERATOR OBJECTIVE)

With an optimal discriminator, the BiGAN encoder–generator objective is expressed through the Jensen–Shannon divergence between their joint measures. The stated deterministic-function assumption applies to later theorems, not to Propositions 1 and 2.

  • Propositions 1 and 2 hold for stochastic or deterministic encoders and generators, whereas Theorems 2 and 3 assume deterministic functions.Under that assumption, the conditionals are represented by delta functions.

A.4 PROOF OF THEOREM 2 (OPTIMAL GENERATOR AND ENCODER ARE INVERSES)

At optimality, the BiGAN encoder and generator invert one another almost everywhere under the data and latent distributions. The proof establishes this by showing the non-inversion regions have measure zero.

  • E = G^-1 almost everywhere for optimal E and G, so G(E(x)) = x for P_X-almost every x and E(G(z)) = z for P_Z-almost every z.
  • The data-space non-inversion region has zero P_X measure, establishing x = G(E(x)) almost everywhere.The argument uses equality of the optimal joint measures P_EX and P_GZ.
  • An analogous argument shows that the latent-space non-inversion region has measure zero under P_Z.

A.5 PROOF OF THEOREM 3 (RELATIONSHIP TO AUTOENCODERS)

The proof connects the optimal BiGAN objective to an autoencoder-like reconstruction loss by analyzing where the joint-measure divergence can be nonzero. This yields the stated ℓ0 autoencoder relationship.

  • The divergence is finite and behaves like an autoencoder reconstruction loss that is nonzero only where G(E(x)) = x.The proof first establishes f > 0 P_EX-almost everywhere and then identifies the relevant overlap region.
  • The KL integrand is zero outside the support of P_GZ because the Radon–Nikodym derivative equals one P_EX-almost everywhere there.Consequently, log f is zero on that region.
  • Within the common support, f is strictly less than one P_EX-almost everywhere, making log f strictly negative there.
  • The optimal encoder–generator objective can be rewritten as an ℓ0 autoencoder loss function.This is the central statement of Theorem 3.
  • The two KL-divergence terms are treated analogously, with both logarithmic terms negative almost everywhere under their respective joint measures.
  • The Jensen–Shannon divergence reduces the analysis to the overlap of the supports of P_EX and P_GZ.Outside either support, the relevant KL contribution is zero.

APPENDIX B LEARNING DETAILS

BiGAN training uses an inverse objective that provides stronger gradient signals, with simultaneous parameter updates adopted for computational efficiency.

  • Learning protocol: The inverse objective Λ provides stronger gradient signals to both the generator G and encoder E while retaining the original objective's fixed-point characteristics.This objective swaps the real and generated labels Y, following an approach observed to strengthen GAN training gradients.
  • Learning protocol: θG and θE are updated in the positive gradient direction of Λ rather than the negative gradient direction of the original objective.
  • Learning protocol: Simultaneous updates of θD, θG, and θE perform similarly to alternating updates, so BiGAN training uses the non-alternating approach for computational efficiency.The same simultaneous-update protocol is also used for the standard GAN experiments.

APPENDIX C MODEL AND TRAINING DETAILS

The appendix specifies evaluation coverage, optimization settings, software frameworks, hardware, and the multilayer architecture used for permutation-invariant MNIST.

  • Scope: The additional details cover the permutation-invariant MNIST and ImageNet evaluations presented in Section 4.
  • Optimization: Adam optimization uses α = 2 × 10^-4, β1 = 0.5, β2 = 0.999, batch size 128, exponential decay to α = 2 × 10^-6, and ℓ2 weight decay of 2.5 × 10^-5.The learning-rate decay starts halfway through training, and weight decay applies to multiplicative weights in linear layers.
  • Software and hardware: BiGANs and baseline feature-learning methods are implemented in Theano, while ImageNet transfer experiments use Caffe and reference implementations from Fast R-CNN and FCN.
  • Permutation-invariant MNIST: For permutation-invariant MNIST, D, G, and E each contain two hidden layers with 1024 units, followed by nonlinearities and a linear prediction layer.The second hidden layer uses parameter-free batch normalization before its nonlinearity.

C.2 IMAGENET

ImageNet experiments use an AlexNet-derived encoder and convolutional generator/discriminator architectures, with specified crop preprocessing and nearest-neighbor analysis of learned features.

  • Model architecture: The ImageNet encoder E follows AlexNet through conv5, removing LRN and applying batch normalization with leaky ReLU during unsupervised training.For supervised evaluation, batch normalization is not used.
  • Model architecture: The discriminator D and generator G generally use four stride-2 5 × 5 convolutions or fractionally strided convolutions, each followed by batch normalization and a rectified nonlinearity.
  • Model architecture: Using AlexNet-like architecture for discriminator D is generally detrimental to generated-image visual fidelity, likely because large kernels and max-pooling discard input information.The discriminator baseline feature-learning experiment is the stated exception.
  • Preprocessing: ImageNet samples are resized with the shorter edge at 72 pixels, randomly cropped to 64 × 64, horizontally flipped with probability 1/2, and scaled to [−1, 1].
  • Nearest neighbors: Figure 5 compares query images with ImageNet training-set nearest neighbors using minimum cosine distance in the BiGAN encoder's fc6 feature space.Because fc6 weights are random, this space is a random projection of the learned conv5 feature space.
  • Timing: 40 minutes per epoch and under 3 days for 100 epochs characterize BiGAN training on a Titan X GPU.Each epoch passes over 1.2 million images.
Loading 1605.09782v7…