PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Autoencoders

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, Brendan Frey

arXiv:1511.05644v2cs.LG

TL;DR

The paper addresses generative modeling and latent representation learning with a probabilistic autoencoder whose aggregated posterior is matched to an arbitrary prior using adversarial training. AAEs support generative modeling and semi-supervised learning, achieving competitive reported results while extending to several representation-learning applications.

  • Problem

    Deep generative models must capture rich data distributions, while semi-supervised learning seeks useful class representations from limited label information.

  • Method

    AAEs combine autoencoder reconstruction with adversarial matching of the aggregated latent posterior to an arbitrary prior distribution.

  • Results

    AAEs achieve competitive generative-modeling and semi-supervised-classification results across MNIST, SVHN, and Toronto Face experiments.

  • Takeaways & Limitations

    AAEs extend probabilistic autoencoders to semi-supervised classification, style-content disentanglement, unsupervised clustering, dimensionality reduction, and data visualization.

Abstract

from arXiv · show

In this paper, we propose the "adversarial autoencoder" (AAE), which is a probabilistic autoencoder that uses the recently proposed generative adversarial networks (GAN) to perform variational inference by matching the aggregated posterior of the hidden code vector of the autoencoder with an arbitrary prior distribution. Matching the aggregated posterior to the prior ensures that generating from any part of prior space results in meaningful samples. As a result, the decoder of the adversarial autoencoder learns a deep generative model that maps the imposed prior to the data distribution. We show how the adversarial autoencoder can be used in applications such as semi-supervised classification, disentangling style and content of images, unsupervised clustering, dimensionality reduction and data visualization. We performed experiments on MNIST, Street View House Numbers and Toronto Face datasets and show that adversarial autoencoders achieve competitive results in generative modeling and semi-supervised classification tasks.

1 Introduction

The paper introduces adversarial autoencoders, which combine reconstruction with adversarial matching of latent codes to a user-specified distribution to create generative models.

  • Unlike earlier MCMC-trained generative models, direct back-propagation methods avoid difficulties caused by slow mixing between modes.The introduction contrasts MCMC-based models with VAEs, GANs, and GMMNs as alternatives trained through back-propagation or moment matching.
  • AAEs turn autoencoders into generative models by combining reconstruction error minimization with adversarial training.The adversarial network distinguishes hidden-code samples from samples drawn from the specified distribution.
  • Adversarial training matches the aggregated posterior of the latent representation to an arbitrary prior distribution.This approach uses a discriminator to compare samples from the autoencoder’s hidden code with samples from the chosen distribution.
  • The encoder maps the data distribution toward the imposed prior, while the decoder maps prior samples back to the data distribution.
  • GAN training alternates between discriminating true and generated samples and updating the generator to fool the discriminator.

2 Adversarial Autoencoders

Adversarial autoencoders combine reconstruction with adversarial regularization to match an autoencoder’s aggregated latent posterior to an arbitrary prior. The resulting decoder defines a generative model, while experiments show improved prior matching, meaningful latent manifolds, label-structured representations, and support for nonparametric priors.

  • Core model: An adversarial autoencoder attaches an adversarial network to the hidden code, matching aggregated posterior q(z) to arbitrary prior p(z) while minimizing reconstruction error.Training alternates reconstruction and regularization phases, with the discriminator distinguishing prior samples from encoded hidden codes and the encoder confusing it.
  • Generative model: After training, the decoder maps samples from the imposed prior p(z) to the data distribution, turning the autoencoder into a generative model.This makes prior sampling the route to generating data through the learned decoder.
  • Core model: The encoder can use deterministic, Gaussian, or universal-approximator posteriors, with input noise enabling sampling from arbitrary q(z|x).Different posterior choices produce different training dynamics; Gaussian and universal-approximator variants introduce stochasticity beyond the data distribution.
  • Relationship to Variational Autoencoders: On MNIST, AAEs matched both spherical Gaussian and mixtures of 10 Gaussians more faithfully than VAEs, whose latent codes showed holes or systematic mode differences.The AAE’s filled coding space supported interpolation along the data manifold, whereas the VAE emphasized matching mixture modes.
  • Relationship to Variational Autoencoders: Unlike VAEs, AAEs require only samples from the prior, enabling imposed distributions without an explicit functional form, including a swiss-roll distribution.The paper demonstrates latent traversal and generated images along the swiss-roll axis.
  • Incorporating Label Information in the Adversarial Regularization: With 10K labeled and 40K unlabeled MNIST examples, assigning mixture components to classes produced class-structured codes while preserving writing style within each component.The same framework maps MNIST onto a swiss-roll prior and generates samples by traversing its main latent axis.

3 Likelihood Analysis of Adversarial Autoencoders

The authors evaluate AAE generative modeling on MNIST and TFD using hold-out likelihood estimates and generated samples. AAEs achieve superior reported log-likelihoods, while the authors caution that these evaluation metrics are flawed.

  • The experiments measure how well AAEs capture MNIST and TFD data distributions using hold-out image likelihood evaluation.The evaluation uses a procedure described in prior work.
  • Generated samples are compared with their nearest training images in pixel-wise Euclidean distance to assess possible copying.The nearest neighbors appear in the final figure column.
  • Table 1 compares AAE log-likelihoods on MNIST and TFD with DBNs, stacked CAEs, Deep GSNs, GANs, and GMMN + AEs.The table reports Parzen-window estimates using 10K or 10M generated samples.
  • AAEs achieve superior log-likelihoods to competing methods in the reported comparisons.The authors note that Parzen-window estimates are lower bounds whose tightness depends on sample count.
  • The authors warn that available likelihood metrics for generative models such as GANs are deeply flawed.They specifically identify problems with 10K- and 10M-sample Parzen-window estimates.

4 Supervised Adversarial Autoencoders

Supervised AAEs separate class-label information from image style by supplying labels to the decoder while regularizing the hidden code. On MNIST and SVHN, fixed hidden codes produce consistent styles as labels vary.

  • The supervised architecture provides a one-hot label vector to the decoder alongside hidden code z to reconstruct each image.This forces information independent of the label into z.
  • On MNIST and SVHN, reconstructed images retain consistent style across each row while the label is systematically changed.The hidden code is constrained to a 15-D Gaussian in the MNIST experiment.
  • For SVHN, the hidden style also captures the unlabeled left-most and right-most digit information.The one-hot vector represents only the central digit.

5 Semi-Supervised Adversarial Autoencoders

Semi-supervised AAEs model images with discrete class and continuous style variables, using separate adversarial regularizers for their latent representations. They outperform VAEs on labeled MNIST settings and nearly match ADGM on SVHN with 1000 labels.

  • The semi-supervised model assumes a Categorical latent class variable y and a Gaussian continuous latent variable z.The model uses unlabeled data together with limited labeled data.
  • The encoder predicts y and z, while the decoder reconstructs images from the one-hot class label and continuous hidden code.Separate adversarial networks regularize the two representations.
  • One adversarial network matches the label representation to a Categorical distribution, while the other imposes a Gaussian distribution on style.The regularizers are intended to keep class labels free of style information and style continuous.
  • On MNIST with 100 or 1000 labels, AAEs perform significantly better than VAEs, match VAT and CatGAN, but trail Ladder networks and ADGM.On SVHN with 1000 labels, the AAE nearly matches ADGM's state-of-the-art classification performance.
  • All AAE models are trained end-to-end, whereas the semi-supervised VAE models require layerwise training.

6 Unsupervised Clustering with Adversarial Autoencoders

AAEs can disentangle discrete class variables from continuous style variables without supervision and use the learned class representation for clustering. On MNIST, clusters reflect both digit identity and discrete writing styles.

  • The unsupervised AAE disentangles discrete class variables from continuous latent style variables using unlabeled data only.
  • For 16-cluster MNIST experiments, the model predicts one-hot cluster assignments without a labeled mini-batch.Cluster heads are generated by fixing style to zero and selecting each one-hot label vector.
  • The learned clusters separate tilted from straight 1s and 6s and divide 2s according to whether they contain a loop.These examples show that clusters capture discrete styles as class labels.
  • The clustering evaluation assigns each cluster the label of its validation example with maximum q(y_i|x_n), then computes test error.
  • The AAE achieves classification error rates of 9.55% with 16 clusters and 4.10% with 30 total labels.

7 Dimensionality Reduction with Adversarial Autoencoders

AAEs provide a parametric approach to dimensionality reduction that uses adversarial regularization to keep similar images’ codes together and avoid manifold fracturing. On MNIST, the method yields clean supervised and unsupervised cluster structure in low-dimensional representations.

  • Motivation and architecture: The final n-dimensional representation adds an n-dimensional cluster-head representation to an n-dimensional style representation.The cluster head is obtained from the label representation, while the style representation is learned separately.
  • Motivation and architecture: AAE dimensionality reduction uses adversarial regularization to attach similar images’ hidden codes and prevent manifold fracturing.The architecture is designed for visualization and uses a parametric encoder for embedding data.
  • Motivation and architecture: The cluster-head cost is zero beyond threshold η and linearly penalizes Euclidean distances below η.This cost encourages separation between cluster heads while leaving sufficiently distant pairs unpenalized.
  • Semi-supervised dimensionality reduction: With 1000 and 100 labels, 2D semi-supervised MNIST representations achieve classification errors of 4.20% and 6.08%, respectively.The digit clusters are cleanly separated, although the 2D constraint limits classification accuracy relative to higher-dimensional cases.
  • Unsupervised dimensionality reduction: Unsupervised 2D MNIST embeddings with 20 clusters separate digits and stylistic sub-clusters, including straight versus tilted 1s and looped versus unlooped 2s.Digit 6 is also divided into three clusters according to tilt.
  • Higher-dimensional representation: With 100 labels in 10 dimensions, the model achieves a 3.90% classification error, versus 1.90% for the concatenated style-and-label representation.The 10D style representation learns a Gaussian distribution and is visualized through a linear map to 2D.

8 Conclusion

The paper presents AAEs as a GAN-based variational inference method for probabilistic autoencoders and reports competitive generative-modeling and semi-supervised classification performance. It also demonstrates applications spanning representation learning, clustering, dimensionality reduction, and visualization.

  • AAEs use the GAN framework as a variational inference algorithm for discrete and continuous latent variables in probabilistic autoencoders.
  • AAEs achieve competitive test likelihoods on real-valued MNIST and Toronto Face datasets.
  • AAEs achieve competitive semi-supervised classification performance on MNIST and SVHN datasets.
  • The paper demonstrates AAEs for disentangling image style and content, unsupervised clustering, dimensionality reduction, and data visualization.

A.1 Likelihood Experiments

The likelihood experiments describe implementation choices and a dimensionality assumption for deterministic and higher-dimensional latent codes. These details constrain how the AAE likelihood experiments are configured.

  • The encoder, decoder, and discriminator each use two 1000-unit ReLU layers, with Euclidean reconstruction cost and mini-batches of 100.The final q(z|x) layer is linear, and weights use Gaussian initialization with standard deviation 0.01.
  • For deterministic q(z|x), the hidden-code dimensionality should match the data’s intrinsic dimensionality because data variation is its only stochastic source.The stated ranges are 5–8 dimensions for MNIST and 10–20 for TFD and SVHN.

A.2.1 MNIST

The MNIST experiments specify shared encoder structure, separate categorical and Gaussian latent representations, and optimization schedules. The setup also includes randomly selected, evenly distributed labels and no additional regularization beyond input dropout.

  • Model configuration: The MNIST model uses shared 1000-unit encoder layers, 10-dimensional style and label representations, Gaussian style priors, and categorical label priors.The encoder, decoder, and discriminators use two 1000-unit ReLU layers.
  • Optimization: All costs are optimized with gradient descent and momentum, using momentum 0.9 for reconstruction and semi-supervised costs and 0.1 for adversarial networks.The reconstruction learning rate starts at 0.01 and is reduced after 50 and 1000 epochs.
  • Dataset and training details: SVHN experiments use about 530K training points and 26K test points, with 20-dimensional style representations and standardized inputs.Inputs are normalized by subtracting the training-set mean and dividing by the training-set standard deviation per dimension.
  • Dataset and training details: Training runs for 1000 epochs with 20% input dropout and no other dropout, ℓ2 weight decay, or Gaussian-noise regularization.Labeled examples are selected randomly while being distributed evenly across the classes.
  • Alternative configuration: A separate configuration uses 5 style dimensions and 30 categorical clusters, with Gaussian style and categorical label distributions.
Loading 1511.05644v2…