PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Latent Autoencoders

Stanislav Pidhorskyi, Donald Adjeroh, Gianfranco Doretto

arXiv:2004.04467v1cs.LGcs.CV

TL;DR

Autoencoders had not fully resolved whether they could match GANs’ generative power while learning disentangled representations. ALAE addresses both goals by learning the latent distribution, adversarially learning the data distribution, and enforcing latent-space reciprocity; StyleALAE reaches 1024 × 1024 face generation comparable with StyleGAN while also reconstructing and manipulating real images. The paper’s evidence includes two backbones, but its MLP experiments show that ALAE does not always provide the best accuracy.

  • Problem

    Autoencoders had not fully established GAN-level generative power together with disentangled representations.

  • Method

    ALAE jointly learns the latent distribution from data, adversarially learns the output distribution, and imposes reciprocity in latent space.

  • Results

    StyleALAE generates 1024 × 1024 face and bedroom images comparable with StyleGAN while enabling real-face reconstruction and manipulation.

  • Takeaways & Limitations

    ALAE’s learned latent distribution and StyleALAE extension support less-entangled representations and image operations unavailable with StyleGAN alone.

  • Takeaways & Limitations

    ALAE does not always provide the best accuracy, with baseline AE variants performing well for 1NN, especially using short features.

Abstract

from arXiv · show

Autoencoder networks are unsupervised approaches aiming at combining generative and representational properties by learning simultaneously an encoder-generator map. Although studied extensively, the issues of whether they have the same generative power of GANs, or learn disentangled representations, have not been fully addressed. We introduce an autoencoder that tackles these issues jointly, which we call Adversarial Latent Autoencoder (ALAE). It is a general architecture that can leverage recent improvements on GAN training procedures. We designed two autoencoders: one based on a MLP encoder, and another based on a StyleGAN generator, which we call StyleALAE. We verify the disentanglement properties of both architectures. We show that StyleALAE can not only generate 1024x1024 face images with comparable quality of StyleGAN, but at the same resolution can also produce face reconstructions and manipulations based on real images. This makes ALAE the first autoencoder able to compare with, and go beyond the capabilities of a generator-only type of architecture.

1. Introduction

The paper introduces ALAE to jointly address autoencoders’ generative power and disentanglement, using learned latent distributions, adversarial data-distribution learning, and latent-space reciprocity. Its StyleALAE variant achieves high-resolution generation while enabling reconstruction and manipulation of real faces.

  • Research gap: Autoencoders had not jointly demonstrated GAN-level generative power and disentangled representations.Prior work addressed these goals separately, with sharp face generation serving as an important testbed for generative success.
  • ALAE approach: ALAE learns the latent distribution from data, learns the output distribution adversarially, and imposes reciprocity in latent space.This combination avoids imposing a fixed latent distribution and avoids relying on data-space reconstruction norms.
  • ALAE approach: The architecture is designed to incorporate GAN advances, including stochasticity sources, loss functions, regularization, and hyperparameter improvements.These components preserve the approach’s compatibility with recent GAN training procedures.
  • Architectures: The authors develop an MLP-based ALAE and StyleALAE, which uses a generator derived from StyleGAN and a companion progressively growing encoder.Both architectures are evaluated for disentanglement, while StyleALAE additionally targets high-resolution image synthesis and inversion.
  • Results: At 1024 × 1024 resolution, StyleALAE produces face and bedroom images comparable with StyleGAN and also reconstructs and manipulates real faces.The encoder makes real-image reconstruction and manipulation possible in addition to generation.

2. Related Work

Related work spans GAN advances, disentanglement-oriented variational autoencoders, hybrid GAN–VAE models, and alternative generative methods. ALAE is positioned among approaches combining generative modeling with latent representations.

  • GAN-based generation: GAN research has pursued high-resolution synthesis through multi-resolution pyramids, hierarchical adversarial objectives, multi-scale architectures, and improved synthesis methods.Examples include LAP-GAN, StackGAN, HDGAN, semantic-label-conditioned GANs, and BigGAN.
  • Variational autoencoders: Variational autoencoders supported research on stable training, interpretable representations, disentanglement control, and quantitative disentanglement metrics.The paper uses established disentanglement metrics to analyze its own approach.
  • Hybrid models: Hybrid approaches have combined GAN and VAE benefits through similarity-based, fully adversarial, or latent-space autoencoding frameworks.The related methods include AAE, VAE/GAN, BiGAN, ALI, VEEGAN, AGE, PIONEER, and IntroVAE.
  • Other generative methods: Autoregressive methods avoid latent representations, whereas flow-based methods retain them while modeling image data distributions.These approaches are mentioned as other promising directions for representing image distributions.

3. Preliminaries

The preliminaries formulate GANs as generator–discriminator systems that match a generated distribution to the data distribution. Training is expressed as a minimax problem over a general concave value function, covering standard and Wasserstein GAN objectives.

  • GAN formulation: A GAN generator G maps a known latent distribution p(z) into a synthetic data distribution q(x), while discriminator D maps data to R.The learning goal is for q(x) to approach the real distribution pD(x).
  • GAN formulation: GAN training finds a minimax solution over the generator–discriminator pair using a value function defined by expectations over real data and latent samples.This formulation corresponds to the Nash equilibrium of the adversarial game.
  • GAN objectives: The general objective uses a concave function f; choosing f(t) = −log(1 + exp(−t)) gives the original GAN, while f(t) = t gives Wasserstein GAN.Thus, the formulation encompasses multiple adversarial training objectives through the choice of f.

4. Adversarial Latent Autoencoders

ALAE modifies the GAN paradigm by decomposing generator and discriminator pathways, learning the latent distribution rather than imposing it, and enforcing latent-space reciprocity. Its adversarial training preserves generative modeling while supporting autoencoding and less entangled representations.

  • Architecture: ALAE decomposes the generator and discriminator into F, G, and E, D networks, respectively, with shared interface latent space W.F maps the input latent distribution to W, while E and G may be stochastic; G can optionally use independent noise η.
  • Reciprocity: ALAE enforces reciprocity by matching the encoder output distribution to the generator input distribution, allowing (G,E) to autoencode latent space W.Latent-space reciprocity permits simple norm-based reconstruction criteria that may be inappropriate in image space.
  • Training objective: The adversarial objective alternates optimization of F,G against E,D to align generated data and latent distributions.The resulting network is called an Adversarial Latent Autoencoder (ALAE).
  • Design rationale: The framework is designed to retain GAN generative properties while incorporating stochasticity and advances in GAN losses, regularization, and hyperparameter methods.Its criteria distinguish data-distribution matching, latent-distribution setting or learning, and the space in which reciprocity is achieved.
  • Latent distribution: Unlike conventional autoencoders, ALAE does not impose a target latent distribution; it learns qE(w) subject only to matching qF(w).The approach specifically avoids forcing F to be the identity map, allowing learning to determine the latent transformation.

5. StyleALAE

StyleALAE adapts a StyleGAN-based generator with a symmetric encoder that extracts multiscale style information and maps it into the intermediate latent space. Progressive growing and MLP-based auxiliary networks support the architecture across increasing image resolutions.

  • Architecture: StyleALAE uses the StyleGAN generator with latent space W playing the role of StyleGAN’s intermediate latent space, alongside a newly designed encoder.The generator corresponds to the right side of the StyleALAE architecture, while the encoder forms the left side.
  • Style encoding: The encoder is symmetric to the generator, extracting style information from corresponding layers through Instance Normalization statistics.These statistics provide per-channel instance averages and standard deviations at multiple levels.
  • Style encoding: The encoder’s information flow mimics multiscale style transfer, with normalized activations continuing through the pipeline after style statistics are extracted.Unlike standard style transfer, the architecture does not use an additional image to provide content.
  • Latent mapping: StyleALAE combines encoder-produced styles and maps them to latent code w through a learnable multilinear map.The styles input to the generator’s AdaIN layers are linearly related to w, and the map uses learnable parameters C_i across N layers.
  • Progressive growing: Progressive growing starts at 4×4-pixel images and smoothly adds blocks to the encoder and generator as resolution increases.F and D are implemented as MLPs; for StyleALAE, F has 8 layers and D has 3 layers.

6. Implementation

ALAE training combines nonsaturating adversarial losses with gradient regularization and alternating updates across discriminator, generator, and latent-space autoencoder components.

  • Adversarial losses and regularization: ALAE uses a nonsaturating SoftPlus loss and gradient regularization, including an R1 zero-centered gradient penalty applied only to real data.The gradient penalty is taken with respect to the parameters of E and D.
  • Training: Algorithm 1 initializes θF, θG, θE, and θD, then repeatedly samples data minibatches and prior codes while applying the three update steps until convergence.The prior samples are drawn from N(0, I).
  • Reconstruction results: Figure 3 compares real MNIST images with BiGAN and ALAE reconstructions using the same MLP architecture for both methods.The figure places real images in the top row, BiGAN reconstructions in the middle, and ALAE reconstructions in the bottom.
  • Training: Each training iteration performs three updates: discriminator networks E and D, generator networks F and G, then latent-space autoencoder networks G and E.The procedure uses alternating updates and Adam with β1 = 0.0 and β2 = 0.99.

7. Experiments

Experiments evaluate ALAE on MNIST, FFHQ, LSUN Bedroom, and CelebA-HQ for representation quality, generation, reconstruction, and disentanglement. StyleALAE produces generations and reconstructions at high resolution, while MNIST analyses favor more disentangled short latent features.

  • MNIST representation learning: ALAE remains more stable than competing approaches when switching from 1NN to linear SVM, suggesting greater disentanglement, especially for short features.For long features, the effect fades because linear separability grows.
  • MNIST representation learning: ALAE does not always provide the best MNIST classification accuracy; baseline AE, especially AE(ℓ2), performs well with 1NN and short features.The paper suggests this may reflect the baseline AE learning a representation closer to a discriminative one.
  • MNIST representation learning: W-space interpolations between the same MNIST digits produce smoother image transitions than Z-space interpolations.The smoother W-space transition is presented as suggesting a lesser degree of entanglement.
  • Image generation and reconstruction: StyleALAE is evaluated on unseen FFHQ images at 1024 × 1024 and on LSUN Bedroom generations and reconstructions at 256 × 256.FFHQ uses 60000 training images and 10000 testing images; LSUN evaluation includes unseen images during training.

8. Conclusions

ALAE is presented as a flexible autoencoder effective with different backbone networks. StyleALAE preserves visual detail while enabling reconstruction and manipulation beyond generator-only capabilities.

  • ALAE is effective with two very different backbone generator-encoder networks.
  • StyleALAE reconstructions of unseen CelebA-HQ samples at 256 × 256 look sharper and less distorted than PIONEER reconstructions.
  • StyleALAE generates and manipulates images beyond StyleGAN alone while maintaining the same level of visual detail.
Loading 2004.04467v1…