PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically Decomposing the Latent Spaces of Generative Adversarial Networks

Chris Donahue, Zachary C. Lipton, Akshay Balsubramani, Julian McAuley

arXiv:1705.07904v3cs.LGcs.AIcs.CVcs.NEstat.ML

TL;DR

GANs generate high-fidelity images but do not explicitly separate latent factors associated with known commonalities. SD-GANs decompose latent codes and train on identity-matched pairs; human and automated evaluations recognize the resulting contrasting face images as depicting the same subject.

  • Problem

    GANs do not explicitly disentangle latent factors according to known commonalities, despite datasets often containing identity labels but few observations per identity.

    Source: §1 · p. 1

  • Method

    SD-GANs divide latent codes into identity and observation components and train Siamese discriminators on real and generated pairs sharing an identity code.

    Source: §1 · p. 1, §2.3 · p. 4

  • Results

    Human annotators and a face verification system recognize stylistically contrasting SD-GAN face pairs as depicting the same subject.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SD-GANs can sample identities not seen in training, unlike conditional GANs, while performing comparably on identity coherence and image diversity.

    Source: §1 · p. 2

    The greater complexity of SD-BEGAN increased training time and limited exhaustive hyperparameter validation; its k = 4 variant was excluded after early mode collapse.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

We propose a new algorithm for training generative adversarial networks that jointly learns latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). By fixing the identity portion of the latent codes, we can generate diverse images of the same subject, and by fixing the observation portion, we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose. Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code. Corresponding samples from the real dataset consist of two distinct photographs of the same subject. In order to fool the discriminator, the generator must produce pairs that are photorealistic, distinct, and appear to depict the same individual. We augment both the DCGAN and BEGAN approaches with Siamese discriminators to facilitate pairwise training. Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm's ability to generate convincing, identity-matched photographs.

1 INTRODUCTION

SD-GANs partition latent codes into identity and observation components, training on pairs that share identity while varying observation. Face and product experiments show identity-matched variation, with human and automated judgments supporting the face results.

1 INTRODUCTION

  • Many datasets provide identity labels but not labels for contingent aspects such as lighting, pose, and background.The paper motivates latent structure that can represent these commonalities and support multiple stages of generation.Source: §1 · p. 1
  • SD-GANs split latent codes into identity and observation components and train on real and generated image pairs sharing an identity.The generator must produce photorealistic, diverse pairs that appear to show the same person when identity is fixed.Source: §1 · pp. 1–2
  • Face experiments show pose, expression, facial hair, color, and lighting can vary across photographs while identity-relevant aspects remain consistent.SD-GANs are also trained on product images with multiple views of each product.Source: §1 · p. 2
  • Human annotators and a face verification system recognize stylistically contrasting SD-GAN face pairs as depicting the same subject.SD-GANs perform comparably to a conditional GAN on identity coherence and image diversity, while also sampling identities absent from training.Source: §1 · p. 2

2 SEMANTICALLY DECOMPOSED GENERATIVE ADVERSARIAL NETWORKS

SD-GANs organize latent variation into identity and observation components, and use matched real and generated pairs to train a discriminator that assesses both realism and identity consistency. Siamese discriminator designs adapt this pairwise objective to DCGAN and BEGAN.

2.1 GAN PRELIMINARIES

  • GAN training alternates discriminator updates on real and generated minibatches with generator updates on sampled latent vectors.The paper also reviews BEGAN's autoencoder-based discriminator and adapts both DCGAN and BEGAN to SD-GAN pairwise training.Source: §2.1 · pp. 2–3

2.2 SD-GAN FORMULATION

  • A conventional GAN models the average over identity rather than disentangling identity from other variation in the data.The intended decomposition makes interpolation and operations on the identity and observation subspaces convenient.Source: §2.2 · p. 3
  • SD-GANs partition latent coordinates into identity and contingent-aspect subspaces, training on real pairs of one identity and generated pairs sharing an identity code.The discriminator rejects generated pairs that are not photorealistic or do not plausibly depict the same subject.Source: §2.2 · p. 3

2.3 SD-GAN DISCRIMINATOR ARCHITECTURE

  • The discriminator encodes both images with shared convolutional weights, reflecting the symmetry of the pairwise identity-matching task.The generator architecture need not change, but the discriminator must jointly score the two images.Source: §2.3 · p. 4, Figure 2 · p. 4
  • SD-BEGAN concatenates the encoded images, maps them through a shared bottleneck, and splits the output into inputs for two decoders.The bottleneck has dimensionality dI + 2dO; its shared intermediate layer provides a mechanism to push apart matched and unmatched pairs.Source: §2.3 · p. 4

3 EXPERIMENTS

Experiments evaluate SD-GANs on face and shoe images, using FaceNet and human judgments to assess identity coherence and image diversity. SD-BEGAN’s face samples are verified comparably to real data, while SD-DCGAN also produces convincing shoe variations.

3 EXPERIMENTS

  • The evaluation uses MS-Celeb-1M faces and Amazon shoe images, with the latter serving as a sanity check where products primarily vary in orientation and distance.The face data has label noise; duplicates were removed, but labels were not cleaned.Source: §3 · pp. 4–5

3.1 TRAINING DETAILS

  • SD-BEGAN’s greater training complexity limited hyperparameter validation, and its k = 4 variant was excluded after early mode collapse.The SD-DCGAN k = 4 variant was also tested.Source: §3.1 · p. 5

3.2 EVALUATION

  • FaceNet evaluates identity coherence alongside ID-Div for within-identity pixel diversity and All-Div for overall sample diversity.The diversity measures use one minus mean MS-SSIM similarity.Source: §3.2.1 · pp. 6–7

3.2.1 QUANTITATIVE

  • FaceNet verified pairs with 87% accuracy on real data and 86% on SD-BEGAN, with SD-GAN samples more diverse in pixel space than AC-DCGAN.The evaluation also reports AUC, FAR, ID-Div, and All-Div.Source: Table 1 · p. 6, §3.2.1 · p. 7
  • SD-GAN model complexity is constant in the number of identities, while an AC-DCGAN for 1M identities would exceed 8GB, with over 97% of parameters in its discriminator softmax.The 10k-identity AC-DCGAN model requires 131MB.Source: §3.2.1 · p. 7

3.2.2 QUALITATIVE

  • Mechanical Turk annotators answered “same person” at the same rate for SD-BEGAN pairs as for real data, but 16% more often for AC-DCGAN pairs than real data.On balanced real pairs, annotators answered “same person” 28% of the time, compared with FaceNet’s 41%.Source: §3.2.2 · p. 7

4 RELATED WORK

Related work explores disentangling style and content, conditional generation, and Siamese learning. SD-GANs differ by modeling contingent factors implicitly, without supervision, and by using a Siamese discriminator for matched-pair generation.

4 RELATED WORK

  • Earlier style-content disentanglement approaches used factored tensor representations, while later work used deep convolutional networks to separate identity and pose manifolds.Related applications include face synthesis and products.Source: §4 · p. 7
  • Mutual-information GANs may disentangle intuitive variation factors, but those factors need not match the factors SD-GANs explicitly disentangle.Chen et al. (2016) describe an unsupervised method.Source: §4 · pp. 7–8
  • Unlike face and product synthesis methods conditioned on explicit pose, lighting, or age factors, SD-GANs model contingent factors implicitly without supervision.The cited methods condition on an identity image and factor labels or vectors.Source: §4 · p. 8
  • Compared with Mathieu et al.’s reconstruction-loss approach, SD-GANs offer a simpler approach that can synthesize higher-resolution color photographs than its 28x28 grayscale experiments.The cited approach uses an encoder-decoder generator and a variational bound on its encoder embedding.Source: §4 · p. 8
  • SD-GANs use a Siamese architecture so the discriminator can distinguish matched from unmatched pairs, framing the approach as a generative counterpart to similarity-learning networks.Liu and Tuzel’s GAN uses shared weights but a different problem formulation and objective.Source: §4 · p. 8

5 DISCUSSION

SD-GANs separate identity variation from observation variation, allowing changes in contingent image factors while apparent identity remains fixed. The paper also estimates latent codes for unseen images and interpolates independently across the identity and observation subspaces.

5 DISCUSSION

  • Varying observation codes changes clothing color, sunglasses, pose, lighting, saturation, and contrast while keeping apparent identity fixed.The discussion also reports that SD-DCGAN samples tend to appear less photorealistic than SD-BEGAN samples.Source: §5 · p. 8
  • For shoes, changing identity while holding observation fixed yields distinct shoes in consistent poses, with identity codes appearing to capture broad shoe categories.Neither original BEGAN nor SD-BEGAN produces diverse shoe images.Source: §5 · p. 8
  • The authors identify disentangling more than one known commonality and applying the approach to identity-conditioned speech synthesis as future directions.Source: §5 · p. 8

A ESTIMATING LATENT CODES

  • For unseen images, the authors estimate latent vectors by minimizing squared distance between the image and the pretrained generator's output.The objective is min_z-hat ||G(z-hat) − x||_2^2, following Lipton & Tripathi (2017).Source: A ESTIMATING LATENT CODES · p. 12
  • SD-BEGAN latent estimates support interpolation across identity and observation subspaces, with identity fixed by row and observation fixed by column.Figure 6 shows this for two image pairs and their source images.Source: A ESTIMATING LATENT CODES · p. 12, Figure 6 · p. 12

B PAIRWISE DISCRIMINATION OF EMBEDDINGS AND ENCODINGS

This section explores pairwise discrimination with identity embeddings and encoder-decoder representations. The experiments are described as exploratory, with qualitatively promising results.

B PAIRWISE DISCRIMINATION OF EMBEDDINGS AND ENCODINGS

  • The AC-GAN baseline's one-hot identity embedding was combined with pairwise discrimination, using real same-identity pairs or a real image paired with a synthetic image of that identity.Other hyperparameters matched the SD-DCGAN experiment; Figure 7 shows the results.Source: B PAIRWISE DISCRIMINATION OF… · p. 12, Figure 7 · p. 12
  • An encoder-decoder approach was also combined with pairwise discrimination, with the discriminator receiving real same-identity pairs or real and generated images.Figure 8 presents the results.Source: B PAIRWISE DISCRIMINATION OF… · pp. 12–13, Figure 8 · p. 13
  • The authors characterize these experiments as exploratory and qualitatively promising, and note that pairwise discrimination with real-real or real-fake pairs had prior examples.They cite Pathak et al. (2016) and Isola et al. (2017).Source: B PAIRWISE DISCRIMINATION OF… · p. 13

C EXPLORATORY EXPERIMENT WITH DR-GANS

The paper adapts DR-GAN, which disentangles pose and illumination, to disentangle identity instead. In this exploratory setup, the generator quickly collapsed to one output per input regardless of observation code.

C EXPLORATORY EXPERIMENT WITH DR-GANS

  • DR-GAN uses an encoder-decoder generator conditioned on an input image, pose code, and noise, and its discriminator predicts image realism and pose.Tran et al. (2017) report disentangling pose and illumination from the remaining representation.Source: C EXPLORATORY EXPERIMENT WITH DR-GANS · p. 13
  • The modified DR-GAN replaced pose prediction with identity prediction and used the encoder's representation as identity input and noise as observation input.The setup matched the AC-DCGAN baseline except that an encoding network replaced the embedding matrix.Source: C EXPLORATORY EXPERIMENT WITH DR-GANS · p. 13
  • The generator quickly produced a single output for each input regardless of observation code, so the experiment was excluded from evaluation.Figure 9 shows generated samples from the setup.Source: Figure 9 · p. 13, C EXPLORATORY EXPERIMENT WITH DR-GANS · p. 14

D IMAGINING IDENTITIES WITH AC-GAN

The authors tested whether AC-GAN could imagine identities using random identity vectors whose entries sum to one. This strategy produced little identity variety compared with the standard one-hot approach and was excluded from evaluation.

D IMAGINING IDENTITIES WITH AC-GAN

E ARCHITECTURE DESCRIPTIONS

The architecture descriptions define the identity and per-identity image dimensions, explain how k is handled in generator implementations, and specify a stacked-channel discriminator variant.

E ARCHITECTURE DESCRIPTIONS

  • k is the number of images generated and observed per identity, while dI is the identity-latent dimensionality and ZO has dimensionality 100 − dI.For the SD-BEGAN discriminator autoencoder with k = 2 and dI = 50, the fc2 bottleneck has output dimensionality 150.Source: E ARCHITECTURE DESCRIPTIONS · p. 14
  • Generator implementations need not be modified to represent k images, because k can be collapsed into the batch size.The tables parameterize generators by k for clarity and symmetry with the discriminators.Source: E ARCHITECTURE DESCRIPTIONS · p. 14
  • For stacked-channel discriminators, the input channel count changes from 3 to 3k and k is set to 1 in the architecture tables.Table 2 describes generator input abstraction, while Tables 3–6 describe the SD-DCGAN and SD-BEGAN architectures.Source: E ARCHITECTURE DESCRIPTIONS · p. 14, Table 2 · p. 15, Table 3 · p. 15
  • The SD-BEGAN discriminator autoencoder decoder is equivalent to the generator architecture but does not share its weights.The architecture descriptions are provided for SD-DCGAN and SD-BEGAN generators and discriminators.Source: Table 3 · p. 15, Table 4 · p. 15, Table 6 · p. 16

F FACE SAMPLES

The face-sample figures show outputs from AC-DCGAN, SD-DCGAN, and SD-BEGAN variants; the captions also report one WGAN evaluation and identify mode collapse for one BEGAN variant.

F FACE SAMPLES

  • The sample matrices hold zI constant across each row and zO constant across each column, with both vectors drawn randomly.The figure set also includes generated samples from SD-DCGAN.Source: F FACE SAMPLES · p. 17, Figure 13 · p. 17
  • Figure 12 presents AC-DCGAN generated samples with four real photographs of the identity on the right.Source: Figure 12 · p. 17
  • The SD-DCGAN sample figures include a stacked-channel discriminator variant and a k = 4 variant.A separate pair of figures shows variants with dI = 25 and dI = 75.Source: Figure 14 · p. 17, Figure 15 · p. 18, Figure 16 · p. 18
  • FaceNet scored AUC .770 and accuracy 68.5% (at τv) on SD-DCGAN samples trained with the Wasserstein GAN loss.The model used RMS-prop with α = 5e−5 and was excluded from Table 1 for brevity.Source: Figure 18 · p. 18
  • The SD-BEGAN variant with k = 4 is captioned as demonstrating mode collapse.The figure set also includes generated samples from SD-BEGAN without that stated condition.Source: Figure 20 · p. 19, Figure 19 · p. 19

G SHOE SAMPLES

The shoe-sample section presents generated examples from SD-DCGAN and SD-BEGAN models trained on the shoes dataset.

G SHOE SAMPLES

Related papers

Loading 1705.07904v3…