PaperlayerView exact revision on arXiv

Source-linked AI summary

Triangle Generative Adversarial Networks

Zhe Gan, Liqun Chen, Weiyao Wang, Yunchen Pu, Yizhe Zhang, Hao Liu, Chunyuan Li, Lawrence Carin

arXiv:1709.06548v2cs.LGstat.ML

TL;DR

Joint-distribution matching across domains is challenging when paired data are difficult to acquire and unsupervised mappings are nonidentifiable. Δ-GAN uses a few paired samples with bidirectional generators and adversarial discriminators; the paper applies it across several domain-pair tasks and reports lower edges2shoes translation MSE than DiscoGAN with paired supervision.

  • Problem

    Fully paired data can be difficult to acquire, while unsupervised domain mappings may be nonidentifiable, leaving the desired joint distribution under-specified.

    Source: §1 · p. 1

  • Method

    Δ-GAN uses two generators for bidirectional conditional mappings and two discriminators that distinguish real pairs from two kinds of fake pairs, trained together adversarially.

    Source: §1 · p. 1, §2.2 · p. 3, §2.4 · p. 4

  • Results

    The paper applies Δ-GAN to semi-supervised classification, image translation, and attribute-based editing; on edges2shoes, MSE is 140.1 for DiscoGAN and 125.3 for Δ-GAN with 10% paired data.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    The paper concludes that Δ-GAN may be employed for a wide range of applications involving semi-supervised joint-distribution matching.

    Source: §5 · p. 8

    The semi-supervised analysis assumes the empirical joint distribution from a few paired samples roughly characterizes the full data joint distribution.

    Source: §2.4 · p. 4

Abstract

from arXiv · show

A Triangle Generative Adversarial Network ($\Delta$-GAN) is developed for semi-supervised cross-domain joint distribution matching, where the training data consists of samples from each domain, and supervision of domain correspondence is provided by only a few paired samples. $\Delta$-GAN consists of four neural networks, two generators and two discriminators. The generators are designed to learn the two-way conditional distributions between the two domains, while the discriminators implicitly define a ternary discriminative function, which is trained to distinguish real data pairs and two kinds of fake data pairs. The generators and discriminators are trained together using adversarial learning. Under mild assumptions, in theory the joint distributions characterized by the two generators concentrate to the data distribution. In experiments, three different kinds of domain pairs are considered, image-label, image-image and image-attribute pairs. Experiments on semi-supervised image classification, image-to-image translation and attribute-based image generation demonstrate the superiority of the proposed approach.

1 Introduction

The introduction motivates joint-distribution matching across domains, where fully paired supervision can be difficult to acquire and unsupervised mappings may be nonidentifiable. It presents Δ-GAN as a semi-supervised framework using a few paired samples, bidirectional generators, and two discriminators, and describes applications across several domain pairs.

1 Introduction

  • Joint-distribution matching supports cross-domain tasks such as image-text captioning and generation [4] and image-to-image translation [5].The introduction notes that the original GAN formulation learns a data distribution in one domain.Source: §1 · p. 1
  • Fully supervised paired data can be difficult to acquire, while unsupervised mappings admit infinitely many possible functions, motivating a few paired examples to guide the desired joint distribution.The introduction names DiscoGAN [7], CycleGAN [8], DualGAN [9], ALI [10], and BiGAN [11] as unsupervised joint-matching approaches.Source: §1 · p. 1
  • Δ-GAN uses two generators for bidirectional domain mappings and two discriminators that distinguish real pairs from two kinds of fake pairs.The discriminators implicitly define a ternary discriminative function, and both generators and discriminators are trained adversarially.Source: §1 · pp. 1–2
  • Compared with Triple GAN [12], Δ-GAN uses two discriminators to define a ternary rather than binary discriminator and combines conditional GAN with ALI rather than two conditional GANs.At an optimal discriminator, Δ-GAN's objective is a symmetric three-distribution JSD; the passage contrasts this with Triple GAN's additional KL term and its density-computation requirement.Source: §1 · p. 2
  • The introduction presents image-label, image-image, and image-attribute pairs for semi-supervised classification, image translation, and attribute-based editing, respectively.It also presents attribute-conditional image generation on COCO [14] as a scalability demonstration.Source: §1 · p. 2

2 Model

Δ-GAN matches real and generated joint distributions using two conditional generators and two cooperating discriminators. Its theory characterizes when the generators recover the data conditionals, while its semi-supervised use relies on a representative set of paired samples.

2.1 Generative Adversarial Networks (GANs)

  • A standard GAN trains a generator to map latent samples to data while a discriminator distinguishes real from generated samples.The generator seeks samples close to real data to fool the discriminator.Source: §2.1 · p. 2

2.2 Triangle Generative Adversarial Networks (∆-GANs)

  • Two generators model opposite conditional distributions, while two cooperating discriminators distinguish real pairs from two kinds of generated pairs.Together, the discriminators implicitly define a ternary discriminative function.Source: §2.2 · p. 3

2.3 Theoretical analysis

  • At equilibrium, the real and generated joint distributions are equal, yielding the true conditional distributions in theory.The semi-supervised setting assumes the few paired samples approximately characterize the full joint distribution.Source: §2.3 · pp. 3–4, §2.4 · p. 4

2.4 Semi-supervised learning

  • The semi-supervised objective combines conditional GAN supervision with BiGAN matching of the two generated joint distributions.Its theoretical analysis assumes full supervision; with only a few pairs, the method assumes their empirical joint distribution approximates the true one.Source: §2.4 · p. 4

2.5 Relation to Triple GAN

  • Unlike Triple GAN’s asymmetric objective, Δ-GAN directly optimizes a symmetric Jensen-Shannon divergence and requires sampling, not an explicit conditional density.Triple GAN’s extra KL term is described as assigning low cost to fake-looking samples.Source: §2.5 · pp. 4–5

2.6 Applications

  • The framework is presented for joint-distribution matching across image-image, image-label, and image-attribute pairs, including translation and attribute-based generation.Image-label and image-attribute models use latent style variables for image generation and a classifier for the reverse conditional.Source: §2.6 · p. 5

3 Related work

Prior work includes supervised conditional GANs and unsupervised methods for cross-domain matching. This work targets semi-supervised joint matching, distinguishes itself from Triple GAN, and describes its framework as complementary to GAN training improvements.

Related approaches

  • Conditional GANs use paired data, while unsupervised approaches include BiGAN/ALI and bidirectional mapping methods such as DiscoGAN, CycleGAN, and DualGAN.Other approaches add reconstruction losses or weight-sharing constraints.Source: §3 · p. 5

Semi-supervised joint matching

  • Among the cited work, Triple GAN also targets semi-supervised joint matching but focuses on image classification, whereas Δ-GAN is presented for a wider range of applications.The paper points to its comparison with Triple GAN for the specific differences.Source: §3 · p. 5

Related approaches

  • GAN stabilization methods are described as orthogonal to Δ-GAN, while supervised joint matching and semi-supervised variational autoencoders are also noted.Examples of stabilization methods include feature matching, Wasserstein GAN, energy-based GAN, and unrolled GAN.Source: §3 · p. 5

4 Experiments

The experiments test Δ-GAN on toy joint matching and image-label, image-image, and image-attribute tasks, including semi-supervised classification, translation, and attribute generation.

4.1 Toy data experiment

  • Both joint distributions learned by Δ-GAN match the real toy-data joint, whereas Triple GAN-s only guarantees their average matches it.The learned distributions are shown alongside the real joint distribution in Figure 2.Source: Figure 2 · p. 6, §4.1 · p. 6

4.2 Semi-supervised classification

  • With 4,000 labels, Δ-GAN achieves the best CIFAR10 performance among competing methods, while generated samples retain row-wise labels and vary style with noise.The reported noise patterns include background and color.Source: Table 1 · p. 6, §4.2 · p. 6, Figure 3 · p. 7

4.3 Image-to-image translation

  • On MNIST-to-MNIST-transpose, Δ-GAN matches the domains well while DiscoGAN fails, and Δ-GAN significantly outperforms Triple GAN and DiscoGAN on classifier accuracy.The classifier used for evaluation achieved 99.4% accuracy on its test set.Source: Table 2 · p. 6, §4.3 · pp. 6–7, Figure 4 · p. 7
  • MSE was 140.1 for DiscoGAN versus 125.3, 113.0, and 66.4 for Δ-GAN with 10%, 20%, and 100% paired data, respectively.With 10% paired data, Δ-GAN generally generates more accurate shoe edge details than DiscoGAN.Source: §4.3 · p. 6, Figure 4 · p. 7

4.4 Attribute-conditional image generation

  • CelebA examples feed predicted attributes into face generation, and changing an input attribute, such as adding sunglasses, edits the generated image.Figure 5 lays out input faces, predicted attributes, and generated faces in successive rows.Source: Figure 5 · p. 7, §4.4 · pp. 7–8
  • On CelebA and COCO, Δ-GAN beats Triple GAN on P@10 and nDCG@10; on COCO, 50% labeled data beats Triple GAN using the full dataset.Table 3 reports attribute-prediction results for both datasets.Source: Table 3 · p. 7, §4.4 · p. 8
  • On COCO, generated images reasonably reflect predicted attributes and vary across noise draws with attributes fixed; training uses 1,000 caption-derived attributes.Figure 7 presents image-to-attribute-to-image results.Source: Figure 7 · p. 8, §4.4 · p. 8

5 Conclusion

The paper presents Δ-GAN for semi-supervised joint distribution matching, learning bidirectional mappings between two domains from a few paired samples. It demonstrates applications across multiple domains and identifies image-caption modeling as a possible future direction.

5 Conclusion

  • Δ-GAN is a GAN framework for semi-supervised joint distribution matching between two domains.It learns bidirectional mappings from a few paired samples.Source: §5 · p. 8
  • The approach learns bidirectional mappings between domains using a few paired samples.Source: §5 · p. 8
  • The paper demonstrates that Δ-GAN may be used for a wide range of applications.Source: §5 · p. 8
  • A proposed future direction is combining Δ-GAN with sequence GAN or textGAN to model image-caption joint distributions.Source: §5 · p. 8

A Detailed theoretical analysis

The analysis characterizes the optimal discriminators and the generator objective. It shows that the global minimum and equilibrium occur when the data joint distribution matches both generator distributions.

A Detailed theoretical analysis

C Additional experimental results

The appendix provides additional experimental results for face-to-attribute-to-face, image editing, image-to-attribute-to-image, and attribute-conditional COCO image generation.

C Additional experimental results

  • Additional results are shown for the face-to-attribute-to-face experiment.Source: Figure 8 · p. 12
  • Additional results are shown for the image editing experiment.Source: Figure 9 · p. 12
  • Additional results are shown for the image-to-attribute-to-image experiment.Source: Figure 10 · p. 13
  • Attribute-conditional image generation results are shown on the COCO dataset.Input attributes are omitted for brevity.Source: Figure 11 · p. 13

D Evaluation metrics for multi-label classification

This section describes Precision@k and normalized Discounted Cumulative Gain at rank k as evaluation metrics for multi-label classification.

D Evaluation metrics for multi-label classification

E Detailed network architectures

The section specifies architecture references for CIFAR10 and edges2shoes and provides architecture tables for other datasets. It also includes attribute-prediction results on CelebA and COCO.

E Detailed network architectures

  • CIFAR10 uses the Triple GAN architecture, edges2shoes uses the pix2pix architecture, and detailed architectures are provided for other datasets.Source: E Detailed network architectures · p. 13
  • Attribute prediction on CelebA and COCO is reported using P@5 and nDCG@5, and separately using P@3 and nDCG@3.Source: Table 4 · p. 14, Table 5 · p. 14
  • Architecture tables cover ∆-GAN models for MNIST, CelebA, and COCO; the COCO table defines Dim as the number of attributes.The captions also define BN as batch normalization and, for CelebA and COCO, lReLU as Leaky ReLU.Source: Table 6 · p. 14, Table 7 · p. 14, Table 8 · p. 14
Loading 1709.06548v2…