PaperlayerView exact revision on arXiv

Source-linked AI summary

Coupled Generative Adversarial Networks

Ming-Yu Liu, Oncel Tuzel

arXiv:1606.07536v2cs.CV

TL;DR

The paper tackles joint distribution learning for multi-domain images without corresponding-image tuples. It introduces CoGAN, which couples GANs through weight sharing and learns from separate marginal samples. Across image-generation tasks, it learns corresponding images and also shows applications to image transformation and unsupervised domain adaptation.

  • Problem

    Existing approaches require tuples of corresponding images across domains, but building such datasets is often challenging.

  • Method

    CoGAN couples one GAN per image domain and shares high-level semantic weights while training adversarially on separate marginal samples.

  • Results

    CoGAN learned corresponding images without correspondence supervision across face and RGBD tasks and significantly outperformed state-of-the-art methods for unsupervised domain adaptation.

  • Takeaways & Limitations

    The framework enables joint multi-domain image learning, image transformation, and unsupervised domain adaptation without corresponding training pairs.

  • Takeaways & Limitations

    CoGAN learning relies on shared high-level representations across the domains; it would fail if such a representation does not exist.

Abstract

from arXiv · show

We propose coupled generative adversarial network (CoGAN) for learning a joint distribution of multi-domain images. In contrast to the existing approaches, which require tuples of corresponding images in different domains in the training set, CoGAN can learn a joint distribution without any tuple of corresponding images. It can learn a joint distribution with just samples drawn from the marginal distributions. This is achieved by enforcing a weight-sharing constraint that limits the network capacity and favors a joint distribution solution over a product of marginal distributions one. We apply CoGAN to several joint distribution learning tasks, including learning a joint distribution of color and depth images, and learning a joint distribution of face images with different attributes. For each task it successfully learns the joint distribution without any tuple of corresponding images. We also demonstrate its applications to domain adaptation and image transformation.

1 Introduction

The paper addresses joint distribution learning for multi-domain images when corresponding-image tuples are difficult to obtain. It introduces CoGAN and evaluates its use for joint image generation, domain adaptation, and image transformation.

  • Motivation: Learning joint distributions can support novel tuples of multi-domain images, including color-depth scenes and faces with different attributes.Applications include movie and game production, image transformation, and domain adaptation.
  • Motivation: Existing approaches require tuples of corresponding images, but constructing such datasets is often challenging and limits their applicability.
  • CoGAN: CoGAN extends GANs to learn joint multi-domain image distributions using separate samples from each marginal distribution, without corresponding training images.
  • CoGAN: Weight sharing between high-level semantic layers encourages the coupled GANs to decode shared semantics consistently while domain-specific layers generate individual images.
  • Evaluation: CoGAN is evaluated on joint image distribution learning and demonstrated for unsupervised domain adaptation and image transformation.

2 Generative Adversarial Networks

GANs learn image distributions through an adversarial game between a generator and discriminator. The generator synthesizes images from random vectors, while the discriminator distinguishes real from synthesized images, with alternating updates during training.

  • GAN framework: A GAN combines a generative model that synthesizes realistic images with a discriminative model that distinguishes real images from synthesized ones.
  • GAN framework: The generator maps a random vector z to an image g(z), while the discriminator estimates whether an input comes from the real distribution pX.
  • Optimization: Training alternates gradient updates for the discriminator and generator parameters using a learning rate and iteration index.The update steps use θf, θg, λ, and t.
  • GAN framework: With sufficient model capacity and training iterations, the generated distribution pG converges to the real image distribution pX.

3 Coupled Generative Adversarial Networks

CoGAN couples two domain-specific GANs through shared high-level representations, enabling joint-distribution learning and corresponding-pair synthesis without correspondence supervision. Generative sharing captures semantics, while discriminator sharing reduces parameters but is not essential.

  • Framework: CoGAN uses a pair of GANs, each synthesizing images in one domain, trained together to generate corresponding cross-domain images without correspondence supervision.The framework requires samples from the marginal distributions rather than paired samples from the joint distribution.
  • Generator coupling: The generators share their first layers, which decode high-level semantics, while their later layers remain separate to materialize those semantics differently across domains.The two generators need not have equal depth or identical output supports.
  • Discriminator coupling: The discriminators share their last layers because these encode high-level features, whereas their first layers extract domain-specific low-level features.This constraint reduces the total parameter count but is not essential for learning a joint distribution.
  • Learning principle: Weight sharing and adversarial training together favor a joint distribution over the product of marginal distributions by constraining how the generators use network capacity.The shared high-level representation encourages realistic image pairs that correspond across domains.
  • Results: In digit experiments, CoGAN generated corresponding digit-edge and digit-negative pairs from the same input noise despite training without corresponding images.Without weight sharing, the two GANs generated unrelated images in the two domains.
  • Results: Pair-generation performance increased with more shared generator layers, showed no correlation with shared discriminator layers, and remained possible without discriminator sharing.Discriminator sharing was nevertheless preferred because it reduces the number of parameters.
  • Limitation: CoGAN depends on shared high-level representations across the domains and fails when no such representation exists for the domains of interest.This assumption defines the method's scope.

4 Experiments

The experiments evaluate CoGAN on digits, faces, and color-depth images without corresponding cross-domain training pairs. Weight sharing produced corresponding samples, including face attributes and RGB-depth relationships.

  • Experimental setup: The experiments used separate, non-overlapping training subsets for the two domains and fixed ADAM-based training hyperparameters for visualization experiments.Training used a mini-batch size of 128 and 25,000 iterations.
  • Digits: CoGAN learned corresponding digit images for both edge-image and negative-image tasks without corresponding training images.The authors attribute this to sharing layers responsible for decoding high-level semantics; without sharing, the GANs generated unrelated images.
  • Faces: The face experiments used CelebFaces Attributes images, with attribute-present faces forming one domain and corresponding attribute-absent faces forming the other.The dataset included large pose variations and background clutter.
  • Faces: CoGAN generated face pairs resembling the same person with and without blond-hair, smiling, or eyeglasses attributes.Attribute changes and transitions between identities were consistent across both domains, despite imbalanced attribute frequencies.
  • Color and depth: CoGAN generated color-depth pairs resembling corresponding RGBD and depth images even though registered images were absent from the two training domains.The authors describe this as unsupervised recovery of appearance-depth correspondence.

5 Applications

The paper applies CoGAN to unsupervised domain adaptation and cross-domain image transformation. It reports improved MNIST-to-USPS classification and describes latent-space optimization for producing corresponding images in another domain.

  • Unsupervised Domain Adaptation: In unsupervised domain adaptation, CoGAN adapts a classifier between digit domains without using label information from the target domain.The method applies the adapted classifier to classify target-domain digits.
  • Unsupervised Domain Adaptation: CoGAN improved unsupervised domain adaptation accuracy from 0.64 to 0.90, a 72% error reduction rate.The comparison concerned adapting classifiers across MNIST and USPS without labeled examples in the new domain.
  • Unsupervised Domain Adaptation: Table 1 compares competing algorithms using classification accuracies for the unsupervised domain adaptation task.The caption identifies classification accuracy as the table’s reported measure.
  • Cross-Domain Image Transformation: Cross-domain transformation finds a latent vector whose first-domain generation minimizes image loss, then feeds that vector to the second generator.The reported Figure 6 transformations used Euclidean loss and L-BFGS optimization.

6 Related Work

The work extends generative image-distribution learning to joint distributions across domains. It relates to multimodal learning and cross-domain generation while removing the need for correspondence annotations during training.

  • Generative image modeling: The related-work discussion places CoGAN within broader neural generative modeling approaches, including VAEs, attention models, moment matching, stochastic back-propagation, and diffusion processes.The paper states that these approaches can learn image distributions from samples.
  • Generative image modeling: CoGAN extends GAN-based image distribution learning to joint distributions of images across domains.The paper distinguishes this focus from prior work aimed at improving single-domain image generation quality.
  • Multimodal learning: Prior multimodal learning and dictionary-learning approaches require correspondence annotations to generate corresponding samples across domains.The paper also distinguishes its focus from prior cross-domain image-generation work on style transformation.

7 Conclusion

The conclusion presents CoGAN as a framework for learning multi-domain joint image distributions from separate marginal samples. It also reports promising image-generation, image-transformation, and unsupervised-domain-adaptation results.

  • Conclusion: CoGAN learned multi-domain joint image distributions using samples drawn separately from the marginal distributions.The mechanism is a weight-sharing constraint on layers responsible for decoding abstract semantics.
  • Conclusion: The paper reports convincing face and RGBD image-generation results alongside promising image-transformation and unsupervised-domain-adaptation results.These are the application areas highlighted in the conclusion.

A.1 Rotation

CoGAN learned a joint distribution linking MNIST digits with their 90-degree rotations, despite training on disjoint domain subsets without corresponding images.

  • A.1 Rotation: The rotation task used disjoint MNIST subsets, with every image in the second domain constructed by rotating digits 90 degrees.The domains had no corresponding training images.
  • A.1 Rotation: CoGAN used fully connected generators and discriminators sharing only the last layer because the domains lacked spatial correspondence.This design addressed relationships between different spatial regions across the two domains.
  • A.1 Rotation: For identical noise inputs, GAN2 generated a digit that was the 90-degree rotated version of GAN1’s digit.The result indicates that CoGAN captured the global in-plane transformation.

A.2 Weight Sharing

The weight-sharing study found that sharing more generative layers improved pair-image generation, whereas discriminator sharing showed no performance correlation but reduced parameters.

  • A.2 Weight Sharing: The study varied weight-sharing layers in both generative and discriminative models while training configurations under the same hyperparameters.It evaluated pair generation over 10000 images and five trials per configuration.
  • A.2 Weight Sharing: The evaluation compared transformed GAN1 images with GAN2 images using average agreed-pixel ratios.The transformation matched the procedure used to generate the second-domain training images.
  • A.2 Weight Sharing: Pair-image generation performance was positively correlated with the number of weight-sharing layers in the generative models.More shared generative layers produced pairs more like true samples from the joint distribution.
  • A.2 Weight Sharing: Pair-image generation performance was uncorrelated with the number of discriminator weight-sharing layers.The authors nevertheless preferred discriminator sharing because it reduces the total number of parameters.

A.3 Comparison with the Conditional Generative Adversarial Nets

The paper compared CoGAN with a conditional GAN trained without corresponding image pairs. CoGAN achieved higher pair-generation scores, while the conditional GAN struggled to produce corresponding cross-domain images.

  • A.3 Comparison with the Conditional Generative Adversarial Nets: The conditional GAN differed from CoGAN by taking a binary domain variable controlling which domain its generator produced.Its generator and discriminator otherwise matched the CoGAN models used in the digit experiments.
  • A.3 Comparison with the Conditional Generative Adversarial Nets: Both models were trained without correspondence supervision using the same training data and hyperparameters.CoGAN was trained for 25000 iterations and used to render 10000 image pairs.
  • A.3 Comparison with the Conditional Generative Adversarial Nets: 0.909 for Task A and 0.778 for Task B were the conditional GAN scores, versus 0.952 and 0.967 achieved by CoGAN.These are average pixel agreement ratio scores reported over five trials with different initializations.
  • A.3 Comparison with the Conditional Generative Adversarial Nets: The conditional GAN had difficulty rendering corresponding images in two domains without paired training examples.The paper’s Figure 8 visualized its pair-generation results.
  • A.3 Comparison with the Conditional Generative Adversarial Nets: Conditional-GAN generation performance changed little after 5000 iterations.This observation concerns the conditional GAN’s generation performance.

C Training Datasets

The experiments used training sets spanning digit transformations, face attributes, and RGB-depth imagery, with figures documenting the source images and generated multimodal outputs.

  • C Training Datasets: The face experiments covered blond hair, smiling, and eyeglasses attributes, pairing each attribute domain with faces lacking that attribute.The CelebFaces Attributes dataset included pose variations and background clutter.
  • C Training Datasets: The paper reports dataset-size statistics for MNIST, face, RGBD, and NYU experiments in Tables 7–10.The tables separately list domain counts for the corresponding training datasets.
  • C Training Datasets: CoGAN’s models use FCONV generators and CONV discriminators, with batch normalization and PReLU processing after convolutional layers.The discriminators also use fully connected final layers.
  • C Training Datasets: RGB-depth experiments used object and indoor-scene datasets, with figures showing color images, depth images, and point clouds.The object figures include point clouds from different viewpoints; indoor-scene figures show RGB and depth outputs.
Loading 1606.07536v2…