PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Cross-Domain Image Generation

Yaniv Taigman, Adam Polyak, Lior Wolf

arXiv:1611.02200v1cs.CV

TL;DR

The paper addresses unsupervised analogy synthesis between separated samples from domains S and T while preserving a function f's output. It introduces a Domain Transfer Network with compound adversarial, f-constancy, and target-domain identity losses, producing visually appealing, identity-preserving transfers and showing promise for unsupervised domain adaptation.

  • Problem

    The paper asks how to learn G: S → T from separated unlabeled samples so that f(x) remains approximately unchanged after transfer.

  • Method

    The Domain Transfer Network composes f with a learned function and uses adversarial, f-constancy, and target-domain identity regularization terms.

  • Results

    Generated face emoji are visually appealing, capture more facial characteristics than manually created emoji, and achieve a median cross-domain identification rank of 16 among 105 distractors.

  • Takeaways & Limitations

    The method supports unsupervised domain adaptation and shows a clear advantage over Ganin et al. (2016) in the reported SVHN-to-MNIST experiment.

  • Takeaways & Limitations

    Unsupervised domain adaptation is demonstrated in only a single experiment, and mapping a set X ⊂ S to a single output in T is left for future work.

Abstract

from arXiv · show

We study the problem of transferring a sample in one domain to an analog sample in another domain. Given two related domains, S and T, we would like to learn a generative function G that maps an input sample from S to the domain T, such that the output of a given function f, which accepts inputs in either domains, would remain unchanged. Other than the function f, the training data is unsupervised and consist of a set of samples from each domain. The Domain Transfer Network (DTN) we present employs a compound loss function that includes a multiclass GAN loss, an f-constancy component, and a regularizing component that encourages G to map samples from T to themselves. We apply our method to visual domains including digits and face images and demonstrate its ability to generate convincing novel images of previously unseen entities, while preserving their identity.

1 INTRODUCTION

The paper formulates unsupervised analogy synthesis between separated domains and proposes a deep network that preserves a function’s representation while generating target-domain samples. It applies this approach to visual transfer, including face-to-emoji generation, where outputs capture facial characteristics.

  • The task is to learn G:S→T from unlabeled samples so that f(x) remains approximately unchanged after transfer.
  • The method combines adversarial target-domain matching, f-constancy, and an identity regularizer on target-domain inputs.The network composes f with a learned function g.
  • The visual experiments use representations from neural networks trained for classification or identity capture as the function f.
  • The main application generates face emoji from facial images using separated, unlabeled source and target samples.
  • Generated face emoji are visually appealing and capture more facial characteristics than emoji created by well-trained human annotators using conventional tools.

2 RELATED WORK

The paper positions unsupervised domain transfer as a distinct problem related to GANs, domain adaptation, inversion, and style transfer. Its key distinction is transferring across target distributions without paired samples while preserving the input’s representation.

  • The authors describe their domain-transfer formulation as novel, naturally occurring, widely applicable, and related to cognitive reasoning.
  • GANs generate target-distribution samples through adversarial training between a generator and discriminator.
  • The paper notes that f-constancy constraints help avoid the uneven mode concentration observed in some GAN-generated samples.
  • Unlike style transfer, the method models a target distribution rather than replicating the style of one or several input images.
  • For photo-to-emoji transfer, style transfer can produce appealing images that do not belong to the emoji domain and compromise target-domain properties.
  • Unlike paired sketch-generation systems, this method operates without matching samples and across distant domains.
  • Domain adaptation and domain transfer can theoretically be reduced to one another, but the paper describes these routes as indirect.Inverting f in the target domain may be hard to learn, making one direction hypothetical.

3 A BASELINE PROBLEM FORMULATION

The baseline formulation learns a mapping that makes generated source samples indistinguishable from target samples while preserving their f-representations. Its adversarial and constancy risks are optimized jointly, but the baseline does not produce desirable results.

  • The objective learns G:S→T by minimizing a combined risk consisting of adversarial and f-constancy terms.
  • The adversarial term requires generated source samples to be indistinguishable from target-domain samples for every discriminator.
  • The f-constancy term requires f to remain invariant under G.The experiments report the simplest MSE form after finding mostly unchanged performance across several distance and metric-learning losses.
  • The baseline optimizes the generator and discriminator adversarially using deep neural networks and training-set summations.
  • The function used in this setup is more accurate on source-domain samples than on target-domain samples, an asymmetry shared across the experiments.
  • Figure 1 distinguishes losses with dashed lines from inputs and outputs with solid lines, and identifies G as the post-training transfer model.

4 THE DOMAIN TRANSFER NETWORK

The Domain Transfer Network composes a learned function with a baseline representation and trains it with multiple losses to transfer samples between domains while preserving task-relevant information. It also trains on target-domain samples and applies the approach to visual transfers including SVHN-to-MNIST and face photos-to-emoji.

  • Training includes generated samples for x ∈ T and requires G to act as the identity on target-domain samples.This target-domain regularization is introduced through LT ID.
  • G is structured as g ◦ f, focusing learning on aspects relevant to the f-constancy objective and adapting f across source and target domains.The composed function is trained on samples from both S and T.
  • The generator loss combines multiclass GAN, f-constancy, target-identity, and total-variation terms weighted by α, β, and γ.The discriminator loss is optimized over D, while the generator loss is optimized over g.
  • An anisotropic total-variation loss slightly smooths the generated image z = [z_ij] = G(x).The implementation uses B = 1; MSE is used for d and d2, and Adam is used for optimization.
  • Figure 2 presents transfers from SVHN to MNIST and from Facescrub face photos to emoji, with inputs in odd columns and outputs in even columns.

5 EXPERIMENTS

DTN is evaluated for transferring SVHN digits to MNIST and face photographs to emoji using unlabeled source and target samples. Across these settings, it produces visually convincing transfers, improves over baselines, preserves identity-related information, and remains sensitive to target-domain class coverage.

  • Experimental settings: DTN transfers substantially different domains, including SVHN digits to MNIST and unlabeled face photographs to emoji.The experiments cover both digit and face-image applications with separated source and target domains.
  • Digits: SVHN to MNIST: Visually appealing digit transfers are obtained despite inaccurate classifiers on both domains.The MNIST classifier is used to evaluate transferred SVHN test samples and method components.
  • Digits: SVHN to MNIST: DTN clearly outperforms the direct-sample baseline; f-constancy and GAN terms are important, while at least one regularizing or constancy term is required.Including f inside G has a dramatic effect on results.
  • Digits: SVHN to MNIST: DTN shows a clear advantage over Ganin et al. for SVHN-to-MNIST domain adaptation on both training samples and unseen SVHN test samples.The adaptation uses transferred samples with a simple nearest-neighbor classifier.
  • Unseen digits: Omitting a digit class from the source is less detrimental than omitting it from the target, while omitting it from both domains performs better than omitting it only from the target.When target-domain examples of “3” are absent, source “3” samples may map to target images of other classes.
  • Faces: photos to emoji: Generated emoji are more informative than manually created emoji but less restrictive, and achieve median rank 16 among 105 distractors in cross-domain identification.Evaluation uses a VGG face descriptor to avoid reusing the DTN training representation for evaluation.
  • Style transfer comparison: Neural style transfer produces visually appealing outputs that do not belong to the emoji target space, whereas DTN can correct missing facial features such as hair.The comparison is shown in Figure 5.

6 DISCUSSION AND LIMITATIONS

The method is asymmetric across domains and depends on the effectiveness of f, which is typically trained on one domain. Reversing the transfer direction produces less appealing results, while unsupervised domain adaptation is demonstrated in only one experiment.

  • The method treats source and target domains differently because f is unlikely to be equally effective in both.The paper notes that f would usually be trained on samples from one domain, although downstream application of D implicitly adapts it.
  • Reversing the roles of S and T yields less appealing results because of f’s asymmetry and the lower information content of the new source domains.
  • Unsupervised domain adaptation is demonstrated in a single experiment, despite promising state-of-the-art results with a simple nearest neighbor classifier.

A FACESCRUB DATASET GENERATIONS

The paper presents the complete Facescrub identity set together with corresponding generated emoji. Figure 6 displays all 80 identities, pairing each input image with its generated result.

  • The complete Facescrub set includes generated emoji corresponding to all shown identities.
  • Figure 6 shows all 80 Facescrub identities, with even columns presenting results for the images in the odd column to their left.

B THE EFFECT OF SUPER-RESOLUTION

The 64 × 64 outputs are upscaled to print quality using the method of Dong et al. (2015), and Figure 7 shows the effect of this postprocessing.

  • The method of Dong et al. (2015) upscales the 64 × 64 output to print quality.Figure 7 presents the effect of applying this postprocessing step.

C THE BASIS ELEMENTS OF THE FACE REPRESENTATION

The paper generates face emoji for the standard basis of the face representation. This basis is treated as the vector space R256.

  • The generator g produces face emoji for the standard basis of the face representation, modeled as the vector space R256.

D DOMAIN TRANSFER IN THE REVERSE DIRECTION

The paper presents domain-transfer results using DTNs in the reverse direction of the main experiments. Figure 9 covers transfers between MNIST and SVHN and between emoji and face-photo domains.

  • Reverse-direction transfer: DTNs are evaluated for domain transfer in the reverse direction of the experiments reported earlier.These results are presented for completion in Fig. 9.
  • Transfer examples: Figure 9 includes transfer from MNIST to SVHN and from emoji to face photos.The figure identifies these as separate visual-domain transfer examples.
  • Figure layout: In Figure 9, inputs appear in odd columns and outputs appear in even columns.This column convention applies to both transfer examples shown.
Loading 1611.02200v1…