PaperlayerView exact revision on arXiv

Source-linked AI summary

XGAN: Unsupervised Image-to-Image Translation for Many-to-Many Mappings

Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Mosseri, Forrester Cole, Kevin Murphy

arXiv:1711.05139v6cs.CV

TL;DR

Unpaired semantic style transfer must map corpus-level styles while preserving shared content, even when domains differ in appearance and content. XGAN learns bidirectional translations through a shared semantic embedding and semantic consistency; the authors report that it outperforms the current baseline on face-to-cartoon translation.

  • Problem

    Semantic style transfer seeks to map between unpaired image collections while preserving shared semantics, without paired examples or supervision on shared content.

    Source: §1 · p. 2

  • Method

    XGAN jointly learns both domain translations through a shared semantic embedding, using domain-adversarial training and feature-level semantic consistency.

    Source: §1 · p. 3, §3 · p. 6

  • Results

    XGAN’s face-to-cartoon experiments outperform the current baseline, and pretrained feature representations can be added as teacher knowledge.

    Source: §6 · p. 15

  • Takeaways & Limitations

    Teacher knowledge acts as a regularizer for embeddings and generated samples, particularly useful for natural-image datasets with abundant pretrained models.

    Source: §6 · p. 15

    Fine-grained attributes such as facial hair are not well captured, and failures remain especially on non-representative inputs.

    Source: §5 · p. 15

Abstract

from arXiv · show

Style transfer usually refers to the task of applying color and texture information from a specific style image to a given content image while preserving the structure of the latter. Here we tackle the more generic problem of semantic style transfer: given two unpaired collections of images, we aim to learn a mapping between the corpus-level style of each collection, while preserving semantic content shared across the two domains. We introduce XGAN ("Cross-GAN"), a dual adversarial autoencoder, which captures a shared representation of the common domain semantic content in an unsupervised way, while jointly learning the domain-to-domain image translations in both directions. We exploit ideas from the domain adaptation literature and define a semantic consistency loss which encourages the model to preserve semantics in the learned embedding space. We report promising qualitative results for the task of face-to-cartoon translation. The cartoon dataset, CartoonSet, we collected for this purpose is publicly available at google.github.io/cartoonset/ as a new benchmark for semantic style transfer.

1 Introduction

The paper frames semantic style transfer as translating between unpaired image collections while preserving shared semantic content, a many-to-many problem with no paired or shared-content supervision. XGAN addresses it with a shared embedding and feature-level semantic consistency for bidirectional translation.

1 Introduction

  • Image-to-image translation spans tasks such as style transfer, colorization, super-resolution, and segmentation, which require understanding information shared between domains.Labels or paired samples often provide this knowledge; grayscale-to-color pairs, for example, can be generated from colored inputs.Source: §1 · p. 1
  • The task maps images into another domain’s style without changing semantics, despite characteristic attributes appearing differently at the pixel level.For example, blonde hair may appear bright yellow in cartoons, while hair style and face shape remain meaningful shared semantics.Source: §1 · p. 2, Figure 1 · p. 2
  • The setting provides two unlabeled image collections with no known cross-domain pairings or supervision for shared semantic content.The collections also differ in both global appearance and content distributions.Source: §1 · p. 2
  • XGAN jointly learns translations in both directions through a shared semantic representation, using semantic consistency to preserve feature-level information rather than pixel-level content.Its semantic consistency loss acts as self-supervision and reduces the need for paired examples.Source: §1 · p. 3

2 Related work

Prior approaches either transfer style from an individual image or impose pixel-level constraints that limit substantial structural changes. XGAN instead draws on feature-level domain adaptation and explicitly constrains a shared representation, distinguishing it from related translation models.

Style Transfer

  • Traditional style transfer transfers texture from a specific image, while patch-based feature matching still produces image-specific results, lacks a meaningful shared representation, and retains pixel structure.These properties limit their suitability for corpus-level style learning and drastic transformations such as face-to-cartoon.Source: §2 · p. 3

Domain adaptation

  • Prior domain-adaptation work maps synthetic images to natural images for training, but considers only pixel-level transformations.XGAN instead learns a shared feature representation to capture semantics rather than pixel information.Source: §2 · p. 4

Unsupervised Image-to-Image translation

  • Cycle-consistency methods jointly learn reverse mappings, but their pixel-level constraint often fails to capture significant structural changes.XGAN’s feature-level semantic consistency allows more flexible transformations.Source: §2 · p. 4
  • UNIT uses a shared embedding bottleneck with pixel-level cycle-consistency, whereas XGAN explicitly adds semantic consistency.UNIT assumes shared high-level layers suffice as a constraint.Source: §2 · p. 4

Domain adaptation

  • DTN is closest in objective and application, but its fixed pretrained encoder is assumed to produce meaningful embeddings in both domains.The paper reports that this assumption is restrictive under large domain shifts.Source: §2 · p. 4

3 Proposed model: XGAN

XGAN jointly learns a domain-invariant embedding for two domains with shared semantics and translates between them using a dual autoencoder. Its weighted training objective combines reconstruction, domain alignment, feature-level semantic consistency, adversarial image generation, and an optional teacher loss.

3 Proposed model: XGAN

  • XGAN learns a shared domain-invariant embedding and uses a dual autoencoder to translate between domains with common semantic content.The two translations are formed by applying one domain’s encoder and the other domain’s decoder.Source: §3 · pp. 4–5
  • The weighted objective combines reconstruction, domain-adversarial alignment, semantic consistency, GAN-based realism, and an optional teacher loss.The hyperparameters ω control the contributions of the five objectives.Source: §3 · p. 5
  • A domain classifier distinguishes encoded images from D1 and D2 while the encoders learn to confuse it, encouraging embeddings to share a subspace.The domain-adversarial loss is especially useful early in training, when generated samples are poor and not representative of the input domains.Source: §3 · pp. 6–7
  • Semantic consistency provides feature-level feedback by encouraging each translated image to retain the input’s learned embedding.The loss is defined for both translation directions and serves as self-supervision without paired examples.Source: §3 · p. 6
  • The GAN loss trains each cross-domain generator against a discriminator so translated samples better match the target domain and improve sample quality.Realistic transformations also help the joint embedding because generated samples are fed back through the encoders for semantic consistency.Source: §3 · p. 6
  • The optional teacher loss distills a pretrained network’s feature knowledge to constrain embeddings to a task-relevant subspace region.It is asymmetric and should not be applied to both domains simultaneously, since the terms could push embeddings in different directions.Source: §3 · p. 7

3.1 Architecture and Training procedure

  • The mirrored convolutional architecture shares the two deepest encoder blocks and the two earliest decoder blocks across domains.Training alternates generator and discriminator updates, uses gradient reversal for domain-adversarial training, and applies Adam with an initial learning rate of 1e-4.Source: §3.1 · p. 7, Table 1 · p. 8

4 The CartoonSet Dataset

CartoonSet is a publicly released dataset introduced for face-to-cartoon research. Its layered, attribute-based construction produces a collection of realistic cartoons unrelated to the source dataset.

4 The CartoonSet Dataset

  • CartoonSet fills the stated lack of a publicly available dataset for frontal-face-to-cartoon research.The authors release it publicly to support research on the task.Source: §4 · p. 8
  • Each cartoon combines 16 components—12 facial and 4 color attributes—rendered in a fixed eight-layer order.The artist-defined artwork handles attribute interactions, such as beard designs that vary with face shape.Source: §4 · p. 8
  • Randomly sampled attributes are filtered to remove unusual colors and unrealistic combinations, yielding approximately 9,000 cartoons.The resulting dataset is described as realistic and unrelated to the source dataset.Source: §4 · p. 8
  • Figure 3 shows random CartoonSet samples.The samples provide a visual view of the dataset's cartoon faces.Source: Figure 3 · p. 9

5 Experiments

Experiments evaluate unpaired frontal-face-to-CartoonSet translation, comparing XGAN with DTN and CycleGAN and examining loss ablations. XGAN produces sensible CartoonSet-style cartoons with semantic similarity to inputs, though non-representative inputs can still fail.

Experiments

  • The evaluation uses 18,054 frontal VGG-Face images and CartoonSet in unpaired training, with 20% of each dataset held out for testing.Images are aligned, background-matted, and resized to 64 × 64; the domains also differ in appearance and content distributions.Source: §5 · p. 9

Comparison to the DTN baseline

  • XGAN produces sensible, crisp CartoonSet-style cartoons with semantic similarity to VGG-Face inputs, while DTN fails to capture the transformation.The qualitative evaluation uses manually selected hyperparameters because unbiased style-transfer metrics are unavailable; some outputs mismatch hair or skin color.Source: Figure 6 · p. 10, §5 · pp. 10–11

Experiments

  • XGAN samples cover both cross-domain mappings, but the reported application uses a discriminator only for face-to-cartoon translation because GAN training can be unstable.The model otherwise appears robust to hyperparameter choice.Source: Figure 7 · p. 11

Comparison to the DTN baseline

  • DTN's fixed joint encoder is considered too restrictive for this setting; finetuning improves sample quality but is hyperparameter-sensitive and prone to mode collapse.The authors suggest the fixed encoder may not adapt well to new scenarios.Source: §5 · p. 12

Comparison to CycleGAN

  • CycleGAN yields poor face-to-cartoon results, consistent with its pixel-level cycle constraint and U-net connections favoring pixel-structure similarity.The paper argues that this emphasis is undesirable for the task's significant domain shift.Source: Figure 8 · p. 12, §5 · p. 12

Ablation study

  • Adding semantic consistency and teacher losses improves sample consistency through complementary embedding constraints, but each addresses different failure patterns.Without the teacher loss, outliers can be more distorted; without semantic consistency, outputs show less variety and preserve rare attributes less well.Source: §5 · pp. 13–15
  • With only reconstruction and domain-adversarial losses, XGAN learns basic semantics such as skin tone, but cross-domain translations are low quality.Low domain-adversarial loss weight can leave embeddings distinguishable and translations unrealistic.Source: §5 · p. 13

Future work

  • The authors identify evaluation on other domain-transfer tasks, where quantitative evaluation is possible, as future work.They note that pixel-level transformations may help learn a meaningful cross-domain representation when combined with semantic consistency.Source: §5 · p. 15

6 Conclusions

XGAN addresses unsupervised domain translation by learning shared semantic representations and applying a semantic consistency loss across both translation directions. The authors report face-to-cartoon results outperforming the current baseline and describe pretrained feature representations as useful regularizers.

6 Conclusions

  • XGAN targets unsupervised translation between structurally different domains by learning a high-level joint semantic representation while discarding local pixel-level dependencies.The model also uses semantic consistency across both domain translations as self-supervision.Source: §6 · p. 15
  • The semantic consistency loss acts as self-supervision on both domain translations.It is described as encouraging semantically consistent translations in the learned representation.Source: §6 · p. 15
  • Face-to-cartoon experiments outperform the current baseline, and pretrained feature representations can regularize learned embeddings and generated samples.The authors note this may be particularly useful for natural-image datasets, where off-the-shelf pretrained models are abundant.Source: §6 · p. 15

7 Appendix

The appendix compares XGAN variants and examines finetuning, domain-adversarial training, and GAN-loss ablations. The full XGAN produces the strongest visual samples, while the experiments also identify instability and failure modes.

7.1 Finetuning the DTN encoder

  • Finetuning DTN’s FaceNet encoder improves visual samples but creates a risk of mode collapse because semantic consistency alone can map domains to one embedding point.The appendix also reports that this finetuned model is sensitive to hyperparameter choice and often collapses to one model.Source: §7.1 · p. 17, Figure 11 · p. 17, §7.2 · p. 17

7.2 Extensive qualitative evaluation

  • The full XGAN yields the best visual samples among the tested models.The reported samples preserve identity characteristics such as face shape, hairstyle, and skin tone, while hair-color mismatches remain a notable error.Source: §7.2 · p. 18, Figure 12 · p. 18
  • With only reconstruction and domain-adversarial losses active, semantics are generally preserved, but samples remain poor quality and can include gender misclassifications.The full XGAN adds components beyond this simpler domain-adaptation setting.Source: §7.2 · p. 18
  • Generated samples often mismatch hair color, and other errors include bald faces gaining light hair and cartoon eyeglasses disappearing in real-face translations.The authors attribute most mistakes to content-distribution shifts between the domains.Source: §7.2 · p. 18

7.3 Failure mode when training with Lrec and Ldann

  • When the domain-adversarial weight is too high in the reconstruction-plus-domain-adversarial setting, the classifier reaches perfect accuracy and cross-domain translation fails.Source: §7.3 · p. 19, Figure 13 · p. 19

7.4 GAN loss ablation experiment

  • With GAN loss inactive, generated samples are noisy and unrealistic; total-variation smoothing makes them more uniform but blurrier over time.The GAN term is used only for the source-to-target path to ease training, despite making training more complex.Source: §7.2 · p. 18, §7.4 · p. 19, Figure 14 · p. 19

Related papers

Loading 1711.05139v6…