PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Truly Unsupervised Image-to-Image Translation

Kyungjune Baek, Yunjey Choi, Youngjung Uh, Jaejun Yoo, Hyunjung Shim

arXiv:2006.06500v2cs.CVcs.LG

TL;DR

Image-to-image translation methods typically require paired images or domain labels, and domain labeling can burden data collection. TUNIT learns pseudo-domains and style codes to translate without either kind of supervision. The authors report that it outperforms baselines across supervision settings and remains robust to cluster-count choices.

  • Problem

    Image-to-image translation methods rely on paired images or domain labels, while obtaining domain labels can be a practical bottleneck.

    Source: §1 · p. 1

  • Method

    TUNIT jointly learns pseudo-domain labels and style representations through clustering and representation learning, then translates images without paired images or domain labels.

    Source: §1 · p. 1, §5 · p. 8

  • Results

    Across various datasets and supervision levels, TUNIT outperforms baselines and is robust to changes in the preset number of clusters.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    TUNIT serves as a strong baseline for developing semi-supervised image translation models.

    Source: §1 · p. 1

Abstract

from arXiv · show

Every recent image-to-image translation model inherently requires either image-level (i.e. input-output pairs) or set-level (i.e. domain labels) supervision. However, even set-level supervision can be a severe bottleneck for data collection in practice. In this paper, we tackle image-to-image translation in a fully unsupervised setting, i.e., neither paired images nor domain labels. To this end, we propose a truly unsupervised image-to-image translation model (TUNIT) that simultaneously learns to separate image domains and translates input images into the estimated domains. Experimental results show that our model achieves comparable or even better performance than the set-level supervised model trained with full labels, generalizes well on various datasets, and is robust against the choice of hyperparameters (e.g. the preset number of pseudo domains). Furthermore, TUNIT can be easily extended to semi-supervised learning with a few labeled data.

1. Introduction

Existing image-to-image translation methods rely on paired images or domain labels, which can burden data collection. TUNIT targets translation without either form of supervision, using clustering and representation learning to separate domains and translate images; experiments report gains across datasets and supervision settings.

1. Introduction

  • The authors identify paired data and domain information as the supervision used by existing translation methods, with domain labeling a practical bottleneck as data and domain counts grow.They note that annotating large datasets such as FFHQ is expensive and domain distinctions can be vague.Source: §1 · p. 1
  • TUNIT combines domain clustering, content and style encoding, and mapping among estimated domains to translate images from a mixed, unlabeled set.Its guiding network supports the unsupervised task and links clustering with translation.Source: §1 · pp. 1–2
  • Across four public datasets, TUNIT outperforms baselines across supervision levels in quantitative and qualitative comparisons.The reported comparisons include unsupervised and semi-supervised settings.Source: §1 · p. 2
  • The paper defines truly unsupervised translation as using neither paired images nor domain labels and presents TUNIT as an end-to-end model for this setting.The authors describe it as the first model to succeed at this task end-to-end, to their knowledge.Source: §1 · pp. 1–2
  • The guiding network's interaction between clustering and translation is reported to help the task.The model jointly trains the guiding network and GAN end-to-end, using GAN feedback to shape domain-separating attributes and style representations.Source: §1 · p. 2
  • TUNIT is insensitive to the preset number of clusters and outperforms the current state-of-the-art semi-supervised image translation model.The paper also presents TUNIT as a strong baseline for semi-supervised translation.Source: §1 · p. 2

2. Related work

Prior image-translation work commonly uses domain labels, while some approaches reduce label requirements through clustering or pretrained classifiers. Related work on unsupervised representation learning offers techniques the paper draws together, though the cited clustering approach has difficulty scaling to image translation.

2. Related work

  • Earlier image-translation models exploit cycle consistency or shared latent spaces to train with domain labels alone.S3GAN and Self-conditioned GAN incorporate clustering for generation with fewer or no labeled data, respectively.Source: §2 · p. 2
  • Prior translation methods use set-level domain supervision, while acquiring domain information can burden data collection and limit applicability to existing datasets and domains.The related-work discussion contrasts label-requiring approaches such as FUNIT with label-cost reduction and pretrained-classifier methods.Source: §2 · p. 2
  • MoCo [3] uses contrastive learning within dictionary learning, while IIC clusters images through mutual-information maximization and even assignment.The paper states that IIC does not scale well when combined with a difficult downstream task such as image-to-image translation.Source: §2 · p. 2

3. Truly Unsupervised Image-to-Image Translation (TUNIT)

TUNIT jointly separates unlabeled images into estimated domains and translates them, using a guiding network that supplies domain labels and style codes to the GAN. Its clustering and style-representation tasks benefit from shared learning, while generator losses promote realism, style matching, and preservation of source content.

3.1. Overview

  • The guiding network estimates a reference image’s domain and supplies its style code and domain estimate to guide the generator and multi-task discriminator.The generator produces the target domain while respecting the reference style and source content.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Learning to produce domain labels and encode style features

  • Mutual-information clustering assigns augmented image pairs consistently while distributing samples across pseudo domains; a separate branch encodes images as style codes for reference-guided translation.The style branch uses contrastive learning; the authors report that combining its loss with the clustering objective improves clustering quality over clustering alone.Source: §3.2 · pp. 3–4

3.3. Learning to translate images

  • The generator uses adversarial, style-contrastive, and reconstruction losses to produce realistic target-domain images, match reference style, and preserve source content.The style-contrastive loss discourages ignoring the reference code, while reconstruction supports preservation of domain-invariant features such as pose.Source: §3.3 · p. 4

3. Truly Unsupervised Image-to-Image Translation (TUNIT)

  • The framework jointly trains the guiding network, generator, and discriminator, with feedback from the generator to the guiding network included as essential to the method.The style-contrastive loss compares the translated image’s style code with the reference code and negative style codes.Source: §3.3 · p. 4

4. Experiments

The experiments evaluate TUNIT on labeled and unlabeled datasets, including comparisons with supervised baselines, robustness to cluster-count choices, and a few-label setting. Results report that TUNIT matches or outperforms supervised configurations, remains effective with excess clusters, and translates across domains in unlabeled data.

4. Experiments

  • The study investigates each TUNIT component and quantitatively and qualitatively compares performance on labeled AnimalFaces-10 and Food-10, using mFID and Density/Coverage to assess generated-image quality, fidelity, and diversity.Source: §4 · pp. 4–5

4.1. Comparative Evaluation on Labeled Datasets

  • Differentiable clustering improves label estimation over off-the-shelf clustering, but its sample diversity remains unsatisfactory; joint TUNIT training addresses this limitation.Figure 3 reports that final model G has better visual fidelity than sequentially trained F and visual quality equivalent to or better than supervised A and B.Source: Figure 3 · p. 5, §4.1 · p. 6
  • TUNIT (G) matches or outperforms supervised model B on mFID and Density/Coverage, and improves AnimalFaces-10 coverage by 0.073.Removing adversarial training feedback to the guiding network changes AnimalFaces-10 mFID from 47.7 to 63.0.Source: Table 1 · p. 5, §4.1 · p. 6

4.2. Analysis on Generalizability

  • TUNIT performs reasonably well when ˆK is at least 7 and remains effective even when ˆK is 100 times the true number of domains.The authors recommend using a sufficiently large ˆK or searching over values on a log scale.Source: §4.2 · p. 7
  • With few labels, TUNIT is comparable to the state-of-the-art semi-supervised method overall and outperforms it by 20% of mFID on small datasets.On large datasets, TUNIT either outperforms or is comparable to the competitors.Source: Table 3 · p. 7, §4.2 · p. 8

4.3. Validation on Unlabeled Dataset

  • On unlabeled datasets, TUNIT reflects reference styles, while FUNIT often fails to make meaningful domain translations or preserve fidelity.On AFHQ Wild, TUNIT’s style space roughly separates images into six species clusters, despite setting ˆK larger than needed.Source: Figure 5 · p. 7, §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

TUNIT performs image translation without paired images or domain labels by jointly learning pseudo labels and style codes. The conclusion reports robustness to the preset number of clusters and comparable results when extended to semi-supervised learning.

5. Conclusion

  • TUNIT translates images without external information by finding pseudo labels and style codes through synergies between clustering and representation learning.Source: §5 · p. 8
  • TUNIT performs unsupervised image translation robustly to hyperparameter changes, including the preset number of clusters.Source: §5 · p. 8
  • TUNIT extends to semi-supervised learning and provides results comparable to the state-of-the-art semi-supervised method.Source: §5 · p. 8

B. Training details

Training begins with the guiding network alone, then proceeds with the full framework. The passage specifies batch sizes, optimizers, and evaluation-weight averaging.

B. Training details

  • The guiding network trains for 65K iterations with generator and discriminator updates frozen, followed by 100K iterations training all networks.Source: B. Training details · p. 11
  • Batch sizes are 32 for 128×128 images and 16 for 256×256 images, and training takes about 36 hours on one Tesla V100 GPU.Source: B. Training details · p. 11
  • The guiding network uses Adam, while the generator and discriminator use RMSprop; evaluation uses exponential moving averages of guiding-network and generator parameters.Source: B. Training details · p. 11

C. Evaluation protocol

Evaluation uses class-wise FID, also called mFID, and D&C. The protocol describes generated-sample construction and feature extraction for these measures.

C. Evaluation protocol

  • Class-wise FID averages individual-class Fréchet distances between real and fake samples embedded by ImageNet-pretrained Inception-V3.Source: C. Evaluation protocol · p. 11
  • For fewer-label experiments, the reported mFID is the mean of the best five values over 100K iterations.Source: C. Evaluation protocol · p. 11
  • D&C is measured using Inception-V3 feature vectors from generated samples, with the number of fake images matched to the number of training images.Source: C. Evaluation protocol · p. 11

D. Difference between the sequential and joint training in Section 4.1

The comparison contrasts end-to-end joint training with sequential training that freezes the guiding network during GAN training. In the sequential setup, the guiding network receives no translation-loss feedback.

D. Difference between the sequential and joint training in Section 4.1

E. Architecture details

The appendix specifies the generator, guiding network, and discriminator architectures. The guiding network uses a shared VGG11-based component followed by separate class and style branches.

E. Architecture details

  • The guiding network uses VGG11 before linear layers and average pooling as its shared component, followed by Eclass and Estyle branches.Source: E. Architecture details · p. 11
  • The Eclass and Estyle branches output ˆK and 128 dimensions, respectively.Source: E. Architecture details · p. 12
  • Tables 4, 5, and 6 provide the generator, guiding-network, and discriminator architectures, respectively.The generator table defines ch as 64 and IN/AdaIN as instance/adaptive instance normalization; the guiding-network table is based on VGG11-BN; the discriminator table defines ˆK as the cluster count.Source: Table 4 · p. 12, Table 5 · p. 12, Table 6 · p. 12

F. Comparison with Swapping autoencoder

The appendix compares TUNIT with other translation approaches, reports user-study and visualization results, and distinguishes the roles of two contrastive losses. Its comparisons cover supervised and unsupervised settings as well as multiple datasets.

F. Comparison with Swapping autoencoder

  • TUNIT captures domain properties that SwAE sometimes misses, and it supports semi-supervised training and cluster-id translation without a reference image.SwAE always requires a reference, whereas TUNIT can translate domains specified by cluster ids via their average styles.Source: F. Comparison with Swapping autoencoder · pp. 12–13, Figure 8 · p. 12

G. Comparison with StarGANv2

  • On AnimalFaces-10, supervised StarGANv2 has mFID 33.67 versus TUNIT’s 47.70; on Food-10, TUNIT has 52.20 versus StarGANv2’s 65.03.The authors report StarGANv2 outperforming TUNIT on AnimalFaces-10, while TUNIT outperforms it on Food-10 despite using no labels.Source: G. Comparison with StarGANv2 · p. 13

H. Perceptual study on disentanglement

I. t-SNE visualization & cluster example im-

J. Additional Comparison with FUNIT: AFHQ, LSUN Car and FFHQ

  • Compared with FUNIT, TUNIT better reflects reference style in AFHQ Cat and LSUN Car, translates AFHQ Wild to the reference species, and preserves FFHQ identity while changing attributes.The FFHQ examples include adding or removing glasses, changing hair color, and changing hairstyle.Source: Figure 13 · p. 16, Figure 14 · p. 16, Figure 15 · p. 16

K.1–K.4

K.5–K.9

L. Difference between equation (2) and equation (4)

  • Removing equation (2) or (4) yields AnimalFaces-10 mFID 86.8 or 93.3, respectively, versus 47.7 originally; the losses serve distinct roles.Equation (2) improves the guiding network’s representation, while equation (4) guides the generator to reflect a reference image’s style.Source: L. Difference between equation (2) and… · p. 26

M. FID and LPIPS on unlabeled dataset

The section examines LPIPS alongside FID for evaluating reference-guided translation. The authors caution that this combination can flag some failures but cannot measure reference-image loyalty.

M. FID and LPIPS on unlabeled dataset

Related papers

Loading 2006.06500v2…