StarGAN v2: Diverse Image Synthesis for Multiple Domains

Yunjey Choi; Youngjung Uh; Jaejun Yoo; Jung-Woo Ha

Source-linked AI summary

StarGAN v2: Diverse Image Synthesis for Multiple Domains

Yunjey Choi, Youngjung Uh, Jaejun Yoo, Jung-Woo Ha

arXiv:1912.01865v2Dec 4, 2019cs.CVcs.LG

TL;DR

Image-to-image translation methods must combine diverse outputs with scalability across multiple domains, but prior approaches typically address only one of these goals. StarGAN v2 introduces a unified style-code framework and reports improved visual quality and diversity across CelebA-HQ and AFHQ, while releasing AFHQ for evaluation.

Problem
Existing image-to-image translation methods have limited diversity or require multiple models to support multiple domains.
Source: §1 · pp. 1–2
Method
StarGAN v2 replaces fixed domain labels with domain-specific style codes generated from latent codes or extracted from reference images.
Source: §1 · p. 2, §2.1 · p. 2
Results
StarGAN v2 generates rich styles across multiple domains and remarkably outperforms previous leading methods, with FIDs of 13.7 on CelebA-HQ and 16.2 on AFHQ.
Source: §3.2 · p. 6, §6 · p. 8
Takeaways & Limitations
StarGAN v2 provides a single framework for diverse multi-domain image synthesis, and AFHQ supplies a dataset for evaluation under large inter- and intra-domain variation.
Source: §1 · p. 2, §6 · p. 8
Directly injecting latent codes into the generator fails to provide as much diversity as expected in multi-domain settings because latent codes do not separate domains.
Source: §3.1 · p. 4

Abstract

from arXiv · show

A good image-to-image translation model should learn a mapping between different visual domains while satisfying the following properties: 1) diversity of generated images and 2) scalability over multiple domains. Existing methods address either of the issues, having limited diversity or multiple models for all domains. We propose StarGAN v2, a single framework that tackles both and shows significantly improved results over the baselines. Experiments on CelebA-HQ and a new animal faces dataset (AFHQ) validate our superiority in terms of visual quality, diversity, and scalability. To better assess image-to-image translation models, we release AFHQ, high-quality animal faces with large inter- and intra-domain differences. The code, pretrained models, and dataset can be found at https://github.com/clovaai/stargan-v2.

1. Introduction

Image-to-image translation must support diverse styles within domains while scaling across many domains. StarGAN v2 addresses both challenges with domain-specific style codes and a unified framework.

Image-to-image translation maps images between visually distinct domains while accounting for each image’s unique style.Styles can include attributes such as makeup, beard, and hairstyle.Source: §1 · p. 1
Existing diversity-focused methods use domain-specific decoders and require K(K-1) generators for K domains.Their two-domain design limits scalability as the number of domains grows.Source: §1 · p. 2
Unified models such as StarGAN scale across domains but produce deterministic outputs because each domain is represented by a fixed label.Given a source image and domain, the fixed label leads to the same output per domain.Source: §1 · p. 2
StarGAN v2 replaces the domain label with domain-specific style codes to generate diverse images across multiple domains.A mapping network converts random Gaussian noise into style codes, while a style encoder extracts codes from reference images.Source: §1 · p. 2
Experiments evaluate component effects, multi-domain scalability, visual quality, and diversity, while introducing AFHQ for large inter- and intra-domain differences.The AFHQ animal-face dataset is released publicly for research.Source: §1 · p. 2

2. StarGAN v2

StarGAN v2 uses a single generator conditioned on domain-specific style codes from either random latent codes or reference images. Its multi-domain modules and training objectives promote diverse synthesis while preserving source characteristics.

The framework trains a single generator to produce diverse images in an arbitrary target domain by reflecting learned domain-specific style vectors.The framework contains a generator, mapping network, style encoder, and discriminator.Source: §2.1 · p. 2, Figure 2 · p. 3
The generator injects style codes through AdaIN, removing the need to provide an explicit domain label while supporting all domains.Style codes come from either the mapping network or style encoder.Source: §2.1 · p. 2
The mapping network transforms randomly sampled latent codes into domain-specific style codes using multiple output branches.Randomly sampling latent vectors and domains produces diverse styles across domains.Source: §2.1 · p. 2
The style encoder extracts a domain-specific style code from a reference image, enabling reference-guided synthesis.Different reference images provide different style codes.Source: §2.1 · p. 2
The discriminator uses multiple domain branches to classify real images from each domain versus generated images.Each branch performs binary real-versus-fake classification for its domain.Source: §2.1 · p. 2, Figure 2 · p. 3
Adversarial, style reconstruction, diversity, and cycle-consistency objectives train realistic, style-faithful, diverse outputs that preserve source characteristics.Cycle consistency encourages reconstruction of the input using its estimated style code.Source: §2.2 · p. 3

3. Experiments

Experiments evaluate StarGAN v2 on CelebA-HQ and AFHQ using visual-quality and diversity metrics, component analyses, and qualitative comparisons. The results show that domain-specific style codes and diversity regularization support high-quality, diverse synthesis across multiple domains.

Component analysis

The component study progresses from deterministic StarGAN configurations to multi-output variants, with configurations (A)–(C) producing one output and (D)–(F) producing multiple outputs.FID and LPIPS are reported for configurations designed to produce multiple outputs; LPIPS is omitted for earlier deterministic variants.Source: §3.1 · p. 4, Figure 3 · p. 4
Direct latent-code injection fails to learn as much meaningful domain-specific style diversity as the proposed style-code design.The analysis attributes this to latent codes lacking domain-separating capability and therefore modeling domain-shared rather than domain-specific styles.Source: §3.1 · p. 4
StarGAN v2 transforms latent codes into domain-specific style codes and combines style reconstruction with diversity regularization to generate diverse styles while preserving source characteristics.Reference-guided results include hairstyle, makeup, beard, and other distinctive styles without hurting source characteristics.Source: Figure 4 · p. 5, §3.1 · p. 6

Latent-guided synthesis

13.7 and 16.2 are StarGAN v2's FIDs on CelebA-HQ and AFHQ for latent-guided synthesis, respectively, more than two times better than the previous leading method.Its LPIPS is also highest on CelebA-HQ, indicating the most diverse results given a single input.Source: §3.2 · pp. 6–7

Reference-guided synthesis

23.8 and 19.8 are StarGAN v2's reference-guided FIDs on the two datasets, about 1.5× and 3.5× better than the previous leading method, respectively.StarGAN v2 also achieves the highest LPIPS, while baselines struggle with domain shifts and mode collapse, especially on AFHQ.Source: §3.2 · p. 7

4. Discussion

StarGAN v2 synthesizes diverse styles across multiple domains through domain-specific style codes and shared modules that promote flexibility and generalization.

Separate multi-head mapping and style-encoder outputs provide domain-specific style codes, letting the generator focus on style information.The domain-specific information is handled before reaching the generator.Source: §4 · p. 8
Learned transformations make StarGAN v2's style space more flexible than the fixed Gaussian style spaces assumed by baseline methods.Source: §4 · p. 8
Shared module components learn domain-invariant features from multiple domains, providing a regularization effect that encourages generalization to unseen samples.Source: §4 · p. 8
On FFHQ images, a model trained on CelebA-HQ captures reference styles and renders them correctly despite the datasets' distribution gap.Source: §4 · p. 8

5. Related work

Prior image-to-image translation methods improve diversity through stochastic or reference-guided mechanisms, but generally address only two domains, while StarGAN v2 demonstrates reference-guided synthesis across datasets.

GAN research treats generated-image diversity as an important objective alongside visual quality, using dedicated losses or architectural designs.Source: §5 · p. 8
Early image-to-image translation methods commonly learn deterministic mappings even when stochastic noise is provided as input.Source: §5 · p. 8
StarGAN v2 successfully extracts reference style codes and synthesizes faithful images on FFHQ using a model trained on CelebA-HQ.Source: Figure 7 · p. 8
Methods using marginal matching, latent regression, diversity regularization, or reference images improve output variation but generally consider only two domains.Source: §5 · p. 8

6. Conclusion

StarGAN v2 addresses diverse translation across multiple target domains and achieves rich styles across domains, alongside releasing AFHQ for evaluation under large variation.

StarGAN v2 targets both diverse images within a target domain and support for multiple target domains.Source: §6 · p. 8
Experimental results show rich styles across multiple domains and markedly better performance than previous leading methods.Source: §6 · p. 8
AFHQ is released as a dataset for evaluating image-to-image translation under large inter-domain and intra-domain variation.Source: §6 · p. 8
AFHQ provides examples of animal faces for the released evaluation dataset.Source: Figure 8 · p. 9

A. The AFHQ dataset

AFHQ is a high-quality animal-face dataset with three domains, substantial breed diversity, and a challenging multi-domain translation setting.

AFHQ contains 15,000 high-quality 512 × 512 images across cat, dog, and wildlife domains.Each domain contributes 5000 images.Source: A. The AFHQ dataset · p. 9
Each AFHQ domain includes diverse images from at least eight breeds, creating substantial within-domain variation.Source: A. The AFHQ dataset · p. 9
The dataset's three domains and breed diversity create a more challenging image-to-image translation problem.Source: A. The AFHQ dataset · p. 9
For each domain, 500 images are selected as a test set.Source: A. The AFHQ dataset · p. 9

B. Training details

Training uses a batch size of eight for 100K iterations, with dataset-specific loss weights and linear decay of the diversity-sensitive weight.

The model is trained with batch size eight for 100K iterations, requiring about three days on a single Tesla V100 GPU.The implementation uses PyTorch.Source: B. Training details · p. 9

C. Evaluation protocol

Evaluation measures both image quality and output diversity by generating ten target-domain outputs per test image from randomly sampled latent vectors.

FID measures discrepancy between translated images and training images in the target domain using ImageNet-pretrained Inception-V3 features.Ten latent vectors are sampled for each test image, and FID is calculated for every source-target domain pair.Source: C. Evaluation protocol · p. 9
LPIPS measures generated-image diversity using pairwise L1 distances between ImageNet-pretrained AlexNet features.For each input, ten outputs produce 45 pairwise distances whose average is reported across test images.Source: C. Evaluation protocol · p. 9

D. Additional results

Additional results show that StarGAN v2 transfers reference styles while preserving source-image structure across both CelebA-HQ and AFHQ.

Style interpolation results are provided through an accompanying video.The paper directs readers to an external video for interpolation results.Source: D. Additional results · p. 9
On CelebA-HQ, StarGAN v2 preserves source identity while transferring reference styles including hairstyle, makeup, beard, and age.The source and reference images are real, while the remaining images are generated.Source: D. Additional results · p. 9, Figure 9 · p. 10
On AFHQ, StarGAN v2 follows reference breed and hair while preserving the source pose.These results are presented as reference-guided image synthesis.Source: D. Additional results · p. 9, Figure 10 · p. 11

E. Network architecture

StarGAN v2 uses a generator, mapping network, style encoder, and multi-task discriminator to synthesize domain-specific styles across multiple domains.

Generator

The generator injects domain-specific style codes through AdaIN layers to translate inputs into styled outputs across domains.Style codes come from either the mapping network or the style encoder, eliminating the need to provide a separate domain label to the generator.Source: E. Network architecture · p. 12

Mapping network

The mapping network uses shared layers followed by K domain-specific branches to transform Gaussian latent codes into style codes.The latent code, hidden layer, and style code dimensions are 16, 512, and 64, respectively.Source: E. Network architecture · p. 12, Table 6 · p. 12

Style encoder

The style encoder uses shared residual blocks followed by K domain-specific fully connected layers to extract fine style features from reference images.The style code dimension D is set to 64, and global average pooling is omitted.Source: E. Network architecture · p. 12, Table 7 · p. 12

Discriminator

The discriminator is a multi-task discriminator with K real/fake classification branches, one for each domain.It uses six pre-activation residual blocks with leaky ReLU and sets the output dimension D to 1.Source: E. Network architecture · p. 12