Source-linked AI summary
StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, Jaegul Choo
TL;DR
Multi-dataset image translation is complicated by partially known labels across datasets. StarGAN uses one generator and discriminator with a mask vector to learn multiple domains, achieving higher visual quality and superiority over baseline models on facial attribute transfer and expression synthesis.
Problem
Multi-dataset image translation is complicated because each dataset provides only partially known domain-label information.
Method
StarGAN uses one generator and discriminator with domain labels and a mask vector to learn mappings across multiple domains and datasets.
Results
Qualitative and quantitative evaluations on facial attribute transfer and facial expression synthesis show superiority over baseline models.
Takeaways & Limitations
StarGAN provides a scalable multi-domain translation model that can utilize multiple datasets with different domain-label sets.
Takeaways & Limitations
The supplied evidence limits the evaluation note to using model outputs from unseen images during training.
Abstract
from arXiv · showhide
Recent studies have shown remarkable success in image-to-image translation for two domains. However, existing approaches have limited scalability and robustness in handling more than two domains, since different models should be built independently for every pair of image domains. To address this limitation, we propose StarGAN, a novel and scalable approach that can perform image-to-image translations for multiple domains using only a single model. Such a unified model architecture of StarGAN allows simultaneous training of multiple datasets with different domains within a single network. This leads to StarGAN's superior quality of translated images compared to existing models as well as the novel capability of flexibly translating an input image to any desired target domain. We empirically demonstrate the effectiveness of our approach on a facial attribute transfer and a facial expression synthesis tasks.
1. Introduction
The introduction frames multi-domain image translation as a challenging extension of two-domain translation because existing approaches require separate models for every domain pair. StarGAN addresses this with one scalable generator, masked domain labels for joint multi-dataset training, and evaluations on facial attribute transfer and expression synthesis.
- Motivation: Multi-domain image-to-image translation changes images according to attributes from multiple domains, enabled by labeled datasets such as CelebA and RaFD.CelebA contains 40 facial-attribute labels, while RaFD contains 8 facial-expression labels.
- Problem: Existing models are inefficient because learning mappings among k domains requires k(k−1) generators, and ineffective because they cannot fully exploit global features shared across domains.For four domains, this corresponds to twelve distinct generator networks.
- StarGAN: StarGAN learns mappings among multiple domains using a single generator that receives both an image and domain information, enabling flexible translation to desired target domains.The model is presented as a scalable approach that trains from images of all domains using a single generator and discriminator.
- Cross-dataset training: A mask vector added to the domain label enables joint training across datasets by ignoring unknown labels and focusing on labels supplied by each dataset.This allows facial-expression synthesis for CelebA images using features learned from RaFD.
- Evaluation: The work reports successful multi-domain translation across different datasets and qualitative and quantitative evaluations on facial attribute transfer and facial expression synthesis.The authors characterize this as the first successful multi-domain image translation across different datasets and report superiority over baseline models.
2. Related Work
Prior work established GANs for diverse vision tasks, conditional generation across classes and domains, and supervised or unpaired image-to-image translation. These approaches motivate StarGAN’s scalable framework for flexibly steering image translation across domains.
- Generative Adversarial Networks: GANs use a discriminator to distinguish real from fake samples and a generator to produce fake samples, supporting many computer vision applications.Applications include image generation, image translation, super-resolution imaging, and face image synthesis.
- Conditional GANs: Conditional GANs incorporate class information to generate samples conditioned on the class.Conditional image generation has also been applied to domain transfer, super-resolution imaging, and photo editing.
- Conditional GANs: Conditional image-generation methods have produced images relevant to text descriptions and supported domain transfer, super-resolution, and photo editing.These applications broaden conditional generation beyond class-conditioned sampling.
- Image-to-Image Translation: Image-to-image translation methods include supervised pix2pix, which combines adversarial and L1 losses and therefore requires paired data.Unpaired frameworks were proposed to reduce the need to obtain paired samples.
3. Star Generative Adversarial Networks
StarGAN trains a single generator and discriminator to translate images among multiple domains conditioned on target labels. It combines adversarial, domain-classification, and reconstruction objectives, and uses masked unified labels to train across datasets with partially known annotations.
- Single-Dataset Framework: A single generator G maps an input image x to G(x, c) conditioned on a randomly generated target-domain label c, while an auxiliary classifier lets one discriminator control multiple domains.The generator learns flexible translations by varying c, and the discriminator produces source and domain-label probability distributions.
- Training Objectives: StarGAN combines adversarial loss for realistic outputs with domain-classification loss that makes generated images belong to the requested target domain.The discriminator classifies real images to their original domains, while the generator minimizes fake-image classification loss for the target domain.
- Training Objectives: An L1 cycle-consistency reconstruction loss encourages translated images to preserve input content while changing domain-related attributes.The same generator first translates the image and then reconstructs it using the translated image and the original domain label.
- Multiple-Dataset Training: StarGAN simultaneously incorporates multiple datasets with different label types, enabling control of all labels at test time despite each dataset containing only partial annotations.CelebA provides attribute labels such as hair color and gender, whereas RaFD provides facial-expression labels.
- Multiple-Dataset Training: An n-dimensional one-hot mask vector m marks the dataset-specific labels that are known, while unknown labels receive zero values so the generator focuses on explicit conditions.The unified label concatenates each dataset’s label vector; in the CelebA and RaFD experiments, n is two.
4. Implementation
StarGAN stabilizes training with a Wasserstein GAN objective and gradient penalty, using λ_gp = 10 throughout. Its generator follows a CycleGAN-adapted convolutional and residual architecture, while a PatchGAN discriminator classifies local patches without normalization.
- Improved GAN Training: StarGAN replaces the original objective with a Wasserstein GAN objective with gradient penalty to stabilize training and improve image quality.The gradient-penalty formulation samples ˆx uniformly along straight lines between real and generated images.
- Improved GAN Training: λ_gp = 10 is used for all experiments.
- Network Architecture: The generator uses two stride-two convolutional layers, six residual blocks, and two stride-two transposed convolutional layers.Instance normalization is applied to the generator, while the discriminator uses no normalization.
- Network Architecture: The discriminator uses PatchGANs to classify whether local image patches are real or fake.
5. Experiments
Experiments show that StarGAN produces higher-quality and more identity-preserving translations than competing methods across facial attribute and expression tasks. Its unified architecture also improves scalability and supports joint learning across multiple datasets.
- Facial attribute transfer: StarGAN achieved higher visual quality than cross-domain models on CelebA and better preserved facial identity than IcGAN.The paper attributes these advantages to multi-task regularization and spatial activation-map representations.
- Facial attribute transfer: StarGAN won the majority of AMT votes in all single- and multi-attribute transfer cases.For gender changes, StarGAN received 39.1% versus 31.4% for DIAT; for the multi-attribute ‘G+A’ case, it received 49.8% versus 20.3% for IcGAN.
- Facial expression synthesis: StarGAN generated the most natural-looking facial expressions while maintaining input identity and facial features, whereas competing outputs were blurry or identity-inconsistent.The paper links this quality to implicit data augmentation from multi-task learning, using 4,000 images rather than 1,000 when training across RaFD domains.
- Facial expression synthesis: StarGAN achieved the lowest classification error on translated RaFD test images, indicating the most realistic facial expressions among the compared methods.The evaluation classified translated images using the referenced expression classifier and used model outputs from unseen training images.
- Scalability: StarGAN required seven times fewer parameters than DIAT and fourteen times fewer than CycleGAN to learn all translations.This scalability results from using one generator–discriminator pair regardless of the number of domains, unlike cross-domain models.
- Joint training across datasets: Jointly trained StarGAN-JNT produced higher-quality facial expressions on CelebA than StarGAN-SNG, while incorrect dataset mask vectors caused expression synthesis to fail.StarGAN-JNT leverages both CelebA and RaFD, and the wrong [1, 0] mask caused age manipulation instead of expression synthesis.
6. Conclusion
StarGAN is presented as a scalable multi-domain image-to-image translation model using a single generator and discriminator. It produces higher visual quality than existing methods, supported by multi-task learning and a simple mask vector for using datasets with different domain-label sets.
- Conclusion: StarGAN performs image-to-image translation among multiple domains with a single generator and discriminator.This architecture provides scalability for multi-domain translation.
- Conclusion: StarGAN generated images of higher visual quality than existing methods [16] [23] [33].The passage attributes this quality to the generalization capability of its multi-task learning setting.
- Conclusion: A simple mask vector enables StarGAN to utilize multiple datasets with different sets of domain labels.This supports training across datasets whose domain-label sets differ.
7. Appendix
The appendix details StarGAN’s joint CelebA–RaFD training setup, network components, and additional 256 × 256 translation results. A two-dimensional mask selects which dataset label the generator should use while ignoring the other.
- Joint CelebA–RaFD training: StarGAN jointly trains on CelebA binary attributes and RaFD categorical attributes using a two-dimensional one-hot mask indicating the valid dataset label.CelebA labels include Black, Blond, Brown, Male, and Young; RaFD labels include Angry, Fearful, Happy, Sad, and Disgusted.
- Joint CelebA–RaFD training: During joint training, the mask directs the generator to focus on the selected dataset label and ignore the other label.The discriminator distinguishes real from fake images and minimizes classification error only for the known label.
- Network architectures: The generator uses instance normalization except at its output layer, while the discriminator uses Leaky ReLU with a negative slope of 0.01.The appendix defines nd as the number of domains and nc as the domain-label dimension, with nc = nd + 2 for joint CelebA–RaFD training.
- Additional results: Figures 9–12 provide additional 256 × 256 images for CelebA attribute transfer and RaFD and CelebA emotional-expression synthesis.All images were generated by a single generator trained on both datasets for seven days on one NVIDIA Pascal M40 GPU.