Source-linked AI summary
Unsupervised Image-to-Image Translation Networks
Ming-Yu Liu, Thomas Breuel, Jan Kautz
TL;DR
Unsupervised image translation must infer a joint distribution from separate marginal distributions, an ill-posed problem because infinitely many joint distributions can match the marginals. The paper assumes corresponding images share a latent representation and builds UNIT from VAE-GAN components with weight sharing and reconstruction objectives. UNIT achieves high-quality translation across diverse tasks and state-of-the-art benchmark domain-adaptation performance, while remaining unimodal and potentially unstable to train.
Problem
Inferring a joint distribution from separate domain marginals is ill-posed because infinitely many joint distributions can produce the same marginals.
Method
UNIT uses a shared-latent-space assumption with VAE-GAN models, weight sharing, adversarial training, and variational reconstruction across two image domains.
Results
UNIT achieves high-quality image translation across various unsupervised tasks and state-of-the-art accuracy on benchmark domain-adaptation datasets.
Takeaways & Limitations
The shared-latent-space constraint provides a basis for unsupervised cross-domain translation and implies the cycle-consistency constraint.
Takeaways & Limitations
The framework is unimodal under its Gaussian latent-space assumption, and training can be unstable because optimization searches for a saddle point.
Abstract
from arXiv · showhide
Unsupervised image-to-image translation aims at learning a joint distribution of images in different domains by using images from the marginal distributions in individual domains. Since there exists an infinite set of joint distributions that can arrive the given marginal distributions, one could infer nothing about the joint distribution from the marginal distributions without additional assumptions. To address the problem, we make a shared-latent space assumption and propose an unsupervised image-to-image translation framework based on Coupled GANs. We compare the proposed framework with competing approaches and present high quality image translation results on various challenging unsupervised image translation tasks, including street scene image translation, animal image translation, and face image translation. We also apply the proposed framework to domain adaptation and achieve state-of-the-art performance on benchmark datasets. Code and additional results are available in https://github.com/mingyuliutw/unit .
1 Introduction
Unsupervised image translation seeks a joint distribution from separate domain marginals, but this is ill-posed without additional assumptions. UNIT addresses the problem with a shared-latent-space VAE-GAN framework and reports strong translation and domain-adaptation results.
- Motivation: Unsupervised translation must infer a joint distribution from marginal image distributions, although infinitely many joint distributions can share those marginals.Additional structural assumptions are therefore required.
- Approach: UNIT assumes corresponding images from two domains map to the same latent representation in a shared-latent space.This assumption supplies the structure used to learn cross-domain correspondences.
- Approach: UNIT models each domain with a VAE-GAN, combining adversarial training, weight sharing, and variational reconstruction to generate corresponding images.The framework uses encoders, generators, and adversarial discriminators for the two domains.
- Results: The framework achieved high-quality image translation across various unsupervised tasks and state-of-the-art accuracy on benchmark domain-adaptation datasets.The paper extends shared-latent Coupled GAN ideas to the UNIT problem and relates the constraint to cycle consistency.
2 Assumptions
UNIT assumes corresponding images share a latent code that can reconstruct both domains, making translation a composition of domain-specific encoding and generation functions. The shared-latent assumption also implies cycle consistency, while a shared intermediate representation separates common scene structure from domain-specific realization.
- Shared-latent assumption: The shared-latent assumption posits a code z that can be computed from either corresponding image and used to recover both images.The assumption introduces shared encoders and generators for the two domains.
- Cross-domain mappings: Translation from X1 to X2 is represented by encoding an X1 image and generating an X2 image, with the reverse direction defined analogously.The learned mappings are compositions of domain-specific encoders and generators.
- Cycle consistency: The shared-latent assumption implies cycle consistency, so translating an image and translating it back can reconstruct the original input.The paper states this implication is one-way: cycle consistency does not imply the shared-latent assumption.
- Intermediate representation: UNIT further assumes a shared intermediate representation h produced from z before domain-specific image formation.For multi-domain translation, z captures compact high-level scene structure, while h represents a particular realization before modality-specific rendering.
3 Framework
The framework combines VAEs and GANs with shared encoder and generator weights to learn bidirectional unsupervised image translation. It jointly trains reconstruction, translation, adversarial, and cycle-reconstruction streams.
- Architecture: UNIT uses six subnetworks—two encoders, two generators, and two adversarial discriminators—to learn translation in both directions.The translation functions assemble an encoder from one domain with a generator from the other.
- Variational autoencoders: Each domain’s encoder–generator pair forms a VAE that encodes an image into a Gaussian latent distribution and reconstructs it through the generator.The latent distribution uses the encoder’s mean with identity covariance, and sampling is made differentiable through reparameterization.
- Shared latent space: Weight sharing links the two VAEs by sharing late encoder layers and early generator layers responsible for high-level representations.The constraint alone does not guarantee identical latent codes for corresponding images, so adversarial training is also used.
- Learning: The framework jointly optimizes VAE, GAN, and cycle-consistency objectives for reconstruction, translation, and cycle-reconstruction streams.Cycle-consistency regularizes the ill-posed unsupervised problem, and the combined training is formulated as a minimax saddle-point problem.
- Generative adversarial networks: Adversarial training makes translated images resemble images in their target domains, while discriminators distinguish real target-domain images from generated ones.The framework applies analogous adversarial processing in both domains.
- Learning: Training alternates discriminator gradient ascent with encoder-and-generator gradient descent, and Figure 2 evaluates configurations using satellite-to-map translation accuracy.The figure varies architectures, hyper-parameters, and weight-sharing or cycle-consistency constraints on the Map dataset.
4 Experiments
Experiments evaluate UNIT through ablations, quantitative translation and adaptation tests, and qualitative translations across street scenes, animals, faces, and synthetic-to-real imagery.
- Performance Analysis: The shallowest discriminator performed worst, while changing the number of VAE weight-sharing layers had little impact.The authors attributed the limited impact to residual blocks and used one sharing layer with five-layer discriminators afterward.
- Performance Analysis: Larger negative-log-likelihood weights generally improved translation accuracy, while KL-term weights of 0.1 consistently performed well.The selected values were λ1 = λ3 = 0.1 and λ2 = λ4 = 100.
- Performance Analysis: 0.600 average pixel accuracy was achieved by the full model, compared with 0.569 without weight sharing and 0.568 without cycle consistency.The ablation averaged accuracy over five trials.
- Qualitative results: Street-scene translation generated realistic images across weather, time-of-day, and season changes, although transformations requiring added detail were harder.The evaluated tasks included sunny↔rainy, day↔night, and summery↔snowy translation.
- Qualitative results: The framework translated dog breeds, cat species, and face attributes, with realistic face translations and region-dependent synthetic-to-real performance.Synthetic-to-real results were better for building, sky, road, and car regions than for human regions.
- Domain Adaptation: 0.9053 accuracy was obtained for SVHN→MNIST, exceeding the previous state-of-the-art result of 0.8488.The method also outperformed Coupled GAN on MNIST↔SVHN.
5 Related Work
Related work includes GANs, VAEs, conditional image translation, and unsupervised approaches. UNIT differs by avoiding corresponding image pairs, translating higher-resolution natural images, and relating cycle consistency to its shared-latent assumption.
- Generative models: GANs, VAEs, and PixelCNN are established deep generative models, while UNIT combines GANs and VAEs for unsupervised image-to-image translation.The paper frames unsupervised translation as conditional image generation without paired examples.
- Image translation: Most earlier conditional image-translation methods used supervised learning with corresponding images, unlike UNIT.The paper also distinguishes UNIT from DTN and other conditional translation approaches.
- Image translation: UNIT extends unsupervised translation to large-resolution natural images and reports better unsupervised domain-adaptation performance than competing approaches.The related-work discussion contrasts this scope with earlier small-resolution face and digit translation.
- Image translation: The paper states that cycle consistency is a natural consequence of its shared-latent-space assumption.Two contemporary works independently introduced cycle consistency for unsupervised image translation.
6 Conclusion and Future Work
The framework translates images between domains without corresponding training pairs, but its current design has unimodal outputs and potentially unstable training. Future work targets both limitations.
- The framework learns cross-domain image translation without corresponding images in the training dataset.
- The Gaussian latent space assumption makes the translation model unimodal.
- Training may be unstable because it involves saddle point searching.
A Network Architecture
The unsupervised image-translation experiments use a documented network architecture with standardized abbreviations for neurons, kernels, strides, transposed convolutions, and residual blocks.
- Table 3 specifies the network architecture used for unsupervised image-to-image translation experiments.
- N denotes neurons, K kernel size, and S stride size in the architecture description.
- DCONV denotes transposed convolutional layers, while RESBLK denotes residual basic blocks.
B Domain Adaptation
The domain-adaptation experiments evaluate MNIST↔USPS and SVHN→MNIST using specified source, target, and test datasets, with architectures adapted from prior methods.
- MNIST↔USPS: The MNIST↔USPS experiments use 60,000 MNIST and 7,291 USPS training images, evaluating on 10,000 MNIST and 2,007 USPS test images.
- MNIST↔USPS: MNIST and USPS images are resized to 28×28, and Coupled GAN is trained under the same setting for fair comparison.
- MNIST↔USPS: The MNIST↔USPS encoder and generator are specified in Table 4, with discriminator architecture specified in Table 5.
- SVHN→MNIST: The SVHN→MNIST experiment uses 531131 SVHN training images as source data and 60000 MNIST training images as target data.
- SVHN→MNIST: The SVHN→MNIST setup uses the MNIST↔USPS encoder and generator, a DTN-like discriminator and classifier, and dropout in every classifier layer.