Source-linked AI summary
Multimodal Unsupervised Image-to-Image Translation
Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz
TL;DR
Unsupervised image-to-image translation must model multimodal target conditionals without paired examples, but existing approaches often produce insufficiently diverse outputs. MUNIT separates shared content from domain-specific style, recombines content with sampled or reference styles, and reports superior quality and diversity to unsupervised methods, comparable to a supervised state-of-the-art approach.
Problem
Unsupervised translation lacks paired examples, while existing deterministic or unimodal approaches fail to represent the multimodal range of target outputs.
Method
MUNIT factorizes images into shared content and domain-specific style, then recombines input content with sampled target styles or a reference-image style.
Results
MUNIT achieves quality and diversity superior to existing unsupervised methods and comparable to a state-of-the-art supervised approach.
Takeaways & Limitations
The content-style decomposition supports diverse multimodal translations and example-guided control of translation style.
Takeaways & Limitations
MUNIT assumes only content is shared across domains while style is domain-specific, an assumption motivated for many-to-many mappings.
Abstract
from arXiv · showhide
Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to the state-of-the-art approaches further demonstrates the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at https://github.com/nvlabs/MUNIT
1 Introduction
MUNIT addresses unsupervised image-to-image translation by modeling the target conditional distribution as multimodal rather than deterministic. It separates shared content from domain-specific style and recombines content with sampled target styles to generate diverse outputs.
- Unsupervised translation learns target-domain conditionals without paired source-target examples.
- Existing methods often assume deterministic or unimodal mappings, so they fail to capture the full range of possible outputs.
- Translation recombines an input content code with a random target-domain style code, making different style codes produce different outputs.
- MUNIT decomposes image representations into domain-invariant content and domain-specific style spaces.
2 Related Works
Prior work addresses unsupervised translation with distribution matching, structural constraints, shared latent spaces, and disentangled representations, but multimodal output diversity remains difficult without paired supervision. MUNIT instead uses partially shared content-style representations, domain-specific encoder-decoder pairs, latent and image reconstruction, and adversarial training.
- Image-to-image translation: GAN-based methods align translated-image distributions with real images, while cycle consistency and related constraints address the ill-posed nature of unsupervised translation.
- Image-to-image translation: Most existing methods lack output diversity; methods generating multiple outputs remain discrete, while BicycleGAN models continuous multimodal distributions but requires paired supervision.
- Learning disentangled representations: MUNIT differs from fully shared latent-space approaches by sharing content while retaining domain-specific style, supporting many-to-many mappings.
- Model: The model uses one encoder-decoder pair per domain and factorizes each latent code into content and style components.
- Model: Translation swaps encoder-decoder pairs, preserves the source content code, and samples a target style code whose nonlinear decoding can yield multimodal outputs.
- Model: Bidirectional reconstruction trains encoders and decoders as inverses through image and latent reconstruction objectives.
- Model: Adversarial objectives match translated images to the target-domain distribution, while style reconstruction encourages diversity and content reconstruction preserves semantic content.
4 Theoretical Analysis
The analysis shows that minimizing MUNIT’s loss matches latent and joint image distributions while yielding a weaker, style-augmented form of cycle consistency suited to multimodal translation.
- At optimality, the total loss is minimized when translated distributions match data distributions and the encoder-decoder pairs are inverses.
- Latent Distribution Matching: Theoretical results show that encoded style distributions match Gaussian priors, while content distributions become domain-invariant.
- Joint Distribution Matching: MUNIT defines two joint distributions involving source images and translated outputs, and matches them at optimality.
- Style-augmented Cycle Consistency: The framework replaces overly strong cycle consistency with style-augmented cycle consistency between image–style joint spaces.The model translates an image to the target domain and back using the original style, recovering the original image at optimality.
5 Experiments
MUNIT combines domain-specific style and domain-invariant content representations in an auto-encoder framework, then evaluates unsupervised translation across diverse image domains. Experiments show that the full model produces diverse, realistic outputs and outperforms unsupervised baselines on reported quality and diversity measures.
- Implementation Details: The auto-encoder separates content and style encoding, then uses AdaIN-conditioned residual blocks and upsampling convolutions to decode translated images.The style encoder uses global average pooling and a fully connected layer; an MLP generates AdaIN parameters from the style code.
- Implementation Details: MUNIT uses GANs with multi-scale discriminators to align translated-image distributions with target-domain images while preserving global structure and realistic details.The framework also uses a domain-invariant perceptual loss, which accelerates training on high-resolution images.
- Results: Both UNIT and CycleGAN fail to generate diverse outputs from the same input, whereas the full MUNIT model produces images that are diverse and realistic without supervision.Removing the content reconstruction loss reduces image quality, while removing the style reconstruction loss causes partial mode collapse.
- Results: The full model achieves quality and diversity comparable to the fully supervised BicycleGAN and significantly better than all unsupervised baselines on edges-to-shoes/handbags.Human preference measures quality, while LPIPS distance measures diversity.
- Results: On animal translation, outputs cover multiple fine-grained target-domain modes while generally preserving pose, and MUNIT obtains the highest CIS and IS scores.The dataset contains house cats, big cats, and dogs, with four fine-grained modes per domain that are unknown during training.
- Results: On street-scene and seasonal translations, MUNIT generates varied renderings involving weather, lighting, shadows, road textures, snow, and foliage.The high-resolution Yosemite experiments also produce outputs with different lighting.
- Results: Example-guided style transfer produces results described as more faithful and realistic than classical style-transfer methods because MUNIT learns the target-domain image distribution with GANs.The comparison includes Gatys et al., Chen et al., AdaIN, and WCT.
6 Conclusions
The paper presents a framework for multimodal unsupervised image-to-image translation. Its model achieves superior quality and diversity to existing unsupervised methods and performance comparable to a state-of-the-art supervised approach.
- Conclusions: MUNIT achieves quality and diversity superior to existing unsupervised methods and comparable to a state-of-the-art supervised approach.The authors identify extending the framework to videos and text as future work.
A Proofs
The proofs characterize MUNIT’s optimal behavior and show that its reconstruction objectives induce distribution matching and style-augmented cycle consistency. They also establish that ordinary cycle consistency collapses multimodal translations to deterministic mappings.
- Latent distribution matching: At optimality, the model matches encoded and generation-time latent distributions, making the content space domain-invariant and matching encoded styles to Gaussian priors.The generation-time content distribution is taken from the other domain, while style codes follow their Gaussian priors.
- Joint distribution matching: The framework matches the joint distributions of source images with their translations in both directions at optimality.This is stated as p(x1, x1→2) = p(x2→1, x2).
- Style-augmented cycle consistency: MUNIT defines a deterministic mapping between image–style joint spaces whose forward and reverse mappings become inverses at optimality.The mapping recombines source content with target style and recovers the original image–style pair in reverse.
- Style-augmented cycle consistency: Style-augmented cycle consistency is implicitly enforced by the bidirectional style, content, and image reconstruction losses.The three reconstruction losses imply the corresponding inverse relations used by the joint-space mapping.
- Deterministic collapse: Under matched marginals and ordinary cycle consistency, both learned conditionals collapse to deterministic delta functions.The result demonstrates that cycle consistency is too strong a constraint for multimodal image translation.
B.1 Hyperparameters
Experiments use Adam with specified momentum parameters, a decaying learning rate, unit batch size, fixed loss weights, and an 8-dimensional style code.
- Optimization: Training uses Adam with β1 = 0.5, β2 = 0.999, and an initial learning rate of 0.0001.The learning rate is halved every 100,000 iterations.
- Training configuration: All experiments use batch size 1, loss weights λx = 10, λc = 1, λs = 1, and style-code dimension 8.The domain-invariant perceptual loss has weight 1 for the street scene and Yosemite datasets.
- Augmentation: Random mirroring is applied during training.
B.2 Network Architectures
The network uses separate content and style encoders with a residual-block decoder, while discriminators use progressively wider strided convolutional blocks.
- Decoder: The decoder contains four residual blocks followed by 128- and 64-filter upsampling layers and a three-channel output convolution.
- Normalization: Instance Normalization is applied to the content encoder.
- Discriminator: Discriminators use 64-, 128-, 256-, and 512-filter strided convolutional blocks.
C Domain-invariant Perceptual Loss
The domain-invariant perceptual-loss experiment compares VGG feature distances for same-domain different-scene pairs and same-scene cross-domain pairs, with and without Instance Normalization.
- Experimental design: The experiment tests whether applying Instance Normalization before feature-distance computation makes VGG distances more domain-invariant.
- Experimental design: Image pairs are sampled either from the same domain but different scenes or from the same scene but different domains.The day ↔ night dataset is used for this comparison.
- Figure interpretation: Figure 11 provides example image pairs for the domain-invariant perceptual-loss experiment.
- Figure interpretation: Figure 12 compares VGG feature-distance histograms without Instance Normalization and after Instance Normalization.Blue denotes same-domain, different-scene pairs; green denotes same-scene, different-domain pairs.