Source-linked AI summary
DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
Hsin-Ying Lee, Hung-Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Singh, Ming-Hsuan Yang
TL;DR
Image-to-image translation lacks aligned training pairs and may have multiple valid outputs for one input. DRIT++ disentangles shared content from domain-specific attributes and uses cross-cycle consistency to learn diverse unpaired translations, which experiments show are realistic and diverse across varied tasks.
Problem
Image-to-image translation must address missing aligned training pairs and multimodal mappings in which one input can correspond to multiple outputs.
Method
DRIT++ separates images into domain-invariant content and domain-specific attribute spaces, then combines encoded content with sampled attributes and trains with cross-cycle consistency on unpaired data.
Results
Qualitative and quantitative experiments show that DRIT++ produces realistic and diverse images across image-to-image translation tasks, including multi-domain translation.
Takeaways & Limitations
The disentangled representation and cross-cycle consistency framework supports diverse image-to-image translation without paired data and extends to multi-domain settings.
Takeaways & Limitations
Limited training data can leave the attribute space under-exploited, while multiple encoders and decoders require substantial memory and limit high-resolution applications.
Abstract
from arXiv · showhide
Image-to-image translation aims to learn the mapping between two visual domains. There are two main challenges for this task: 1) lack of aligned training pairs and 2) multiple possible outputs from a single input image. In this work, we present an approach based on disentangled representation for generating diverse outputs without paired training images. To synthesize diverse outputs, we propose to embed images onto two spaces: a domain-invariant content space capturing shared information across domains and a domain-specific attribute space. Our model takes the encoded content features extracted from a given input and attribute vectors sampled from the attribute space to synthesize diverse outputs at test time. To handle unpaired training data, we introduce a cross-cycle consistency loss based on disentangled representations. Qualitative results show that our model can generate diverse and realistic images on a wide range of tasks without paired training data. For quantitative evaluations, we measure realism with user study and Fréchet inception distance, and measure diversity with the perceptual distance metric, Jensen-Shannon divergence, and number of statistically-different bins.
1 Introduction
Image-to-image translation must handle unpaired data and multimodal mappings. DRIT++ addresses both by disentangling shared content from domain-specific attributes and using cross-cycle consistency, producing diverse and realistic translations across dual- and multi-domain tasks.
- Motivation: Image-to-image translation is challenging because aligned pairs are difficult to obtain and one input can have multiple valid outputs.Noise injection may fail to increase variation because generators can ignore the noise and collapse modes.
- Approach: DRIT++ embeds images into a domain-invariant content space and a domain-specific attribute space for diverse unpaired translation.Content represents information shared across domains, while attributes model within-domain variation for the same content.
- Approach: A content discriminator discourages domain-specific cues in content features, while latent regression encourages invertible attribute-to-output mappings.These objectives support disentanglement between shared content and domain-specific variation.
- Approach: Cross-cycle consistency enables training with unpaired data by swapping attribute vectors across unaligned images and reconstructing the original inputs.The method performs cross-domain mappings before reconstruction using disentangled representations.
- Experiments: Extensive experiments report diverse and realistic outputs, including randomly sampled and example-guided translations across dual- and multi-domain tasks.Realism is evaluated with user studies and FID, while diversity uses perceptual distance metrics.
2 Related Work
Prior I2I methods either require paired data, produce one output per input, or assume particular latent-space structures. Disentangled representation methods instead model shared and variation-specific factors to support diverse unpaired translation.
- Paired translation: Pix2pix produces high-quality image-to-image translations but requires paired training data.Its conditional GAN formulation models the mapping using aligned examples.
- Unpaired translation: CycleGAN, DiscoGAN, and UNIT enable training with unpaired data through cycle consistency, but generate one output conditioned on each input image.These methods address paired-data dependence without modeling multiple outputs for a single input.
- Disentangled and multi-domain methods: Concurrent disentangled-representation methods learn diverse image-to-image translations from unpaired data, while other approaches extend translation to multiple domains.The cited multi-domain methods are described as achieving one-to-one mappings among domains.
- Disentangled representations: Unsupervised disentangled-representation research models factors of data variation, including class-related and class-independent components.InfoGAN achieves disentanglement by maximizing mutual information between latent variables and data variation.
3 Disentangled Representation for I2I Translation
The framework disentangles shared content from domain-specific attributes to enable multimodal translation without paired data. Cross-cycle consistency and auxiliary losses support dual-domain and shared-network multi-domain translation.
- Architecture: The model uses content encoders, attribute encoders, generators, and discriminators for both visual domains.Content encoders map images to a shared space, while attribute encoders map them to domain-specific spaces; generators combine both representations.
- Disentangled representations: Content representations capture cross-domain information, whereas attribute representations encode domain-specific variation.The framework assigns common information to C and remaining domain-specific information to AX and AY.
- Disentanglement: Weight sharing and a content discriminator encourage content encoders from both domains to produce domain-invariant representations.The discriminator distinguishes domain membership, while the encoders learn representations whose domain membership cannot be distinguished.
- Cross-cycle consistency: Cross-cycle consistency swaps attributes between non-corresponding images, performs forward and backward translation, and reconstructs the originals.The procedure uses two translation stages and swaps attribute representations during both directions.
- Additional losses: Additional losses support realistic generation, self-reconstruction, latent invertibility, Gaussian attribute priors, and improved diversity.These include domain adversarial, self-reconstruction, latent regression, KL, and mode-seeking regularization terms.
- Multi-domain translation: The framework extends to multiple domains using shared networks, one-hot domain codes, and a discriminator that also classifies domains.Forward and backward translation remain similar to the dual-domain setting.
4 Experimental Results
Experiments evaluate DRIT++ on diverse, unpaired image-to-image translation tasks, including random and example-guided attribute transfer, multi-domain translation, and realism-diversity trade-offs. Results indicate that the method improves diversity and visual quality across these settings.
- Qualitative evaluation: Random attribute sampling produces diverse outputs across Photo→Monet, winter→summer, photograph→portrait, and cat→dog translations.The sample grids use one input column and five outputs generated from random attribute vectors.
- Qualitative evaluation: DRIT++ produces more diverse and realistic winter→summer samples than baseline methods, whereas Cycle/Bicycle outputs have limited diversity and visible artifacts.Removing the content discriminator also causes variations to collapse mainly into global color differences and introduces domain-specific detail failures.
- Qualitative evaluation: Mode-seeking regularization substantially alleviates mode collapse, particularly for challenging dog-to-cat shape-variation translation.The regularizer improves diversity while maintaining visual quality.
- Experimental setup: DRIT++ extends DRIT with mode-seeking regularization for sample diversity and a multi-domain formulation while retaining the two-domain model as a special case.The multi-domain algorithm is backward compatible with the two-domain and original DRIT methods.
- Attribute transfer: Disentangled content and attribute representations enable example-guided inter-domain and intra-domain attribute transfer, although intra-domain transfer is not explicitly trained.The shared content space allows content features encoded from either domain to condition synthesis.
- Multi-domain translation: Using one generator, DRIT++ performs diverse translation among three domains comprising real images and two artistic styles, and among four weather domains.The evaluated settings include real images, Monet, Ukiyoe, and sunny, cloudy, snowy, and foggy conditions.
- Quantitative evaluation: DRIT++ performs well on all reported realism and diversity metrics and compares favorably with state-of-the-art and baseline methods.The evaluation uses user studies, FID, perceptual distance, Jensen-Shannon divergence, and statistically-different bins.
Multi-domain model on two-domain translation
The multi-domain model is evaluated in the two-domain setting and on high-resolution translation, with ablations examining content alignment and multi-scale generation. Results show strong metric performance, improved content-space alignment, and more realistic high-resolution outputs, subject to data and memory limitations.
- Two-domain translation: Two-domain translation is a special case of the multi-domain problem, and the multi-domain model performs well against the two-domain model across reported metrics.The comparison uses the experimental settings of Tables 2 and 3.
- Content alignment: The content discriminator substantially reduces the distance between latent content representations from the two domains.The effect is quantified in Table 4 and visualized with t-SNE in Figure 13.
- High-resolution translation: A multi-scale generator-discriminator architecture supports translation of 720×360 street-scene images by applying an additional adversarial loss to generated low-resolution images.Intermediate generator features produce the low-resolution images used by the additional discriminator.
- High-resolution translation: The multi-scale architecture generates more photo-realistic images than simply adding two generator layers on the GTA5→Cityscape task.The comparison reports FID and LPIPS scores in Table 7 and Figure 16.
- Ablation studies: Ablations evaluate the content discriminator, latent regression loss, and mode-seeking regularization as separate components of the proposed algorithm.These component effects are summarized in Table 5.
- Limitations: The method is limited by under-sampled attribute spaces, large domain-characteristic differences, and memory-intensive cross-cycle training that constrains high-resolution applications.Human-figure translation can fail when relevant portraits are absent from the Monet training collection.
5 Conclusions
The framework enables diverse image-to-image translation from unpaired data by disentangling shared content from domain-specific variation. It also produces realistic and diverse images and achieves competitive domain-adaptation performance.
- The model disentangles latent representations into a content space for cross-domain information and a domain-specific attribute space for diverse variations.
- A content discriminator and cross-cycle consistency loss support representation disentanglement and cyclic reconstruction without paired training data.
- Qualitative and quantitative evaluations show that the model produces realistic and diverse images, while domain adaptation performance is competitive with state-of-the-art methods.