Source-linked AI summary
Diverse Image-to-Image Translation via Disentangled Representations
Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, Ming-Hsuan Yang
TL;DR
Image-to-image translation lacks aligned training pairs and often has multiple valid outputs for one input. The paper disentangles domain-invariant content from domain-specific attributes and uses cross-cycle consistency to train with unpaired data. It reports diverse, realistic images and competitive unsupervised domain-adaptation performance on MNIST-M and LineMod-related datasets.
Problem
Image-to-image translation must handle both unavailable aligned training pairs and multimodal mappings in which one input has multiple possible outputs.
Method
The model separates domain-invariant content and domain-specific attributes, then combines encoded content with sampled or transferred attributes using cross-cycle consistency for unpaired training.
Results
The model produces diverse and realistic images and achieves competitive performance for unsupervised domain adaptation on MNIST-M and Cropped LineMod.
Takeaways & Limitations
Disentangled representations enable multimodal image-to-image translation without aligned training pairs while supporting an unsupervised domain-adaptation application.
Takeaways & Limitations
The method can fail for attribute vectors in under-sampled regions and when domain characteristics differ substantially, such as human figures absent from Monet collections.
Abstract
from arXiv · showhide
Image-to-image translation aims to learn the mapping between two visual domains. There are two main challenges for many applications: 1) the lack of aligned training pairs and 2) multiple possible outputs from a single input image. In this work, we present an approach based on disentangled representation for producing diverse outputs without paired training images. To achieve diversity, we propose to embed images onto two spaces: a domain-invariant content space capturing shared information across domains and a domain-specific attribute space. Our model takes the encoded content features extracted from a given input and the attribute vectors sampled from the attribute space to produce diverse outputs at test time. To handle unpaired training data, we introduce a novel cross-cycle consistency loss based on disentangled representations. Qualitative results show that our model can generate diverse and realistic images on a wide range of tasks without paired training data. For quantitative comparisons, we measure realism with user study and diversity with a perceptual distance metric. We apply the proposed model to domain adaptation and show competitive performance when compared to the state-of-the-art on the MNIST-M and the LineMod datasets.
1 Introduction
Image-to-image translation must address both unpaired training data and multiple valid outputs per input. The paper proposes disentangled content and attribute representations with cross-cycle consistency, producing diverse realistic translations and competitive domain-adaptation results.
- Motivation: Unpaired image-to-image translation is difficult because aligned pairs are scarce, while single inputs can have multiple valid outputs.Simply injecting noise often fails because generators may ignore it or collapse modes.
- Approach: The framework separates domain-invariant content from domain-specific attributes to generate diverse outputs from unpaired data.Content features capture shared information across domains, while attributes model within-domain variation.
- Approach: A content discriminator and cross-cycle consistency loss support representation disentanglement and training without aligned image pairs.Cross-domain attribute swapping is followed by reconstruction of the original inputs.
- Results: The model generates images that experiments describe as both diverse and realistic and compares favorably with existing image-to-image translation models.Realism is evaluated with a user study and diversity with perceptual distance metrics.
- Results: The method achieves competitive results on unsupervised domain adaptation for MNIST-M and Cropped LineMod.The paper applies the translation model to domain adaptation without requiring source-domain annotations for image-to-image translation.
2 Related Work
Prior image-to-image translation methods include conditional GANs for paired data and cycle-consistency approaches for unpaired data, but typically produce one output per input. This work instead uses disentangled representations to support multimodal unpaired translation and attribute-controlled generation.
- Image-to-image translation: Conditional GAN methods model image-to-image mappings but require paired training data.Pix2pix is presented as a representative paired-data approach.
- Image-to-image translation: CycleGAN, DiscoGAN, and UNIT use cycle consistency to train image-to-image translation with unpaired data.These methods address the paired-data requirement through cycle-consistency regularization.
- Image-to-image translation: Unpaired methods commonly generate a single output conditioned solely on the input image, while added noise often fails to increase variation.Mode collapse can prevent noise vectors from controlling generated outputs.
- Disentangled representations: Disentangled-representation research models factors of data variation, including unsupervised approaches using mutual information or adversarial losses.The paper relates its content adversarial loss to prior efforts separating shared and varying components.
- Position of this work: The proposed method learns multimodal mappings with unpaired data and generates outputs from random or transferred attribute vectors.It differs from related domain-adaptation methods by not using source-domain annotations for image-to-image translation.
3 Disentangled Representation for I2I Translation
The framework disentangles shared content from domain-specific attributes, enabling multimodal translation between unpaired visual domains. It combines these representations through cross-cycle consistency and additional losses to support diverse, realistic outputs.
- Disentangled representations: Content encoders, attribute encoders, generators, and discriminators translate images using content features combined with target-domain attributes.Attribute vectors are regularized toward N(0, 1), enabling stochastic sampling at test time.
- Disentangled representations: The model embeds images into a shared domain-invariant content space and separate domain-specific attribute spaces.Content captures information shared across domains, while attributes represent within-domain variation.
- Disentangled representations: Weight sharing and a content discriminator encourage content representations from the two domains to occupy a common, domain-independent space.The content adversarial objective trains encoded content features whose domain membership cannot be distinguished.
- Cross-cycle consistency: The cross-cycle consistency loss swaps attribute representations between non-corresponding images, translates twice, and reconstructs the original images.This constraint uses disentangled content and attribute representations rather than assuming a one-to-one mapping between domains.
- Additional losses: Additional losses support self-reconstruction, Gaussian attribute regularization, realistic image generation, and invertible mappings between latent attributes and outputs.The full objective combines these terms with the content adversarial and cross-cycle losses.
- Generation and evaluation: Random attribute sampling produces varied translations across artistic, seasonal, and portrait-related mappings, while interpolation tests continuity in the learned attribute space.The sample-results figure shows five generated-output columns for each input, and the interpolation experiment assesses generalization beyond memorized visual information.
4 Experimental Results
Experiments evaluate DRIT’s disentangled representations for diverse, realistic unpaired translation, attribute transfer, reconstruction, and domain adaptation. Results show benefits from the content discriminator and diverse generation, while performance remains bounded by training-data coverage and domain similarity.
- Implementation: DRIT uses content and attribute encoders, a generator, and an 8-dimensional attribute vector in its implementation.The content encoder uses convolutional and residual blocks, while the attribute encoder uses convolutional and fully connected layers.
- Qualitative Evaluation: DRIT supports random-sampled and example-guided attribute transfer, including both inter-domain and intra-domain transfer.Intra-domain transfer is not explicitly used during training.
- Qualitative Evaluation: Without the content discriminator, generated diversity contains artifacts and mainly appears as global color variation rather than domain-related detail.The comparison also reports ill-quality outputs for the Cycle/Bicycle baseline because its pseudo-paired data are not uniformly ideal.
- Qualitative Evaluation: Attribute interpolation verifies continuity in the learned attribute space and indicates generalization beyond memorized trivial visual information.The interpolation experiment uses two given attributes to generate intermediate images.
- Realism and Diversity: CycleGAN performs favorably in realism but has limited diversity, while the experiments report that DRIT’s content discriminator is necessary for its results.Realism is evaluated by pairwise user study and diversity with LPIPS on Yosemite winter-to-summer translation.
- Reconstruction: DRIT performs comparably with BicycleGAN on edge-to-shoes reconstruction despite training without paired data, and the content discriminator contributes greatly to disentangled encoding quality.BicycleGAN requires paired training data for this comparison.
- Limitations: The attribute space is not fully exploited with limited training data, and sampled vectors in under-sampled regions can cause translation failures.The method also has difficulty when domain characteristics differ significantly, such as human figures absent from Monet collections.
- Domain Adaptation: 86.93% classification accuracy is achieved by Ours, ×1, compared with 74.5% for CycleGAN; Ours reaches 90.21% at ×3 and 91.54% at ×5.The reported adaptation results cover MNIST to MNIST-M and Synthetic Cropped LineMod to Cropped LineMod; the method does not outperform PixelDA but uses no label information during training.
5 Conclusions
The paper presents a disentangled framework for diverse image-to-image translation with unpaired data. It separates shared content from domain-specific variation, uses cross-cycle consistency for unpaired training, and reports realistic, diverse outputs with competitive domain-adaptation performance.
- Conclusions: The framework separates latent representations into shared content and domain-specific attribute spaces to model diverse variations for the same content.A content discriminator facilitates this disentanglement.
- Conclusions: Cross-cycle consistency enables cyclic reconstruction when paired training data are unavailable.The loss operates on the disentangled representations.
- Conclusions: Qualitative and quantitative results show realistic and diverse generated images, while domain-adaptation experiments achieve competitive performance against state-of-the-art methods.The conclusion covers both unpaired translation and its domain-adaptation application.