Source-linked AI summary
Image-to-image translation for cross-domain disentanglement
Abel Gonzalez-Garcia, Joost van de Weijer, Yoshua Bengio
TL;DR
The paper addresses how to disentangle shared and domain-specific factors while performing multimodal image-to-image translation. It proposes bidirectional GAN translation with cross-domain autoencoders, achieving diverse translation, domain-specific manipulation, and cross-domain retrieval across synthetic and realistic datasets.
Problem
Image translation and representation disentanglement are usually treated separately, limiting control over shared and domain-specific variation across image domains.
Method
The model uses bidirectional GAN-based translation, partitions representations into shared and exclusive parts, and adds cross-domain autoencoders to align latent spaces and enforce disentanglement.
Results
The model supports diverse sample generation, domain-specific image transfer and interpolation, and cross-domain retrieval, with better translation and retrieval results reported across the evaluated datasets.
Takeaways & Limitations
Cross-domain disentanglement provides a representation that supports multimodal translation and manipulation across domains using paired images without labeled data.
Abstract
from arXiv · showhide
Deep image translation methods have recently shown excellent results, outputting high-quality images covering multiple modes of the data distribution. There has also been increased interest in disentangling the internal representations learned by deep methods to further improve their performance and achieve a finer control. In this paper, we bridge these two objectives and introduce the concept of cross-domain disentanglement. We aim to separate the internal representation into three parts. The shared part contains information for both domains. The exclusive parts, on the other hand, contain only factors of variation that are particular to each domain. We achieve this through bidirectional image translation based on Generative Adversarial Networks and cross-domain autoencoders, a novel network component. Our model offers multiple advantages. We can output diverse samples covering multiple modes of the distributions of both domains, perform domain-specific image transfer and interpolation, and cross-domain retrieval without the need of labeled data, only paired images. We compare our model to the state-of-the-art in multi-modal image translation and achieve better results for translation on challenging datasets as well as for cross-domain retrieval on realistic datasets.
1 Introduction
The paper introduces cross-domain disentanglement to separate shared factors from domain-specific variation while improving image-to-image translation control. Its bidirectional GAN-based model supports diverse translation and cross-domain retrieval without labeled data.
- Motivation and contribution: Disentangled representations can isolate nuisance factors, allowing models to marginalize irrelevant variation and control task-relevant transformations.The motivation includes factors such as illumination, shadows, viewpoint, and object orientation.
- Motivation and contribution: Cross-domain disentanglement separates representation factors into shared information and domain-exclusive variation.The shared component captures factors common to both domains, while exclusive components capture factors particular to each domain.
- Capabilities: The approach generates diverse samples conditioned on an input image and explicitly models variation in both domains.This contrasts with methods that produce deterministic outputs or model variation only in the output domain.
- Capabilities: The shared representation enables cross-domain retrieval, while disentanglement also supports domain-specific image transfer and interpolation.These capabilities are presented as applications of separating common and exclusive factors.
- Method: The model uses bidirectional image translation with GANs and cross-domain autoencoders to enforce the proposed representation structure.The authors combine multiple losses with the new cross-domain autoencoder component.
2 Cross-domain disentanglement networks
The architecture splits latent representations into shared and exclusive parts, translating with shared content plus noise while using cross-domain autoencoders and adversarial constraints to enforce disentanglement. It replaces skip connections with a larger shared bottleneck and uses reconstruction and adversarial objectives for consistency and multimodal outputs.
- Representation structure: Each image representation is partitioned into shared information and exclusive information for domains X and Y.For colored digits, shared content is the digit, while exclusive content includes digit or background color.
- Image translation modules: Translation modules G and F use encoder-decoder architectures, but decoders translate using the shared representation combined with random noise.The noise represents the missing exclusive factors and enables multiple plausible translations for one input.
- Disentanglement constraints: A GRL adversarially discourages exclusive features EX from containing information that could generate images in domain Y.The reversed gradient affects encoder units involved in generating exclusive features.
- Disentanglement constraints: Small N(0, 0.1) noise is added to encoder outputs because shared-representation matching can otherwise encourage SX toward zero.The authors report that this modification leads to better results.
- Architectural bottleneck: The architecture avoids skip connections and enlarges the shared latent spatial dimensions from 1 × 1 × 512 to 8 × 8 × 512.This reduces the bottleneck while ensuring the decoder receives only shared information from the encoder.
- Training objectives: The model uses WGAN-GP adversarial losses and omits paired-image reconstruction from the generator loss to preserve multimodal output coverage.Generated outputs need not match the paired target because they should cover multiple modes of the output distribution.
- Cross-domain autoencoders: Cross-domain autoencoders address the missing constraint that translated images preserve the corresponding input concept across domains.Image translation constraints alone do not prevent different concepts from mapping to the same shared latent representation.
- Cross-domain autoencoders: Cross-domain autoencoders reconstruct x as x′ = Fd(SY, EX), using the other domain’s shared representation with X-exclusive information.This provides an additional incentive for domain-specific information to remain in EX.
3 Related work
Prior work explores disentangled representations for controlling image variation and improving multimodal image translation. The paper’s related methods include latent-code reconstruction, conditional VAE-GANs, and content-style factorization.
- Disentangled representation methods aim to separate factors of variation, including through generative models, mutual-information objectives, sub-manifolds, and representation arithmetic.
- BicycleGAN combines latent-code reconstruction with a conditional GAN and conditional VAE to achieve one-to-many image translations.
- Concurrent approaches disentangle image representations into content and style for multimodal outputs, while related work separates foreground, background, and pose.
- Figure 3 illustrates the paper’s model generating samples from random exclusive representations and forming visual analogies by combining shared and exclusive parts.
4 Experiments
The experiments assess cross-domain disentanglement on synthetic and realistic data, multimodal translation, retrieval, and component ablations. The representation supports domain-specific manipulation and retrieval, while quantitative results show stronger translation performance in both directions for cars and a mixed result for chairs.
- Representation disentangling on MNIST variations: The MNIST variations extend disentanglement evaluation to cross-domain representations, using colorized digits or colored backgrounds with standard train and test splits.The evaluation uses 50K training images and 10K test images.
- Representation disentangling on MNIST variations: Interpolating the exclusive representation varies domain-specific factors while preserving shared content; interpolating the shared representation preserves domain-specific properties.The examples demonstrate controlled variation by fixing one representation part while interpolating another.
- Representation disentangling on MNIST variations: Shared features retrieve same-digit images across both domains in 46% of cases, whereas pixel distances retrieve almost 100% from the query domain and exclusive features retrieve visually similar images regardless of digit.These results use top-neighbor retrieval from the union of both domains.
- Many-to-many image translation: The method outperforms pix2pix and BicycleGAN on car translation in both directions under LPIPS evaluation, producing samples perceptually closer to the ground truths.For each test image, three samples are generated and compared with both possible ground truths before selecting the minimum distance.
- Many-to-many image translation: Car analogies transfer one car’s orientation while preserving properties such as style or color, and the method covers many output modes while remaining unimodal when only one viewpoint is available.The viewpoint experiment uses frontal images in one domain and all non-frontal viewpoints in the other.
- Ablation study: Removing cross-domain autoencoders significantly reduces visual-analogy performance, while removing the GRL increases shared information in exclusive features more strongly as their size grows.The GRL-related recall increase is 0.7% for E=1x8, 8% for E=1x128, and 25% for E=8x8x256; its effect is limited in the current configuration.
5 Conclusions
The paper introduces cross-domain disentanglement and applies it across synthetic and real-image datasets to support multiple image-translation tasks.
- The model separates representations into one shared part and two domain-exclusive parts.
- The approach supports diverse sample generation, cross-domain retrieval, domain-specific image transfer, and interpolation.
- Experiments cover datasets with different complexity, including synthetic data and real images.
- The paper introduces many-to-many image translation using disentangled representations to address limitations of existing approaches.
A Network architecture and hyperparameters
The experiments use a pix2pix-weighted bidirectional translation architecture with common, shared, and exclusive components. Its implementation specifies convolutional encoders and decoders, adversarial discriminators, and dataset-specific sample counts.
- Hyperparameters: Training uses pix2pix weights wGAN = 1 and wL1 = 100, while the exclusive decoder loss uses wEx = 0.1.The authors report that the lower exclusive-loss weight supports disentanglement without excessively changing generated-image quality.
- Encoder: The common encoder has five 4x4, stride-2 convolutional layers with 64, 128, 256, and 512 filters, LeakyReLU 0.2, and batch normalization.
- Encoder: The shared encoder adds a 4x4, stride-2 convolution with 512 filters followed by a fully connected layer with 8 output units.
- Decoder: The generator decoder uses five convolutional layers with 512, 256, 128, 64, and 3 filters, ReLU, batch normalization, and training-only dropout of 0.5 on its first three layers.
- Decoder: The exclusive generator decoder matches the generator decoder architecture but places a Gradient Reversal Layer at its input.
- Discriminators: WGAN-GP and GAN discriminators use three stride-2 convolutional layers with 64, 128, and 256 channels, followed by distinct one-output layers.The WGAN-GP discriminator uses a linear fully connected output, whereas the GAN discriminator uses a sigmoid convolutional output and batch normalization.
- Datasets: The experiments include MNIST-CD/CB with 15 samples, 3D cars with 900, 3D chairs with 75, Facades with 200, and Maps with 200.