Source-linked AI summary
A Unified Feature Disentangler for Multi-Domain Image Translation and Manipulation
Alexander H. Liu, Yen-Cheng Liu, Yu-Ying Yeh, Yu-Chiang Frank Wang
TL;DR
Existing cross-domain models provide limited attribute manipulation and can become costly across multiple domains. UFDN uses adversarially trained unified encoder-generator networks to disentangle domain-invariant and domain-specific factors, enabling continuous translation and manipulation. Its learned representation shows significant improvements in unsupervised domain adaptation, although the paper identifies constraints around domain disentanglement and unsupervised domain-vector manipulation.
Problem
Existing cross-domain models have limited capacity for manipulating specific attributes, while multi-domain disentanglement designs can require high computational costs.
Method
UFDN uses a unified encoder-generator architecture with adversarial training to learn domain-invariant features while disentangling domain and attribute factors.
Results
UFDN performs continuous multi-domain image translation and manipulation, and its latent representation achieves significant improvements in unsupervised domain adaptation.
Takeaways & Limitations
The framework provides a single unified model for describing, translating, and manipulating data across multiple domains.
Takeaways & Limitations
Without explicit domain information or pixel-space adversarial learning, domain factors may not be disentangled reliably and manipulated outputs are not guaranteed to be satisfactory.
Abstract
from arXiv · showhide
We present a novel and unified deep learning framework which is capable of learning domain-invariant representation from data across multiple domains. Realized by adversarial training with additional ability to exploit domain-specific information, the proposed network is able to perform continuous cross-domain image translation and manipulation, and produces desirable output images accordingly. In addition, the resulting feature representation exhibits superior performance of unsupervised domain adaptation, which also verifies the effectiveness of the proposed model in learning disentangled features for describing cross-domain data.
1 Introduction
The paper introduces UFDN, a unified framework for disentangling domain-invariant and domain-specific factors across multiple domains. It supports continuous image translation and manipulation while producing features effective for unsupervised domain adaptation.
- Motivation: Existing cross-domain models have limited capacity for manipulating specific feature attributes when recovering data across domains.Prior work typically learns mappings between domains or a joint representation, but does not adequately support attribute-specific manipulation.
- Motivation: Existing disentanglement designs incur high computational costs when handling more than two domains or multiple feature attributes.This motivates a unified representation model for multi-domain data.
- Method: UFDN uses unified encoder and generator networks with adversarial training to derive a domain-invariant latent feature space from multiple domains.The generator recovers or translates data across domains, while domain-specific factors such as domain codes and attributes are disentangled.
- Results: UFDN supports continuous cross-domain translation by interpolating domain codes and achieves promising multi-domain image-to-image translation results.The framework can synthesize and manipulate outputs in target and intermediate domains.
- Results: The derived disentangled representation achieves significant improvements over state-of-the-art methods in unsupervised domain adaptation.This provides evidence that the learned features are useful beyond image translation and manipulation.
- Method: UFDN treats data domains and image attributes as latent factors to be disentangled within a single unified framework.This design enables multi-domain image translation rather than separate domain-pair models.
2 Related Work
Related work covers representation disentanglement, image-to-image translation, and unsupervised domain adaptation. The paper positions UFDN as a unified approach for manipulating multi-domain data while addressing limitations in prior models.
- Representation Disentanglement: Representation-disentanglement research seeks interpretable deep features, while prior cross-domain work relied on architecture choices that could limit practical use.The paper identifies a need for unified disentanglement across multiple domains.
- Image-to-Image Translation: Image-to-image translation methods learn mappings across domains using paired or unpaired data and related cross-domain consistency objectives.This line of work provides context for UFDN’s multi-domain translation capability.
- Unsupervised Domain Adaptation: Unsupervised domain adaptation classifies target-domain samples using labeled source data and unlabeled target data while seeking invariance to domain shift.Adversarial training is a representative strategy for learning domain-invariant features in this setting.
3 Unified Feature Disentanglement Network
UFDN learns domain-invariant features across multiple domains by disentangling domain information and preserving image recovery through adversarial learning. The unified architecture supports translation, manipulation, and related cross-domain tasks without pairwise training data.
- UFDN uses a unified encoder and generator to derive domain-invariant representations from multiple domains, rather than only a pair of domains.The representation is learned by disentangling domain information from the latent space.
- 3.1 Self-supervised feature disentanglement: The VAE-style pathway encodes an image as z, combines it with domain vector vc, and reconstructs the image through generator G.The reconstruction objective includes image recovery and a KL divergence penalty toward the latent prior.
- 3.1 Self-supervised feature disentanglement: Adversarial domain classification trains feature discriminator Dv to predict domain codes while encoder E confuses those predictions, removing domain-specific information from z.The domain vector can be one-hot, concatenated one-hot, or real-valued.
- 3.2 Adversarial learning in pixel space: Pixel-space discriminator Dx improves synthesized-image quality while distinguishing real from fake images and classifying reconstructed and translated outputs by domain.Its domain-classification objective implicitly maximizes mutual information between the domain vector and synthesized image.
- 3.2 Adversarial learning in pixel space: UFDN alternately updates E, G, Dv, and Dx, and does not require pairwise training data for multi-domain disentangled representation learning.The resulting representation is also used for unsupervised domain adaptation.
- 3.3 Comparison with state-of-the-art cross-domain visual tasks: UFDN supports multi-domain translation with continuous domain-vector manipulation and can translate randomly sampled latent identities into supplied domains.Figure 3 covers sketch, photo, and paint domains, plus translations from randomly generated identities.
4 Experiment Results
Experiments evaluate UFDN on multi-domain face translation and unsupervised digit adaptation, showing continuous domain manipulation, competitive translation quality, and domain-invariant representations.
- Multi-domain image translation: UFDN translates among sketch, photo, and paint domains while manipulating the domain vector continuously.The framework also generates images from randomly sampled latent representations and applies supplied domain vectors.
- Multi-domain image translation: UFDN outperformed E-CDRD in translation quality and matched StarGAN on human-face image translation.Table 2 evaluates recovered images using SSIM, MSE, and PSNR; pairwise information was not used during training.
- Multi-domain image translation: Fixed domain-invariant representations allow domain, gender, and smiling attributes to be manipulated independently.The experiment expands the domain vector with annotated attributes and uses a fixed representation across outputs.
- Unsupervised domain adaptation: For SVHN→MNIST, UFDN reduced classification error from 7.6% to 5% while using 60K SVHN images instead of UNIT’s 531K.The model also surpassed the state of the art for MNIST→USPS and SVHN→MNIST.
- Unsupervised domain adaptation: t-SNE representations clustered digits by class rather than domain, and manipulating the domain vector transformed SVHN images into MNIST.An auxiliary digit classifier was jointly trained with UFDN using the domain-invariant representation.
- Ablation study: Removing self-supervised feature disentanglement lets the generator ignore the domain vector, while removing pixel-space adversarial learning weakens photo–paint differences.The full UFDN produced clearer domain changes than the ablated variants.
5 Conclusion
UFDN learns disentangled, domain-invariant features across multiple domains through a unified encoder–generator architecture with adversarial learning.
- Conclusion: UFDN combines a unified encoder and generator with adversarial learning to learn disentangled representations across domains.The resulting model supports image translation and manipulation and enables unsupervised domain adaptation.