Source-linked AI summary
One-Sided Unsupervised Domain Mapping
Sagie Benaim, Lior Wolf
TL;DR
Unsupervised domain mapping seeks analog target-domain samples from unmatched source and target datasets, but prior convincing approaches learn mappings in both directions. The paper learns only G_AB by preserving cross-domain pairwise distances, including distances between parts of one image, and reports better numerical results than circularity. Its conclusion is that raw RGB distances can suffice for forming analogies between related domains.
Problem
Unsupervised mapping must learn G_AB between unmatched domains, while recent convincing approaches simultaneously recover the inverse G_BA.
Method
The method trains a single mapping by preserving pairwise distances across domains, with a related constraint also applicable to different parts of the same image.
Results
The distance-based loss empirically outperforms circularity, and combining both constraints often yields further improvements.
Takeaways & Limitations
Raw RGB values can provide the minimal information needed to form analogies between related domains.
Takeaways & Limitations
Circularity is only approximate because mappings that discard target-domain-changing details, such as zebra stripes, cannot recover the exact input.
Abstract
from arXiv · showhide
In unsupervised domain mapping, the learner is given two unmatched datasets $A$ and $B$. The goal is to learn a mapping $G_{AB}$ that translates a sample in $A$ to the analog sample in $B$. Recent approaches have shown that when learning simultaneously both $G_{AB}$ and the inverse mapping $G_{BA}$, convincing mappings are obtained. In this work, we present a method of learning $G_{AB}$ without learning $G_{BA}$. This is done by learning a mapping that maintains the distance between a pair of samples. Moreover, good mappings are obtained, even by maintaining the distance between different parts of the same sample before and after mapping. We present experimental results that the new method not only allows for one sided mapping learning, but also leads to preferable numerical results over the existing circularity-based constraint. Our entire code is made publicly available at https://github.com/sagiebenaim/DistanceGAN .
1 Introduction
Unsupervised image translation ranges from paired supervision to fully unsupervised two-way mappings. This work introduces one-sided unsupervised mapping by enforcing cross-domain distance correlation, with optional within-image constraints.
- Problem and setting: Image translation maps a source-domain image to an analogous target-domain image using GAN-based generators.The generated image is conditioned on the input image rather than random noise.
- Prior approaches: Existing approaches use paired samples, semantic consistency, shared weights, or jointly learned mappings with circularity, depending on the available supervision.Circularity requires mapping a sample to the other domain and back to recover the original sample.
- Proposed approach: One-sided unsupervised mapping is learned by enforcing high correlation between matching pairwise distances in the source and target domains.The constraint avoids learning the inverse mapping and empirically produces better numerical results than circularity.
- Proposed approach: The distance constraint can compare pairs of samples or different parts of the same image, such as its top and bottom halves.Training batches already contain multiple samples, but per-image constraints can further reduce the information needed for alignment.
- Prior approaches: Unsupervised mapping methods require only samples from two domains, unlike supervised methods that require matched source-target pairs.Related approaches include one-sided reconstruction-based methods, CoGAN-style shared weights, and feature-constancy constraints.
2 Preliminaries
Unsupervised domain mapping learns an analogue-preserving function between unmatched domains, traditionally alongside an inverse mapper. The proposed one-sided approach constrains corresponding pairwise distances, including distances between parts of the same image, and is motivated by evidence that such distances correlate across domains.
- Problem: Unsupervised domain mapping learns G_AB from unlabeled samples in domains A and B, while prior approaches also learn the inverse G_BA.The target is an analogue sample in domain B for each source sample in domain A.
- Distance constraint: The method constrains pairs of inputs with similar distances to produce output pairs with similar distances, enabling one-directional mapping without learning G_BA.The constraint compares ∥x − x′∥ with ∥G_AB(x) − G_AB(x′)∥.
- Empirical justification: A nonnegative linear transformation approximates CycleGAN’s handbag-to-shoe mapper, with 99.5% of transformation elements below 0.01 and mean row sum 1.06 (SD 0.08).The transformation produces each output pixel as a linear combination of input pixels, suggesting that much of the mapping behaves like a fixed spatial transformation.
- Motivation: Pairwise distances can remain correlated because many mappings alter only limited image information, such as texture, gender, hair color, or glasses.Examples include horse-to-zebra texture changes and relatively minor facial modifications.
- Within-image distances: Distances between the left and right halves of the same image also show very significant correlation after translation for horses-to-zebras and Cityscapes image-label pairs.This tests whether successful alignment can use information from different parts of one sample rather than image pairs.
- Optimization: The proposed loss minimizes the sum of absolute distance differences, spreading error more uniformly than maximizing normalized distance correlation.Source-domain distances are fixed, so correlation maximization lets distances with large absolute values dominate optimization.
3 Unsupervised Constraints on the Learned Mapping
The section reviews unsupervised constraints for mapping between unmatched domains, then introduces distance-based and self-distance constraints to reduce ambiguity while enabling one-sided learning.
- Adversarial constraints: Adversarial training matches the translated source distribution to the target distribution, but permits many possible mappings between samples.The authors describe circularity, target-domain identity, and distance constraints as additional ways to restrict this space.
- Circularity constraints: Circularity requires translating a sample to the other domain and back to recover the original, typically using mappings in both directions.The circularity loss is usually applied symmetrically, while exact recovery can be unrealistic when translation removes domain-specific information such as zebra stripes.
- Distance constraints: The proposed distance-preserving map matches pairwise distances across domains up to scale, further reducing admissible mappings.Its loss uses the absolute difference between distances in the two domains; means and standard deviations of training-set distances are precomputed.
- Distance constraints: The distance constraint avoids the model-collapse problem described for circularity, where distinct source samples map to one target sample.Pairwise distance preservation prevents two different samples from domain A from collapsing to the same sample in domain B.
- Self-distance constraints: A self-distance constraint compares distances between two parts of one image, allowing one-sample-at-a-time training and addressing minibatch-size-one settings.The operators L and R return the left and right image halves, and analogous statistics are computed for domains A and B.
- Training objective: The combined objective can include adversarial, circularity, distance, and self-distance terms, but one-sided mapping activates the adversarial term and either distance-based term.The experiments use trade-off weights, do not jointly test distance and self-distance constraints, and employ DiscoGAN or CycleGAN architectures.
4 Experiments
Experiments compare distance-based, circularity-based, self-distance, and combined mappings across image-translation tasks using visual, numerical, descriptor, classifier, and user-study evaluations. Distance-based methods generally improve over baselines, while combining constraints is task-dependent.
- Experimental setup: The experiments compare DiscoGAN and CycleGAN baselines with one-sided distance mapping, combined constraints, and self-distance variants.The evaluations cover rotated cars and heads, CelebA attributes, shoes-to-handbags, horses-to-zebras, and SVHN-to-MNIST translation.
- DiscoGAN-based models: Lower RMSE is obtained with pairwise distance mapping than with DiscoGAN for car-to-car translation; combining constraints improves further, while self-distance performs best.The reported RMSE measures the difference between source and translated car angles.
- DiscoGAN-based models: Distance mapping outperforms the other methods for car-to-head translation, whereas combining distance and circularity is less competitive and self-distance is worse than circularity.The authors treat mirror and 90° rotation biases as mapping ambiguities and remove mean error before computing RMSE.
- CelebA: For CelebA, pairwise distance mapping has fewer visual failure cases than DiscoGAN, and self-distance also appears to outperform DiscoGAN.Self-distance uses distances between the top and bottom halves of face images.
- CelebA: CelebA pair-distance mappings produce more similar input-output faces by VGG cosine similarity, while combining pair-distance with circularity gives the best results but requires both mappings.Self-distance is slightly worse than pair-distance but generally better than DiscoGAN; separation accuracy remains similar to DiscoGAN.
- User study: In user studies, the proposed method is preferred for shoes-to-handbags in 65% of cases, handbags-to-shoes in 87%, and male-to-female fit in 88%.For male-to-female realness, the methods receive similar scores: 51% for the proposed method versus 49% for DiscoGAN.
- CycleGAN-based models: On horses-to-zebras, all methods fail for zebra-to-horse translation, although distance plus cycle appears most promising subjectively.The combined method does not consistently dominate across datasets and directions.
- Numerical evaluation: For SVHN-to-MNIST, pairwise distance improves over the baseline but remains below the semi-supervised result, and combining distance with circularity performs poorly.The authors attribute the remaining gap to irrelevant information in SVHN images.
5 Conclusion
The paper proposes an unsupervised distance-based loss for learning a single mapping without its inverse, and reports that it empirically outperforms circularity loss.
- The distance-based loss learns a single unsupervised mapping without learning its inverse.
- The loss is applied directly to raw RGB image values rather than image descriptors or low-layer network activations.
- The experiments show that the distance-based loss empirically outperforms the circularity loss.
A Experiments with the DiscoGAN architecture
Experiments with the DiscoGAN architecture evaluate losses on CelebA and show mappings across several visual transformations, including hair, eyeglasses, handbags, shoes, and edges.
- Table 5 reports eight losses for each method on the CelebA dataset.
- The distance loss is measured for DiscoGAN even though it is not part of DiscoGAN’s loss.
- The distance method is run in both directions at once to permit circularity computation.
- The image experiments use the same network architecture as DiscoGAN across multiple mapping experiments.
- The experiments include black-to-blond hair, eyeglasses-to-no-eyeglasses, handbags-to-shoes, and edges-to-shoes conversions, including reverse directions.
B Experiments with the CycleGAN architecture
Experiments with the CycleGAN architecture cover bidirectional horse–zebra translation, SVHN-to-MNIST digit translation, and label-to-cityscapes image translation.
- The CycleGAN experiments include horse-to-zebra and zebra-to-horse image translation.
- The experiments also present examples of transforming SVHN images into MNIST digits.
- The CycleGAN architecture is used for translation from labels to Cityscapes images.