Source-linked AI summary
Image to Image Translation for Domain Adaptation
Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, Kyungnam Kim
TL;DR
The paper addresses unsupervised domain adaptation when labeled source data must transfer to an unlabeled target domain affected by domain shift. It uses shared domain-agnostic features constrained by reconstruction and cycle-consistent image translation, and reports state-of-the-art performance across digit classification, Office classification, and driving-scene segmentation.
Problem
Unsupervised adaptation must transfer models from labeled source domains to unlabeled target domains despite differing data distributions and expensive target annotations.
Method
The framework learns domain-agnostic latent features while enforcing domain-specific reconstruction and cycle consistency through unpaired image-to-image translation.
Results
The method outperforms state of the art for digit classification and semantic segmentation, with further gains when combined with DenseNet.
Takeaways & Limitations
A unified framework can jointly perform image-to-image translation, source discrimination, and domain adaptation while encompassing many recent methods as special cases.
Takeaways & Limitations
For simple digit datasets, translation classification loss is insufficient because decoders can learn digit permutations; for large images, cycle consistency is computationally expensive.
Abstract
from arXiv · showhide
We propose a general framework for unsupervised domain adaptation, which allows deep neural networks trained on a source domain to be tested on a different target domain without requiring any training annotations in the target domain. This is achieved by adding extra networks and losses that help regularize the features extracted by the backbone encoder network. To this end we propose the novel use of the recently proposed unpaired image-toimage translation framework to constrain the features extracted by the encoder network. Specifically, we require that the features extracted are able to reconstruct the images in both domains. In addition we require that the distribution of features extracted from images in the two domains are indistinguishable. Many recent works can be seen as specific cases of our general framework. We apply our method for domain adaptation between MNIST, USPS, and SVHN datasets, and Amazon, Webcam and DSLR Office datasets in classification tasks, and also between GTA5 and Cityscapes datasets for a segmentation task. We demonstrate state of the art performance on each of these datasets.
1. Introduction
The paper frames unsupervised domain adaptation as transferring knowledge from labeled source data to unlabeled target data despite domain shift. It proposes a unified framework combining domain-agnostic features, domain-specific reconstruction, and cycle consistency, demonstrated across classification and segmentation tasks.
- Motivation: Domain shift arises because target-domain data follow a different distribution from the labeled source domain.This makes transferring learned representations to unlabeled or sparsely labeled target data difficult.
- Motivation: Synthetic-to-real segmentation suffers when models trained on GTA5 are applied to Cityscapes because image characteristics differ substantially.The paper uses autonomous-driving scenes to motivate the cost of dense pixel annotation and the need for adaptation.
- Proposed framework: The framework combines domain-agnostic feature extraction, domain-specific reconstruction, and cycle consistency to constrain unsupervised domain adaptation.Feature distributions should be indistinguishable across domains, while features must reconstruct images in both domains and support well-behaved mappings.
- Scope and contribution: The shared framework jointly supports image-to-image translation, source discrimination, and target-domain adaptation across digit classification and driving-scene segmentation.The authors also show that many existing domain-adaptation methods can be viewed as special cases of the framework.
2. Related Work
Related domain-adaptation methods align source and target representations using varied architectures and losses. The paper unifies these recurring ingredients into a generic framework that also addresses image-to-image translation.
- Existing approaches: Visual domain-adaptation methods commonly map source and target domains into a shared space where their distributions are aligned.Methods differ in architectures and losses, including MMD, correlation maximization, and adversarial confusion.
- Common design patterns: Prior work identifies source discrimination, domain-agnostic embeddings, reconstructability of domain-specific images, and adversarial learning as recurring design patterns.These patterns collectively motivate the framework proposed here.
- Proposed unification: The proposed method unifies these components into a generic framework that simultaneously addresses domain adaptation and image-to-image translation.This extends the shared-space perspective by requiring the embedding to preserve information sufficient for domain-specific reconstruction.
- Related unified frameworks: ADDA provides another unifying framework organized around generative versus discriminative bases, parameter sharing, and adversarial-training choices.Its authors observed that modeling image distributions may not be strictly necessary when embeddings are domain invariant.
3. Method
The framework learns a joint latent space for labeled source and unlabeled target images, regularizing it with classification, reconstruction, adversarial, translation, and cycle-consistency objectives. These components support domain-agnostic representations while preserving information needed for prediction and image reconstruction.
- Latent space: The method maps source and target images into a joint latent space Z using CNN encoders, with a classifier mapping latent representations to labels or annotations.The latent representation may be a high-dimensional vector for image-level tasks or a feature map for pixel-level tasks.
- Label prediction: Supervised classification loss trains source representations to predict known source annotations, including image-level classification or pixel-level segmentation labels.The classifier is trained through h(fx(xi)) = ci using an appropriate loss such as cross entropy.
- Domain-specific reconstruction: Decoders reconstruct source and target images from latent features, constraining Z to preserve core domain information while discarding structured domain noise.The source and target decoders map Z to X and Y, respectively, using a pixel-wise image loss such as L1.
- Domain-agnostic features: An adversarial latent-space discriminator distinguishes features from domains X and Y, while the encoders are regularized toward domain-agnostic representations that fool it.The discriminator uses binary domain labels to classify whether a latent feature originated from the source or target domain.
- Translation and cycle consistency: Translation adversarial losses classify cross-domain decoded images as real or fake, and cycle-consistency losses require translated images to return to their original domains through re-encoding and decoding.The framework combines these losses with the other objectives and optimizes them end-to-end using SGD with adaptive learning rates and alternating discriminator updates.
- Implementation: Weight sharing between encoders and early decoder layers further constrains learned features, while Improved Wasserstein and least-squares GAN variants stabilize domain discriminators.The least-squares GAN is used for the latent Z discriminator because both real and fake distributions change during training.
4. Experiments
Experiments evaluate the framework across digit, Office, and driving-scene domain adaptation tasks, showing strong performance and illustrating how feature alignment complements image translation.
- Digit datasets: The method is evaluated on MNIST, USPS, and SVHN using labeled source domains and unlabeled target domains.Experiments cover MNIST→USPS, USPS→MNIST, and SVHN→MNIST, with MNIST→SVHN translation examples shown.
- Digit datasets: The method consistently outperforms prior digit-adaptation methods, with DenseNet combined with the method significantly outperforming the prior state of the art.Comparisons use a modified LeNet encoder and also a deeper DenseNet encoder.
- Digit datasets: Full adaptation aligns source and target feature distributions, whereas image translation alone can map them to different latent regions.The full model combines image translation with a feature-distribution adversarial loss; translation also makes features richer.
- Office dataset: On the Office dataset, the method performs best on four of six domain-adaptation tasks.The two tasks where it is not best involve large domain shifts and little source training data.
- Driving-scene segmentation: For GTA5→Cityscapes semantic segmentation, the method outperforms the previous method on all but three categories and is 5% better overall.Using Dilated DenseNets increases the state of the art by 8.6%.
5. Implementation Details
Implementation combines pretrained convolutional encoders, domain-specific decoders, image and feature discriminators, and task-specific training choices across classification and segmentation settings.
- Digit implementation: Images from MNIST and USPS are upsampled to 32×32, SVHN is converted to grayscale, and all images are normalized to [−1, 1].
- Digit implementation: Digit experiments use a modified LeNet encoder, transposed-convolution decoders, convolutional image discriminators, and fully connected feature discriminators.A deeper DenseNet encoder was also tested and improved performance for all methods.
- Training design: For simple digit datasets, the translation classification loss is ineffective because decoders can learn digit permutations that reconstruct the original input.For example, a digit may be translated to another digit and then translated back.
- Segmentation implementation: For driving-scene segmentation, encoders use dilated ResNet34 or DenseNet121 with output stride 8 and ImageNet initialization.Additional dilation layers are used to remove checkerboarding artifacts.
- Training design: The translated classification loss is backpropagated only through the second encoding step to prevent encoders and decoders from hiding information in translated images.
6. Conclusion
The paper presents a unified unsupervised domain-adaptation framework that combines image translation, source discrimination, and domain adaptation, reporting state-of-the-art adaptation results.
- The framework encompasses many recent domain-adaptation methods as special cases.
- The implementation outperforms the state of the art for digit classification and semantic segmentation of driving scenes.With DenseNet, it significantly outperforms the current state of the art.