Source-linked AI summary
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, Dilip Krishnan
TL;DR
Synthetic training data is inexpensive to annotate but often fails to generalize to real images. PixelDA addresses this gap with an unsupervised GAN-based pixel transformation that preserves source content while making images target-domain-like, outperforming prior methods across adaptation scenarios and generalizing to unseen object classes.
Problem
Models trained on rendered images often fail to generalize to real images, while collecting large well-annotated real-image datasets is prohibitively expensive.
Method
PixelDA uses an unsupervised GAN-based pixel-space transformation to make source images resemble the target domain while maintaining their original content.
Results
PixelDA outperforms previous work across unsupervised domain-adaptation scenarios, including more than halving pose-estimation error in the challenging Synthetic Cropped Linemod to Cropped Linemod scenario.
Takeaways & Limitations
PixelDA decouples domain adaptation from task-specific architectures, supports differing test-time label spaces, and generalizes adaptation to object classes unseen during training.
Takeaways & Limitations
The evaluation excludes the Office dataset because of significant high-level variations attributed to label pollution, and hyperparameter optimization lacks a universally applicable procedure.
Abstract
from arXiv · showhide
Collecting well-annotated image datasets to train modern machine learning algorithms is prohibitively expensive for many tasks. One appealing alternative is rendering synthetic data where ground-truth annotations are generated automatically. Unfortunately, models trained purely on rendered images often fail to generalize to real images. To address this shortcoming, prior work introduced unsupervised domain adaptation algorithms that attempt to map representations between the two domains or learn to extract features that are domain-invariant. In this work, we present a new approach that learns, in an unsupervised manner, a transformation in the pixel space from one domain to the other. Our generative adversarial network (GAN)-based method adapts source-domain images to appear as if drawn from the target domain. Our approach not only produces plausible samples, but also outperforms the state-of-the-art on a number of unsupervised domain adaptation scenarios by large margins. Finally, we demonstrate that the adaptation process generalizes to object classes unseen during training.
1. Introduction
Synthetic data offers automatically generated annotations but often transfers poorly to real images, motivating unsupervised domain adaptation. PixelDA learns an unsupervised pixel-space transformation that preserves source content while producing target-domain-like images, with added flexibility and strong results.
- Motivation: Synthetic datasets can provide virtually unlimited labeled data, but models trained on synthetic images often fail to generalize to real images.This motivates transferring knowledge from labeled source data to unlabeled target data.
- Approach: PixelDA uses a GAN-based architecture to transform source images into target-domain-like images without corresponding source–target image pairs.The transformation is learned in pixel space while maintaining the original image content.
- Advantages: PixelDA maps images at the pixel level, allowing the task-specific architecture to change without retraining the domain-adaptation component.This decouples adaptation from the inference architecture.
- Advantages: PixelDA handles test-time target label spaces that differ from the label space used during training.Previous task-coupled approaches constrain source and target label spaces to match.
- Data Augmentation: PixelDA can generate virtually unlimited stochastic target-like samples by conditioning on source images and noise vectors.This extends beyond adaptation from a finite set of source and target examples.
- Results: Across object classification and pose-estimation scenarios, PixelDA outperforms prior unsupervised adaptation methods while generating images visually similar to the target domain.The adapted pixel outputs are also more interpretable than domain-adapted feature vectors.
2. Related Work
Prior unsupervised domain-adaptation work largely learns domain-invariant features or feature-level mappings. Related GAN approaches generate images or corresponding pairs, but PixelDA conditions on source images and noise for pixel-level adaptation.
- Unsupervised Domain Adaptation: Unsupervised domain adaptation remains an open theoretical and practical problem, with CNN methods emphasized because of their empirical performance.The review focuses primarily on CNN-based approaches.
- Unsupervised Domain Adaptation: DANN learns domain-invariant features by jointly using task classification and adversarial domain classification objectives.Shared layers are optimized to classify source tasks while confusing the domain classifier.
- Feature-Level Adaptation: Feature-level adaptation methods learn mappings between domains while keeping the feature-extraction pipeline fixed during optimization.CORAL is a recent CNN-based example of this approach.
- Generative Adversarial Networks: Image-conditioned GANs have been used for super-resolution and clothing generation, with clothing generation trained from corresponding image pairs.These methods differ from PixelDA in their conditioning or pairing requirements.
- Generative Adversarial Networks: Coupled GANs generate corresponding source–target image pairs by sharing selected generator and discriminator weights across domains.This is described as the work most similar to PixelDA.
3. Model
PixelDA transforms labeled source images into target-style images with a GAN while preserving task-relevant content, then trains a task-specific classifier on the adapted data. The design combines adversarial, task, and content-similarity objectives, while keeping domain adaptation decoupled from classifier architecture.
- Model: PixelDA decouples pixel-level adaptation from the task-specific classifier, allowing the classifier architecture to change without retraining the adaptation component.The classifier’s architecture is omitted from the general model description because it varies by task.
- Model: PixelDA assumes domain differences are primarily low-level, such as noise, resolution, illumination, and color, rather than object types or geometry.This assumption defines the setting in which pixel-level transformation is intended to operate.
- Model: PixelDA maps a source image and noise vector to an adapted image that appears drawn from the target domain while retaining its source label.The generated dataset can be created at any size and used to train the task-specific classifier.
- Learning: The discriminator distinguishes target-domain images from generated images, while the generator is trained through a generative adversarial objective.The model alternates discriminator and task-parameter updates with generator updates.
- Learning: The task-specific classifier is trained on both adapted and non-adapted source images to preserve label assignments and stabilize training.Training only on adapted images can allow class assignments to shift while still optimizing the objective.
- Content–similarity loss: The content-similarity loss penalizes differences between generated and source foreground pixels, encouraging consistent object shape while allowing appearance changes.The masked PMSE compares pixel pairs in the foreground and avoids spending modeling capacity on absolute color or intensity.
4. Evaluation
The evaluation spans digit and LineMod domain-adaptation scenarios, combining quantitative benchmarks with qualitative checks of pixel transformation, background robustness, memorization, and unseen-class generalization.
- Evaluation setup: The evaluation covers MNIST, MNIST-M, USPS, and synthetic-to-real LineMod scenarios for object classification and pose estimation.The study uses qualitative and quantitative evaluation across several unsupervised domain-adaptation settings.
- Quantitative results: PixelDA outperforms prior work on MNIST to MNIST-M and MNIST to USPS, and more than halves mean angle error on Synthetic Cropped Linemod to Cropped Linemod.The reported comparisons include better performance than previous work, state-of-the-art results, and a pose-error reduction relative to the previous state of the art.
- Qualitative results: Qualitative results show target-like backgrounds and foreground photometric adaptation, while depth images receive a plausible noise model.For MNIST-M, the model generates varied backgrounds and learns the inversion process; for LineMod, it adapts RGB appearance and depth noise.
- Model analysis: With black or random ImageNet backgrounds, PixelDA improves over Source-only models in RGB-only experiments.The background-variation experiments use random ImageNet crops and omit depth because equivalent background images are unavailable.
- Model analysis: Nearest-neighbor comparisons indicate that generated images differ from target training images, suggesting the model does not simply memorize the target dataset.The procedure compares generated samples with pixel-space nearest neighbors from the target training set.
- Model analysis: Training on six of eleven LineMod object classes is used to evaluate performance on the five classes withheld from training.After fixing the generator, the classifier is evaluated on 6,060 samples from unobserved objects and on the full target test set.
5. Conclusion
PixelDA is presented as a state-of-the-art unsupervised domain adaptation method that outperforms prior work across scenarios, including substantially reducing pose-estimation error in a challenging setting.
- PixelDA outperforms previous work across a set of unsupervised domain adaptation scenarios.
- PixelDA more than halves pose-estimation error versus the previous best result on Synthetic Cropped Linemod to Cropped Linemod.
- GAN-based training is stabilized with task-specific and content-similarity losses.
- PixelDA decouples domain adaptation from the task-specific architecture and supports visualization through adapted image outputs.
A. Additional Generated Images
Interpolation between random noise vectors shows that the model separates style from content in MNIST-M images.
- Linear interpolation between random noise vectors demonstrates style–content separation in MNIST-M.Rows retain the same digit, while columns share the same noise vector.
B. Model Architectures and Parameters
The experiments use convolutional generator and discriminator architectures with alternating optimization and shared regularization and noise settings.
- The generator and discriminator architectures vary filters, residual blocks, regularization, and layer counts across experiments.
- Training alternates discriminator and task-classifier optimization with generator optimization.
- The default setup decays learning rate by 0.95 every 20,000 steps and uses 3x3 convolutional kernels.
- The discriminator receives Gaussian noise after every layer and uses dropout with 90% keep probability.
- The 10-dimensional input noise vector is sampled uniformly from U(−1, 1).
- Several architectural conventions follow the DCGAN paper.
- All parameters use L2 weight decay of 1e−5, Leaky ReLU leakiness 0.2, Gaussian initialization with standard deviation 0.02, and ADAM β1 = 0.5.
B.1 USPS Experiments
The USPS experiments use equal unit weights for discriminator, generator, and task-classifier losses, without a similarity loss between synthetic and generated images.
- The generator and discriminator are identical to the MNIST-M experiments in the USPS setting.
- The discriminator loss weight is 1.0.
- The generator loss weight is 1.0.
- The task-classifier loss weight in the generator step is 1.0.
- No similarity loss is used between synthetic and generated images.
B.2 MNIST-M Experiments (Paper Table 1)
The MNIST-M experiments use a residual-block generator and convolutional discriminator, with weighted adversarial and task-classification losses and no similarity loss.
- The generator uses 6 residual blocks with 64 filters each, while the discriminator uses four convolutions with 64, 128, 256, and 512 filters.
- The discriminator, generator, and task-classifier loss weights are 0.13, 0.011, and 0.01, respectively.
- The model uses no similarity loss between synthetic and generated images.
B.3 LineMod Experiments
The LineMod experiments evaluate a residual-block GAN with task-related objectives, masked and unmasked parameter settings, and measured single-image transfer speed.
- Experimental setup: Domain transfer takes 30 ms per LineMod example on a single K80 GPU, averaged over 1,000 runs.All experiments use a cluster of 10 TensorFlow workers.
- Architecture: The LineMod generator has 4 residual blocks with 64 filters each, while its discriminator follows the structure depicted in Figure 2 and uses 35% dropout keep probability.
- Task objective: When T is a classifier, optimizing the task loss variationally maximizes mutual information between predicted class and equivalent source and generated samples.The classifier T approximates the posterior distribution in this formulation.
D. Deep Reconstruction-Classification Networks
The reported MNIST-to-USPS comparison gives 95.9% accuracy for the method versus 91.80% for Ghifary et al., while the figures show generated samples, nearest neighbors, and task-classifier architectures.
- 95.9% accuracy is reported on MNIST → USPS, compared with 91.80% for Ghifary et al.The authors report being unable to reproduce comparable performance using published code and their own implementation.
- Figure 2 presents generated images and depth channels in its left four columns, with corresponding L2 nearest neighbors in the right four columns.
- Figure 3 depicts task-classifier architectures for each dataset and uses the same task classifiers as prior work for fair comparisons.The MNIST-M classifier is used for USPS, MNIST, and MNIST-M classification.
- At test time, the target-domain classifier combines shared parameters with private parameters belonging to the generated-images classifier.The private layers allow high-level layers to be shared when synthetic and generated images have different channels.