PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, Dilip Krishnan

arXiv:1612.05424v2cs.CV

TL;DR

Creating large, well-annotated image datasets is prohibitively expensive, motivating synthetic training data. PixelDA uses a GAN to adapt source images to the target domain without paired examples, and outperforms previous work across unsupervised domain adaptation scenarios.

  • Problem

    Creating large, well-annotated computer-vision datasets is prohibitively expensive, motivating synthetic data as an alternative for model training.

    Source: §1 · p. 1

  • Method

    PixelDA uses a GAN to transform source-domain images to appear target-domain-like while maintaining content, without paired examples from the two domains.

    Source: §1 · p. 2

  • Results

    PixelDA outperforms previous work across unsupervised adaptation scenarios; on Synthetic Cropped Linemod to Cropped Linemod, it more than halves pose-estimation error versus the previous best.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Pixel-level adaptation can be separated from the task-specific inference architecture, allowing that architecture to be changed without retraining the adaptation component.

    Source: §1 · p. 2

    The evaluation excludes Office, a commonly used visual domain-adaptation dataset for classification, because of significant high-level variations attributed to label pollution.

    Source: §4 · p. 5

Abstract

from arXiv · show

Collecting well-annotated image datasets to train modern machine learning algorithms is prohibitively expensive for many tasks. One appealing alternative is rendering synthetic data where ground-truth annotations are generated automatically. Unfortunately, models trained purely on rendered images often fail to generalize to real images. To address this shortcoming, prior work introduced unsupervised domain adaptation algorithms that attempt to map representations between the two domains or learn to extract features that are domain-invariant. In this work, we present a new approach that learns, in an unsupervised manner, a transformation in the pixel space from one domain to the other. Our generative adversarial network (GAN)-based method adapts source-domain images to appear as if drawn from the target domain. Our approach not only produces plausible samples, but also outperforms the state-of-the-art on a number of unsupervised domain adaptation scenarios by large margins. Finally, we demonstrate that the adaptation process generalizes to object classes unseen during training.

1. Introduction

Synthetic training data is costly to annotate, and models trained on it often fail to generalize to real images. PixelDA learns an unpaired pixel-space transformation that makes source images resemble the target domain while preserving their content.

Introduction

  • PixelDA uses a GAN to transform source images into target-like images while preserving content, without requiring paired examples from the two domains.Source: §1 · p. 2
  • Unlike task-coupled adaptation, PixelDA lets users change the inference architecture without retraining adaptation and supports target test labels that differ from training labels.Source: §1 · p. 2
  • A task-specific loss on source and generated images plus pixel-similarity regularization helps prevent mode collapse and reduce performance variance across random initializations.Source: §1 · p. 2
  • Conditioning on source images and stochastic noise enables virtually unlimited target-like samples, while PixelDA’s adapted images are more interpretable than adapted feature vectors.Source: §1 · p. 2
  • PixelDA outperforms state-of-the-art unsupervised domain adaptation methods on object classification and pose estimation across multiple datasets, while generating target-like images.The evaluated tasks involve objects in the foreground of source- and target-domain images.Source: §1 · p. 2

2. Related Work

The review centers on CNN-based unsupervised domain adaptation, contrasting feature-alignment approaches with this paper’s pixel-space transformation of source images to resemble the target domain. It also distinguishes the method from image-conditioned GANs and style-transfer work.

Related Work

  • The review focuses on CNN methods, which prior work found empirically superior for unsupervised domain adaptation [14] [31] [41] [45].Source: §2 · p. 2
  • DANNs [13] [14] and [3] learn domain-invariant features adversarially, whereas this method transforms source images to resemble the target domain.Tzeng et al. [45] and Long et al. [31] replace domain-loss maximization with MMD minimization; Ghifary et al. [?] combine source task loss with target reconstruction, while Bousmalis et al. [5] separate shared and domain-private components using reconstruction, similarity, and difference losses.Source: §2 · p. 2
  • Feature-level mapping methods keep the feature-extraction pipeline fixed during adaptation, including CNN-based CORAL [41] and earlier non-CNN approaches.Source: §2 · p. 2
  • Ledig et al. [28] and Yoo et al. [47] use image-conditioned GANs for super-resolution and clothing-image generation, unlike this method’s image-and-noise conditioning and domain-adaptation task.Source: §2 · p. 3
  • Liu and Tuzel [30] use coupled GANs with shared generator and discriminator weights to generate corresponding image pairs for unsupervised adaptation.Style transfer by Gatys et al. [15] and Johnson et al. [24] targets the style of a single image, whereas this work models an entire image domain.Source: §2 · p. 3

3. Model

PixelDA adapts labeled source images to look like target-domain images, separating pixel-level adaptation from task-specific classification. Its training combines adversarial domain matching with task supervision, and can use a foreground content-similarity loss when masks are available.

3. Model

  • PixelDA transforms labeled source images into target-like samples independently of task-specific classification, allowing an off-the-shelf classifier to train on adapted data.The method assumes domain differences are primarily low-level, such as noise, resolution, illumination, and color, rather than object types or geometry.Source: §3 · p. 3

3.1. Learning

  • The generator conditions on a source image and noise, while a discriminator distinguishes generated from target images and a classifier assigns task-specific labels.Unlike the standard GAN formulation, the generator uses both a noise vector and a source image as input.Source: §3.1 · p. 3
  • Training the classifier on both source and adapted images prevents class-label shifts and stabilizes training.Training on adapted images alone can achieve similar performance, but may require many runs with different initializations because of model instability.Source: §3.1 · pp. 3–4
  • The generator and discriminator are convolutional networks, and training alternates between updating the discriminator and classifier, then updating the generator.The generator uses residual connections and maintains the original image resolution.Source: §3.1 · p. 4

3.2. Content–similarity loss

  • For rendered single-object data, z-buffer masks enable a foreground-only content loss that anchors generated images and stabilizes minimax training.The masked pairwise mean squared error (PMSE) [11] penalizes pixel-pair differences rather than absolute input-output differences; it preserves overall shape without fixing color or intensity, while encouraging consistent foreground changes.Source: §3.2 · pp. 4–5

4. Evaluation

The evaluation tests PixelDA across digit and synthetic-to-real object adaptation, using qualitative image inspection and quantitative task performance. It reports gains on benchmark tasks and examines robustness, generalization, training stability, and semi-supervised use.

4. Evaluation

  • Evaluation covers MNIST→USPS, MNIST→MNIST-M, and synthetic-to-real LineMod, comparing target-test performance with source-only and target-only baselines; Office is omitted because of label pollution.For quantitative comparisons, methods share the same task architecture and use about 1,000 labeled target examples for hyperparameter validation.Source: §4 · p. 5, §4.2 · p. 6

4.1. Implementation Details

  • G is a resolution-preserving convolutional residual network conditioned on 10-dimensional uniform noise; experiments use batches of 32, and task CNNs match prior protocols [14] [5].The discriminator is a convolutional network whose depth depends on image resolution.Source: §4.1 · p. 6

4.2. Quantitative Results

  • PixelDA beats prior work on MNIST→MNIST-M and reaches state-of-the-art on MNIST→USPS; on synthetic-to-real LineMod, it more than halves the prior state-of-the-art mean angle error.The digit comparisons are classification accuracy; the LineMod comparison is pose error. Tables 1 and 2 report the quantitative results.Source: Table 1 · p. 6, Table 2 · p. 7, §4.2 · p. 7

4.3. Qualitative Results

  • Generated MNIST-M digits acquire varied backgrounds and learn the digit-mask color inversion, while adapted LineMod images gain realistic RGB backgrounds, adjusted foreground photometrics, and plausible depth noise.The qualitative inspection compares generated samples with their nearest neighbors in the target training set.Source: §4.3 · p. 7

4.4. Model Analysis

  • PixelDA improves over source-only training with black or random ImageNet RGB backgrounds, and nearest-neighbor checks show generated LineMod images are not copies of target training images.Background experiments use RGB images only because equivalent backgrounds are not available for the depth channel.Source: Figure 4 · p. 6, §4.4 · p. 7, Table 3 · p. 7
  • Training on half of LineMod’s object classes lets the model generalize to the unobserved classes at test time.The model is trained on six of the 11 objects, then evaluated on the objects excluded from training.Source: §4.4 · p. 8, Table 4 · p. 8
  • Task and content-similarity losses do not improve overall performance, but reduce run-to-run variability across ten training runs.The study measures variability as the standard deviation of performance across runs with different random initializations and the same hyperparameters.Source: §4.4 · p. 8, Table 5 · p. 8

4.5. Semi-supervised Experiments

  • With 1,000 labeled target examples, the semi-supervised model outperforms both target-only and synthetic-plus-target baselines, and also exceeds its fully unsupervised performance.The labeled examples are added as extra training data for classifier T.Source: Table 6 · p. 8, §4.5 · p. 8

5. Conclusion

PixelDA uses a GAN-based approach for unsupervised domain adaptation, outperforming previous work across several scenarios and more than halving pose-estimation error in a challenging synthetic-to-real setting. It is stabilized by task-specific and content-similarity losses, decouples adaptation from task architecture, and visualizes adapted outputs.

5. Conclusion

  • In Synthetic Cropped Linemod to Cropped Linemod, PixelDA more than halves pose-estimation error versus the previous best result and outperforms prior work across multiple adaptation scenarios.The method uses a GAN-based technique stabilized by task-specific and novel content-similarity losses.Source: §5 · p. 8
  • PixelDA decouples domain adaptation from the task-specific architecture.The conclusion also notes that visualizations of adapted image outputs make the model easy to understand.Source: §5 · p. 8

Supplementary Material · A. Additional Generated Images

Linear interpolation between random noise vectors on MNIST-M shows that the model separates style from content.

A. Additional Generated Images

  • In MNIST-M, each row keeps the same digit while each column keeps the same noise vector, demonstrating style-content separation through linear interpolation.The figure shows interpolation between two random noise vectors.Source: Figure 1 · p. 11

B. Model Architectures and Parameters

The section describes the generator and discriminator design choices and gives shared hyperparameters for reproducing the experiments.

B. Model Architectures and Parameters

B.1 USPS Experiments

The USPS experiments reuse the MNIST-M generator and discriminator, set the discriminator, generator, and task-classifier loss weights to 1.0, and omit similarity loss between synthetic and generated images.

B.1 USPS Experiments

  • The USPS experiments use the same Generator and Discriminator as the MNIST-M experiments.Source: §B.1 · p. 12
  • The discriminator, generator, and task-classifier losses in the generator step each have weight 1.0.Source: §B.1 · p. 12
  • The USPS experiments include no similarity loss between synthetic and generated images.Source: §B.1 · p. 12

B.2 MNIST-M Experiments (Paper Table 1)

The MNIST-M experiments use a residual-block generator and a four-layer convolutional discriminator, with specified weights for discriminator, generator, and task-classifier losses.

B.2 MNIST-M Experiments (Paper Table 1)

  • The generator uses 6 residual blocks with 64 filters each, while the discriminator has 4 convolutions with 64, 128, 256, and 512 filters.The discriminator has the same overall structure as Figure 2.Source: §B.2 · p. 12
  • The discriminator loss weight is 0.13 and the generator loss weight is 0.011.Source: §B.2 · p. 12
  • The task-classifier loss weight in the generator step is 0.01, and there is no similarity loss between synthetic and generated images.Source: §B.2 · p. 12

B.3 LineMod Experiments · C. InfoGAN Connection

The task-classification objective can be viewed as variational mutual-information maximization in the spirit of InfoGAN, while LineMod experiments specify model, runtime, and masked versus unmasked training configurations.

B.3 LineMod Experiments

  • LineMod domain transfer took 30 ms per example on one K80 GPU, averaged over 1,000 runs; experiments used 10 TensorFlow workers.The generator used four residual blocks with 64 filters each, the discriminator followed Figure 2, and dropout keep probability was 35%.Source: §B.3 · p. 12
  • The base learning rate was 2.2e−4, decayed by 0.75 every 95,000 steps; discriminator and generator loss weights were 0.004 and 0.011, respectively.Task classification and pose loss weights were 1.0 and 0.2, respectively.Source: §B.3 · p. 13
  • Without masked loss, the task-classifier loss weight during the generator step was 0, the classifier was not trained on synthetic images, and no similarity loss was used.Source: §B.3 · p. 13
  • The base learning rate was 2.6e−4, decayed by 0.75 every 95,000 steps; discriminator and generator loss weights were 0.0088 and 0.011.Task classification and pose loss weights were 1.0 and 0.29, respectively.Source: §B.3 · p. 13
  • With masked loss, the task-classifier loss weight during the generator step was 0, the classifier was not trained on synthetic images, and MPSE loss weight was 22.9.Source: §B.3 · p. 13

C. InfoGAN Connection

  • The task-classification objective variationally maximizes mutual information between predicted classes and generated and equivalent source images, akin to InfoGAN [7].The classification loss can be rewritten using the conventions of [7].Source: C. InfoGAN Connection · p. 13
  • Maximizing that mutual information regularizes adaptation to preserve class-specific appearance and source-image content, preventing objects from being transformed to resemble another class.The classifier T approximates the posterior distribution used in the mutual-information formulation.Source: C. InfoGAN Connection · p. 13

D. Deep Reconstruction-Classification Networks

The section reports higher MNIST →USPS accuracy than Ghifary et al. [?] and describes task classifiers with shared and private parameters.

D. Deep Reconstruction-Classification Networks

  • 95.9% accuracy on MNIST →USPS, versus Ghifary et al.’s [?] 91.80%.Attempts to reproduce Ghifary et al.’s results using their published code and the authors’ own implementation did not achieve comparable performance.Source: D. Deep Reconstruction-Classification… · p. 13
  • Task classifiers use shared parameters and image-set-specific private parameters, with private layers accommodating different channel counts.The MNIST-M classifier is used for USPS, MNIST, and MNIST-M; at test time, the target classifier combines shared parameters with private parameters from the generated-images classifier.Source: Figure 3 · p. 15

Related papers

Loading 1612.05424v2…