PaperlayerView exact revision on arXiv

Source-linked AI summary

Generate To Adapt: Aligning Domains using Generative Adversarial Networks

Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, Rama Chellappa

arXiv:1704.01705v4cs.CV

TL;DR

Unsupervised domain adaptation must address distribution shifts when labeled source data does not match an unlabeled target domain. The paper couples supervised classification with an adversarial generator-discriminator framework to learn a shared feature space, and reports strong performance across DIGITS, OFFICE, and synthetic-to-real settings. Its approach is designed to remain effective even when image generation quality is poor.

  • Problem

    Domain adaptation seeks predictors that work across source and target distributions using labeled source data and unlabeled target data, because domain-specific label distributions can harm performance on unseen data.

  • Method

    The method jointly learns an encoder-classifier embedding with a generator-discriminator pair, using supervised source labels and adversarial gradients from unlabeled target data.

  • Results

    The approach is reported to perform well across DIGITS, OFFICE, and synthetic-to-real adaptation, with experiments demonstrating broad applicability across three difficulty regimes.

  • Takeaways & Limitations

    Using adversarial image generation to shape the feature embedding can support domain adaptation without relying completely on successful generated images or standalone GAN-based augmentation.

  • Takeaways & Limitations

    The paper assumes labeled source data and unlabeled target data over a shared input space, and notes future work is needed for more challenging domains such as RGB-D recognition and medical imaging.

Abstract

from arXiv · show

Domain Adaptation is an actively researched problem in Computer Vision. In this work, we propose an approach that leverages unsupervised data to bring the source and target distributions closer in a learned joint feature space. We accomplish this by inducing a symbiotic relationship between the learned embedding and a generative adversarial network. This is in contrast to methods which use the adversarial framework for realistic data generation and retraining deep models with such data. We demonstrate the strength and generality of our approach by performing experiments on three different tasks with varying levels of difficulty: (1) Digit classification (MNIST, SVHN and USPS datasets) (2) Object recognition using OFFICE dataset and (3) Domain adaptation from synthetic to real data. Our method achieves state-of-the art performance in most experimental settings and by far the only GAN-based method that has been shown to work well across different datasets such as OFFICE and DIGITS.

1. Introduction

The paper addresses unsupervised visual domain adaptation by learning embeddings that reduce source–target distribution shift using unlabeled target data. Its joint generative-discriminative approach couples supervised classification with adversarial image generation and is evaluated across several adaptation settings.

  • Motivation: Unlabeled target data is used to mitigate mismatched label distributions that can make models trained on one domain perform poorly on unseen domains.The introduction highlights synthetic-to-real transfer as a setting where labeled real data may be unavailable.
  • Approach: The method learns a robust shared embedding by combining supervised source classification with an adversarial image-generation procedure driven by source and target data.The approach directly learns a joint feature space intended to reduce the distance between source and target distributions.
  • Approach: During training, source embeddings support label prediction and image generation, while unlabeled target embeddings receive updates through adversarial gradients because target labels are unavailable.The encoder is updated from classifier gradients for source data and generative gradients for both domains.
  • Approach: After training, classification uses only the encoder-classifier pair, with the adversarial stream removed from the test pipeline.The adversarial branch is a training mechanism rather than part of test-time prediction.
  • Results: The approach is reported to outperform related adaptation methods and to work across DIGITS, OFFICE, and synthetic-to-real experiments, including settings where standalone image generation can fail.The authors contrast direct feature-space adaptation with GAN-based data augmentation and retraining.

2. Related Work

Prior visual domain adaptation methods align representations using discrepancy losses, adversarial embedding losses, or pixel-space image translation. The paper distinguishes its approach by using GANs to provide adaptation gradients in feature space rather than relying on successful cross-domain image generation for augmentation.

  • Representation alignment: Earlier methods sought domain-invariant representations through feature reweighting, selection, or explicit transformations aligning source and target distributions.Deep learning approaches extend this goal by optimizing domain-discrepancy losses alongside the task loss.
  • Discrepancy-based methods: MMD-based methods minimize distribution discrepancy in task-specific layers or projected reproducing-kernel spaces, while related methods align joint distributions across multiple layers.Examples include DDC, DAN, RTN, and Joint Adaptation Networks.
  • Adversarial methods: Adversarial embedding methods train a domain classifier to distinguish source from target while encouraging the feature extractor to confuse it and preserve label-prediction performance.RevGrad implements this objective by reversing gradients from the domain classifier.
  • Pixel-space methods: Pixel-space approaches use GANs for cross-domain image mapping or generation, often treating generated images as a data-augmentation resource for adaptation.These methods differ from feature-space adversarial alignment.
  • Paper's distinction: The proposed method uses GAN-derived gradients to make embeddings domain adaptive, so it remains effective when image generation is difficult, such as on OFFICE with few samples per class.It therefore does not completely depend on successful image generation.

3. Approach

The approach jointly learns an embedding, classifier, generator, and discriminator for unsupervised domain adaptation, using labeled source data and unlabeled target data to align domains. Its adversarial framework transfers target-domain information into a feature space while preserving class consistency.

  • Problem formulation: Unsupervised domain adaptation learns a predictor from labeled source data and unlabeled target data, with an embedding F and classifier C modeled as deep networks.The embedding is intended to capture domain shift between source and target distributions during training.
  • Model design: The method modifies an Auxiliary Classifier GAN so generated images use encoder embeddings, random noise, and class labels, while target samples receive a fake-class encoding.The discriminator predicts both whether an image is real and its class distribution.
  • Optimization: The iterative procedure alternates updates to D, G, F, and C, with α and β balancing classification against source and target adversarial losses.The authors report that the approach is not overly sensitive to α and β, although their values depend on the application and dataset size.
  • Joint training: Source images are encoded for label prediction and generation, while the encoder receives both classifier gradients and adversarial gradients from the generator-discriminator framework.The generator is updated with adversarial and classification losses to produce realistic class-consistent source images.
  • Target adaptation: For unlabeled target images, the discriminator evaluates only the real-versus-fake objective, and its gradients update F to transfer target-distribution information into the embedding.This target update is intended to bring source and target distributions closer in the learned feature space.

4. Experiments and Results

Experiments evaluate the approach across digits, OFFICE, synthetic-to-real, VISDA, and component ablations, spanning increasing domain shift and dataset complexity. The method performs strongly across these settings, including cases where image generation is difficult.

  • Experimental settings: Three adaptation regimes—DIGITS, OFFICE, and synthetic-to-real—cover low, moderate, and high domain shifts with increasing data complexity.The paper characterizes complex datasets as having high image variability and limited samples.
  • DIGITS experiments: 32.1% performance gain improves SVHN→MNIST accuracy from 60.3% to 92.4% over the source-only model.The method also outperforms other methods by at least 10.4% in this setting.
  • OFFICE experiments: The OFFICE experiments obtain state-of-the-art performance across settings, including hard transfers A→W, A→D, W→A, and D→A.The dataset contains 31 classes across Amazon, Webcam, and DSLR domains, with 2817, 795, and 498 images respectively.
  • Synthetic-to-real experiments: 12.3 percentage points improve synthetic-to-real accuracy from 38.1% to 50.4%, while outperforming all compared methods.The experiment uses CAD renderings as source data and a PASCAL VOC subset as target data.
  • VISDA challenge: VISDA experiments show significant performance gains over the baseline model on a synthetic-to-real classification challenge.VISDA uses synthetic CAD renderings for training and real images from Microsoft COCO and YouTube Bounding Box datasets for adaptation and evaluation.
  • Ablation study: The auxiliary classifier C2 is needed for the full performance benefit beyond the improvement obtained from the real/fake classifier C1 alone.Without C2, the authors observed missing modes and mismatched mappings to images of incorrect classes.

5. Conclusion and Future Work

The paper addresses unsupervised visual domain adaptation with a joint adversarial-discriminative approach that transfers target-distribution information into the learned embedding. Experiments across three tasks support its generality and versatility.

  • The proposed approach uses a generator-discriminator pair to transfer target-distribution information into the learned embedding.
  • Experiments on three different tasks demonstrate the approach’s superiority over existing methods and broader applicability.
  • Future work includes stronger encoder architectures and more challenging applications such as RGB-D object recognition and medical imaging.

6. Network Architectures and Hyperparameters

The experiments use task-specific encoder, classifier, generator, and discriminator architectures, with pretrained feature extractors for OFFICE and synthetic-to-real adaptation. Hyperparameters vary across digit and non-digit settings.

  • Network architectures: The architecture descriptions and detailed network specifications are provided in Figure 3.
  • Digits experiments: Digit experiments use DigF, DigC, DigG, and DigD architectures, with models trained from scratch using Adam optimization.SVHN → MNIST uses DigF1 and DigC1; other digit experiments use DigF2 and DigC2.
  • OFFICE experiments: OFFICE experiments use a pretrained ResNet50 encoder truncated to a 2048-dimensional embedding, alongside task-specific classifier, generator, and discriminator networks.Adam uses learning rate 0.0004 and momentum 0.7; the noise dimension is 128.
  • Synthetic to Real experiments: Synthetic-to-real experiments use a pretrained VGG16 encoder truncated to a 4096-dimensional embedding and reuse OFFICE hyperparameter settings.The classifier, generator, and discriminator use SynC, OsG, and OsD architectures, respectively.

7. Noise Analysis

The noise analysis studies how generator noise dimensionality affects transfer accuracy for SVHN → MNIST across training epochs. Performance is broadly stable, with extreme dimensions slightly less effective.

  • Noise formulation: The generator input concatenates the feature embedding, noise vector z, and one-hot class label l.The noise vector is sampled from N(0, 1) and has dimensionality d.
  • Noise dimensionality study: Transfer accuracy for SVHN → MNIST is evaluated across training epochs while varying noise dimensionality over {32, 64, 128, 256, 512}.
  • Findings: The approach is not overly sensitive to noise dimensionality because all tested values achieve similar average performance.

8. Generation visualization

Visualization compares generated images in digit and OFFICE settings, using source and target inputs. Digits yield better generation quality, while OFFICE generations remain class-consistent but exhibit mode collapse.

  • Visualization setup: Figure 5 compares generated images from SVHN → MNIST and Office A → W experiments using source and target inputs.The top set uses source-dataset samples, while the bottom set uses target-dataset images.
  • Observed generation behavior: Generation quality is better in the digit experiments than in the OFFICE experiments.
  • Observed generation behavior: The generator produces source-like images for both source and target inputs in a class-consistent manner.
  • Observed generation behavior: OFFICE generations exhibit mode collapse, yet the method remains effective despite severe mode collapse and poor generation quality.

9. Synthetic to Real adaptation with ResNet

This synthetic-to-real experiment evaluates the method with a ResNet-50-initialized feature network and reports accuracy over five independent runs. The method reaches 46.5% performance, exceeding the VGG16 baseline by 16.3%.

  • The feature network was initialized with a pretrained ResNet-50 model trained on ImageNet.
  • Accuracy was reported as mean ± standard deviation over five independent runs on the synthetic-to-real dataset.
  • 30.2% performance was achieved by the model trained only on the source domain.
  • 46.5% performance was achieved by the proposed method, 16.3% above the VGG16 baseline and better than the other compared approaches.
Loading 1704.01705v4…