PaperlayerView exact revision on arXiv

Source-linked AI summary

From source to target and back: symmetric bi-directional adaptive GAN

Paolo Russo, Fabio Maria Carlucci, Tatiana Tommasi, Barbara Caputo

arXiv:1705.08824v2cs.CV

TL;DR

Unsupervised domain adaptation must exploit labeled source images when target annotations are unavailable, while prior inverse mappings did not generate target-to-source images. SBADA-GAN jointly learns symmetric image transformations with self-labeling and class consistency, integrates two classifiers at test time, and exceeds prior state-of-the-art results in four of six settings. The paper reports realistic reconstructions and identifies complementary directional features, with self-labeling helping under moderate domain shifts.

  • Problem

    Unsupervised adaptation lacks annotated target data, and prior inverse mappings operated through adapted feature spaces without generating new source-domain images.

  • Method

    SBADA-GAN jointly learns source-to-target and target-to-source image transformations with adversarial and classification losses, target self-labeling, and a class consistency constraint.

  • Results

    SBADA-GAN produces realistic reconstructions and exceeds previous state-of-the-art classifiers in four of six unsupervised adaptation settings.

  • Takeaways & Limitations

    Bidirectional mapping avoids choosing one adaptation direction beforehand, while the two classifiers provide complementary predictions and self-labeling helps under moderate domain shifts.

  • Takeaways & Limitations

    The generative process does not explicitly reduce overall distribution shift, and methods such as DAass, DTN, and DSN perform better on the large-gap SVHN→MNIST setting.

Abstract

from arXiv · show

The effectiveness of generative adversarial approaches in producing images according to a specific style or visual domain has recently opened new directions to solve the unsupervised domain adaptation problem. It has been shown that source labeled images can be modified to mimic target samples making it possible to train directly a classifier in the target domain, despite the original lack of annotated data. Inverse mappings from the target to the source domain have also been evaluated but only passing through adapted feature spaces, thus without new image generation. In this paper we propose to better exploit the potential of generative adversarial networks for adaptation by introducing a novel symmetric mapping among domains. We jointly optimize bi-directional image transformations combining them with target self-labeling. Moreover we define a new class consistency loss that aligns the generators in the two directions imposing to conserve the class identity of an image passing through both domain mappings. A detailed qualitative and quantitative analysis of the reconstructed images confirm the power of our approach. By integrating the two domain specific classifiers obtained with our bi-directional network we exceed previous state-of-the-art unsupervised adaptation results on four different benchmark datasets.

1. Introduction

SBADA-GAN treats source-to-target and target-to-source image mappings as complementary, jointly combining adversarial adaptation, self-labeling, and class consistency. Its two classifiers are integrated at test time, and the architecture exceeds prior state-of-the-art results in four of six adaptation settings.

  • Motivation: Unsupervised adaptation is difficult because labeled source data may coexist with an entirely unannotated target domain.Existing approaches commonly reduce source-target distribution shift or map source images into the target domain.
  • Approach: SBADA-GAN unifies source-to-target and target-to-source image transformations because either direction may be easier depending on the application.The paper argues that the two directions are complementary rather than alternative, producing a more robust and general system.
  • Approach: The architecture uses adversarial losses for both transformations and classification losses on original source images and transformed target-like source images.These losses encourage images to acquire the other domain’s style while retaining supervised source classification.
  • Approach: A class consistency loss requires source images mapped to the target and back to return to their ground-truth class.Unlike standard reconstruction, it constrains class identity rather than image appearance and helps align the two generators.
  • Inference: At test time, classifiers for the two directions predict from original target images and source-like transformed target images, then integrate their predictions.The two models are trained on complementary domain-specific inputs.
  • Results: Four of six unsupervised adaptation settings exceed previous state-of-the-art classifiers while the architecture produces realistic image reconstructions.An ablation study also examines component importance and hyperparameter robustness.

2. Related Work

Prior work includes GAN foundations, distribution- and reconstruction-based domain adaptation, adversarial feature alignment, and image generation for transferring visual style. These approaches motivate SBADA-GAN’s bidirectional image-level adaptation and pseudo-labeling design.

  • GANs: GANs combine a generator, which synthesizes samples resembling real data, with a discriminator, which distinguishes real from generated samples.Conditional GANs additionally provide class annotations to both modules.
  • Domain Adaptation: Domain adaptation methods commonly minimize source-target distribution differences, reconstruction errors, or inconsistencies among similarly related target samples.Other work associates source and target samples by class or assigns target pseudo-labels through asymmetric tri-training.
  • Domain Adaptation: Adversarial feature-based methods treat domain invariance as binary classification and jointly optimize domain and source classification objectives.Some alternatives separate domain-specific and shared feature components or recombine objectives in a second step.
  • Image Generation for Domain Adaptation: Image-generation methods synthesize samples that preserve content while adopting a reference style, including GAN-based adaptation of source images toward target-domain appearance.Generated data can serve as training input for classifiers evaluated on real images.

3. Method

SBADA-GAN performs unsupervised cross-domain classification with symmetric source-to-target and target-to-source image transformations, classifiers, discriminators, self-labeling, and class-consistency training. At test time, it combines predictions from classifiers applied to original and source-like target images.

  • Bidirectional architecture: SBADA-GAN maps source images toward the target domain and target images toward the source domain using symmetric generators, discriminators, and classifiers.The source-to-target branch trains Ct on transformed source images, while the target-to-source branch uses Cs on source data and transformed target images.
  • Target-to-source adaptation: The target-to-source branch uses Cs to pseudo-label source-like transformed target images, then backpropagates their self-labeling loss to improve Gts.The pseudo-labeled images are used transductively by Cs, and the resulting classification feedback guides the target-to-source generator.
  • Class consistency: Class consistency requires a source image transformed through Gst and then Gts to retain its ground-truth class under Cs, aligning both generators.Unlike image reconstruction, the constraint preserves class identity rather than overall appearance and jointly optimizes the two mappings.
  • Source-to-target adaptation: The source-to-target branch jointly optimizes adversarial discrimination and classification on transformed source samples to produce target-like training data.Ct receives generated target-like images, while Dt distinguishes real target images from transformed source images and guides Gst adversarially.
  • Inference: At test time, Ct classifies original target images and Cs classifies source-like target transformations; their outputs are linearly combined for the final prediction.The two classifiers make different errors and confidence rankings, so the ensemble exploits their complementary predictions.
  • Optimization: The complete objective combines six weighted losses, and the reported ablation study finds the system robust to changes in loss-weight values.Each directional branch contains adversarial and classification-related losses, with the overall interaction controlled by nonnegative weights.

4. Evaluation

SBADA-GAN is evaluated across six digit and traffic-sign adaptation settings, with bidirectional classification, image generation, self-labeling, and class consistency assessed quantitatively and qualitatively.

  • Quantitative Results: SBADA-GAN improves over the state of the art in four out of six settings, with gains reaching +8 percentage points in MNIST→SVHN.The MNIST→SVHN result addresses adaptation from grayscale handwritten digits to colorful street-view house numbers.
  • Quantitative Results: In SVHN→MNIST, SBADA-GAN ranks eighth among thirteen baselines and is on par with ADDA despite using five times fewer fully-connected neurons.Its classifiers are also shallower than DRCN, with fewer convolutional layers.
  • Quantitative Results: In Synth Signs→GTSRB, SBADA-GAN is slightly worse than DAass but outperforms all other competing methods.Its performance remains robust to hyperparameter variations, whereas DAass drops significantly with untuned fixed parameters.
  • Qualitative Results: The generated images mimic the chosen domain style, including the difficult transformation from MNIST digits to colorful SVHN house numbers.This qualitative evaluation complements the classification results.
  • Qualitative Results: Mean SSIM indicates that generated images reproduce both the target style and the perceptual variability of the chosen domain.The evaluation randomly pairs generated images within each class and averages results across classes.
  • Ablation and Robustness Study: Self-labeling provides the largest improvement in the progressive component study, but its benefit disappears in MNIST↔SVHN when pseudo-label accuracy is about 65%.In the other settings, source-classifier accuracy on source-like transformed target images reaches 80−90%; unreliable pseudo-labels do not hinder overall performance in MNIST↔SVHN.
  • Ablation and Robustness Study: Class consistency preserves digit identity while producing realistic target-style samples, unlike removing consistency or using image-based cycle consistency alone.The comparison uses MNIST→SVHN, the strongest domain shift considered in the ablation.
  • Ablation and Robustness Study: Changing the class-consistency weight ν across [0.1, 1, 10] changes accuracy by at most 0.6 percentage points, while changing classifier-loss weights changes it by at most 0.2.Halving batch size from 32 to 16 reduces performance by about 0.2 percentage points.

5. Conclusion

SBADA-GAN combines simultaneous source-to-target and target-to-source mappings with self-labeling and class consistency, using both classifiers at test time. The bi-directional design, generator alignment, and self-labeling are associated with improved stability, reconstruction quality, and adaptation performance.

  • SBADA-GAN maps source samples into the target domain and target samples into the source domain, then uses both classifiers at test time.
  • Self-labeling regularizes the source-trained classifier, while class consistency improves architectural stability and reconstructed-image quality in both domains.
  • The bi-directional mapping avoids choosing a single transfer direction and combines complementary features learned by the two network directions.
  • Class consistency aligns the image generators so that transfers in both directions can influence each other.
  • Self-labeling boosts performance under moderate domain shift without hindering performance when domain gaps are large.

A. SBADA-GAN network architecture

The SBADA-GAN implementation uses symmetric GAN components with residual generators, convolutional discriminators, shared preprocessing conventions, and classifier architectures inherited from prior work. Training omits discriminator noise injection and dropout because the protocol is stable.

  • SBADA-GAN consists of two symmetric GANs, each using an architecture analogous to PixelDA.
  • The generators are convolutional residual networks with four residual blocks, each containing two 64-feature convolutional layers.
  • The noise input z has N_z elements sampled from a standard normal distribution, with N_z = 5 in all experiments.
  • The discriminators use two convolutional layers, average pooling, and a final convolution producing a scalar output.
  • Generator and discriminator convolutions use batch normalization except for the generator’s final convolution, while classifiers follow the structures in.
  • The implementation uses ReLU in generators and classifiers, leaky ReLU with slope 0.2 in discriminators, and zero-centered image rescaling.Generator inputs are rescaled to [−0.5, 0.5], whereas generated images and classifier or discriminator inputs are rescaled to [−127.5, 127.5].
  • The stable training protocol uses neither injected discriminator noise nor dropout layers.

B. Experimental Settings

The experiments evaluate seven domain-adaptation dataset configurations spanning handwritten digits, street-view numbers, and traffic signs. Training, validation, and testing splits are specified separately for each setting, with labeled validation subsets used for classifier-combination weights.

  • MNIST→MNIST-M uses 50k MNIST training images, 10k validation images, and all 60k MNIST-M training images for testing.A labeled subset of 1k MNIST-M images validates the classifier-combination weights.
  • USPS→MNIST resizes USPS images to 28 × 28 pixels, uses USPS training and validation splits, and tests on 60k MNIST training images.A labeled 1k-sample subset is used for validation purposes.
  • MNIST→USPS uses 50k MNIST training and 10k validation images, tests on all 9,298 USPS images, and reserves 1k USPS images for validation.
  • SVHN→MNIST uses 73,257 SVHN training and 26,032 validation samples, tests on 60k MNIST images, and validates with 1k labeled MNIST images.The remaining SVHN data are disregarded for this experiment.
  • MNIST→SVHN uses 50k MNIST training and 10k validation images, tests on 99,289 SVHN samples, and uses 1k labeled SVHN images for validation.
  • Synth Signs→GTSRB trains on 90k synthetic-sign images, validates on 10k, and tests on 51,839 resized GTSRB samples.A labeled subset of 1k GTSRB images is used for validation.

C. Distribution Visualizations

The distribution visualizations preprocess images with scaling and PCA before projecting them into two dimensions using t-SNE. For MNIST→MNIST-M, the generated data successfully cover the target space, extending the behavior observed elsewhere.

  • The visualization pipeline scales images to [−1, 1], reduces Width×Height vectors to 64 dimensions with PCA, and applies t-SNE to obtain a 2-dimensional projection.
  • The MNIST→MNIST-M visualization shows generated data faithfully covering the target space, consistent with successful mapping.

D. Robustness experiments

SBADA-GAN remains accurate across tested loss-weight settings and smaller batches, while mapped source images faithfully cover the target space. Compared with DAass, it is reported as more robust to hyperparameter choices despite using a GAN-based approach.

  • Qualitative analysis: The mapped source images faithfully cover the target space across all visualized settings in the t-SNE analysis.Figure 6 compares source, target, and source-mapped-to-target images.
  • Comparison with DAass: The robustness experiment evaluates SBADA-GAN and DAass on Synth. Signs → GTSRB while varying their respective loss weights.Figure 7 reports the SBADA-GAN and DAass loss-weight settings used in this comparison.
  • SBADA-GAN robustness: SBADA-GAN’s classification accuracy changes by less than 0.2 percentage point when varying classification or class-consistency loss weights.The discriminative-loss and self-labeling weights were fixed while β, µ, or ν varied over [0.1, 1, 10].
  • SBADA-GAN robustness: Reducing the SBADA-GAN batch size from 32 to 16 leaves overall accuracy almost unchanged.The supplied passage reports that performance remains almost unchanged under this smaller random batch size.
  • Comparison with DAass: SBADA-GAN is reported as more robust than DAass to loss-weight hyperparameters while retaining high accuracy with small random batches.DAass uses walker and visit losses to encourage associative similarity and broader target-sample visitation across domains.
Loading 1705.08824v2…