PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Unsupervised Deep Domain Adaptation

Garrett Wilson, Diane J. Cook

arXiv:1812.02849v3cs.LGstat.ML

TL;DR

Supervised deep learning assumes matched training and testing distributions, while target labels may be costly or unavailable. This survey examines single-source unsupervised deep domain adaptation methods, their components, results, theory, applications, and future directions. It concludes that several method categories appear promising, while direct comparisons, combinations, bidirectional adaptation, and novel datasets require further research.

  • Problem

    Supervised deep learning assumes matching training and testing distributions, but adaptation often must use unlabeled target data and reduce costly target-label requirements.

  • Method

    The survey compares single-source homogeneous unsupervised deep domain adaptation approaches, examining their methods, common and unique components, results, theory, applications, and research directions.

  • Results

    Several method categories appear promising, with empirical results varying across datasets, data types, and tuning requirements.

  • Takeaways & Limitations

    Further research is required for direct comparisons, novel method combinations, improved bidirectional adaptation, and use on novel datasets and applications.

  • Takeaways & Limitations

    The surveyed datasets are relatively small compared with common deep-learning datasets and are mostly computer-vision datasets, potentially limiting some methods and applications.

Abstract

from arXiv · show

Deep learning has produced state-of-the-art results for a variety of tasks. While such approaches for supervised learning have performed well, they assume that training and testing data are drawn from the same distribution, which may not always be the case. As a complement to this challenge, single-source unsupervised domain adaptation can handle situations where a network is trained on labeled data from a source domain and unlabeled data from a related but different target domain with the goal of performing well at test-time on the target domain. Many single-source and typically homogeneous unsupervised deep domain adaptation approaches have thus been developed, combining the powerful, hierarchical representations from deep learning with domain adaptation to reduce reliance on potentially-costly target data labels. This survey will compare these approaches by examining alternative methods, the unique and common elements, results, and theoretical insights. We follow this with a look at application areas and open research directions.

1 INTRODUCTION

The introduction frames unsupervised deep domain adaptation as a response to performance loss under domain shifts and limited target labels. It positions this survey as a comparison of single-source, homogeneous deep adaptation methods, their theory, applications, and research directions.

  • Motivation: When training and testing distributions differ, source-trained classifiers will likely lose performance on the target domain.
  • Motivation: Unsupervised domain adaptation uses labeled source data and only unlabeled target data during training.
  • Motivation: Domain adaptation can reduce costly target annotation, such as the approximately 1.5 hours required to annotate each Cityscapes image.
  • Deep adaptation: Deep networks learn powerful hierarchical representations but can still experience performance drops from domain shifts.
  • Survey scope: The survey addresses gaps in prior coverage by comparing single-source homogeneous unsupervised deep adaptation methods, components, results, theory, applications, and future directions.

2 BACKGROUND

The survey situates domain adaptation within transfer learning and focuses on single-source, typically homogeneous settings where labeled source data support evaluation on a different target domain. It also introduces GANs and adversarial training as foundations for later domain adaptation methods.

  • 2.1 Transfer Learning: Transfer learning trains on a source domain or task and evaluates on a different but related target domain or task.
  • 2.1.2 Domain Adaptation: Domain adaptation is transductive transfer learning in which the target task remains the same as the source while the domain differs.Homogeneous adaptation retains the domain feature space; heterogeneous adaptation changes it.
  • 2.1.3 Related Problems: In this survey, unsupervised domain adaptation uses labeled source data and unlabeled target data, while semi-supervised and supervised variants include labeled target data.
  • 2.1.3 Related Problems: Domain adaptation differs from domain generalization because target examples are available during adaptation, whereas domain generalization tests on an unseen target domain.
  • 2.2 Generative Adversarial Networks: GANs train a generator and discriminator adversarially to generate synthetic data resembling a training distribution.The discriminator distinguishes real from generated samples while the generator seeks to make generated samples appear real.
  • 2.2.1 Training: GAN training can suffer from convergence difficulty, mode collapse, and vanishing gradients, while evaluation and model comparison can also be difficult.The survey notes proposed remedies including architectural, objective, ensemble, and MMD-based modifications.

3 METHODS

The surveyed methods primarily seek domain-invariant representations by aligning source and target data through divergences, reconstruction, or adversarial objectives. They vary in alignment mechanisms, feature-extractor weight sharing, and whether alignment is statistical, learned, or reconstruction-based.

  • 3 METHODS: Recent unsupervised domain adaptation methods align source and target distributions, map between domains, separate normalization statistics, use ensembles, or adjust decision boundaries.
  • 3 METHODS: Most methods learn domain-invariant features whose distributions are intended to match across source and target domains.The target representation is then passed to the task classifier.
  • 3 METHODS: Methods differ in alignment components and in whether feature extractors share none, some, or all weights between domains.
  • 3.1 Divergence-based Methods: Distribution-alignment methods minimize divergences including MMD, correlation alignment, contrastive domain discrepancy, Wasserstein distance, and graph matching losses.
  • 3.1 Divergence-based Methods: MMD compares distributions through differences between smooth-function means, while CORAL aligns second-order feature statistics such as covariances.
  • 3.2 Reconstruction-based Methods: Reconstruction methods learn representations that classify labeled source data and reconstruct target data or both source and target data.
  • 3.1.3 Adversarial: Adversarial methods commonly use a domain classifier that predicts source versus target, while the feature extractor learns representations that prevent reliable domain classification.Other variants learn an approximate Wasserstein distance or use a GAN as the alignment component.

3.2 Domain Mapping

Domain mapping adapts source data toward the target domain through pixel- or feature-level translation, often using conditional GANs. These methods vary in training direction, classifier integration, and consistency constraints, while assuming domain differences are primarily low-level.

  • Domain Mapping: Conditional GANs can translate source images toward the target distribution, providing an alternative to learning domain-invariant features.The generator performs pixel-level adaptation by making source inputs resemble target-domain images.
  • Domain Mapping: Image-to-image translation supports supervised, semi-supervised, and unsupervised settings depending on whether corresponding cross-domain images are available.Pix2pix is a commonly used supervised method, while CycleGAN is a commonly used unsupervised method.
  • Domain Mapping: Image-to-image translation methods assume that domain differences are primarily low-level, while feature-level mapping has been found particularly helpful for structured domain adaptation.Feature-level mapping makes source features resemble target features rather than explicitly making them domain invariant.
  • Domain Mapping: For classification, source-to-target translation enables a classifier trained on labeled translated images, whereas target-to-source translation uses the source-domain classifier after mapping target inputs.The two configurations correspond to the alternatives illustrated in Figure 5.
  • Domain Mapping: Independent mapping and classification can rename classes, so semantic consistency losses and joint training can preserve class assignments.PixelDA is cited as an example of jointly training the mapping and classification models.
  • Domain Mapping: A combined two-direction approach can linearly combine predictions from target-domain and source-domain classifiers, but it requires hyperparameter training.The source-to-target GAN is not used during testing, whereas the target-to-source GAN is used during testing.

3.3 Normalization Statistics

Normalization-statistics methods treat network weights and normalization statistics as carrying different kinds of knowledge. AdaBN adjusts target-domain statistics, while AutoDIAL learns alignment through coupled source-target processing.

  • Normalization Statistics: AdaBN performs domain adaptation by replacing source batch-normalization statistics with target-domain statistics.The method assumes layer weights encode task knowledge while batch-normalization statistics encode domain knowledge.
  • Normalization Statistics: AdaBN is simple, parameter free, and complementary to other adaptation methods.Its adaptation changes normalization statistics without learning new network weights from target data.
  • Normalization Statistics: AutoDIAL extends AdaBN by using target data to learn network weights through coupled source- and target-domain parameters.It introduces domain alignment layers before each batch-normalization layer.
  • Normalization Statistics: AutoDIAL mixes source and target inputs by a learnable amount before batch normalization, allowing the network to learn how much alignment each layer needs.The resulting statistics are computed over mixtures of source and target data rather than either domain alone.

3.4 Ensemble Methods

Ensemble methods exploit prediction or parameter diversity to improve confidence and adaptation. Unsupervised variants use temporal ensembles, teacher-student networks, agreement-based pseudo-labels, or class-wise confidence balancing.

  • Ensemble Methods: Ensembles can outperform a single model by averaging outputs or voting when their members make different mistakes.The improvement comes with increased computation because multiple models must be evaluated.
  • Ensemble Methods: Self-ensembling evaluates a single model through predictions or weights accumulated at multiple training points.Past predictions can be recorded, or network weights can be maintained with a running average.
  • Ensemble Methods: Teacher-student adaptation feeds differently augmented images to two networks, training the student with gradient descent while updating the teacher separately.The method extends self-ensembling to unsupervised domain adaptation.
  • Ensemble Methods: Asymmetric tri-training assigns target pseudo-labels when two source-trained networks agree with sufficient confidence, then trains a third network on those labels.The networks share a feature extractor, and an additional loss handles ensemble diversity.
  • Ensemble Methods: Confidence-based pseudo-labeling can transfer mainly easy classes, so class-wise weighting is added to normalize confidence levels and balance the class distribution.This adjustment was proposed for semantic segmentation.

3.5 Target Discriminative Methods

Target-discriminative methods shape representations or decision boundaries using assumptions about unlabeled target structure. They combine local smoothness, entropy, adversarial generation, or classifier disagreement to make target predictions more discriminative.

  • Target Discriminative Methods: The cluster assumption places samples in separate same-label clusters, implying that decision boundaries should lie in low-density regions.This assumption motivates several target-discriminative domain adaptation methods.
  • Target Discriminative Methods: VADA and Co-DA combine variational adversarial training with conditional entropy loss to regularize target predictions.The combination is used because either component alone has a stated limitation: overfitting or insufficient local Lipschitz behavior.
  • Target Discriminative Methods: GAGL uses generated images labeled as “fake” to repurpose the task classifier as a GAN discriminator that moves decision boundaries toward lower-density regions.The method alternates generated inputs with source or target data during feature learning.
  • Target Discriminative Methods: Adversarial dropout regularization treats disagreement between two dropout-based classifiers as a discriminator for adversarially training target-discriminative features.The two classifiers share the feature extractor but can produce different predictions because dropout is stochastic.

3.6 Combinations

Recent domain adaptation methods increasingly combine complementary components rather than relying on a single adaptation strategy. These combinations target multimodal data, exploit classifier disagreement, and integrate mapping, feature learning, normalization, or target-discriminative objectives.

  • Combined methods integrate domain mapping with domain-invariant feature learning either sequentially or jointly, while domain-specific batch normalization is widely used.
  • Other combinations pair target-discriminative methods with self-ensembling or combine target-discriminative and domain-invariant features with a sliced Wasserstein metric.
  • MADA uses a separate discriminator for each class, weighting each discriminator’s loss by the classifier’s softmax probabilities for unlabeled target samples.This design combines adversarial feature learning with ensemble-style class-specific discrimination to better handle multimodal data.
  • MCD combines adversarial feature learning, ensemble classifiers, and target-discriminative features by measuring disagreement between two task-specific classifiers.The shared feature extractor and classifier discrepancy indicate when target samples lie outside the source-domain support.

4 COMPONENTS

The surveyed systems combine task objectives with distributional, adversarial, consistency, similarity, confidence, and architectural choices. They also vary in how networks share weights, train components, and adapt across representation levels.

  • 4 COMPONENTS: Table 1 categorizes methods by adaptation type, loss functions, generator use, and feature-extractor weight sharing.
  • 4 COMPONENTS: Distance losses align distributions or ensemble outputs, while similarity-based methods instead organize embeddings or classify by proximity to source examples.
  • 4 COMPONENTS: Multi-network methods promote diversity by encouraging different classifier or feature-extractor representations through weight-product norms or orthogonality constraints.
  • 4 COMPONENTS: Cycle consistency reconstructs a source image after source-to-target-to-source translation, avoiding the need for corresponding images across domains.
  • 4 COMPONENTS: Semantic consistency preserves classifier or pixel-level class assignments between an original source image and its mapped target output.
  • 4 COMPONENTS: Nearly all methods use a task loss, generally cross-entropy, although prototype, nearest-neighbor, batch-normalization, and application-agnostic translation methods are exceptions.
  • 4 COMPONENTS: Adversarial losses train discriminators or critics to make domain or generated outputs indistinguishable, supporting invariant features, realistic images, or target-discriminative features.
  • 4 COMPONENTS: Methods also introduce task-specific objectives, confidence-based target filtering, selective weight sharing, staged training, and multi-level adaptation.Multi-level adaptation combines pixel-level domain mapping with feature-level invariant learning to address different magnitudes of domain shift.

5 RESULTS

Results across classification and sentiment-analysis benchmarks are promising but difficult to compare directly because studies use different architectures, tuning, augmentation, data amounts, and initializations. Performance varies by method and dataset, with adversarial and carefully tuned approaches often prominent.

  • 5 RESULTS: Published accuracies require caution when compared across papers because architectures, hyperparameters, augmentation, data amounts, and random initializations differ.The methods may also perform differently in other application areas.
  • 5 RESULTS: CAN performs highest on the Office dataset, WDGRL generally leads on Amazon reviews, and SBADA-GAN achieves the highest accuracy on MNIST→MNIST-M.
  • 5 RESULTS: CyCADA increases accuracy from 54% to 82% on a synthetic season-adaptation dataset by combining adversarial feature learning with domain mapping.
  • 5 RESULTS: Adversarial techniques may help on particular datasets or data types, reduce tuning requirements, or augment promising non-adversarial methods.JAN-A outperformed JAN on average on the Office dataset, while the survey identifies further combinations as possible improvements.
  • 5 RESULTS: 97.0% versus 81.7% was reported for French et al. and Co-DA, respectively, on MNIST→SVHN; French et al.’s result required problem-specific augmentation and hyperparameter tuning.Without that tuning, French et al. achieved 37.5%, whereas Co-DA did not require similarly problem-specific tuning.

6 THEORY

The survey presents theoretical bounds for when unsupervised domain adaptation can help, emphasizing source error, domain discrepancy, and labeling differences. These results also show that adaptation is not guaranteed to improve target performance.

  • Theoretical Bounds: Target error can be bounded using source error, source–target divergence, and the error of an ideal predictor shared across domains.The empirical source error is directly estimable, whereas domain divergence requires specialized estimation.
  • Theoretical Bounds: A second upper bound replaces ideal-predictor dependence with marginal-distribution discrepancy and the distance between optimal source and target labeling functions.This formulation explicitly separates feature-distribution mismatch from labeling-function mismatch.
  • Semi-Supervised Adaptation: Semi-supervised adaptation uses labeled source and target data through an α-error, whose optimal weighting depends on empirical errors, domain divergence, and sample counts.The survey contrasts this setting with unsupervised adaptation, which uses labeled source data and unlabeled target data.
  • Theoretical Insights: When optimal predictor error is large, no hypothesis trained on the source domain can perform well on both domains.When this error is small, domain-invariant methods instead target source error and H∆H-divergence.
  • Theoretical Insights: Aligning feature distributions can increase target error when marginal label distributions differ significantly, and target accuracy may decline despite increasing source accuracy.This effect was experimentally observed during adaptation among MNIST, USPS, and SVHN.
  • Semi-Supervised Adaptation: Using source data is not always beneficial: with enough target data it may increase test-time error, while insufficient source data can make source examples unhelpful.The optimal weighting changes with the relative availability of source and target examples.

7 APPLICATIONS

Domain adaptation has been applied across computer vision, natural language processing, time-series analysis, robotics, medicine, and related domain-generalization settings. Its motivation includes reducing the human effort required to label target-domain data.

  • Motivation: Domain adaptation can reduce target-labeling effort in labor-intensive tasks such as semantic segmentation.Each pixel-level annotated Cityscapes image required approximately 1.5 hours to complete.
  • Computer Vision: Computer-vision applications include synthetic-to-real adaptation, robotics, semantic segmentation, medical imaging, object detection, person re-identification, and depth estimation.Examples span synthetic numbers or signs, autonomous navigation, chest X-rays, CT-to-X-ray segmentation, and MRI segmentation.
  • Natural Language Processing: Natural-language applications include sentiment analysis, text classification, relation extraction, sequence labeling, question answering, sentence specificity, and neural machine translation.The survey includes both supervised-target and weakly or semi-supervised variants among these applications.
  • Time-Series Data: Time-series applications include health-data transfer across age groups, speech recognition, driving-maneuver prediction, anomaly detection, and inertial tracking.The survey also notes domain-generalization work for sleep-stage classification using radio time-series data.
  • Related Settings: Domain-invariant feature-learning methods have also been used for domain generalization across multiple source domains and unseen target domains.An adversarial domain classifier was used to generalize sleep-stage models to new people or environments.
  • Related Applications: Adversarial techniques related to domain adaptation have been applied to object-detection robustness, sound synthesis, and machine-learning security.Examples include adversarial spatial dropout, adversarial spatial transformation, physical-model cloning, and robustness to adversarial examples.

8 RESEARCH DIRECTIONS

The survey identifies unresolved challenges in difficult adaptation directions, class imbalance, hyperparameter selection, data scale, and fair method comparison. It also proposes broader method combinations, unified implementations, and expansion beyond single-source homogeneous settings.

  • Bi-Directional Adaptation: MNIST→SVHN remains substantially harder than the reverse direction, with general-purpose results below the 97.0% achieved using problem-specific unsupervised tuning.Reported results range from 70.7% to 99.3% for SVHN→MNIST, while the highest untuned MNIST→SVHN result cited is 81.7%.
  • Hyperparameter Tuning: Cross-problem hyperparameter tuning remains open because reverse validation can be computationally costly or poorly aligned with test accuracy.Problem-specific pixel-intensity matching is possible with domain knowledge but does not generalize broadly.
  • Combining Method Components: Aligning joint or conditional feature–label distributions may help with multi-modal data or differing label proportions, complementing common feature-distribution alignment.The survey presents this as a direction for incorporating key aspects of other methods.
  • Balancing Classes: Class-balance terms can improve training stability and help avoid degenerate minima on challenging transfers such as MNIST→SVHN.The survey notes that class balancing may matter differently across datasets.
  • Experimental Comparison: Published method comparisons are difficult because experiments use different datasets, architectures, hyperparameters, and augmentation settings.Unified implementations could clarify which components drive performance and facilitate combinations of methods.
  • Data Requirements: GAN-based methods may be limited by the large amounts of source or target data they require, especially on small datasets.The survey calls for modifications that address this data-scale constraint.
  • Broader Settings: Future work should examine multiple source or target domains, heterogeneous feature spaces, and alternative supervision levels beyond the surveyed single-source homogeneous setting.The survey identifies these extensions as potentially capable of improving performance.

9 CONCLUSIONS

Unsupervised domain adaptation extends deep networks trained with source labels to smaller target datasets that may lack labels. The survey organizes existing methods, connects empirical observations to theory, and identifies priorities for further research.

  • Unsupervised domain adaptation adapts deep networks to possibly smaller target datasets that may not have target labels.
  • The survey categorizes approaches into domain-invariant feature learning, domain mapping, normalization statistics-based, and ensemble-based methods.
  • These approaches contain both unique and common elements, while theoretical results provide insight into empirical observations.
  • Further research is needed for direct comparisons, novel method combinations, improved bi-directional adaptation, and applications to novel datasets and tasks.
Loading 1812.02849v3…