Source-linked AI summary
A Brief Review of Domain Adaptation
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, Hamid R. Arabnia
TL;DR
Domain adaptation addresses the mismatch between training and test distributions that can degrade performance when labeled data are available only in the source domain. The paper reviews unsupervised domain adaptation by defining its settings, categorizing domain and category gaps, and surveying shallow and deep approaches. It concludes by presenting methods that align distributions or learn transferable representations for target-domain prediction.
Problem
Training and test data may come from different distributions, making direct application of a model trained on labeled source data prone to performance degradation.
Method
The paper reviews unsupervised domain adaptation, categorizes its settings, and presents shallow and deep methods for reducing source-target disparity.
Results
The review presents shallow and deep domain adaptation approaches that address domain shifts through distribution alignment and transferable feature learning.
Takeaways & Limitations
Domain adaptation provides a framework for using labeled source data and unlabeled target data when feature, label, or probability distributions differ across domains.
Takeaways & Limitations
The review focuses on closed-set unsupervised domain adaptation, and prior-shift solutions requiring labeled target data are outside its scope.
Abstract
from arXiv · showhide
Classical machine learning assumes that the training and test sets come from the same distributions. Therefore, a model learned from the labeled training data is expected to perform well on the test data. However, This assumption may not always hold in real-world applications where the training and the test data fall from different distributions, due to many factors, e.g., collecting the training and test sets from different sources, or having an out-dated training set due to the change of data over time. In this case, there would be a discrepancy across domain distributions, and naively applying the trained model on the new dataset may cause degradation in the performance. Domain adaptation is a sub-field within machine learning that aims to cope with these types of problems by aligning the disparity between domains such that the trained model can be generalized into the domain of interest. This paper focuses on unsupervised domain adaptation, where the labels are only available in the source domain. It addresses the categorization of domain adaptation from different viewpoints. Besides, It presents some successful shallow and deep domain adaptation approaches that aim to deal with domain adaptation problems.
1 Introduction
Domain adaptation addresses performance degradation when training and test domains differ, while transfer learning and related paradigms reuse knowledge across domains with varying assumptions.
- Classical machine learning assumes training and test data share similar joint probability distributions.Real-world domain shifts can arise from changing statistics over time or collecting samples from different sources.
- Domain adaptation learns from labeled source data to generalize to a target domain by minimizing differences between domain distributions.
- Domain adaptation is a special case of transfer learning in which domains differ while tasks remain unchanged.
- Semi-supervised classification assumes labeled and unlabeled samples come from equivalent distributions, unlike domain adaptation and transfer learning.
- Multi-task learning improves generalization across related tasks by simultaneous training, common information, parameter sharing, and feature transformation.
- Multi-view learning uses complementary information from distinctive feature sets, including audio+video, image+text, and text+text.
- Domain generalization trains on multiple annotated source domains without target samples during training, whereas domain adaptation requires target data for alignment.
2 Notations and definitions
The paper formalizes domains through feature spaces, label spaces, and joint distributions, then defines unsupervised domain adaptation as target-oriented learning with unlabeled target data.
- A domain comprises an input space X, an output space Y, and a joint probability distribution p(x, y).The feature space is a subset of R^d, while the label space may be binary or multi-class.
- Source data contain feature-label pairs, while target samples have associated labels that are unknown in unsupervised domain adaptation.
- Naively transferring knowledge between related but differently distributed domains can negatively affect target performance.
- Domain adaptation reduces source-target disparity and trains a generalized classifier for target samples despite distributional shift.
- Source-risk minimization uses a loss function to measure disagreement between classifier predictions and true labels.
- Source and target joint distributions are denoted PS(x, y) and PT(x, y), respectively.
- Domain adaptation relaxes the classical assumption that training and test data are drawn from the same distribution.
3 Categorization of Domain Adaptation
The review categorizes domain adaptation by category gaps and domain shifts, focusing on closed-set unsupervised adaptation with identical feature and label spaces but differing distributions.
- Conventional domain adaptation keeps feature and label spaces unchanged while allowing probability distributions to vary between domains.
- Different marginal distributions and label spaces are termed domain gaps and category gaps, respectively.
- Category-gap settings include closed-set, open-set, partial, and universal domain adaptation.
- Universal domain adaptation identifies shared labels before aligning distributions in the common label set and training a classifier on matched source data.
- The review focuses on closed-set unsupervised domain adaptation using labeled source data and unlabeled target data under distribution shift.
- Closed-set adaptation has identical feature and label spaces, while joint probability distributions may differ.
- Domain shift is categorized into prior shift, covariate shift, and concept shift.
- Under prior shift, class priors differ while posterior distributions remain equivalent, and solving it requires labeled source and target data.
4 Approaches
Domain adaptation approaches are broadly divided into shallow and deep architectures, with shallow methods commonly aligning distributions through instance-based or feature-based techniques.
- Shallow domain adaptation methods mainly use instance-based and feature-based techniques to align domain distributions.
- Distribution alignment can minimize distances such as maximum mean discrepancy, Wasserstein metric, and correlation alignment.
4.1 Instance-Based Adaptation
Instance-based adaptation addresses domain shifts by reweighting labeled source samples to better represent the target distribution. Its methods include density-ratio estimation, constrained weighting, and KL-divergence minimization.
- Instance-based adaptation minimizes target risk using labeled source data when source and target joint distributions differ.
- Prior shift: Under prior shift, conditional distributions remain equal while class priors differ, but solving this case requires labeled data in both domains.
- Covariate shift: Importance weighting compensates for covariate shift by weighting source samples with the target-to-source density ratio w(x) = pT(x)/pS(x).
- Covariate shift: Indirect density-ratio estimation is challenging in high-dimensional feature spaces, motivating direct optimization of sample weights.
- Weight estimation: KMM estimates weights by matching weighted source features to target features in an RKHS under bounded-weight and normalization constraints.
- Weight estimation: KLIEP directly minimizes KL-divergence between the target distribution and the importance-weighted source distribution to estimate instance weights.
4.2 Feature-Based Adaptation
Feature-based adaptation learns transformed representations that reduce source–target discrepancy while preserving useful data structure. Its approaches include common subspaces, distribution alignment, feature-discrepancy measures, and reconstruction-based transfer.
- Feature-based methods transform source data into representations that minimize domain gaps while preserving the original data’s underlying structure.
- Subspace-based adaptation: Subspace-based adaptation constructs a shared intermediate representation by reducing discrepancy between low-dimensional source and target subspaces.
- Subspace-based adaptation: Grassmann-manifold methods represent source and target subspaces as points and can sample intermediate subspaces along a geodesic path.
- Subspace-based adaptation: Subspace distribution alignment extends basis alignment by aligning both subspace distributions and their bases.
- Transformation-based adaptation: Feature transformation methods use MMD, KL-divergence, or Bregman divergence to measure and reduce cross-domain dissimilarity.
- Reconstruction-based adaptation: Reconstruction-based methods learn projections and coefficients that reconstruct source samples in an intermediate representation while modeling noise and outliers.
- Reconstruction-based adaptation: LTSL reconstructs each target sample from a neighborhood of source samples to transfer locality and discriminative properties.
4.3 Deep Domain Adaptation
Deep domain adaptation uses neural networks to learn transferable representations despite distribution shifts, addressing the need for abundant labels in deep learning. Reviewed approaches align domains through marginal matching, reconstruction, adversarial objectives, or visual translation.
- Deep neural networks learn generalized feature representations but typically require abundant labeled data for training.
- Marginal-distribution adaptation: Deep Adaptation Network matches marginal distributions across domains through multiple adaptation layers while assuming conditional distributions remain unchanged.
- Autoencoder-based adaptation: Autoencoder-based approaches minimize reconstruction error to learn invariant and transferable representations across source and target domains.
- Adversarial adaptation: Adversarial domain adaptation minimizes distribution discrepancy by learning transferable features that make domain discrimination difficult.
- Adversarial adaptation: Gradient Reversal Layer methods train on labeled source data and unlabeled target data while reversing gradients during backpropagation.
- Visual adversarial adaptation: Visual adversarial methods reduce domain shifts through pixel-level, feature-level, or combined representations using generative adversarial networks.