Source-linked AI summary
DeepJDOT: Deep Joint Distribution Optimal Transport for Unsupervised Domain Adaptation
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, Nicolas Courty
TL;DR
Domain adaptation must transfer classifiers from labeled source data to unlabeled target data whose generating process differs, while preserving discriminative information. DeepJDOT jointly aligns deep representations and labels using optimal transport, and the paper reports favorable performance across visual adaptation benchmarks. Its full coupling is computationally infeasible at large sample sizes, motivating stochastic optimization.
Problem
Unsupervised domain adaptation seeks to learn a target-domain classifier from labeled source data and unlabeled target data under differing data-generating processes.
Method
DeepJDOT jointly learns source-target deep representations and a classifier by minimizing the discrepancy between joint feature-label distributions with optimal transport.
Results
DeepJDOT outperformed state-of-the-art methods on cross-domain digits and Office-Home adaptation and provided satisfactory results on VisDA-2017 adaptation.
Takeaways & Limitations
DeepJDOT provides an efficient stochastic optimal-transport framework that is simple to integrate into modern deep-learning systems.
Takeaways & Limitations
Computing a full transport coupling is infeasible for large sample sizes, and future work includes multi-domain evaluation.
Abstract
from arXiv · showhide
In computer vision, one is often confronted with problems of domain shifts, which occur when one applies a classifier trained on a source dataset to target data sharing similar characteristics (e.g. same classes), but also different latent data structures (e.g. different acquisition conditions). In such a situation, the model will perform poorly on the new data, since the classifier is specialized to recognize visual cues specific to the source domain. In this work we explore a solution, named DeepJDOT, to tackle this problem: through a measure of discrepancy on joint deep representations/labels based on optimal transport, we not only learn new data representations aligned between the source and target domain, but also simultaneously preserve the discriminative information used by the classifier. We applied DeepJDOT to a series of visual recognition tasks, where it compares favorably against state-of-the-art deep domain adaptation methods.
1 Introduction
Unsupervised domain adaptation transfers knowledge from labeled source data to unlabeled target data whose generating process differs. DeepJDOT jointly adapts representations and classification through optimal transport, and performs favorably across visual adaptation tasks.
- Domain shifts arise when source and target datasets share characteristics such as classes but differ in backgrounds, colorspaces, devices, or other data-generating factors.
- Unsupervised domain adaptation learns a target-domain classifier from labeled source examples and unlabeled target observations.
- JDOT addresses domain adaptation by aligning joint feature-label distributions rather than only matching feature distributions.
- DeepJDOT jointly learns domain embeddings and a classifier in one CNN using an optimal-transport loss over deep representations and labels.
- DeepJDOT compares favorably against several recent state-of-the-art competitors on a series of visual domain adaptation tasks.
2 Related works
Related work develops domain adaptation through shared representations and optimal transport, with later deep methods adding architectural or distribution-matching mechanisms. Optimal-transport approaches provide geometric alignment but historically face scalability constraints.
- Unsupervised domain adaptation methods seek representations that allow one classifier to operate across domains despite domain shifts.
- Earlier approaches align data spaces, while deep-learning methods use weight sharing, reconstruction, MMD, and association-based losses between source and target domains.
- Optimal transport has been used to learn transformations between domains, after which classifiers are trained on transported source representations.
- Exact optimal-transport solutions over all samples restrict these methods to small or medium-sized problems.
3 Optimal transport for domain adaptation
Optimal transport compares distributions by finding a minimum-cost coupling, and JDOT extends this idea to joint feature-label distributions. DeepJDOT addresses JDOT’s scaling and input-space representation issues with stochastic deep-layer alignment.
- Optimal transport compares probability distributions by finding a coupling with minimum displacement cost under a specified sample dissimilarity function.
- The transport optimization uses a nonnegative pairwise cost matrix and a coupling matrix with prescribed marginals, but scales super-quadratically with sample size.
- JDOT directly learns a classifier within the transport cost by aligning joint feature-label distributions instead of only feature distributions.
- JDOT matches samples using representation distance and classification loss, with unknown target labels replaced by classifier predictions.
- DeepJDOT’s batch coupling matrix uses latent representations and labels from both domains to form the global loss.
- DeepJDOT computes small stochastic couplings during CNN optimization and performs transport between deeper semantic representations rather than image-space inputs.
4 Proposed method
DeepJDOT jointly learns a deep embedding, classifier, and optimal-transport coupling to align source and target domains while preserving discriminative information. A stochastic minibatch approximation makes the optimization scalable to large datasets and compatible with modern deep-learning frameworks.
- DeepJDOT formulation: DeepJDOT jointly optimizes an embedding function and classifier while minimizing Wasserstein distance between embedded source and target joint distributions.The embedding maps inputs into a latent space, while the classifier maps latent representations to target-domain labels.
- DeepJDOT formulation: The method reinstates source-domain loss to reduce catastrophic forgetting while learning a classifier for the target domain.The formulation otherwise emphasizes target-domain classification and disregards source-sample performance.
- Stochastic optimization: Full optimal-transport coupling becomes infeasible for large sample sizes because it requires excessive memory and time.The coupling matrix is quadratic in the number of samples, motivating stochastic optimization.
- Stochastic optimization: DeepJDOT approximates the objective with randomly sampled source and target minibatches, then alternates coupling estimation with stochastic updates to the embedding and classifier.With fixed network parameters, the coupling is solved for each minibatch; with fixed coupling, the network parameters are updated.
- Stochastic optimization: The sparse coupling aligns source and target samples with similar activations and labels while simultaneously learning the classifier in both domains.This coupling also performs label propagation between source and target samples.
5 Experiments and Results
DeepJDOT is evaluated across digit, Office-Home, and VisDA domain-adaptation tasks, with comparisons against multiple baselines and ablations of its objective. It generally performs competitively or better, while embeddings and ablations indicate that joint distribution alignment, deep-layer costs, and source-domain information are important.
- Experimental setup: DeepJDOT is evaluated on digit classification, Office-Home, and VisDA-2017 domain-adaptation tasks using comparisons with numerous non-adversarial, adversarial, and image-generation methods.The digit experiments use MNIST, USPS, MNIST-M, and SVHN, while Office-Home and VisDA provide additional visual recognition benchmarks.
- Digit classification: DeepJDOT achieves accuracies comparable to or higher than current state-of-the-art methods across the evaluated digit adaptation tasks.It outperforms first-block competitors by large margins except for similar DANN performance on MNIST→USPS, and performs strongly in challenging SVHN→MNIST and MNIST→MNIST-M settings.
- t-SNE embeddings: DeepJDOT aligns source and target embeddings while producing near-perfect class separation on MNIST→MNIST-M, unlike the less-aligned embeddings of DANN and StochJDOT.The visualization also motivates computing the optimal-transport ground metric in deeper CNN layers.
- Ablation study: Ablations show that removing either the joint distribution component or feature-space distribution component reduces accuracy in at least one evaluated adaptation direction.The authors attribute the JDOT-only drop partly to overfitting noisy propagated labels and conclude that the components provide complementary information.
- Office-Home: On Office-Home, DeepJDOT outperforms other models on every task except product→clipart.Competitive-method figures are reported from the corresponding prior work.
- VisDA-2017: On VisDA-2017, DeepJDOT outperforms DeepCORAL and DAN for all classes except knife, ranks sixth by mean accuracy, and ranks third by source-to-target accuracy difference.Performance depends on source-model capacity, and the authors note negative transfer for car.
6 Conclusions
DeepJDOT learns a common, discriminative latent space for source and target distributions by minimizing joint deep feature/label discrepancy with optimal transport. It outperformed state-of-the-art methods on cross-domain digits and Office-Home, achieved satisfactory VisDA-2017 results, and motivates future multi-domain and richer-cost evaluations.
- DeepJDOT minimizes joint deep feature/label distribution discrepancy with optimal transport to learn a common latent space conveying discriminative information across domains.
- Future work will evaluate DeepJDOT in multi-domain scenarios and explore more complex cost functions incorporating representation and classifier-label similarities.