Source-linked AI summary
Co-regularized Alignment for Unsupervised Domain Adaptation
Abhishek Kumar, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, William T. Freeman, Gregory Wornell
TL;DR
Unsupervised domain adaptation must learn from labeled source data and unlabeled target data, but aligning marginal feature distributions may leave class-conditional distributions misaligned. The paper proposes co-regularized alignment across diverse feature spaces and reports improved benchmark performance, including substantial gains on harder tasks.
Problem
Marginal feature alignment does not guarantee aligned class-conditional distributions, limiting reliable target-domain prediction from labeled source and unlabeled target examples.
Method
Co-DA constructs diverse feature embeddings, aligns source and target distributions separately in each, and encourages agreement between their target-class predictions.
Results
Co-DA improves performance on several domain adaptation benchmarks, with substantial gains on more challenging tasks and new state-of-the-art results in some cases.
Takeaways & Limitations
The generic co-regularized alignment approach can improve domain adaptation methods that use domain alignment.
Takeaways & Limitations
The method relies on a finite disparity-control hyperparameter because allowing unlimited disparity can destabilize feature maps and harm alignment.
Abstract
from arXiv · showhide
Deep neural networks, trained with large amount of labeled data, can fail to generalize well when tested with examples from a \emph{target domain} whose distribution differs from the training data distribution, referred as the \emph{source domain}. It can be expensive or even infeasible to obtain required amount of labeled data in all possible domains. Unsupervised domain adaptation sets out to address this problem, aiming to learn a good predictive model for the target domain using labeled examples from the source domain but only unlabeled examples from the target domain. Domain alignment approaches this problem by matching the source and target feature distributions, and has been used as a key component in many state-of-the-art domain adaptation methods. However, matching the marginal feature distributions does not guarantee that the corresponding class conditional distributions will be aligned across the two domains. We propose co-regularized domain alignment for unsupervised domain adaptation, which constructs multiple diverse feature spaces and aligns source and target distributions in each of them individually, while encouraging that alignments agree with each other with regard to the class predictions on the unlabeled target examples. The proposed method is generic and can be used to improve any domain adaptation method which uses domain alignment. We instantiate it in the context of a recent state-of-the-art method and observe that it provides significant performance improvements on several domain adaptation benchmarks.
1 Introduction
Domain adaptation addresses failures when models trained on source data encounter target data with different distributions, especially when target labels are unavailable. The paper proposes co-regularized domain alignment to improve class-conditional alignment and reports gains across benchmark tasks.
- Models trained on source-domain data can fail to generalize to target-domain observations drawn from a different distribution.
- Unsupervised domain adaptation learns a target-domain predictor from labeled source examples and unlabeled target examples.
- Domain alignment extracts domain-invariant, discriminative features, but aligning marginal feature distributions does not ensure correct class-conditional alignment.
- The proposed method constructs diverse feature embeddings, aligns each source-target pair, and co-regularizes their target predictions to make the alignments agree.
- Evaluated on digit and object recognition benchmarks, the method produced significant improvements over state-of-the-art performance.
2 Formulation
Unsupervised domain adaptation aligns source and target feature distributions, but marginal alignment alone may not align class-conditionals. Co-regularized alignment addresses this by learning diverse feature spaces whose target predictions agree while each space is aligned separately.
- 2.1 Domain Alignment: Small H∆H-distance does not ensure low target error unless a hypothesis performs well on both domains, requiring suitable class-conditional alignment.
- 2.1 Domain Alignment: Domain alignment seeks domain-invariant features that retain discriminative information by bringing source and target feature distributions closer.
- 2.2 Co-regularized Domain Alignment: Marginal feature alignment can incorrectly align regions belonging to different classes, a failure that target labels would be difficult to detect without.
- 2.2 Co-regularized Domain Alignment: Co-regularized alignment learns two diverse feature generators and predictors, aligns each source-target pair, and penalizes disagreement on unlabeled target predictions.
- 2.2 Co-regularized Domain Alignment: The method trims possible alignments by rejecting alignments lacking an agreeing counterpart while retaining desirable class-conditional alignments.
- 2.2.1 Algorithmic Instantiation: Its objective combines source cross-entropy, distribution-distance losses, target prediction disagreement, and a diversity term for the feature generators.
- 2.2.1 Algorithmic Instantiation: Diversity is encouraged by separating minibatch feature means, while a finite ν prevents the feature maps from diverging and harming alignment.
3 Related Work
Co-DA differs from prior domain-alignment and classifier-disagreement approaches by using diverse feature generators whose alignments are regularized through target-prediction agreement.
- Domain alignment: Prior domain-alignment methods are widely used in domain adaptation, while Saito et al. optimize a proxy for H∆H-distance with two classifiers sharing one feature generator.Their procedure alternates between maximizing target-prediction disagreement and minimizing it through feature-generator updates.
- Co-regularized alignment: Co-DA instead learns diverse feature generators that induce different source-target alignments and correct each other’s errors through target-prediction agreement.Unlike the related approach, Co-DA maximizes diversity at the feature-generator outputs rather than at final classifier predictions.
- Related learning paradigms: Co-training and co-regularization reduce the hypothesis space by combining diversity with agreement, whereas ensemble methods have a different motivation.The distinction is that co-regularization uses agreement to constrain compatible predictors, not merely to combine multiple models.
4 Experiments
Experiments instantiate Co-DA within VADA and evaluate it across several domain-adaptation benchmarks using matched protocols and targeted comparisons. Co-DA improves VADA substantially on difficult settings, with gains varying by adaptation direction and dataset.
- Experimental setup: Co-DA is evaluated by extending VADA on the same datasets and implementation basis, with comparisons against VADA, DIRT-T, and a no-diversity variant.The evaluation covers MNIST, SVHN, MNIST-M, Synthetic Digits, CIFAR-10, and STL.
- MNIST→SVHN: ∼81.7% target accuracy improves VADA’s 73.3% on MNIST→SVHN with instance normalization and surpasses VADA+DIRT-T’s 76.5%.Without instance normalization, Co-DA improves VADA from 47.5% to 52%, while Co-DAbn reaches 55.3%; DIRT-T after Co-DA reaches 88% with instance normalization.
- Experimental setup: Table 1 reports target-domain test accuracy, including separate accuracies for Co-DA’s two classifiers and variants differing in batch normalization or stochastic layers.Co-DA accuracies are reported at 80k iterations, while VADA and DIRT-T results come from prior work.
- SVHN→MNIST: ∼4% improvement over VADA is obtained on SVHN→MNIST with instance normalization, while performance is already close to 99%.Without instance normalization, the improvement is about ∼1%; DIRT-T after Co-DA provides no significant additional boost over VADA+DIRT-T.
- MNIST→MNIST-M: ∼2% improvement over VADA is obtained on MNIST→MNIST-M with instance normalization, compared with about ∼1% without it.This follows the improvement pattern reported for SVHN→MNIST.
- Syn-DIGITS→SVHN: ∼1% improvement over VADA on Syn-DIGITS→SVHN brings Co-DA close to a fully supervised target model without data augmentation.The comparison concerns target-domain accuracy and is reported without data augmentation.
- Diagnostic analyses: During MNIST→SVHN training, Co-DA classifiers initially disagree strongly but converge to agreement, while VADA begins more accurately and eventually falls below Co-DA.A separate kNN evaluation uses PCA-reduced source features to assess target-domain accuracy for VADA and Co-DA.
- CIFAR↔STL: 3% improvement over VADA is observed on STL→CIFAR for both with- and without-instance-normalization settings.The paper identifies STL→CIFAR as the harder direction because CIFAR has more labeled examples than STL.
5 Conclusion
The paper proposes co-regularized domain alignment for unsupervised domain adaptation and reports improved benchmark performance, including substantial gains on challenging tasks. It also identifies diversity-loss design and theoretical analysis as open directions.
- The method improves a state-of-the-art domain adaptation method on commonly used benchmarks.
- Substantial gains on challenging tasks establish new state-of-the-art results in those cases.
- More effective diversity losses and theory for deep-network co-regularization remain open research directions.