Source-linked AI summary
Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shift
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, Liang Lin
TL;DR
The paper addresses multi-source UDA when source domains differ and may not share categories. It proposes DCTN, which combines weighted source predictions with alternating adversarial and discriminative adaptation, and reports state-of-the-art results across most evaluation protocols. The paper also finds that DCTN resists negative transfer under category shift.
Problem
Existing UDA commonly assumes one source distribution, while multi-source settings may contain domain shifts and non-consistent source categories.
Method
DCTN uses multi-way adversarial adaptation to estimate source-specific perplexity scores and pseudo-labeled target data to update multi-source classifiers and features.
Results
DCTN achieves state-of-the-art results in most evaluation protocols and positive transfer gains in all tested category-shift situations.
Takeaways & Limitations
DCTN provides one framework for ordinary multi-source adaptation and category shift, learning transferable and discriminative features from multiple sources.
Takeaways & Limitations
The category-shift formulation permits source category sets to differ but retains a target category set related to their union.
Abstract
from arXiv · showhide
Unsupervised domain adaptation (UDA) conventionally assumes labeled source samples coming from a single underlying source distribution. Whereas in practical scenario, labeled data are typically collected from diverse sources. The multiple sources are different not only from the target but also from each other, thus, domain adaptater should not be modeled in the same way. Moreover, those sources may not completely share their categories, which further brings a new transfer challenge called category shift. In this paper, we propose a deep cocktail network (DCTN) to battle the domain and category shifts among multiple sources. Motivated by the theoretical results in \cite{mansour2009domain}, the target distribution can be represented as the weighted combination of source distributions, and, the multi-source unsupervised domain adaptation via DCTN is then performed as two alternating steps: i) It deploys multi-way adversarial learning to minimize the discrepancy between the target and each of the multiple source domains, which also obtains the source-specific perplexity scores to denote the possibilities that a target sample belongs to different source domains. ii) The multi-source category classifiers are integrated with the perplexity scores to classify target sample, and the pseudo-labeled target samples together with source samples are utilized to update the multi-source category classifier and the feature extractor. We evaluate DCTN in three domain adaptation benchmarks, which clearly demonstrate the superiority of our framework.
1. Introduction
The paper argues that practical adaptation must handle multiple heterogeneous sources and possible category shift, then proposes DCTN with alternating adversarial and discriminative adaptation. Experiments across three benchmarks report state-of-the-art performance across most transfer tasks.
- Motivation: Most deep UDA methods assume one source distribution, although practical datasets may combine labeled data from multiple heterogeneous sources.The paper cites Amazon, Flickr, and ImageNet as examples of diverse sources.
- Motivation: Multi-source adaptation is challenging because source domains can differ, their information is complementary, and their categories may not align.The authors argue that simply combining sources or eliminating every source-target discrepancy can be inappropriate.
- Method: DCTN represents the target distribution through weighted source distributions and combines source-specific classifiers with source-target perplexity scores.The network includes a feature extractor, multi-source category classifier, and multi-source domain discriminator.
- Method: DCTN alternates multi-way adversarial learning with discriminative fine-tuning using labeled source data and pseudo-labeled target data.The adversarial step reduces domain discrepancies, while discriminative adaptation supports classifiable features and partially prevents negative transfer.
- Results: The method achieves state-of-the-art results across most transfer tasks on three domain-adaptation benchmarks.The paper evaluates both vanilla and category-shift settings.
2. Related Work
Related work spans single-source UDA, multi-source adaptation, and neighboring transfer-learning settings. The paper positions DCTN within theoretical work showing that target prediction can combine source-specific hypotheses using distribution weights.
- Single-source UDA: Single-source UDA methods reduce source-target discrepancy using distribution measures, including shallow and deep approaches.Examples include TCA, JDA, DAN, WMMD, and RTN.
- Multi-source adaptation: Multi-source adaptation accounts for domain shifts among sources and has developed through source-specific ensembles, shallow models, and theoretical learning bounds.The related work traces MDA from A-SVM to later theoretical analyses.
- Multi-source adaptation: The distribution weighted combining rule states that an ideal target hypothesis can combine source hypotheses according to relationships between the target and each source.This principle motivates using multiple source-specific classifiers for target prediction.
- Related transfer settings: Continual transfer learning trains across multiple tasks and domains sequentially, whereas domain generalization trains on multiple labeled domains without unlabeled target samples.Both are identified as related but distinct branches of transfer learning.
3. Problem Setup
The problem setup distinguishes vanilla MDA, where sources share categories, from category shift, where source category sets may differ. It formalizes category relations through public and private classes while retaining unlabeled target samples for evaluation.
- Vanilla MDA: Vanilla MDA assumes multiple source distributions whose samples share a common category set.The setup denotes the different underlying source distributions as p_sj(x, y).
- Data and evaluation: Target samples are drawn without observed labels, while a labeled target test set evaluates adaptation performance.The source and target datasets form the training ensemble, and target test samples measure adaptation.
- Category Shift: Category shift allows categories from different source domains to differ, relaxing the shared-category assumption of vanilla MDA.This is introduced as a new MDA protocol.
- Category Shift: Source category sets may overlap through public classes while also containing private classes, with their intersection contained in their union.The setup defines category shift when the source category intersection is not equal to their union.
- Relation to Open Set DA: Open set DA differs by collapsing uncommon source-target classes into an unknown category, whereas category shift models specific disaligned categories across multiple sources.The paper notes that open set DA can be extended toward the category-shift setting.
4. Deep Cocktail Network
DCTN adapts across multiple sources by combining source-specific category predictions according to target-source perplexity scores, while accommodating category shifts without restructuring its subnetworks. Its alternating pipeline first performs multi-way adversarial adaptation, then fine-tunes features and classifiers using labeled source data and pseudo-labeled target data.
- Architecture: DCTN handles vanilla and category-shifted multi-source adaptation without changing the structures or parameters of its three subnets.Category shifts instead modify the class combination used by the target classification operator.
- Architecture: The feature extractor maps all source and target images into a common feature space using adversarial learning to capture domain-invariant features and target-source relations.The extractor is built on deep convolutional networks.
- Multi-way adversarial adaptation: Source-specific discriminators distinguish each source from the target and provide target-source perplexity scores used to weight source-specific predictions.Each target instance activates all source-specific discriminators, whereas source-j instances activate only discriminator j.
- Target classification: The target classification operator reweights each source classifier’s softmax predictions by its perplexity score, sums the weighted class confidences, and selects the highest-confidence class.Only sources containing a class contribute that class’s source-specific prediction, enabling category-shift handling.
- Alternating adaptation: After pre-training, DCTN alternates multi-way adversarial adaptation with discriminative fine-tuning of the feature extractor and category classifier using labeled sources and pseudo-labeled target images.The classifier is fixed when optimizing the combined objective to provide stable gradients.
- Training considerations: The feature extractor’s adversarial optimization can oscillate because source and target distributions change simultaneously, motivating a multidomain confusion objective for more stable training.DCTN also uses target samples to learn discriminative features and reduce negative transfer from mismatched categories.
5. Experiments
DCTN is evaluated across three multi-source domain-adaptation benchmarks in vanilla and category-shift settings, using alternating adversarial and discriminative adaptation. Results show strong transfer performance and resistance to category-shift degradation and negative transfer.
- Experimental setup: DCTN is evaluated on Office-31, ImageCLEF-DA, and Digits-five, with experiments covering vanilla and category-shift settings.The implementation includes a feature extractor, category classifier, domain discriminator, confidence threshold, and adversarial iteration threshold.
- Training procedure: The training algorithm alternates multi-way adversarial adaptation with target discriminative adaptation using source-labeled and target pseudo-labeled samples.The algorithm updates the domain discriminator and feature extractor during adversarial adaptation, then estimates target confidence and updates the feature extractor and category classifier.
- Vanilla setting: DCTN achieves the best results on Office-31 tasks A,W→D and A,D→W, performs compellingly on D,W→A, and attains state-of-the-art results on all ImageCLEF-DA tasks.The method also outperforms conventional multi-source adaptation baselines by large margins.
- Vanilla setting: DCTN improves source-combine performance by 6.0% in Digits-five despite involving multiple source-domain shifts.The evaluated shifts include mm, mt, sy, up →sv and mt, sv, sy, up →mm, with comparisons against DAN and single-source transfer combinations.
- Category shift setting: Under category shift, DAN accuracy drops by −6.4 and −4.0 in Overlap and by −18.7 and −18.5 in Disjoint across Office-31 and ImageCLEF-DA, whereas DCTN obtains positive transfer gains in all situations.The evaluation constructs Overlap and Disjoint source organizations with private and public classes, then compares accuracy degradation and transfer gain.
- Further analysis: Ablation results indicate that adversarial and pseudo-label adaptations cooperate: removing pseudo-labels reaches a performance bottleneck, while removing the adversary substantially lowers pseudo-label accuracy.Testing and pseudo-label accuracies converge during alternative learning, and classification, adversarial, and testing-error losses gradually converge.
6. Conclusion
The paper introduces category shift for multi-source unsupervised domain adaptation and proposes DCTN to learn transferable, discriminative features across ordinary and category-shift settings.
- DCTN addresses multi-source adaptation when source classes are non-consistent, formalized as the category shift protocol.
- The framework learns transferable and discriminative features from multiple sources.
- DCTN applies to both ordinary multi-source adaptation and category shift.
- DCTN achieves state-of-the-art results in most evaluation protocols.