Multi-source Distilling Domain Adaptation

Sicheng Zhao; Guangzhi Wang; Shanghang Zhang; Yang Gu; Yaxian Li; Zhichao Song; Pengfei Xu; Runbo Hu; Hua Chai; Kurt Keutzer

Source-linked AI summary

Multi-source Distilling Domain Adaptation

Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, Kurt Keutzer

arXiv:1911.11554v2Nov 22, 2019cs.LGcs.CVstat.ML

TL;DR

Multi-source domain adaptation remains challenging because many existing methods focus on a single source despite practical settings involving multiple sources. MDDA separately adapts and distills source classifiers while weighting their predictions, and outperforms state-of-the-art multi-source methods in most benchmark cases.

Problem
Existing domain-adaptation methods largely focus on single-source settings, whereas the paper targets unlabeled-target adaptation from multiple labeled sources.
Method
MDDA separately pre-trains source classifiers, adversarially maps the target into each source space, fine-tunes relevant source samples, and weights aggregated predictions.
Results
MDDA outperforms state-of-the-art multi-source methods in most cases, improving over the best source-combined method by 3.1% on Digits-five and 0.5% on Office-31.
Takeaways & Limitations
The results support using source-specific adaptation and relevance-weighted prediction aggregation for multi-source domain adaptation.
Takeaways & Limitations
The experiments assume homogeneous domains sharing categories, while heterogeneous, open-set, and category-shift adaptation are left for future work.

Abstract

from arXiv · show

Deep neural networks suffer from performance decay when there is domain shift between the labeled source domain and unlabeled target domain, which motivates the research on domain adaptation (DA). Conventional DA methods usually assume that the labeled data is sampled from a single source distribution. However, in practice, labeled data may be collected from multiple sources, while naive application of the single-source DA algorithms may lead to suboptimal solutions. In this paper, we propose a novel multi-source distilling domain adaptation (MDDA) network, which not only considers the different distances among multiple sources and the target, but also investigates the different similarities of the source samples to the target ones. Specifically, the proposed MDDA includes four stages: (1) pre-train the source classifiers separately using the training data from each source; (2) adversarially map the target into the feature space of each source respectively by minimizing the empirical Wasserstein distance between source and target; (3) select the source training samples that are closer to the target to fine-tune the source classifiers; and (4) classify each encoded target feature by corresponding source classifier, and aggregate different predictions using respective domain weight, which corresponds to the discrepancy between each source and target. Extensive experiments are conducted on public DA benchmarks, and the results demonstrate that the proposed MDDA significantly outperforms the state-of-the-art approaches. Our source code is released at: https://github.com/daoyuan98/MDDA.

Introduction

Domain shift makes direct transfer from labeled sources to unlabeled targets unreliable, while existing domain adaptation methods often overlook practical multi-source settings. MDDA addresses this by modeling source–target relationships, distilling source classifiers with target-relevant samples, and weighting their aggregated predictions by domain similarity.

Motivation: Limited labeled data and visual-data variance restrict the scalability and applicability of supervised deep learning models.Obtaining abundant labels is labor-intensive and expensive.
Domain adaptation: Domain shift causes performance decay when models trained on labeled source domains are directly transferred to unlabeled target domains.This motivates unsupervised domain adaptation methods.
Multi-source challenge: Most domain adaptation methods focus on a single source and may produce suboptimal solutions when multiple labeled sources have different distributions.This practical setting motivates multi-source domain adaptation techniques.
MDDA framework: MDDA has four stages: separately pre-training source classifiers, adversarially mapping the target into each source feature space, distilling classifiers with target-relevant samples, and aggregating source predictions.The method explores relationships among different sources and the target throughout these stages.
Source aggregation: MDDA weights source predictions by source–target discrepancy to emphasize relevant sources and suppress irrelevant ones.The resulting features are designed to be discriminative for the task and indiscriminate with respect to domains.
Contributions: MDDA aims to improve target inference by fine-tuning and aggregating source classifiers according to source–target relationships.The paper contrasts this with symmetric mapping of multiple sources and the target into a common space, which it says can cause oscillation from simultaneous changes.

Related Work

Related work covers deep single-source and multi-source unsupervised domain adaptation methods, which typically align domains in shared architectures and combine source classifiers. MDDA instead uses source-specific feature spaces, selected-sample distillation, and weighted prediction aggregation.

Single-source UDA: Deep SUDA methods commonly use two-stream architectures for source and target representations, jointly optimizing source task loss with an additional domain-shift loss.The additional loss may be a discrepancy loss.
Multi-source UDA: Representative deep MDA methods include MDAN, DCTN, and MMN, which use a shared feature extractor to map multiple sources and the target into one space.MDAN and DCTN train discriminators for source-target pairs, while target predictions are uniformly averaged across source classifiers.
MDDA: Unlike prior MDA methods, MDDA uses unshared source-specific extractors, asymmetrically matches target features, distills classifiers with selected samples, and applies a novel prediction-weighting strategy.These components distinguish MDDA from shared-space mapping and uniform averaging across source classifiers.

Problem Definition

The problem considers unsupervised adaptation from multiple labeled source domains to one unlabeled target under shared feature and class spaces but differing distributions. MDDA addresses this by learning source-specific representations and classifiers, selecting target-relevant source samples, and weighting source predictions by source-target discrepancy.

Problem Setting: The setting comprises M labeled source domains and one fully unlabeled target domain, with source samples drawn from p_i(x, y) and target samples from p_T(x, y).
Assumptions: The homogeneity assumption places all domains in the same feature space while allowing their data distributions to differ.
Assumptions: The closed-set assumption requires all domains to share the same class-label space, and the goal is to correctly predict target-domain samples from labeled sources and unlabeled target data.
Scope and Limitations: The method can be extended to heterogeneous, open-set, and category-shift adaptation, but these extensions are left for future work.
MDDA Framework: MDDA explores source-target relationships by pre-training separate source classifiers, aligning each target representation adversarially, distilling target-relevant source samples, and aggregating source predictions.

Experiments

Experiments evaluate MDDA for multi-source domain adaptation in visual classification, covering digit recognition and object classification.

MDDA is evaluated on multi-source domain adaptation tasks spanning digit recognition and object classification.

Experimental Settings

Experiments use the Digits-five and Office-31 benchmarks, compare MDDA against source-only and single-source adaptation standards, and report an 84.2% Office-31 accuracy. The setups specify domain-combination strategies, network architectures, and α=10 for Eq. (5).

Benchmarks: Digits-five comprises five digit-image domains, with 25,000 training and 9,000 testing images sampled for mt, mm, sv, and sy, plus all 9,298 up images.The domains are MNIST, MNIST-M, SVHN, Synthetic Digits, and USPS.
Benchmarks: Office-31 contains 4,110 images across 31 categories and three domains: Amazon, Webcam, and DSLR.Amazon images are downloaded from amazon.com, while Webcam and DSLR images are captured with web and digital SLR cameras.
Baselines: Baselines include Source-only and single-source DA methods spanning conventional models such as TCA and GFK and deep methods such as DDC, DRCN, and RevGrad.Source-only trains on source domains and tests directly on the target, serving as a lower bound of domain adaptation.
Evaluation strategies: For source-only and single-source DA, evaluations use source-combined and single-best strategies.Source-combined merges all source domains, whereas single-best adapts each source separately and selects the best target-test result.
Results: 84.2% accuracy is achieved by MDDA on the Office31 classification task, attaining state-of-the-art performance.The result is reported for multisource unsupervised domain adaptation on Office31.
Implementation: Digits-five uses a three-convolutional-layer and two-fully-connected-layer encoder with one fully connected classifier, while Office-31 uses AlexNet with its last layer as classifier.Following Gulrajani et al. (2017), α in Eq. (5) is set to 10.

Comparison with the State-of-the-art

The comparison shows that domain adaptation generally outperforms source-only transfer, with Source-combined DA usually exceeding Single-best DA. Multi-source methods perform particularly well, and MDDA surpasses state-of-the-art alternatives in most cases through domain weighting and selective source-sample distillation.

Source-only transfer performs worst in most adaptation settings because domain shift reduces the model’s transferability from source to target.The differing joint distributions of images and labels between source and target domains hinder direct transfer, even with more training samples.
Almost all adaptation methods outperform source-only methods, while Source-combined DA performs better on average than Single-best DA.After adaptation, domain-invariant representations allow Source-combined DA to benefit from more training data.
Multi-source DA generally outperforms other adaptation standards by bridging shifts between sources and the target as well as across source domains.This improvement reflects the complementarity of different source domains, including in comparisons such as MDDA versus ADDA and MDAN versus DANN.
3.1% and 0.5% are MDDA’s performance improvements over the best Source-combined method on Digits-five and Office-31, respectively.MDDA also achieves 3.3% and 4.8% improvements over DCTN on Digits-five and Office-Home, and 0.4% and 0.9% over MDAN on those datasets, respectively.
MDDA weights domains using Wasserstein distance and distills selective source samples to fine-tune each source classifier for better target-feature adaptation.The weighting emphasizes domains closer to the target, while selective sample distillation further adapts the source classifiers.

Interpretability and Ablation Study

MDDA’s visualizations show that adversarial adaptation aligns source and target features and shifts attention toward discriminative object regions. Ablation results verify the contributions of source distilling and similarity-based prediction weighting.

Feature Alignment: After adversarial adaptation, target features become denser and fit the source domain more tightly, indicating improved source-target distribution alignment.
Weighting Ablation: 6.6% and 1.1% improvements are achieved by the proposed weighting strategy over uniform weighting on Digits-five and Office31, respectively.The strategy weights sources according to their relative similarity to the target based on Wasserstein distance.
Source-Distilling Ablation: 0.3% and 0.5% average accuracy improvements result from source distilling on Digits-five and Office31, respectively, while removing distilling reduces performance in most cases.The comparison evaluates fine-tuning with versus without distilled source samples.
Source-Distilling Ablation: Source distilling yields different gains across sources, with larger improvements for sources having larger domain discrepancies from the target, such as mt to sy and up to sy.For the Wasserstein Distance based ADDA example with sy as target, the reported improvement for these sources is 2.5%.
Interpretability: Grad-CAM visualizations show that adaptation shifts attention from background regions toward discriminative object regions, including the ring binder and laptop.

Conclusion · 31 benchmarks

MDDA is presented as an effective multi-source domain adaptation approach that separately models source domains, aligns target features, distills source data, and weights predictions. Across Digits-five and Office-31, it improves over DCTN by 3.3% for digit classification and 0.4% for object classification.

Conclusion: MDDA separately pre-trains a feature extractor and classifier for each source domain to explore labeled-source discriminability.The approach maintains source-specific models before adaptation.
Conclusion: Adversarial discriminative adaptation matches the target feature distribution to each source distribution.This is one of the approach’s stated adaptation mechanisms.
Conclusion: Source distilling selects and fine-tunes the pre-trained source classifiers for target adaptation.The conclusion identifies source distilling as the mechanism for refining classifiers.
Conclusion: MDDA uses a novel weighting mechanism to combine source-specific predictions.The supplied conclusion introduces weighting as part of the model, without specifying its formula.
Conclusion: 3.3% performance improvements are reported over DCTN for digit classification.This result is attributed to extensive experiments on the Digits-five benchmark.
Conclusion: Future work will extend MDDA to challenging vision tasks such as scene segmentation.The authors also plan to investigate combining generative and discriminative pipelines for multi-source domain adaptation.
Conclusion: Future studies will investigate combining generative and discriminative pipelines for multi-source domain adaptation.This is presented as a planned research direction.