PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Adaptive Ensemble Learning

Kaiyang Zhou, Yongxin Yang, Yu Qiao, Tao Xiang

arXiv:2003.07325v3cs.CV

TL;DR

DAEL targets generalization from multiple source domains under both multi-source UDA and DG, where domain shift challenges performance on a target domain. It uses a shared CNN with source-specialized classifier experts trained collaboratively through pseudo-target supervision, including pseudo-labels for unlabeled UDA targets. Experiments report large-margin state-of-the-art improvements across three UDA and two DG datasets.

  • Problem

    Domain shift makes neural networks degrade when target and training distributions differ, creating a need to generalize from multiple source domains under UDA and DG.

  • Method

    DAEL combines a shared CNN feature extractor with source-specialized classifier heads and collaboratively trains their ensemble using pseudo-target experts or confident pseudo-labels.

  • Results

    DAEL outperforms the current state of the art by a large margin on three multi-source UDA datasets and two DG datasets.

  • Takeaways & Limitations

    DAEL provides one collaborative ensemble-learning framework for both multi-source UDA and DG.

Abstract

from arXiv · show

The problem of generalizing deep neural networks from multiple source domains to a target one is studied under two settings: When unlabeled target data is available, it is a multi-source unsupervised domain adaptation (UDA) problem, otherwise a domain generalization (DG) problem. We propose a unified framework termed domain adaptive ensemble learning (DAEL) to address both problems. A DAEL model is composed of a CNN feature extractor shared across domains and multiple classifier heads each trained to specialize in a particular source domain. Each such classifier is an expert to its own domain and a non-expert to others. DAEL aims to learn these experts collaboratively so that when forming an ensemble, they can leverage complementary information from each other to be more effective for an unseen target domain. To this end, each source domain is used in turn as a pseudo-target-domain with its own expert providing supervisory signal to the ensemble of non-experts learned from the other sources. For unlabeled target data under the UDA setting where real expert does not exist, DAEL uses pseudo-label to supervise the ensemble learning. Extensive experiments on three multi-source UDA datasets and two DG datasets show that DAEL improves the state of the art on both problems, often by significant margins. The code is released at \url{https://github.com/KaiyangZhou/Dassl.pytorch}.

I. INTRODUCTION

DAEL addresses multi-source UDA and DG by collaboratively training domain-specialized classifier experts whose ensemble uses complementary information for an unseen target. It combines a shared CNN feature extractor with multiple classifier heads and reports large-margin state-of-the-art improvements across both settings.

  • Domain shift causes neural networks to degrade when test and training distributions differ, motivating UDA and DG research.
  • DAEL learns source-domain experts collaboratively so their ensemble can exploit complementary information for the target domain.
  • A DAEL model uses a shared CNN feature extractor and multiple classifier heads, each specializing in one source domain.
  • Each source domain becomes a pseudo-target in turn, with its expert supervising an ensemble of non-experts from the other sources.
  • For unlabeled UDA targets, the most confident expert supplies a pseudo-label to supervise ensemble learning.
  • DAEL outperforms the current state of the art by a large margin on three multi-source UDA datasets and two DG datasets.

II. RELATED WORK

Prior work largely uses distribution alignment, adversarial alignment, augmentation, or conventional ensembles to address domain shift. DAEL instead collaboratively trains domain-specific classifiers and uses each source as a pseudo-target.

  • Unsupervised domain adaptation: UDA methods commonly reduce source-target feature discrepancy using distance metrics, optimal transport, graph matching, or domain discriminators.
  • Unsupervised domain adaptation: Multi-source UDA methods extend domain alignment by considering source-target pairs, whereas DAEL collaboratively learns domain-specific classifiers.
  • Domain generalization: DG methods often learn domain-invariant features by minimizing distances between source domains.
  • Domain generalization: Data augmentation methods seek to avoid source-data overfitting through adversarial input gradients, generation networks, or feature mixing.
  • Ensemble methods: Conventional ensemble learning combines multiple learners for the same problem, while DAEL trains classifier experts collaboratively to promote generalizable features.

III. METHODOLOGY

DAEL unifies multi-source UDA and DG through a shared CNN with domain-specific classifier heads trained collaboratively. Source-domain experts supervise ensembles of non-experts, while unlabeled target data use confidence-filtered pseudo-labels.

  • Model: DAEL learns a multi-expert model with a shared CNN feature extractor and classifier heads specialized to individual source domains.Each expert specializes in one source domain, while the ensemble averages all experts at test time.
  • Collaborative ensemble learning: For source-domain training, each domain acts as a pseudo-target whose expert supervises the strongly augmented ensemble of non-experts.The non-expert ensemble is trained to match the corresponding expert using mean-squared error.
  • Collaborative ensemble learning: For unlabeled target data, the most confident source expert generates a pseudo-label for the strongly augmented ensemble prediction.Pseudo-labels are retained only when the maximum expert confidence reaches the threshold ϵ, fixed to 0.95.
  • Learning objective: The full objective combines domain-specific cross-entropy, collaborative regularization, and a weighted unlabeled-target loss for UDA.For DG, the unlabeled-target loss is removed because target data are unavailable during training.
  • Collaborative ensemble learning: Collaborative learning updates each expert using information from other experts, facilitating exploitation of complementary information.The paper contrasts this with individual learning, whose update depends only on the expert’s own prediction.

A. Experiments on Domain Adaptation

DAEL achieves the strongest reported multi-source UDA performance across Digit-5, DomainNet, and miniDomainNet, including near-oracle accuracy on Digit-5. Its gains are especially pronounced on difficult domains and relative to M3SDA.

  • The experiments use target-domain classification accuracy, repeated three times with mean accuracy and standard deviation reported.Baseline results come from published papers or reimplementations when source code is available.
  • DAEL outperforms the second-best methods by 3.51% on Digit-5, 2.2% on DomainNet, and 6.21% on miniDomainNet in average accuracy.
  • DAEL achieves near-oracle performance on Digit-5, reaching 96.47% versus the oracle’s 97.00%.MNIST-M and SVHN are identified as particularly difficult domains because of complex, blurred, and cluttered visual characteristics.
  • DAEL exceeds M3SDA by 11.62% on MNIST-M and 4.06% on SVHN.
  • On DomainNet and miniDomainNet, DAEL achieves the best performance among all evaluated methods.The reported DomainNet comparison gives DAEL a 2.2% average margin over CMSS.
  • DAEL is superior to M3SDA on all three datasets, consistent with the difficulty of aligning each target domain separately to highly varied source domains.

B. Experiments on Domain Generalization

DAEL achieves the best reported domain-generalization performance on PACS and Office-Home. It consistently outperforms distribution-alignment, self-supervised, augmentation, and domain-shift simulation competitors.

  • Evaluation follows a leave-one-domain-out protocol, using one unseen test domain and the remaining three domains for training.
  • DAEL achieves the best results on both PACS and Office-Home, with clear margins over all competitors.
  • DAEL improves over CCSA and MMD-AAE by at least 4% on PACS and at least 1.2% on Office-Home.
  • DAEL outperforms JiGen by 2.9% on PACS and 4.9% on Office-Home.
  • DAEL also achieves clear improvements over CrossGrad and Epi-FCR.The comparison attributes DAEL’s advantage over Epi-FCR to collaborative ensemble learning; Epi-FCR uses additional domain-specific and domain-agnostic extractors and classifiers.

C. Analysis

The analysis shows that collaborative ensemble learning and the weak-strong augmentation strategy are important to DAEL’s performance, while DAEL improves feature alignment and robustness across domains.

  • Ablation study: Each of Lcr and Lu improves performance, and combining them gives the best result, confirming their complementarity.
  • Collaborative ensemble or individual expert training?: Collaborative learning improves over individual expert training by aggregating information from different experts and exploiting source complementarity.
  • Learning an ensemble of classifiers or a single classifier?: The ensemble classifier is essential because a single classifier may erase domain-specific knowledge useful for target recognition.
  • Using expert’s prediction or real label in Lcr?: Using the expert’s prediction in Lcr is slightly better than using the real label because its soft distribution encodes class relations.
  • Augmentation strategy: 61.73% accuracy with weak augmentation for pseudo-labels and strong augmentation for prediction falls to 54.32% when the augmentations are swapped and 58.05% when both are strong.
  • Visualization of features: DAEL produces clearer class clusters and smaller source-target feature discrepancy than Source-only in the t-SNE visualization.

V. CONCLUSION

DAEL is presented as a unified approach for generalizing from multiple source domains in both DG and UDA settings. Its current reliance on task-specific augmentation leaves a stated direction for future work.

  • DAEL provides a unified framework for multi-source domain generalization and unsupervised domain adaptation using collaborative ensemble learning.
  • In DG, DAEL uses source-domain experts without target data; in UDA, it uses pseudo-labels for unlabeled target data.
  • DAEL shows promising out-of-distribution generalization on PACS and Office-Home and uses the same collaborative strategy across DG and UDA.

APPENDIX A IMPLEMENTATION DETAILS

The implementation uses SGD with momentum and cosine annealing, with backbone, batch, learning-rate, and training-duration choices varying across adaptation and generalization experiments.

  • Domain adaptation uses SGD with momentum and cosine annealing, with CNN backbones and training schedules varying between Digit-5 and DomainNet.
  • Domain generalization uses ResNet18, SGD with momentum, cosine annealing, 40 epochs, and batches of 30 images across source domains.
  • The Lu term is discarded in domain generalization because no target data is available for training.

APPENDIX B PSEUDO-CODE

The pseudo-code combines domain-specific expert learning with collaborative consistency training, and adds unlabeled-target supervision when target data is available.

  • DAEL requires labeled source mini-batches, an optional unlabeled target mini-batch, source experts, weak and strong augmentations, and λu.
  • For each source, the expert receives weakly augmented images and contributes cross-entropy loss against the source labels.
  • The non-expert ensemble processes strongly augmented source images and is trained with an MSE consistency loss against the corresponding expert prediction.
  • When target data is available, pseudo-labels and instance masks supervise the all-expert ensemble through an additional weighted cross-entropy loss.
Loading 2003.07325v3…