Source-linked AI summary
Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data
Jiaxing Huang, Dayan Guan, Aoran Xiao, Shijian Lu
TL;DR
Unsupervised domain adaptation traditionally requires labeled source data, creating privacy, portability, and transmission concerns. The paper instead adapts source-trained models without source data using historical contrastive learning, whose instance- and category-level designs preserve source hypothesis while learning target representations. Across varied visual tasks and learning setups, HCL consistently outperforms state-of-the-art techniques.
Problem
Unsupervised domain adaptation requires source-domain data, whereas UMA seeks to adapt source-trained models to target distributions without accessing that data.
Method
Historical contrastive learning contrasts current-model target embeddings with historical-model keys and re-weights pseudo labels by prediction consistency.
Results
HCL consistently outperforms state-of-the-art methods across a variety of visual tasks and learning setups.
Takeaways & Limitations
HCL learns instance- and category-discriminative target representations while preserving the source hypothesis in source-free adaptation.
Abstract
from arXiv · showhide
Unsupervised domain adaptation aims to align a labeled source domain and an unlabeled target domain, but it requires to access the source data which often raises concerns in data privacy, data portability and data transmission efficiency. We study unsupervised model adaptation (UMA), or called Unsupervised Domain Adaptation without Source Data, an alternative setting that aims to adapt source-trained models towards target distributions without accessing source data. To this end, we design an innovative historical contrastive learning (HCL) technique that exploits historical source hypothesis to make up for the absence of source data in UMA. HCL addresses the UMA challenge from two perspectives. First, it introduces historical contrastive instance discrimination (HCID) that learns from target samples by contrasting their embeddings which are generated by the currently adapted model and the historical models. With the historical models, HCID encourages UMA to learn instance-discriminative target representations while preserving the source hypothesis. Second, it introduces historical contrastive category discrimination (HCCD) that pseudo-labels target samples to learn category-discriminative target representations. Specifically, HCCD re-weights pseudo labels according to their prediction consistency across the current and historical models. Extensive experiments show that HCL outperforms and state-of-the-art methods consistently across a variety of visual tasks and setups.
1 Introduction
The paper studies unsupervised model adaptation without source data, motivated by privacy, portability, and transmission concerns. It proposes historical contrastive learning to adapt models while preserving source hypothesis and learning discriminative target representations.
- Unsupervised model adaptation adapts source-trained models to target distributions without accessing source-domain data.
- UMA carries forward a portable source-trained model that is usually smaller and more efficiently transmitted than source data, while alleviating privacy and intellectual-property concerns.
- Historical contrastive learning uses current and historical models to address missing source supervision while preserving the source hypothesis.
- HCID contrasts current-model queries with historical-model keys to learn instance-discriminative target representations, while HCCD re-weights pseudo labels by prediction consistency for category discrimination.
- Extensive experiments show HCL consistently outperforms state-of-the-art methods across varied visual tasks and setups.
2 Related Works
Related work covers unsupervised model adaptation, domain adaptation, memory-based learning, and contrastive learning. The paper positions HCL as a memory mechanism for UMA that contrasts historical and current models to compensate for absent labeled source data.
- Prior unsupervised model adaptation methods target specific visual tasks while adapting source-trained models without source-domain data.
- HCL uses a memory mechanism that contrasts historical and currently evolved models, presented as the first exploration of memory-based learning for UMA.
- Unlike UMA, domain adaptation methods require labeled source data and commonly use adversarial alignment or iterative pseudo-label self-training.
- Existing memory-based learning uses historical hypotheses or models to regularize current models and produce stable predictions in supervised or semi-supervised settings.
- Many related methods do not transfer well to UMA because the absence of labeled source supervision can cause collapse or provide little adaptation benefit.
- Contrastive learning creates discriminative representations by matching a query with its designated positive key and separating it from negative keys.
- Figure 2 frames HCL as combining instance-level HCID with category-level HCCD for complementary target representation learning.
3 Historical Contrastive Learning
Historical Contrastive Learning (HCL) adapts models using current and historical model representations, combining instance-level contrast with category-level pseudo-label discrimination. Its historical comparisons preserve source hypothesis while learning discriminative target representations.
- Historical Contrastive Learning: HCL uses historical models to memorize source hypothesis while adapting representations from unlabeled target data.The method contrasts historical and currently evolved models to address the absence of source data.
- Historical Contrastive Instance Discrimination: HCID contrasts current-model queries with historical-model keys, pulling positive pairs together and pushing negative keys apart.The query is encoded by the current model, while keys are generated by historical encoders; reliability reweights each key's similarity loss.
- Historical Contrastive Instance Discrimination: HCID learns instance-discriminative target representations intended to generalize to unseen data, with InfoNCE as a special case when m = 0 and reliability is fixed.In practice, multiple historical models can be used to distill well-learnt historical embeddings.
- Historical Contrastive Category Discrimination: HCCD generates pseudo labels and reweights self-training using prediction consistency across current and historical models.Consistent predictions receive greater influence, while poorly learnt samples receive reduced influence in the weighted cross-entropy loss.
- Historical Contrastive Category Discrimination: HCCD operates at category level to learn category-discriminative target representations aligned with visual recognition objectives.Its pseudo labels are based on model predictions for unlabeled target samples.
- Theoretical Properties: HCID and HCCD are modeled through maximum-likelihood formulations and are each reported convergent under certain conditions.HCID uses Expectation Maximization, whereas HCCD uses Classification Expectation Maximization.
4 Experiments
Experiments evaluate HCL across semantic segmentation, object detection, image classification, and alternative adaptation setups. HCL consistently outperforms UMA methods, complements existing approaches, and produces discriminative representations and improved qualitative results.
- Experimental scope: Experiments cover semantic segmentation, object detection, and image classification under UMA.The evaluated tasks include GTA5 →Cityscapes, SYNTHIA →Cityscapes, Cityscapes →Foggy Cityscapes, Cityscapes →BDD100k, VisDA17, and Office-31.
- Main results: HCL outperforms state-of-the-art UMA methods across semantic segmentation, object detection, and image classification.The reported segmentation results use mIoU, while the detection and classification experiments also show clear improvements over state-of-the-art UMA methods.
- Ablation and representation analysis: HCID and HCCD provide complementary self-supervision signals, and their combination produces the best segmentation.HCID focuses on instance-level discrimination, whereas HCCD focuses on category-level discrimination using pseudo category labels.
- Complementarity: HCL improves existing UMA methods consistently when incorporated as “+HCL”.The experiments identify HCL as complementary to existing UMA methods.
- Ablation and representation analysis: HCL achieves instance-discriminative and category-discriminative representations and better qualitative segmentation than UR and SFDA.The feature visualization compares HCL with UR and SFDA on GTA →Cityscapes, while qualitative illustrations report improved segmentation results.
- Generalization: HCL achieves competitive performance consistently in partial-set and open-set adaptation.Table 8 evaluates these two learning setups on Office-Home.
5 Conclusion
The paper presents HCL for UMA, using historical source hypotheses through instance- and category-level contrastive discrimination. Experiments across visual tasks and learning setups show consistent outperformance of state-of-the-art techniques.
- Conclusion: HCL uses historical source hypotheses to address the absence of source data in UMA.The method adapts source-trained models to target data while preserving the source hypothesis.
- Conclusion: Historical contrastive instance discrimination and category discrimination learn discriminative target representations while preserving source hypothesis.These are the two designs through which HCL implements historical contrastive learning.
- Conclusion: Extensive experiments across visual tasks and learning setups show that HCL consistently outperforms state-of-the-art techniques.The conclusion reports this result across the paper’s evaluated settings.