Source-linked AI summary
Contrastive Test-Time Adaptation
Dian Chen, Dequan Wang, Trevor Darrell, Sayna Ebrahimi
TL;DR
Test-time adaptation must adapt a source-trained model to unlabeled target data without source access. AdaContrast jointly applies target-domain contrastive learning with online nearest-neighbor pseudo-label refinement, achieving state-of-the-art benchmark performance and improved calibration, while remaining memory-efficient and relatively insensitive to hyper-parameters.
Problem
Test-time adaptation seeks to adapt a source-trained model to unlabeled target data when source data are unavailable, requiring target representation learning and classifier construction without ground-truth labels.
Method
AdaContrast jointly trains self-supervised target-domain contrastive learning with per-batch pseudo-label refinement using soft nearest-neighbor voting and a memory queue.
Results
AdaContrast achieves state-of-the-art test-time adaptation performance on major benchmarks, including 86.8% average and 84.5% overall accuracy on VisDA-C.
Takeaways & Limitations
The method combines memory efficiency, hyper-parameter insensitivity, and better model calibration with strong test-time adaptation performance.
Takeaways & Limitations
The method is presented for closed-set domain adaptation in image classification, and improved calibration could still lead to inappropriate trust in deployed systems.
Abstract
from arXiv · showhide
Test-time adaptation is a special setting of unsupervised domain adaptation where a trained model on the source domain has to adapt to the target domain without accessing source data. We propose a novel way to leverage self-supervised contrastive learning to facilitate target feature learning, along with an online pseudo labeling scheme with refinement that significantly denoises pseudo labels. The contrastive learning task is applied jointly with pseudo labeling, contrasting positive and negative pairs constructed similarly as MoCo but with source-initialized encoder, and excluding same-class negative pairs indicated by pseudo labels. Meanwhile, we produce pseudo labels online and refine them via soft voting among their nearest neighbors in the target feature space, enabled by maintaining a memory queue. Our method, AdaContrast, achieves state-of-the-art performance on major benchmarks while having several desirable properties compared to existing works, including memory efficiency, insensitivity to hyper-parameters, and better model calibration. Project page: sites.google.com/view/adacontrast.
1. Introduction
Test-time adaptation addresses domain shift when only a source model and unlabeled target data are available. AdaContrast jointly uses target-domain contrastive learning and online pseudo-label refinement, achieving strong benchmark performance with practical empirical merits.
- Test-time adaptation adapts a source-trained model to unlabeled target data without accessing source data, addressing domain shift under privacy and bandwidth constraints.
- TTA must learn target representations without ground-truth labels and build a target classifier using only the source classifier as a proxy.
- AdaContrast jointly optimizes self-supervised contrastive learning and pseudo labeling to exploit pairwise target-sample information while reusing source knowledge.
- Online pseudo-label refinement uses soft nearest-neighbor voting in target feature space, producing more accurate labels through per-batch aggregation and a relatively small memory queue.
- 86.8% average accuracy and 84.5% overall accuracy on VisDA-C surpass the previous state of the art by +3.8% and +6.2%, respectively.
- Contrast achieves state-of-the-art 67.8% accuracy averaged over 7 domain shifts.
2. Related Work
Related work in source-free and test-time adaptation uses entropy minimization, pseudo labeling, and self-supervised learning, but existing approaches have limitations in modeling sample relations and calibration.
- Unsupervised domain adaptation seeks to close the performance gap when a source model is deployed on a different target domain without target annotations.
- Source-free and test-time adaptation methods operate with only a source model and unlabeled target data, including entropy minimization in TENT and pseudo labeling combined with entropy minimization in SHOT.
- Existing entropy-minimization objectives do not model relations among samples and can disrupt target-model calibration through direct entropy optimization.
- Pseudo labeling treats predicted labels or cluster assignments for unlabeled samples as ground-truth-like supervision, while on-target adaptation adds contrastive target-domain learning to teacher-student training.
3. Method
AdaContrast adapts a source-initialized classifier to unlabeled target data by refining pseudo labels online and jointly optimizing self-training with target-domain contrastive learning. Its design uses memory queues, source-initialized encoders, weak-strong consistency, and regularization to improve target adaptation.
- Online pseudo label refinement: AdaContrast initializes the target model with source parameters and produces target pseudo labels during adaptation to reuse source knowledge while bootstrapping toward the target domain.Pseudo labels are predicted and refined per batch rather than only after each epoch.
- Online pseudo label refinement: For each weakly augmented target image, nearest neighbors in a memory queue are retrieved using feature-space cosine distance, and their probabilities are averaged before an argmax produces the pseudo label.The queue stores weakly augmented target features and predicted probabilities and is updated online with the current mini-batch.
- Joint self-supervised contrastive learning: The contrastive task uses source-initialized target and momentum encoders with two strong augmentations, following a MoCo-style query-key construction without projection heads.Source initialization provides an informative feature space, allowing contrastive learning to converge in few epochs.
- Joint self-supervised contrastive learning: Pseudo labels exclude same-class pairs from contrastive negatives, while the weak-image label supervises the strong-image prediction and diversity regularization reduces reliance on false labels.Historical pseudo labels are stored with contrastive features to support same-class negative-pair exclusion.
- Joint self-supervised contrastive learning: The refined pseudo labels are used jointly in contrastive learning and self-training, so target representation learning and classifier adaptation are optimized in a multi-task objective.The contrastive objective is applied to target data during test-time adaptation rather than only during pretraining.
4. Experiments
Experiments evaluate AdaContrast on VisDA-C and DomainNet-126, comparing benchmark performance, calibration, hyper-parameter sensitivity, and algorithmic components. Results show strong adaptation performance, improved calibration, robustness across settings, and gains from online pseudo-label refinement and joint contrastive learning.
- Experimental setup: Experiments use VisDA-C and DomainNet-126, including seven DomainNet-126 domain shifts and VisDA-C train→val adaptation.The study compares AdaContrast with unsupervised domain adaptation and test-time adaptation methods.
- Benchmark results: 3.8% higher per-class average accuracy than SHOT is achieved on VisDA-C, with 87.2% after an additional knowledge-distillation stage.AdaContrast also reaches 78.7% average accuracy in online adaptation.
- Benchmark results: +18.9% averaged performance over MCC and +10.1% over TENT are reported on DomainNet-126, with the best result on 4 of 7 domain shifts.AdaContrast achieves the highest averaged performance in this comparison.
- Calibration: AdaContrast reports 0.65% ECE and 8.20% MCE, versus SHOT’s 2.97% ECE and 39.16% MCE on VisDA-C.The reliability diagram shows SHOT as over-confident, while AdaContrast aligns more closely with y = x.
- Hyper-parameter sensitivity: Using M = 2048, less than 4% of the full queue size, reaches 86.3% per-class average accuracy, on-par with M = 55388.Performance remains around 84.5% overall and 86.7% per-class average across broad choices of M and N.
- Ablation studies: Online pseudo-label refinement raises performance to 64.7% on DomainNet-126 and 86.5% on VisDA-C at 1x learning rate, while joint contrastive learning recovers VisDA-C 10x accuracy from 10.0% to 84.3%.The joint objective is reported to stabilize the feature space against compounding pseudo-label errors and hyper-parameter choices.
5. Limitations
Domain adaptation can improve model robustness when models are deployed to new domains, but its benefits and harms may both be amplified in larger AI applications. AdaContrast’s improved calibration may support more reliable systems while also creating a risk of inappropriate trust in deployment.
- Domain adaptation methods can increase the robustness of models deployed to new domains.
- Their deployment can amplify both benefits and harms in larger AI applications.
- Improved model calibration may support more reliable systems but could lead to inappropriate trust.
6. Conclusion
The paper introduces AdaContrast for closed-set image-classification domain adaptation, combining target-domain contrastive learning with pseudo labeling and online nearest-neighbor refinement. It reports stronger benchmark performance alongside hyper-parameter insensitivity, better calibration, and no need for global memory banks.
- AdaContrast is a test-time adaptation approach for closed-set domain adaptation in image classification.
- It starts from a source-domain pretrained model and combines target-domain contrastive learning with pseudo labeling.
- Online nearest-neighbor soft voting refines per-batch predictions into more accurate pseudo labels.
- AdaContrast surpasses existing test-time adaptation approaches on major domain-adaptation benchmarks.
- The method also offers hyper-parameter insensitivity, better model calibration, and no need for global memory banks.