PaperlayerView exact revision on arXiv

Source-linked AI summary

Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation

Xiangyu Yue, Zangwei Zheng, Shanghang Zhang, Yang Gao, Trevor Darrell, Kurt Keutzer, Alberto Sangiovanni Vincentelli

arXiv:2103.16765v1cs.CV

TL;DR

FUDA addresses adaptation when source labels are scarce and target labels are unavailable, while prior instance-wise self-supervision has weaknesses in semantic structure learning and abnormal matching. PCS uses in-domain prototypical contrastive learning and cross-domain prototypical self-supervision in an end-to-end framework. It improves mean classification accuracy over different domain pairs across four benchmarks.

  • Problem

    FUDA limits source supervision to a very small labeled subset, making discriminative feature learning difficult; instance-wise methods also fail to encode semantic structure and can be sensitive to abnormal matches.

  • Method

    PCS unifies in-domain prototypical self-supervision, cross-domain instance-to-prototype matching, adaptive prototype-classifier learning, and mutual-information maximization.

  • Results

    PCS outperforms previous best methods in 47 of 52 domain-pair settings and improves mean classification accuracy across Office, Office-Home, VisDA, and DomainNet.

  • Takeaways & Limitations

    PCS provides an end-to-end FUDA framework that jointly learns discriminative features, semantic structures, and cross-domain alignment.

  • Takeaways & Limitations

    The approach addresses few-shot adaptation with scarce labeled source data and no labeled target data; prior instance-wise matching is described as sensitive to abnormal samples.

Abstract

from arXiv · show

Unsupervised Domain Adaptation (UDA) transfers predictive models from a fully-labeled source domain to an unlabeled target domain. In some applications, however, it is expensive even to collect labels in the source domain, making most previous works impractical. To cope with this problem, recent work performed instance-wise cross-domain self-supervised learning, followed by an additional fine-tuning stage. However, the instance-wise self-supervised learning only learns and aligns low-level discriminative features. In this paper, we propose an end-to-end Prototypical Cross-domain Self-Supervised Learning (PCS) framework for Few-shot Unsupervised Domain Adaptation (FUDA). PCS not only performs cross-domain low-level feature alignment, but it also encodes and aligns semantic structures in the shared embedding space across domains. Our framework captures category-wise semantic structures of the data by in-domain prototypical contrastive learning; and performs feature alignment through cross-domain prototypical self-supervision. Compared with state-of-the-art methods, PCS improves the mean classification accuracy over different domain pairs on FUDA by 10.5%, 3.5%, 9.0%, and 13.2% on Office, Office-Home, VisDA-2017, and DomainNet, respectively. Our project page is at http://xyue.io/pcs-fuda/index.html

1. Introduction

The paper addresses FUDA, where scarce source labels make discriminative feature learning and domain adaptation difficult. PCS uses prototypical self-supervision to learn semantic, discriminative, and domain-aligned features end to end.

  • 1. Introduction: FUDA uses only an extremely small fraction of labeled source samples, while all remaining source and target samples are unlabeled.
  • 1. Introduction: Limited source labels make learning discriminative features difficult in both source and target domains.
  • 1. Introduction: Instance-wise cross-domain self-supervision can separate semantically similar instances and is sensitive to abnormal cross-domain matches.
  • 1. Introduction: PCS performs in-domain prototypical self-supervision to encode semantic structure and cross-domain instance-to-prototype matching for robust knowledge transfer.
  • 1. Introduction: PCS unifies prototype learning, adaptive cosine-classifier updates, entropy-based mutual-information maximization, and classification loss in a single-stage framework.

2. Related Work

Related work spans feature-distribution, statistical, adversarial, pixel-level, and entropy-based domain adaptation, alongside self-supervised representation learning. PCS is situated at the intersection of self-supervision and domain adaptation.

  • 2. Related Work: UDA methods align domains using discrepancy criteria, joint distributions, second-order statistics, adversarial learning, image translation, or minimax entropy.
  • 2. Related Work: The PCS framework combines in-domain and cross-domain self-supervision with prototype-based feature processing and adaptive classifier learning.
  • 2. Related Work: Self-supervised learning uses auxiliary pretext tasks, with contrastive learning achieving state-of-the-art representation-learning performance.
  • 2. Related Work: Self-supervision-based domain adaptation incorporates self-supervised losses into task networks to learn domain-invariant or shared representations.

3. Approach

PCS is an end-to-end FUDA framework that learns discriminative, domain-aligned representations and a target-effective cosine classifier from few-shot labeled source data and unlabeled samples. It combines in-domain prototypical contrastive learning, cross-domain instance-prototype self-supervision, and adaptive prototype-classifier learning.

  • Problem setting: FUDA uses very few labeled source samples, while the remaining source and all target samples are unlabeled.The model is trained on labeled and unlabeled source data plus unlabeled target data, then evaluated on the target domain.
  • In-domain prototypical contrastive learning: Instance discrimination separates samples but does not encode semantic structure, motivating prototypical contrastive learning.PCS performs clustering and prototype-based contrastive learning separately within source and target domains to avoid incorrect cross-domain clustering.
  • Cross-domain instance-prototype SSL: Cross-domain instance-prototype self-supervision matches instances to prototypes rather than individual instances across domains.This addresses domain-gap mismatches that can map instances to different-class samples in instance-instance matching.
  • Adaptive prototypical classifier learning: PCS uses a cosine classifier whose class weights represent class prototypes and are updated adaptively using few-shot labels and high-confidence predictions.Source estimates are favored early because they are more robust, while target estimates become more representative later in training.
  • Adaptive prototypical classifier learning: Mutual-information maximization combines prediction diversity with confident per-sample outputs across source and target data.The framework maximizes expected prediction entropy while minimizing entropy for individual network outputs.
  • PCS learning for FUDA: The overall PCS objective unifies in-domain prototypical contrastive learning, cross-domain instance-prototype self-supervision, and adaptive prototype-classifier learning.The framework is trained jointly with classification and self-supervised objectives rather than as a separate fine-tuning stage.

4. Experiments

Experiments evaluate PCS across four benchmarks and show strong performance under few-shot source-label settings. Ablations and feature visualizations support contributions from its components, discriminative representations, and domain alignment.

  • Experimental Setting: Experiments use Office, Office-Home, VisDA-2017, and DomainNet with few-shot or low-percentage labeled source data.Office and DomainNet use 1-shot and 3-shot settings; Office-Home uses 3% and 6%, while VisDA-2017 uses 0.1% and 1%.
  • Results on FUDA: PCS outperforms previous best methods in 47 of 52 domain-pair settings across the evaluated benchmarks.The reported improvements are 10.5% and 3.4% on Office, 4.3% and 4.2% on Office-Home, 9.0% and 0.7% on VisDA, and 13.2% and 3.6% on DomainNet.
  • Ablation Study and Analysis: Each PCS component contributes to final Office adaptation results without performance degradation.The ablation study reports that PCS still outperforms previous methods even without MIM.
  • Ablation Study and Analysis: PCS features cluster samples by class and aggregate more closely than ImageNet pre-training and CDS in t-SNE visualizations.The visualizations qualitatively associate these patterns with more discriminative features and better semantic structure.
  • Sample Efficiency: 76.1% accuracy is achieved with one labeled source image per class, compared with 77.4% in the fully-labeled setting.The one-shot setting uses 31 labeled source images, whereas the fully-labeled setting uses 498; reducing labels by 94% degrades performance by only 1.3%.

5. Conclusion

The paper studies FUDA with few labeled source samples and no labeled target samples, and proposes PCS to combine prototypical self-supervision with adaptive prototype-classifier learning. Extensive benchmark experiments demonstrate PCS’s superiority over previous methods and establish a new state of the art for FUDA.

  • FUDA uses only a few labeled source samples while all target samples remain unlabeled.
  • PCS combines in-domain and cross-domain prototypical self-supervised learning with adaptive prototype-classifier learning.
  • Extensive experiments on multiple benchmark datasets demonstrate PCS’s superiority over previous best methods.

A. Proof of Equation (13)

The objective promotes diversified class predictions and confident individual predictions, and is shown to equal mutual-information maximization. A moving average of predictions is used in the objective.

  • PCS maximizes entropy of expected predictions to promote diversified outputs across classes.
  • Entropy minimization encourages high-confidence predictions for individual samples.
  • The combined entropy objective is shown to equal maximizing mutual information between inputs and outputs.
  • The objective uses ˆp0, defined as a moving average of p(y|x; θ).

C. Additional Implementation Details

The implementation specifies training, temperature, clustering, prototype-update, and evaluation settings for PCS. Additional figures describe target accuracy under cosine-classifier and weighted-kNN evaluation.

  • Training uses batch sizes of 64 for source and target self-supervision and 32 for classification loss, with SGD and weight decay 5e−4.The learning-rate ratio between linear and convolution layers is 1 : 0.1.
  • PCS stability is evaluated using averaged accuracy and standard deviation across three runs for Office 1-shot and 3-shots settings.
  • Figure 6a plots target accuracy with a cosine classifier against training epochs, while Figure 6b uses weighted kNN.
  • Spherical k-means uses nc and 2nc clusters, while new cluster centroids and classifier weights are computed each epoch.Classifier weights use source images during the first five epochs, and tw is set near half the average images per class.

D. Stability Analysis of PCS

PCS shows steadier target-accuracy improvement and faster convergence than the compared methods during training. In the cosine-classifier plot, PCS plateaus around Epoch 3 while several baselines peak later or decline earlier.

  • The stability analysis averages accuracy and standard deviation over three random-seed runs in Office 1-shot and 3-shots settings.
  • PCS target accuracy increases more steadily and robustly than other methods.
  • PCS performance increases smoothly until the end of training, unlike several CDS variants that begin decreasing early.CDS decreases from Epoch 3 with the cosine classifier; CDS and CDS+ENT decrease from Epoch 1 with weighted kNN, while CDS+ENT+MME decreases from training onset.
  • PCS converges faster, plateauing around Epoch 3 while CDS+ENT and CDS+ENT+MME reach best performance at Epoch 9 and 10.

E. Quantitative Feature Analysis

Cross-domain feature evaluations show that PCS improves weighted-kNN performance, aligns source and target class prototypes, and produces more discriminative semantic structure than comparison methods. Retrieval and visualization analyses further assess cross-domain alignment and feature quality.

  • Weighted-kNN evaluation: Weighted kNN predicts target classes by cosine similarity to source-memory-bank features and weighted voting over the top k neighbors.Neighbor weights are αi = exp(si/τ), with τ = 0.07 and k = 200.
  • Weighted-kNN evaluation: Both cross-domain prototypical self-supervision and the full PCS framework outperform previous approaches in cross-domain weighted-kNN evaluation.The comparison covers standalone cross-domain SSL methods and FUDA methods.
  • Cross-domain retrieval: Figure 8 compares closest cross-domain source neighbors for target queries using CDS and PCS on Office-Home Real-to-Art adaptation.Retrieval uses pairwise cosine similarity between the target query feature and source-domain features.
  • Prototype alignment: PCS aggregates source, target, and classifier prototypes more closely for each class than ImageNet pre-training and CDS.The t-SNE plots encode class by color, domain by shape, and prototype domain by cyan, red, or black.
  • Prototype quality: PCS learns an embedding space with better semantic structure, indicated by the least summed pairwise cosine similarities among prototypes.
  • Cross-domain retrieval: CDS retrieval features are biased toward wrong attributes such as color, texture, and other visual clues.

H. More Ablation Study Results

On Office-Home, ablations show that adding each PCS component improves final adaptation accuracy without performance degradation.

  • Each PCS component improves final adaptation accuracy without performance degradation on Office-Home.The ablation results support the effectiveness of all components in the framework.

I. Performance Comparison with UDA Methods using Full Source Labels

PCS achieves the best results against other UDA methods on Office and Office-Home even when the source domain is fully labeled, suggesting broader applicability.

  • PCS achieves the best results among UDA methods on Office and Office-Home with fully labeled source domains.The comparisons are reported in Tables 13 and 12, respectively.
  • The fully labeled-source results suggest that PCS could apply to a wider range of domain adaptation settings.
Loading 2103.16765v1…