PaperlayerView exact revision on arXiv

Source-linked AI summary

Source-Free Unsupervised Domain Adaptation: A Survey

Yuqi Fang, Pew-Thian Yap, Weili Lin, Hongtu Zhu, Mingxia Liu

arXiv:2301.00265v2cs.CVcs.AIcs.LG

TL;DR

Source-data access limits conventional unsupervised domain adaptation because of privacy, storage, transmission, and computation constraints. This survey systematically reviews SFUDA, organizing methods by source-model accessibility and learning strategy, and covering comparisons, datasets, generalizability techniques, challenges, and future directions. It concludes that SFUDA remains in an early stage and identifies challenges and promising directions for further research.

  • Problem

    Existing unsupervised domain adaptation studies highly depend on source data, whose access is often limited by privacy, storage, transmission, and computation constraints.

  • Method

    The survey organizes SFUDA into white-box and black-box groups, further subdivides them by learning strategy, and reviews their methods, datasets, techniques, challenges, and future directions.

  • Results

    The survey provides a comprehensive review and taxonomy of recent SFUDA studies, including comparisons between white-box and black-box approaches.

  • Takeaways & Limitations

    SFUDA is still in its early stages, with the survey offering challenges and potential future directions for advancing the field.

  • Takeaways & Limitations

    Black-box methods relying on pseudo-label denoising are greatly reduced in utility when predictors provide only one-hot hard predictions instead of probabilities.

Abstract

from arXiv · show

Unsupervised domain adaptation (UDA) via deep learning has attracted appealing attention for tackling domain-shift problems caused by distribution discrepancy across different domains. Existing UDA approaches highly depend on the accessibility of source domain data, which is usually limited in practical scenarios due to privacy protection, data storage and transmission cost, and computation burden. To tackle this issue, many source-free unsupervised domain adaptation (SFUDA) methods have been proposed recently, which perform knowledge transfer from a pre-trained source model to unlabeled target domain with source data inaccessible. A comprehensive review of these works on SFUDA is of great significance. In this paper, we provide a timely and systematic literature review of existing SFUDA approaches from a technical perspective. Specifically, we categorize current SFUDA studies into two groups, i.e., white-box SFUDA and black-box SFUDA, and further divide them into finer subcategories based on different learning strategies they use. We also investigate the challenges of methods in each subcategory, discuss the advantages/disadvantages of white-box and black-box SFUDA methods, conclude the commonly used benchmark datasets, and summarize the popular techniques for improved generalizability of models learned without using source data. We finally discuss several promising future directions in this field.

1 INTRODUCTION

Domain shift can degrade transfer from source to target domains, while practical constraints limit access to source data. This survey addresses that gap by systematically organizing SFUDA methods, comparisons, datasets, generalizability techniques, challenges, and future directions.

  • Domain shift causes performance degradation when source and target domains have distribution gaps.
  • Source-data access is often limited by privacy protection, storage and transmission costs, and computational burden.
  • SFUDA transfers a pre-trained source model to unlabeled target data without accessing source data.
  • The survey responds to a need for a more current and systematic SFUDA review than earlier coverage provided.
  • The survey classifies SFUDA into white-box and black-box categories, with finer subdivisions based on learning strategies.
  • It compares the two SFUDA groups, summarizes datasets and generalizability techniques, and discusses category-specific challenges and future directions.

2 WHITE-BOX SOURCE-FREE UNSUPERVISED DOMAIN ADAPTATION

White-box SFUDA adapts a pre-trained source model using unlabeled target data while keeping its training parameters accessible. Existing methods use either source-data generation or direct model fine-tuning.

  • White-box SFUDA assumes source data and labels are inaccessible, but the pre-trained source model’s training parameters remain available.
  • Its goal is to learn a target model for improved target inference from the source model and unlabeled target samples.
  • White-box methods divide into Data Generation Method and Model Fine-Tuning Method categories.

2.1 Data Generation Method

SFUDA data-generation methods create source-like images, proxy source samples, or synthetic source data so standard UDA can transfer knowledge without accessible source data. Their trade-offs involve fidelity, diversity, computation, and applicability across domain shifts.

  • Taxonomy: SFUDA data-generation methods comprise batch normalization statistics transfer, surrogate source data construction, and GAN-based image generation.These strategies generate source-like or proxy data before applying cross-domain adaptation.
  • Batch Normalization Statistics Transfer: BN statistics transfer matches source and target batch-normalization statistics to preserve target content with source style before UDA.Source-model BN statistics provide style information, while adaptation is performed between generated source-like data and target data.
  • Surrogate Source Data Construction: Surrogate source construction selects appropriate target samples to approximate a proxy source domain, then applies conventional UDA.Alternating adaptation and sample-transport learning can refine the approximated source domain and target labels.
  • GAN-based Image Generation: GAN-based methods use a predefined label and random noise to synthesize source data with guidance from the pretrained source model.Conditional GANs can generate labeled proxy source data, while other generators translate target images into source style.
  • Challenges and Insight: BN transfer is efficient but weak at preserving content, making it better suited to domains whose contextual image structure differs little.The survey cautions that large content changes, such as natural-image to cartoon shifts, may reduce adaptation performance.
  • Challenges and Insight: Surrogate source methods are computation-efficient, but their samples may poorly represent the original source domain and require informative target-sample selection.GAN methods produce more diverse images but require extra frameworks and computation, and surrogate methods generally outperform GAN generators in reported comparisons.
  • Challenges and Insight: Surrogate source construction generally outperforms distribution-generation methods, while distribution-generation approaches usually outperform GAN-based image generation.The survey attributes this pattern to the greater accuracy and stability of source distributions derived directly from existing target data.

2.2 Model Fine-Tuning Method

Model fine-tuning methods adapt pre-trained source models using unlabeled target data without generating source-like data. The survey organizes them into five strategy-based subcategories and highlights their mechanisms and limitations.

  • Taxonomy: Five fine-tuning strategies are identified: self-supervised knowledge distillation, domain alignment via statistics, contrastive learning, uncertainty-guided adaptation, and hidden structure mining.These methods exploit unlabeled target data while fine-tuning a pre-trained source model.
  • Self-Supervised Knowledge Distillation: Self-supervised knowledge distillation transfers source knowledge through mean-teacher frameworks that enforce consistency across target-data augmentations.Some approaches use Transformer modules, style diversification, or multiple teacher-student networks.
  • Domain Alignment via Statistics: Domain alignment via statistics uses stored batch-normalization statistics to approximate inaccessible source distributions and reduce source-target discrepancy.Batch-normalization mean and variance can support distribution matching, including with KL divergence.
  • Contrastive Learning: Contrastive learning pulls similar-class target samples together and pushes dissimilar-class samples apart using generated pseudo-labels.Some methods adaptively divide target instances into source-similar and source-dissimilar sets for class-aware cross-set alignment.
  • Uncertainty-Guided Adaptation: Uncertainty-guided adaptation uses uncertainty estimates from techniques such as Monte Carlo Dropout or entropy to guide predictions and filter noisy pseudo-labels.Uncertainty-based denoising can reduce prediction differences, improve robustness to noise, and retain more reliable target supervision.
  • Challenges and Insights: Statistics-based methods are lightweight and potentially suitable for real-time deployment but require batch-normalization layers, while contrastive and hidden-structure methods face noise, memory, efficiency, and data-scale challenges.Hidden-structure mining can also be unsuitable for non-Euclidean data because many methods construct geometry in Euclidean space.

3 BLACK-BOX SOURCE-FREE UNSUPERVISED DOMAIN ADAPTATION

Black-box SFUDA adapts to an unlabeled target domain when source data and detailed source-model parameters are unavailable, using only hard or soft source predictions. The survey groups these methods into self-supervised knowledge distillation, pseudo-label denoising, and generative distribution alignment, while identifying important limitations and future combinations.

  • Setting and Taxonomy: Black-box SFUDA uses only hard or soft predictions from a source model because source data and detailed model parameters are inaccessible.The survey divides these approaches according to how they exploit noisy target predictions.
  • Self-Supervised Knowledge Distillation: Self-supervised knowledge distillation transfers source knowledge by enforcing output consistency between a source teacher and a customized target student.Methods may use prediction memory banks, exponential moving averages, structural regularization, or temporal information for video adaptation.
  • Pseudo-Label Denoising: Pseudo-label denoising rectifies noisy target labels using procedures such as noise-rate estimation, progressive filtering, and MixMatch-based exploitation of easy and hard target groups.These methods can suppress noise accumulation by progressively incorporating target representations.
  • Challenges and Insights: Pseudo-label denoising may underperform self-supervised distillation when it neglects target structure, and its utility is greatly reduced when black-box predictors provide only one-hot hard predictions.The survey suggests combining denoising with distillation, especially for class-imbalance scenarios.
  • Generative Distribution Alignment: Generative distribution alignment generates a reference source distribution for target alignment using approaches including variational autoencoders and surrogate source-domain construction.The category addresses inaccessible source distributions through generative rather than direct source-data alignment.
  • Challenges and Insights: Black-box adaptation also benefits from robust source predictors, potentially improved by increasing source-data diversity or using soft probability labels.These strategies are presented as ways to improve the accuracy and generalization of source-model predictions.

4 DISCUSSION

This section compares white-box and black-box SFUDA, highlighting trade-offs among adaptation performance, privacy, flexibility, and computational cost, and summarizes generalizability techniques.

  • Comparison of White-Box and Black-Box SFUDA: White-box SFUDA can mine more source knowledge, such as batch statistics, enabling more effective domain adaptation than black-box methods.
  • Comparison of White-Box and Black-Box SFUDA: White-box methods may expose private information through source-model attacks, whereas black-box methods better protect privacy but may degrade cross-domain performance.
  • Comparison of White-Box and Black-Box SFUDA: Black-box methods support task-specific, lightweight target architectures, benefiting users with limited computational resources.
  • Comparison of White-Box and Black-Box SFUDA: 0.83s versus 3.17s and 22.43s: one black-box method reportedly required less computational time than two competing white-box methods.
  • Comparison of White-Box and Black-Box SFUDA: Selecting between white-box and black-box SFUDA requires trading off performance, confidential-information protection, and computational or memory costs.
  • Useful Strategies for Improved Generalizability: Common generalizability techniques include entropy minimization, diversity enforcement, label smoothing, regularization, and adaptive pseudo-label thresholds.

5 FUTURE OUTLOOK

The survey identifies future directions spanning multisource and federated adaptation, test-time settings, category shifts, architecture design, modality changes, source retention, and limited-label adaptation.

  • Multi-Source/Target Domain Adaptation: Multisource data-free adaptation remains challenged by negative transfer, while federated approaches must address computation, communication, and privacy costs.
  • Test-Time Domain Adaptation: Test-time adaptation improves online efficiency and avoids target training data, but current methods may require batches or fail under large distribution shifts.
  • Open/Partial/Universal-Set Domain Adaptation: The survey focuses on close-set adaptation, whereas open-set, partial-set, and universal-set category shifts remain less explored.
  • Black-Box Architecture Design: Black-box adaptation may require manually designed target architectures that are unsuitable for the target domain, motivating automatic architecture design.
  • Cross-Modality Domain Adaptation: Cross-modality adaptation and multimodal source-free learning remain relatively under-studied beyond predominantly single-modality research.
  • Source Performance Retention: Source-domain performance is often neglected, leaving catastrophic forgetting as an ongoing challenge despite proposed mitigation strategies.
  • Semi-Supervised Source-Free Adaptation: Semi-supervised source-free adaptation with limited labeled target data still has substantial room for improvement.

6 CONCLUSION

The survey reviews recent SFUDA progress through a white-box/black-box taxonomy, comparisons, generalizability techniques, datasets, challenges, and future directions.

  • The paper classifies SFUDA studies into white-box and black-box groups, further organizing each by learning strategy and discussing category-specific challenges.
  • It compares the groups, summarizes adaptation techniques and datasets, and identifies promising future research directions in this still-early field.
Loading 2301.00265v2…