PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Domain Adaptation through Self Supervision

Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Kate Saenko

arXiv:2002.07953v3cs.CV

TL;DR

Universal domain adaptation must accommodate arbitrary, unknown category shifts rather than assuming a known source-target overlap setting. DANCE learns target structure with self-supervised neighborhood clustering and uses entropy separation to align known target features or reject unknowns. Across open-set, open-partial, partial, and other universal settings, it outperforms the source-only model in all settings and many state-of-the-art baselines.

  • Problem

    Universal domain adaptation seeks to handle closed-set, open-set, partial, and mixed category shifts without knowing the target-source category overlap beforehand.

  • Method

    DANCE combines self-supervised neighborhood clustering of target samples with entropy separation that aligns targets to source prototypes or rejects them as unknown.

  • Results

    DANCE outperforms the source-only model in all settings and state-of-the-art baselines in many settings, while extracting discriminative features for unknown classes without target supervision.

  • Takeaways & Limitations

    DANCE provides a universal domain-adaptation framework applicable without prior knowledge of the specific category shift.

  • Takeaways & Limitations

    Entropy separation assumes known and unknown target samples can be separated by a threshold, but that threshold may be ambiguous and change with domain shift.

Abstract

from arXiv · show

Unsupervised domain adaptation methods traditionally assume that all source categories are present in the target domain. In practice, little may be known about the category overlap between the two domains. While some methods address target settings with either partial or open-set categories, they assume that the particular setting is known a priori. We propose a more universally applicable domain adaptation framework that can handle arbitrary category shift, called Domain Adaptative Neighborhood Clustering via Entropy optimization (DANCE). DANCE combines two novel ideas: First, as we cannot fully rely on source categories to learn features discriminative for the target, we propose a novel neighborhood clustering technique to learn the structure of the target domain in a self-supervised way. Second, we use entropy-based feature alignment and rejection to align target features with the source, or reject them as unknown categories based on their entropy. We show through extensive experiments that DANCE outperforms baselines across open-set, open-partial and partial domain adaptation settings. Implementation is available at https://github.com/VisionLearningGroup/DANCE.

1 Introduction

Universal domain adaptation must handle unknown category overlap without knowing the category-shift setting in advance. DANCE addresses this by learning target structure through self-supervised neighborhood clustering and using entropy separation for source alignment or unknown rejection.

  • Problem: Universal DA covers closed-set, open-set, partial, and mixed open-partial category shifts when the target label set is unknown.Using a method designed for the wrong setting can cause catastrophic misalignment, such as aligning unknown target classes with known source classes.
  • Problem: Source-focused alignment may fail to learn target features that distinguish unknown categories from known ones.Such discriminative features may not exist among the source categories, while prior self-supervision did not exploit target cluster structure.
  • DANCE: DANCE uses neighborhood clustering to self-supervise target feature learning rather than relying only on source-category supervision.The method harnesses cluster structure specific to the target domain.
  • DANCE: DANCE combines target clustering with entropy separation to preserve useful source features, align target examples, or reject them as unknown.Figure 1 depicts clustering neighboring target examples and entropy-based consideration of source alignment.

2 Related Work

Related domain-adaptation methods generally assume a particular relationship between source and target categories. Self-supervised approaches can discover target neighborhoods, but some alternatives require knowing the target cluster count.

  • Domain Adaptation: Closed-set adaptation assumes every target example belongs to a source class and commonly minimizes source-target feature-distribution distance.Methods use domain classifiers, pseudo-labels, or clustering to reduce distribution shift.
  • Domain Adaptation: Partial adaptation treats target classes as a subset of source classes, while open-set adaptation handles target classes absent from the source.Partial methods importance-weight source examples; open-set methods identify unknown target examples but assume unknowns are present.
  • Self-Supervised Learning: Self-supervised learning uses unlabeled images for surrogate tasks, including jigsaw solving, instance discrimination, and cluster-index prediction.Applying cluster-index prediction to universal DA is challenging because it requires knowing the number of target clusters.
  • Self-Supervised Learning: Instance-discrimination methods can discover neighborhoods and cluster samples without specifying the number of clusters.This provides a related route for learning target structure in universal domain adaptation.

3 DANCE: Domain Adaptive Neighborhood Clustering via Entropy optimization

DANCE addresses universal domain adaptation by learning clustered target features while aligning target samples with known source classes or rejecting them as unknown. It combines neighborhood clustering, entropy separation, domain-specific batch normalization, and source classification in a unified objective.

  • Universal domain adaptation must handle closed-set, open-set, partial, and mixed category shifts without knowing the setting beforehand.
  • Neighborhood Clustering: DANCE uses neighborhood clustering to self-supervise target feature learning by associating each target point with a nearby target sample or source prototype.The objective minimizes the entropy of similarity distributions over target samples and source prototypes.
  • Neighborhood Clustering: A memory bank stores target features absent from the current mini-batch, while prototype vectors are included to compute similarities across target samples and source classes.The memory is updated with current mini-batch features and retains older features for subsequent similarity calculations.
  • DANCE improves over the source-only model in all evaluated universal settings and achieves the strongest average rank among the compared baselines.Table 1 averages accuracy across Office, OC, OH, and VisDA domains and adaptation scenarios.
  • Entropy Separation loss: Entropy separation pushes target samples toward known-class prototypes when classifier entropy is low and away from known classes when entropy is high.The threshold is set using the number of source classes, with a confidence margin excluding ambiguous samples from the separation loss.
  • Training with Domain Specific Batch Normalization: Domain-specific batch normalization provides weak alignment by separately processing source and target batches, while avoiding potentially harmful strong distribution alignment.The final objective combines source classification, neighborhood clustering, and entropy separation losses using a shared weighting parameter λ.

4 Experiments

Experiments evaluate DANCE across all Universal DA sub-cases and multiple object-classification benchmarks, using common evaluation protocols and comparisons. DANCE improves over source-only adaptation broadly, performs especially strongly in open-set and open-partial settings, and learns target features that cluster known and unknown examples.

  • Experimental Settings: Experiments compare DANCE across closed-set, partial, open-set, and open-partial domain adaptation on four object-classification datasets.The evaluation follows established settings for each Universal DA sub-case and includes Office, OfficeHome, VisDA, Caltech, and ImageNet analyses.
  • Universal Comparison: DANCE is the only method that improves over source-only performance in all Universal DA settings and performs best on open-set and open-partial adaptation.It also performs best in partial and closed adaptation for OfficeHome and VisDA, while average accuracy and rank exceed other baselines.
  • CDA and PDA: DANCE significantly improves accuracy over source-only in closed-set and partial adaptation, while remaining comparable to specialized baselines in partial adaptation.In closed-set adaptation, it is comparable to some baselines and superior in OfficeHome; in partial adaptation, it is comparable to ETN.
  • Feature Visualization: DANCE separates target known features from unknown features while clustering many unknown examples by their original classes without unknown-class supervision.The t-SNE visualization uses black points for known examples and different colors for unknown classes; same-class unknown examples cluster together.
  • Sensitivity to Category Shift: DANCE outperforms other methods even with many outlier source or target classes, while accuracy improves as its two losses decrease.Figure 4 examines learning curves, increasing unknown classes, and increasing source classes in VisDA partial and related settings.
  • Ablation by Clustering Unknown Examples: DANCE preserves or improves unknown-class accuracy, whereas baseline methods substantially worsen unknown classification; neighborhood clustering and entropy separation both contribute.The ablation uses OfficeHome with 15 known and 50 unknown classes and trains a linear classifier using one labeled target example per category.
  • The Number of Unknown Classes: As the number of unknown target classes increases, all methods decline, but DANCE consistently performs better and remains robust to this change.The analysis reports both all-class accuracy and unknown-class AUROC in Office open-set adaptation from Amazon to Caltech.

5 Conclusion

DANCE addresses universal domain adaptation through neighborhood clustering and entropy separation, handling arbitrary category shifts while learning discriminative features for unknown target classes.

  • DANCE introduces neighborhood clustering and entropy separation as self-supervision-based components for arbitrary category shifts.These components support universal domain adaptation without prior knowledge of the category-shift setting.
  • DANCE outperforms the source-only model in all settings and state-of-the-art baselines in many settings.
  • DANCE extracts discriminative representations for unknown-class examples without target-domain supervision.

Broader Impact

The work aims to reduce supervised data collection through knowledge transfer from auxiliary datasets, with potential benefits for resource-limited, privacy-constrained, and sim2real applications.

  • DANCE can train deep neural networks with less supervision by transferring knowledge from auxiliary datasets.
  • The proposed method may reduce data-gathering effort for applications where large related datasets are available.
  • Potential applications include settings with limited resources, privacy-law constraints, or easily generated simulations but difficult-to-collect real data.
  • The authors note risks including criminal misuse, adversarial attacks, and limited interpretability.

A Dataset Detail

Experiments use Caltech-256, OfficeHome, and VisDA with category subsets assigned as shared, source-private, or unknown classes across partial, open-set, and open-partial settings.

  • Caltech-256: Caltech-256 experiments assign 10 shared classes, 21 source-private classes for PDA, and 11 unknown classes for OSDA.OPDA uses 10 shared, the next 10 source-private, and the remaining 11 unknown classes.
  • OfficeHome: OfficeHome contains four domains and 65 classes, with settings defined by alphabetical class subsets.PDA uses 25 shared classes; OSDA uses 15 shared classes; OPDA uses 10 shared, 5 source-private, and the remainder unknown.
  • VisDA: VisDA contains 12 classes spanning synthetic and real-image domains.The synthetic domain has 152,397 renderings, while the real domain has 55,388 images.

B Implementation Detail

Implementation details specify hardware, optimization settings, released-code baselines, evaluation metrics, checkpoint selection, and run-to-run deviation reporting.

  • Hardware: Each experiment uses one TITAN X Pascal GPU with 12GB and takes about two hours.
  • DANCE settings: DANCE training uses batch size 36, temperature 0.05, 10,000 iterations, Nesterov SGD, and an initial learning rate of 0.01.
  • Baselines: Universal comparisons use released implementations of ETN, UAN, and STA, with their provided hyperparameters when tuning produced no improvements.
  • Baselines: Setting-specific comparisons include ETN and results reported in the original papers, with “NA” indicating unavailable results.
  • Metrics: Table B evaluates open-set and open-partial adaptation using OS for all classes and OS* for known classes.
  • Evaluation protocol: Reported performance uses fixed-iteration checkpoints for fair comparison, although the best checkpoint is selected for each setting elsewhere.
  • Variability: Office and VisDA standard deviations are calculated across three runs, and DANCE shows low deviations.

C Supplemental Results

Supplemental experiments examine detailed ODA/OPDA results, self-supervision alternatives, variability, and hyper-parameter sensitivity. DANCE performs well across unknown-aware metrics and settings, while its rejection behavior depends on ρ.

  • Detailed ODA and OPDA results: DANCE performs well on both OS* and OS metrics in detailed ODA and OPDA results.ETN sometimes achieves higher OS* but has much weaker unknown-sample recognition reflected in OS.
  • Comparison with Jigsaw: DANCE outperforms a jigsaw-puzzle replacement in almost all settings, supporting clustering-based self-supervision on target samples.The comparison replaces DANCE’s neighborhood clustering loss with jigsaw puzzle loss on the target domain.
  • Results with standard deviations: Three-run results include standard deviations, which are described as decent for DANCE.The main paper reports averaged accuracy, while the supplement provides variability across three runs.
  • Sensitivity to hyper-parameters: Increasing ρ assigns more examples as known and decreases performance on unknown examples.The sensitivity analysis varies λ, the margin m, and ρ; ρ is determined from the number of known classes.
Loading 2002.07953v3…