PaperlayerView exact revision on arXiv

Source-linked AI summary

OVANet: One-vs-All Network for Universal Domain Adaptation

Kuniaki Saito, Kate Saenko

arXiv:2104.03344v4cs.CV

TL;DR

Universal domain adaptation must classify known target categories while rejecting unknown ones without knowing the category shift or unknown-sample proportion. OVANet learns a rejection threshold from source inter-class distances with one-vs-all classifiers, adapts them through open-set entropy minimization, and outperforms baselines across varied settings. Its evaluation also notes that H-score weights known and unknown recognition equally, which can overemphasize unknown accuracy when unknown instances are scarce.

  • Problem

    Universal domain adaptation lacks a practical way to reject unknown target classes when their number and properties are unavailable without target supervision.

  • Method

    OVANet trains one-vs-all open-set classifiers with hard-negative sampling on source data, then adapts them to unlabeled target data using open-set entropy minimization.

  • Results

    OVANet outperforms baselines across universal domain adaptation benchmarks with varied proportions of shared and unknown classes.

  • Takeaways & Limitations

    The framework provides a simple threshold-learning approach that is applicable across diverse universal adaptation and semi-supervised settings.

  • Takeaways & Limitations

    H-score weights known and unknown recognition equally and can overemphasize unknown accuracy when unknown instances are scarce.

Abstract

from arXiv · show

Universal Domain Adaptation (UNDA) aims to handle both domain-shift and category-shift between two datasets, where the main challenge is to transfer knowledge while rejecting unknown classes which are absent in the labeled source data but present in the unlabeled target data. Existing methods manually set a threshold to reject unknown samples based on validation or a pre-defined ratio of unknown samples, but this strategy is not practical. In this paper, we propose a method to learn the threshold using source samples and to adapt it to the target domain. Our idea is that a minimum inter-class distance in the source domain should be a good threshold to decide between known or unknown in the target. To learn the inter-and intra-class distance, we propose to train a one-vs-all classifier for each class using labeled source data. Then, we adapt the open-set classifier to the target domain by minimizing class entropy. The resulting framework is the simplest of all baselines of UNDA and is insensitive to the value of a hyper-parameter yet outperforms baselines with a large margin.

1. Introduction

Universal domain adaptation addresses unknown category shifts between labeled source and unlabeled target domains, where unknown samples must be rejected without target supervision. OVANet learns this rejection threshold from source inter-class distances using one-vs-all classifiers and adapts them through target entropy minimization.

  • Motivation: Universal domain adaptation handles uncertain category shifts while transferring knowledge from labeled source data to unlabeled target data.The target may contain unknown categories, and their prevalence and properties are unavailable without manual labeling.
  • Motivation: Existing methods manually set unknown-rejection thresholds through validation, prior unknown-sample ratios, or synthesized negatives.These strategies require target information, accurate ratio estimates, or tuning that may not reflect real unknown data.
  • OVANet: OVANet uses the minimum inter-class distance between source categories as a threshold for distinguishing known from unknown target samples.Samples outside every class margin are treated as unknown, while samples within a class margin can receive that class label.
  • OVANet: The method trains one-vs-all classifiers with hard-negative sampling to learn class boundaries and efficiently estimate the relevant source distances.Each classifier treats its class as positive and other classes as negative, emphasizing the nearest negative class.
  • Evaluation: Open-set entropy minimization adapts the learned classifiers to unlabeled target samples, while experiments vary the proportions of shared and unknown classes.The evaluation reports that OVANet outperforms baselines that use the unknown-sample ratio.

2. Related Work

Related work covers domain adaptation under differing label spaces, open-set recognition, and one-vs-all classifiers. OVANet differs by learning open-set boundaries from hard negative classes and adapting them with open-set entropy minimization.

  • Domain Adaptation: Universal domain adaptation combines closed-set, open-set, and partial domain adaptation settings when the category shift is unknown in advance.Open-set DA includes target-private classes, partial DA includes source-private classes, and UNDA handles mixtures of these cases.
  • Domain Adaptation: Prior universal and open-set adaptation methods compute confidence scores and reject samples below thresholds that are validated or predetermined.Such threshold selection is described as unrealistic or dependent on unavailable target information.
  • Open-set Recognition: Open-set recognition methods seek anomaly measurements or representations that distinguish known from unknown test samples.Recent approaches also use contrastive learning to learn representations for detecting out-of-distribution samples.
  • One-vs-All Classification: OVANet trains open-set classifiers around positive and nearest negative class boundaries, rather than relying only on generic anomaly measurements.Its hard-negative focus targets classes that are similar to the positive class.

3. OVANet

OVANet combines one-vs-all open-set classifiers with a closed-set classifier to identify known and unknown target samples without manually setting a rejection threshold. It trains open-set boundaries using hard negatives, adapts them through target entropy minimization, and uses both classifiers during testing.

  • 3.1. Hard Negative Classifier Sampling (HNCS): Hard-negative classifier sampling trains each class classifier with samples from similar but different classes as negatives.The loss uses the positive class and the hardest negative class.
  • 3.1. Hard Negative Classifier Sampling (HNCS): OVANet uses |Ls| one-vs-all sub-classifiers, each distinguishing its corresponding class as an in-lier from outliers.Each sub-classifier outputs probabilities for the sample being an in-lier or outlier.
  • 3.2. Open-set Entropy Minimization (OEM): Open-set entropy minimization adapts the source-trained open-set classifiers to unlabeled target samples by minimizing the average entropy across classifiers.This increases confidence in known-versus-unknown predictions while accounting for target-domain characteristics.
  • 3.2. Open-set Entropy Minimization (OEM): Unlike closed-set entropy minimization, OVANet can keep unknown target instances unknown because its open-set classifiers explicitly represent the unknown concept.The method aligns known target samples to source samples while retaining unknown samples as unknown.
  • 3.3. Learning: OVANet shows stable H-score and accuracy across different openness settings, while baselines can degrade substantially.In Office experiments, the number of unknown classes varies while the number of known classes remains fixed at 10.
  • 3.3. Learning: At test time, the closed-set classifier identifies the nearest known class, and that class’s open-set score determines whether the sample is known or unknown.The framework jointly trains the feature extractor, open-set classifiers, and closed-set classifier; it has one hyper-parameter, λ.

4. Experiments

Experiments across four benchmarks and varied domain-adaptation settings show that OVANet is consistently superior or comparable to baselines, without per-setting hyper-parameter optimization. Ablations and analyses support the roles of open-set entropy minimization, hard-negative selection, and combined open- and closed-set classifiers.

  • Robustness to openness: OVANet remains stable as the number of unknown OfficeHome classes varies, whereas baseline performance can degrade substantially under some openness settings.Figure 6 fixes 10 shared and 5 source-private classes while varying target-private classes.
  • Overall results: OVANet is superior or comparable to baselines across four benchmark datasets and diverse settings without optimizing the hyper-parameter for each setting.The evaluation covers Office, OfficeHome, VisDA, and DomainNet.
  • Overall results: More than 10 points separate OVANet from existing methods in VisDA and OfficeHome H-score results, while OVANet uses no tuned hyper-parameter.CMU selects optimal threshold hyper-parameters separately for each dataset, whereas OVANet achieves the best H-score without tuning.
  • Ablation and separation analysis: Open-set entropy minimization improves both closed-set accuracy and unknown-sample rejection, while hard-negative classifier selection improves the learned separation.The hard-negative benefit is especially evident in DomainNet, which contains more known classes and therefore more irrelevant negative classes.
  • Ablation and separation analysis: Using both open-set and closed-set classifiers is necessary because open-set classifiers alone do not distinguish known classes sufficiently for discriminative classification.Removing the closed-set classifier causes a large performance degradation.
  • Semi-supervised learning: OVANet outperforms baselines on all reported metrics in semi-supervised NAbird experiments with 300 known and 255 unknown categories.Half of each known class is labeled, while the remaining known samples and all unknown-class samples are unlabeled.

5. Conclusion

OVANet trains a One-vs-All classifier for each class and uses its output to distinguish known from unknown samples. The framework is presented as simple, broadly effective across settings, and applicable to semi-supervised learning.

  • OVANet trains a One-vs-All classifier for each class to decide whether target samples are known or unknown.
  • The framework is described as the simplest of all UNDA methods while showing strong performance across diverse settings.
  • Extensive evaluation demonstrates OVANet’s applicability to semi-supervised settings.

A. Experimental Details

The experiments specify implementation settings for UNDA, semi-supervised learning, and open-set recognition, including data selection, model backbones, and training losses.

  • Implementation: UNDA experiments use a source and target batch size of 36, with learning rates of 0.01 for new layers and 0.001 for ResNet50 backbone layers.The learning rate uses inverse decay scheduling, while VGGNet follows the learning-rate setup of prior work.
  • Category Selection: Known and unknown categories are selected using benchmark defaults or alphabetical ordering, with NAbird assigning the first 300 classes as known.The remaining NAbird classes are treated as unknown.
  • Semi-supervised Learning: The semi-supervised experiment matches UNDA implementation, replacing the source with labeled data and the target with unlabeled data.ResNet50 is used as the backbone.
  • Open-Set Recognition: Open-set recognition trains models from scratch with WideResNet and follows FixMatch implementation, adding contrastive loss relative to OVAN w/o OEM.
  • Training Objective: The overall training loss combines source loss with λ times instance discrimination loss: Lall = E (xi,yi)∼D Lsrc(xi, yi) + λ E xi∼D Lsimc(xi).The instance discrimination loss is from SimCLR, and λ is set to 0.1.

B. Additional Results of UNDA

Additional experiments report OVANet’s performance across open-partial domain adaptation and uncertainty analyses. The results show strong comparative performance, while the AUROC analysis identifies cases where its output is not the best uncertainty calibration.

  • Open-partial DA on OfficeHome: OVANet outperforms baselines with a large margin in all OfficeHome open-partial domain adaptation scenarios.
  • Detailed metrics on Office: Office open-set results report accuracy over all samples, H-score, and unknown-sample rejection accuracy.
  • AUROC of Entropy and OVANet: OVANet’s output often outperforms softmax output in AUROC, although entropy is sometimes better at calibrating uncertainty.The analysis trains a model without open-set entropy minimization to compare entropy, predicted-category softmax, and one-vs-all outputs.
Loading 2104.03344v4…