PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Effectiveness of Image Rotation for Open Set Domain Adaptation

Silvia Bucci, Mohammad Reza Loghmani, Tatiana Tommasi

arXiv:2007.12360v1cs.CV

TL;DR

OSDA must handle domain shift while rejecting target classes absent from the source, but existing evaluation and reproducibility practices present concerns. ROS uses rotation recognition in two stages for target separation and domain alignment, and experiments on Office-31 and Office-Home report stronger performance than competing methods while motivating HOS as a balanced metric.

  • Problem

    OSDA involves different source and target class sets, requiring separation of shared known classes from target-private unknown classes before alignment.

  • Method

    ROS uses rotation recognition to separate known and unknown target samples, align known target samples with source data, and classify unknowns.

  • Results

    ROS outperforms competing methods on Office-31 and Office-Home, achieving the highest average performance and outperforming the state of the art on reported domain shifts.

  • Takeaways & Limitations

    HOS balances known-class recognition with unknown rejection, while the experiments underscore the need for reproducible OSDA evaluation.

  • Takeaways & Limitations

    The OSDA formulation assumes target classes include all source classes plus additional private classes, with differing source and target distributions.

Abstract

from arXiv · show

Open Set Domain Adaptation (OSDA) bridges the domain gap between a labeled source domain and an unlabeled target domain, while also rejecting target classes that are not present in the source. To avoid negative transfer, OSDA can be tackled by first separating the known/unknown target samples and then aligning known target samples with the source data. We propose a novel method to addresses both these problems using the self-supervised task of rotation recognition. Moreover, we assess the performance with a new open set metric that properly balances the contribution of recognizing the known classes and rejecting the unknown samples. Comparative experiments with existing OSDA methods on the standard Office-31 and Office-Home benchmarks show that: (i) our method outperforms its competitors, (ii) reproducibility for this field is a crucial issue to tackle, (iii) our metric provides a reliable tool to allow fair open set evaluation.

1 Introduction

OSDA addresses domain shift when target data include classes absent from the source, requiring unknown-sample isolation before adaptation. ROS uses rotation-based self-supervision for separation and alignment, while introducing an evaluation metric and examining reproducibility.

  • OSDA relaxes closed-set assumptions by allowing target samples from unknown classes, making their isolation important before reducing domain shift.
  • ROS uses a two-stage rotation-based method to separate known and unknown target samples, then reduce domain shift between source and known target data.Stage I predicts relative rotations for separation; Stage II applies rotation again during alignment.
  • The method introduces rotation recognition for both cross-domain robustness and novelty detection in OSDA.
  • ROS is evaluated on Office-31 and Office-Home alongside existing OSDA methods, with the study highlighting reproducibility concerns.The paper also proposes a new metric intended to represent open-set performance more appropriately.
  • The authors provide a PyTorch implementation and replication instructions for their experiments.

2 Related Work

Related work frames ROS within self-supervised learning, anomaly detection, and domain adaptation. OSDA combines domain alignment with known/unknown separation because source and target domains may contain different categories.

  • The ROS schematic contains Stage I for known/unknown separation and Stage II for domain alignment.
  • ROS separates target samples and aligns those predicted as known, structuring the task as anomaly detection followed by closed-set domain adaptation.
  • Self-supervised learning creates supervision from manipulated unlabeled data, including rotation recognition, to learn transferable feature representations.Other examples include predicting patch position, colorizing grayscale images, and inpainting removed patches.
  • OSDA extends domain adaptation to settings where target data contain source-shared classes plus private classes treated as unknown.
  • OSDA methods include adversarial boundary learning, two-stage similarity-based separation and alignment, and universal adaptation approaches.

3 Method

ROS addresses OSDA’s two central challenges by separating known and unknown target samples before aligning known target data with the source. It uses rotation-based self-supervision in both stages, with different rotation classifiers and objectives.

  • Problem: OSDA combines a domain gap with a category gap, requiring shared target classes to be aligned while private target classes are rejected as unknown.Matching all source and target data can align unknown samples with source data and cause negative transfer.
  • Stage I: known/unknown separation: ROS first separates known and unknown target samples using relative and multi-rotation recognition trained on source data.Relative rotation compares an original image with its rotated counterpart, while multi-rotation combines semantic and rotation labels into 4 × |Cs| classes.
  • Stage I: known/unknown separation: The Stage I network uses an encoder with multi-rotation and semantic heads, trained with semantic cross-entropy plus rotation cross-entropy and center loss.Center loss encourages low intra-class variation and separation between different classes, supporting rotation-based novelty detection.
  • Stage I: known/unknown separation: The normality score combines rotation recognition and prediction entropy, assigning high values to likely known samples and low values to likely unknown samples.The score uses the more reliable of the rotation and entropy components and separates the target dataset using its average normality score without an additional parameter.
  • Stage II: domain alignment: Stage II aligns predicted-known target samples with the source while using predicted-unknown samples to extend the semantic classifier with an unknown class.Its classifier outputs |Cs| + 1 semantic classes and its rotation head outputs four rotation classes; the objective combines supervised and unsupervised semantic losses with a rotation loss.
  • Stage II: domain alignment: After Stage II training, the rotation head is discarded and target labels are predicted with the extended semantic classifier.The unsupervised entropy loss involves target samples recognized as known and places the decision boundary through low-density regions.

4 On reproducibility and open set metrics

The paper treats reproducibility and metric design as central evaluation issues in OSDA. It re-runs existing methods and proposes HOS, a harmonic-mean metric that balances known-class recognition with unknown rejection.

  • Motivation: OSDA is a young field, making reproducible experiments and valid evaluation metrics especially important.The paper states that OSDA was first introduced in 2017 and emphasizes these concerns as the field develops.
  • Reproducibility: The authors re-run state-of-the-art OSDA methods using public implementations and reported parameters instead of copying reported results.They describe this practice as valuable for assessing reproducibility and compare reproduced outcomes with the original papers.
  • Open set metrics: HOS uses the harmonic mean of known-class accuracy OS* and unknown accuracy UNK, penalizing large performance gaps between them.Unlike OS, HOS is high only when both known recognition and unknown rejection are strong, independently of |Cs|.

5 Experiments

ROS is evaluated against existing OSDA methods on Office-31 and Office-Home, including benchmark comparisons, openness tests, reproducibility checks, metric analysis, and ablations. Across these experiments, ROS generally performs best, while the analyses emphasize unknown-sample separation, reproducibility, and balanced evaluation.

  • Benchmark comparison: ROS outperforms the state of the art on 13 of 18 domain shifts and achieves the highest average performance on both Office-31 and Office-Home.Its HOS improvement reaches 2.2% over OSBP, the second-best method.
  • Known/unknown separation: ROS’s advantage is mainly associated with known/unknown separation: its average OS* is similar to competitors, but its average UNK is significantly higher.The t-SNE comparison shows more confused known and unknown features for OSBP than for ROS.
  • Reproducibility: Reproduced OS accuracy is 1.3% to 4.9% lower than originally published results despite using authors’ implementations and information.The study identifies discrepancies across OSDA papers and calls for more complete reproducibility information.
  • Metric analysis: HOS better reflects OSDA performance than OS because it is high only when known-class recognition and unknown rejection are both high.UAN illustrates the issue: OS reaches 72.5% on Office-Home and 91.4% on Office-31 while UNK is 0.0% for most Office-Home shifts.
  • Ablation analysis: Ablations show that removing center loss, using absolute instead of relative rotation, or omitting rotation or entropy scores significantly reduces Stage I performance.The reduced variants still outperform STA’s multi-binary classifier in the reported Stage I comparison.
  • Openness analysis: ROS maintains consistent performance as openness increases, whereas STA and OSBP deteriorate because they cannot recognize unknown samples effectively.Office-Home settings use 25, 10, or 5 known classes, with openness values of 0.62, 0.85, and 0.92.

6 Discussion and conclusions

ROS uses rotation prediction to separate known and unknown target samples before aligning predicted-known samples with the source, while HOS balances known recognition and unknown rejection. Experiments report stronger benchmark performance, steady performance as openness increases, and reproducibility problems in existing OSDA results.

  • Method: ROS separates target samples into known and unknown, then aligns predicted-known target samples with source samples using rotation-prediction variants.The two-stage design applies self-supervision to both target separation and domain alignment.
  • Evaluation: HOS is the harmonic mean of known-class recognition accuracy and unknown-sample rejection, addressing OS's diminishing unknown contribution as known classes increase.The metric is intended to assess both components of open-set performance.
  • Results: ROS outperforms competing methods on Office-31 and Office-Home and maintains steady performance under increasing openness.HOS is described as crucial for correctly assessing known and unknown samples in this evaluation.
  • Reproducibility: Re-running existing OSDA methods exposed failures to reproduce published results, highlighting a reproducibility issue for the field.The paper connects this issue to broader concerns about reproducibility in machine learning.

A Implementation Details

The implementation appendix specifies ROS architectures, optimization settings, training schedules, and backbone-dependent Stage II choices for Office-31 and Office-Home. Experiments use ResNet-50 and VGGNet configurations with repeated runs.

  • Architectures: ROS experiments use ResNet-50 and VGGNet backbones on Office-31 and Office-Home, with dataset- and backbone-specific encoder fine-tuning.ResNet-50 updates the last convolutional block, while VGG-19 updates the last two fully connected layers.
  • Architectures: The semantic classifiers C1 and C2 output known classes and known-plus-unknown classes, respectively, while rotation classifiers R1 and R2 use 4×|Cs| and 4 outputs.All classifier layers are learned from scratch in the described ResNet-50 configuration.
  • Training stages: Stage II learning rates and initialization differ by backbone: ResNet-50 inherits Stage I, whereas VGGNet does not because inheritance produced lower results.The extra unknown-class learning rate is twice the known-class rate for ResNet-50 and 1.5 times it for VGGNet.
  • Training settings: For Office-Home ResNet-50, the center-loss weight for rotation classifier R1 is reduced to λ1,2 = 0.001, while other listed weights remain higher.The appendix says the same parameter set is essentially used across settings, supporting cross-dataset and cross-architecture generalization.

B Reproducibility Study

The reproducibility study compares published and rerun OS accuracy for existing OSDA methods using provided code and documented settings. It identifies discrepancies and ambiguities in papers, implementations, hyperparameters, and unavailable variants.

  • Study design: The extended study compares reported OS accuracy for STA, OSBP, and UAN on Office-31 and Office-Home using the metric shared across all works.Table 7 reports averages over three runs across sub-domains and indicated backbones.
  • Study design: Competitor evaluations follow published dataset class partitions, including shared, private-source, and private-target classes for Office-31 and Office-Home.The appendix describes alphabetic class selection for the replicated settings.
  • Transparency: The appendix documents competitor code, hyperparameters, and training procedures to improve transparency of the rerun comparisons.It states that implementation details were summarized for complete transparency.
  • Replication settings: The study uses repository-specific training configurations for STA, OSBP, and UAN, including different iteration counts, learning rates, batch sizes, and backbone availability.These settings are described separately for Office-31, Office-Home, and available backbones.
  • Reproducibility limitations: STA has unresolved implementation ambiguities involving learning-rate selection, feature-extractor training, similarity aggregation, and missing VGGNet code.The paper-code mismatch and absent VGGNet implementation prevent reproducing that variant.

C Extended Openness Analysis

The extended openness analysis adds a lower-openness setting with 40 known classes and reports the same trend as the main analysis. The ablation focuses on ROS and its two strongest competitors.

  • Setting: The added setting uses 40 known classes with openness O = 0.38 and class IDs {0-39, 15-54, 25-64}.This extends the main paper's openness analysis toward lower openness.
  • Results: Figure 6 confirms the trend observed in the main paper, while the ablation compares ROS with OSBP and STA.UAN is excluded because of its low UNK and HOS results.

D Sensitivity analysis of the hyper-parameters

ROS is robust to hyper-parameter changes across the evaluated settings, and its performance advantage is not explained by tuning. The same hyper-parameters are used across all 18 domain pairs.

  • ROS is not very sensitive to hyper-parameter values; only λ2,1 changes HOS by more than 1.0.The analysis uses Office-31 with ResNet-50 and reports HOS variation above 1.0 only for λ2,1.
  • Setting the entropy weight to 0.1 is safe without hyper-parameter tuning.This matches the practice used in.
  • ROS outperforms OSBP (HOS=83.7) regardless of the specific hyper-parameters.The comparison is presented as evidence that the superior performance results from algorithmic novelty rather than hyper-parameter tuning.
  • The same hyper-parameters are used for all 18 domain pairs, demonstrating robustness across datasets.
  • ROS has a comparable number of parameters to competing approaches.λ1,1 and λ2,2 are constrained to the same value, leaving three parameters and two training-iteration parameters overall.

E Other Self-Supervised Tasks and Further Ablation

The ablations test whether rotation recognition is preferable to alternative self-supervised tasks and whether specific ROS design choices matter. Rotation recognition performs best in the reported comparisons, while multi-rotation and entropy loss provide measurable benefits.

  • Further Ablation: The experiments compare self-supervised tasks across the two ROS stages and include further ablations.
  • Other Self-Supervised Tasks: Rotation recognition alone outperforms both translation classification and the combination of translation classification with jigsaw-puzzle solving.Translation classification is evaluated for Stage I and jigsaw solving for Stage II on Office-31 with ResNet-50.
  • Further Ablation: The multi-rotation task is crucial for Stage I, where standard 4-Class rotation decreases AUC-ROC by 28.6%.The relative-rotation anchor is retained in the 4-Class experiment.
  • Further Ablation: Removing the entropy loss reduces performance by 1.9 percentage points on average.The effect is more evident for large domain gaps, including A→W and W→A.

F Time analysis

The time analysis evaluates training runtime on Office-31(A-W) with ResNet-50, while the accompanying procedure computes normality scores used to generate the known-target dataset. ROS is reported to be twice as fast as OSBP in the HOS comparison.

  • Time analysis: ROS is even twice as fast as OSBP in the runtime comparison for HOS performance.The analysis is conducted on Office-31(A-W) with ResNet-50.
  • Normality Score Pseudo-code: The normality-score procedure uses trained networks E and R1 to process the unlabeled target dataset.It initializes the known-target dataset and computes scores for target samples.
  • Normality Score Pseudo-code: Algorithm 1 computes normality scores and generates the known target dataset Dknw.
  • Normality Score Pseudo-code: The rotation score constructs a vector from selected elements of z for each class, while the entropy score is computed as z · log(z)/log(|Cs|).
  • Normality Score Pseudo-code: For each target sample, the procedure averages entropy scores, sums rotation scores, takes the maximum rotation value, and sets ηj to max(o, 1 − h).The resulting normality score is used to distinguish known from unknown target samples.
Loading 2007.12360v1…