PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-World Semi-Supervised Learning

Kaidi Cao, Maria Brbic, Jure Leskovec

arXiv:2102.03526v3cs.LGcs.CV

TL;DR

Semi-supervised learning commonly assumes that unlabeled test data contains only previously seen classes, but open-world data may also contain unseen classes. The paper formalizes this setting and proposes ORCA, an end-to-end method using an uncertainty adaptive margin to classify seen classes and discover novel ones. ORCA consistently outperforms baselines, including 25% improvement on seen and 96% improvement on novel ImageNet classes.

  • Problem

    Open-world SSL addresses the mismatch in which unlabeled test data contains both seen classes and an unknown number of unseen classes requiring classification or discovery.

  • Method

    ORCA is an end-to-end framework using an uncertainty adaptive margin to control intra-class variance while classifying seen classes and forming novel classes.

  • Results

    ORCA consistently outperforms alternative baselines across image and single-cell evaluations, with 25% and 96% improvements on seen and novel ImageNet classes.

  • Takeaways & Limitations

    The results support evaluating machine-learning models in the more realistic open-world setting rather than only under traditional closed-world assumptions.

  • Takeaways & Limitations

    The method includes assumptions about class shift between labeled and unlabeled data and uses maximum entropy regularization because prior class distributions are generally unavailable.

Abstract

from arXiv · show

A fundamental limitation of applying semi-supervised learning in real-world settings is the assumption that unlabeled test data contains only classes previously encountered in the labeled training data. However, this assumption rarely holds for data in-the-wild, where instances belonging to novel classes may appear at testing time. Here, we introduce a novel open-world semi-supervised learning setting that formalizes the notion that novel classes may appear in the unlabeled test data. In this novel setting, the goal is to solve the class distribution mismatch between labeled and unlabeled data, where at the test time every input instance either needs to be classified into one of the existing classes or a new unseen class needs to be initialized. To tackle this challenging problem, we propose ORCA, an end-to-end deep learning approach that introduces uncertainty adaptive margin mechanism to circumvent the bias towards seen classes caused by learning discriminative features for seen classes faster than for the novel classes. In this way, ORCA reduces the gap between intra-class variance of seen with respect to novel classes. Experiments on image classification datasets and a single-cell annotation dataset demonstrate that ORCA consistently outperforms alternative baselines, achieving 25% improvement on seen and 96% improvement on novel classes of the ImageNet dataset.

1 INTRODUCTION

The paper introduces open-world semi-supervised learning, where unlabeled data may contain both previously seen and unseen classes. It proposes ORCA, an end-to-end method that discovers novel classes while classifying seen ones, outperforming baselines across evaluations.

  • 1 INTRODUCTION: Open-world SSL requires classifying unlabeled instances into seen classes or forming and assigning them to novel classes.Unlike closed-world SSL, the unlabeled test set may contain an unknown number of unseen classes.
  • 1 INTRODUCTION: Open-world SSL differs from robust SSL, which rejects novel instances, and novel class discovery, which assumes all unlabeled classes are novel.Existing approaches therefore do not directly address the combined recognition-and-discovery setting.
  • 1 INTRODUCTION: ORCA uses an uncertainty adaptive margin to reduce the intra-class variance gap between seen and novel classes during training.The method addresses bias caused by learning discriminative features for seen classes faster than for novel classes.
  • 1 INTRODUCTION: ORCA is an end-to-end framework that classifies seen classes while grouping similar unlabeled instances into newly discovered classes.It does not require the number of novel classes to be known ahead of time.
  • 1 INTRODUCTION: 25% and 96% improvements on ImageNet were achieved for seen and novel classes, respectively, over alternative baselines.The evaluation covers image classification datasets and a single-cell annotation dataset.

2 RELATED WORK

Open-world SSL extends related settings by combining recognition of seen classes with discovery of novel classes in unlabeled data. The paper contrasts this setting with SSL, robust SSL, novel class discovery, open-set recognition, and generalized zero-shot learning.

  • 2 RELATED WORK: Novel class discovery clusters unlabeled data composed entirely of classes disjoint from the labeled data, so it does not recognize seen classes.Open-world SSL allows unlabeled data to contain both seen and novel classes.
  • 2 RELATED WORK: Closed-world SSL assumes labeled and unlabeled data come from the same set of classes.Robust SSL relaxes this assumption by allowing novel-class instances in unlabeled data.
  • 2 RELATED WORK: Open-set recognition treats novel classes inductively and requires rejecting their instances, whereas open-world SSL discovers and assigns instances to individual novel classes.The paper extends open-set methods with a baseline that discovers classes among rejected instances.
  • 2 RELATED WORK: Generalized zero-shot learning assumes auxiliary attributes that uniquely describe both seen and novel classes.The paper identifies this prior-knowledge requirement as restrictive for practical application.

3 PROPOSED APPROACH

Open-world SSL requires a model to classify unlabeled instances into seen classes or discover and assign them to novel classes. ORCA addresses this setting with uncertainty-adaptive margins, pairwise pseudo-labeling, and regularization.

  • 3.1 OPEN-WORLD SEMI-SUPERVISED LEARNING SETTING: Open-world SSL extends semi-supervised learning by requiring unlabeled instances to be classified into seen classes or assigned to newly formed novel classes.The setting includes labeled data from seen classes and unlabeled data containing both seen and an unknown number of unseen classes.
  • 3.3 SUPERVISED OBJECTIVE WITH UNCERTAINTY ADAPTIVE MARGIN: Uncertainty-adaptive margins control the faster learning of seen classes, reducing intra-class variance differences that otherwise bias pseudo-labels toward seen classes.Standard cross-entropy updates seen-class gradients but not novel-class gradients, creating classifier imbalance and seen-class bias.
  • 3.2 OVERVIEW OF ORCA: ORCA uses an embedding network and classification heads for previously seen classes plus expected novel classes, with unused heads left inactive when the novel-class count is unknown.Predictions are obtained by selecting the highest-scoring head; a sufficiently large number of novel-class heads can be initialized when the count is unavailable.
  • 3.2 OVERVIEW OF ORCA: The ORCA objective combines a supervised objective with uncertainty-adaptive margin, a pairwise objective, and a regularization term.The three components respectively address supervised classification, pseudo-label-based pairwise learning, and prevention of degenerate class assignments.
  • 3.4 PAIRWISE OBJECTIVE: The pairwise objective groups same-class instances and generates unlabeled-data pseudo-labels using only the most confident positive pairs, updated online during training.The method uses labeled annotations for labeled pairs and generated pseudo-labels for unlabeled pairs; noisy pseudo-labels are considered detrimental to cluster learning.
  • 3.5 REGULARIZATION TERM: A KL-divergence regularizer discourages assigning all instances to one class, while maximum-entropy regularization is used when the prior label distribution is unknown.The authors report that this regularization does not negatively affect ORCA performance with unbalanced data distributions.

4 EXPERIMENTS

ORCA is evaluated across image and single-cell datasets using adapted baselines, ablations, and settings with unknown novel-class counts. It consistently outperforms alternatives, while uncertainty adaptive margins improve training behavior and pseudo-label quality.

  • Experimental setup: ORCA is evaluated on CIFAR-10, CIFAR-100, ImageNet, and a cross-tissue single-cell annotation dataset, with existing methods extended for open-world SSL.The evaluation includes novel class discovery, SSL, and open-set recognition baselines.
  • Benchmark evaluation: ORCA consistently outperforms baselines, improving seen-class accuracy by 21% on CIFAR-100 and 25% on ImageNet.On novel classes, improvements are 51% on CIFAR-100, 96% on ImageNet, and 104% on the single-cell dataset.
  • Ablation and mechanism: The uncertainty adaptive margin helps ORCA reduce novel-class intra-class variance, whereas ORCA-ZM learns seen classes quickly but performs poorly on novel classes.Without the adaptive margin, learning-rate decay can expose overfitting and noisy-pseudolabel issues associated with variance differences.
  • Ablation and mechanism: ORCA surpasses other baselines’ final performance after only 12 epochs, while adaptive margins improve pseudo-label quality and remain robust to λ.These findings support the importance of the uncertainty adaptive margin.
  • Unknown novel-class count: With an estimated 124 classes, ORCA finds 114 novel clusters and achieves a 97% improvement over RankStats.It automatically prunes unused classification heads and performs only slightly worse than when the novel-class count is known.
  • Objective ablation: Both ORCA’s supervised objective and regularization term are essential components, while the pairwise objective enables novel-class discovery.Removing either the supervised objective or regularization term degrades performance on CIFAR-100.

5 CONCLUSION

The paper defines open-world SSL as handling seen and novel classes in unlabeled test data, and proposes ORCA to control their intra-class variance during training. Experiments report large-margin improvements over alternative baselines and motivate more realistic open-world evaluation.

  • 5 CONCLUSION: Open-world SSL requires assigning unlabeled instances to seen classes or forming and assigning them to novel classes.Novel classes may appear in unlabeled test data.
  • 5 CONCLUSION: ORCA uses an uncertainty adaptive margin to control intra-class variance for seen and novel classes during training.The method is presented as the paper’s solution to open-world SSL.
  • 5 CONCLUSION: ORCA outperforms alternative baselines by a large margin across the reported experiments.The conclusion describes ORCA as effectively solving the open-world SSL setting.
  • 5 CONCLUSION: The work advocates shifting machine-learning evaluation from the traditional closed-world setting toward open-world evaluation.This recommendation is framed as a response to the presence of novel classes in unlabeled test data.

A ADDITIONAL RELATED WORK

Open-world SSL differs from related settings in how it treats novel target classes and whether feature-distribution shift is assumed. Margin-loss research provides context for using margins to regulate class variation.

  • Universal domain adaptation: Universal domain adaptation treats target-only classes as unknowns to reject, whereas open-world SSL aims to discover individual novel classes.Open-world SSL also does not assume feature-distribution shift between labeled and unlabeled data.
  • Margin loss: Margin-based losses adjust intra- and inter-class variation to improve classification accuracy.Prior work includes large-margin softmax, angular softmax, circle loss, and additive margin softmax.

B IMPLEMENTATION DETAILS

The implementation evaluates ORCA and adapted baselines across image, single-cell, and open-world SSL settings using dataset-specific training configurations and evaluation protocols.

  • Computational cost: ORCA adds no computational overhead over comparable methods, with CIFAR experiments taking less than an hour and ImageNet experiments taking a few hours.The implementation uses PyTorch and NVIDIA RTX 2080 Ti hardware.
  • Evaluation metrics: Performance is measured with accuracy on seen classes and accuracy plus normalized mutual information on novel classes.Novel-class accuracy uses Hungarian optimal assignment, while joint accuracy matches both seen and novel classes.
  • Image datasets: On image datasets, experiments use modified ResNet-18 or ResNet-50 backbones with standard augmentation and stochastic-gradient training.CIFAR experiments use a modified ResNet-18, whereas ImageNet uses ResNet-50.
  • Single-cell dataset: The single-cell dataset contains 93,718 cells from 50 mouse cell types across 23 organs and uses a two-layer fully connected backbone without pretraining or augmentation.The dataset is highly unbalanced, with skewness 3.8 and class sizes ranging from 479 to 13,268 examples.
  • Baseline comparisons: ORCA is compared with novel class discovery, semi-supervised, and open-set recognition methods adapted to the open-world SSL setting.These adaptations address missing seen-class recognition or novel-class detection capabilities in the original methods.

C ADDITIONAL RESULTS

Additional experiments test ORCA under reduced supervision, varying class mixtures and hyperparameters, alternative pretraining and margin strategies, and unbalanced data. Across these analyses, its reported advantages persist, while performance reflects trade-offs between seen and novel classes.

  • Reduced labeled data: ORCA retains substantial improvements over baselines when only 10% of examples in each seen class are labeled.This evaluation covers all four benchmark datasets.
  • Different novel-class ratios: ORCA’s seen-class performance remains stable across different seen/novel class ratios, while novel-class performance decreases as the proportion of novel classes increases.Even with 90% novel classes, ORCA outperforms baselines evaluated with 50% novel classes.
  • Fixed negative margin: A margin value of 0.5 achieves the best performance among the tested fixed negative margins and is used for ORCA-FNM experiments.The comparison is conducted on CIFAR-100.
  • Sensitivity analysis: Higher η1 slightly improves seen-class performance, lower η2 improves seen-class performance, and η1 = η2 = 1 is optimal for novel classes.η1 weights the supervised objective, while η2 weights maximum-entropy regularization.
  • Uncertainty regularization: A slightly larger λ improves novel-class accuracy at the cost of lower seen-class accuracy, whereas smaller λ favors seen classes.λ is the uncertainty regularizer associated with matching intra-class variance between labeled and unlabeled data.
  • Pseudo-label quality: The uncertainty adaptive margin improves generated pseudo-label quality compared with zero-margin and fixed-negative-margin approaches.The experiment evaluates pseudo-label accuracy during training on CIFAR-100.
  • Unbalanced data: Maximum-entropy regularization consistently improves performance over the non-regularized model even under a 10:1 long-tailed class imbalance.The evaluation uses unbalanced CIFAR-10 and CIFAR-100 data and reports accuracy and NMI.
  • Pretraining and margin variants: ORCA only slightly degrades when pretrained on labeled data alone, benefits further from RotationNet pretraining, and underperforms its uncertainty adaptive margin with entropy or linear scheduling.These comparisons are reported on CIFAR-100.
Loading 2102.03526v3…