Source-linked AI summary
Automatically Discovering and Learning New Visual Categories with Ranking Statistics
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, Andrew Zisserman
TL;DR
The paper asks how to discover novel visual classes when only other classes have labels and the new classes remain unlabelled. It combines self-supervised representation learning, rank-statistics transfer, and joint optimization of labelled recognition and unlabelled clustering. On standard benchmarks, the combination outperforms existing methods for the same task.
Problem
The task is to discover new visual classes in unlabelled images using labelled examples from other classes, despite the absence of labels for the new classes.
Method
The method self-supervises a shared representation on labelled and unlabelled images, uses rank statistics to generate pairwise pseudo-labels, and jointly optimizes labelled and unlabelled objectives.
Results
On standard benchmarks, the combination of these ideas results in much better performance than existing methods that solve the same task.
Takeaways & Limitations
Rank statistics provide robust clustering, while joint incremental optimization can reinforce labelled recognition and unlabelled clustering without forgetting.
Takeaways & Limitations
The approach assumes that labelled and unlabelled classes are disjoint and that new classes have some degree of visual similarity to known classes.
Abstract
from arXiv · showhide
We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. This setting is similar to semi-supervised learning, but significantly harder because there are no labelled examples for the new classes. The challenge, then, is to leverage the information contained in the labelled images in order to learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data. In this work we address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labeled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) we use rank statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data. We evaluate our approach on standard classification benchmarks and outperform current methods for novel category discovery by a significant margin.
1 INTRODUCTION
The paper addresses automatic discovery of new visual classes without labels by combining unbiased representation learning, rank-based transfer, and joint supervised–unsupervised optimization. Evaluations on public benchmarks show large-margin gains over applicable existing methods.
- Motivation: New-class discovery remains a clustering problem even when supervised learning provides effective recognition of known classes.Features learned from labelled classes may not represent new classes adequately, and unlabelled images still require clustering.
- Method: Self-supervised pre-training on labelled and unlabelled images reduces bias toward the labelled classes before supervised fine-tuning.Labels are introduced only after pre-training to learn the known-class classifier and fine-tune the deepest CNN layers.
- Method: Rank statistics transfer information from labelled images by comparing unlabelled representation vectors through shared maximally activated components.The resulting noisy pairwise pseudo-labels train a similarity function for clustering unlabelled images.
- Method: Joint optimization uses labelled targets and unlabelled pseudo-labels together, while incremental learning lets information flow between the two subsets.This design avoids the forgetting issue associated with a sequential approach.
- Results: The method outperforms all applicable existing techniques by a large margin on several public benchmarks.The authors present this benchmark result as evidence of the approach’s effectiveness.
2 METHOD
The method learns a representation for known and novel classes by combining self-supervised pretraining, rank-based transfer, and joint labelled–unlabelled optimization. It uses pairwise pseudo-labels and consistency training to cluster unlabelled images while retaining labelled-class knowledge.
- Problem: The problem assumes disjoint labelled and unlabelled classes, with the number of unlabelled classes known in advance.The method seeks a representation that recognizes known classes and discovers new ones.
- Self-supervised learning: Supervised bootstrapping can over-specialize the representation to labelled classes, producing poor features for novel classes.The method therefore avoids using labels during representation initialization.
- Self-supervised learning: Self-supervised pretraining on the union of labelled and unlabelled images initializes a less label-biased CNN representation before fine-tuning the known-class classifier.RotNet predicts one of four image-rotation angles, and only the classifier head and final representation macro-block are updated during fine-tuning.
- Transfer learning via rank statistics: Rank statistics transfer information to unlabelled clustering by assigning pairwise pseudo-labels when images share the same top-k representation dimensions.The experiments use k = 5; similar pairs receive sij = 1 and dissimilar pairs receive sij = 0.
- Transfer learning via rank statistics: A shared representation supports comparison of unlabelled images, whose descriptors are trained with a binary cross-entropy loss and mapped to Cu novel-class outputs.The maximum output index provides each unlabelled image's predicted novel-class label.
- Joint training on labelled and unlabelled data: Jointly optimizing labelled cross-entropy, unlabelled pairwise BCE, and consistency losses avoids sequential training's forgetting risk and reduces augmentation-induced variability in ranking labels.The consistency term uses MSE and a ramp-up coefficient ω(t).
3 EXPERIMENTS
Experiments evaluate novel-category discovery across five benchmark datasets, using clustering accuracy on unlabelled classes and an incremental-learning setting. The method’s components improve clustering, separate novel-class representations during training, and support recognition of both old and new classes.
- Datasets and metrics: Experiments use CIFAR-10, CIFAR-100, SVHN, OmniGlot, and ImageNet, with clustering accuracy (ACC) measured on unlabelled classes.OmniGlot and ImageNet results average over 20 and 3 evaluation splits, respectively.
- Ablation study: Removing BCE causes CIFAR-10 ACC to fall from 90.4% to 26.2%, while removing consistency, cross-entropy, or self-supervision lowers ACC by 7.8%, 5.7%, or 1.0%.The ablation evaluates clustering on the unlabelled data, using the same points for training and testing without augmentation for cluster assignment.
- Novel category discovery: The method substantially outperforms k-means and prior KCL, MCL, and DTC methods on CIFAR-10, CIFAR-100, and SVHN.The comparisons use ACC on the unlabelled set.
- Novel category discovery: Training progressively separates initially overlapping CIFAR-10 unlabelled-class clusters in t-SNE representations.The unlabelled examples are dogs, frogs, horses, ships, and trucks, with colors denoting ground-truth labels.
- Novel category discovery: On OmniGlot and ImageNet, the method achieves 95.0% and 82.5% ACC, compared with 89.0% and 78.8% for Han et al. (2019).Metric learning is used for these datasets because self-supervision provided no obvious gains in this setting.
- Incremental learning: Incremental learning yields additional gains of 1.3, 2.0, and 0.2 percentage points on CIFAR-10, CIFAR-100, and SVHN, while the method performs well on both old and new classes.The joint classifier is evaluated on a testing set entirely disjoint from training data.
4 RELATED WORK
The paper situates novel category discovery at the intersection of semi-supervised learning, transfer learning, clustering, and rank statistics, while distinguishing its setting from standard approaches.
- Semi-supervised learning: Unlike standard semi-supervised learning, novel category discovery assumes unlabelled images belong to classes different from the labelled data.SSL typically propagates labels within the same class set, whereas this task transfers information from known classes to new ones.
- Transfer learning: Unlike conventional transfer learning, this work transfers information from a labelled dataset to an unlabelled target dataset.Most transfer-learning settings use fully annotated source and target data.
- Clustering: Clustering methods partition unlabelled collections, but the task is often ill-posed because multiple partitioning criteria can be equally valid.
- Novel category discovery: The closest prior methods learn embeddings from labelled data before fine-tuning or clustering on unlabelled data, including KCL and MCL.Han et al. use metric learning, while Hsu et al. introduce similarity-prediction-based clustering methods.
- Rank statistics: The paper introduces rank statistics for novel category discovery with deep neural networks, extending earlier uses of rank statistics for vector similarity.
5 CONCLUSIONS
The paper concludes that self-supervised representation learning, rank-based comparison, and joint incremental optimization together improve novel category discovery. For larger, more diverse datasets, supervised pretraining can replace self-supervision while rank statistics remain beneficial.
- Conclusions: Self-supervised bootstrapping trades representation quality for generality, producing a better overall solution for novel category discovery.
- Conclusions: Rank statistics compare noisy image descriptors robustly, resulting in robust clustering.
- Conclusions: Jointly optimizing labelled recognition and unlabelled clustering can reinforce both tasks while avoiding forgetting in an incremental-learning setup.
- Conclusions: On standard benchmarks, combining these ideas yields much better performance than existing methods for the same task.
- Conclusions: For larger datasets with more classes and diverse data, self-supervision can be bypassed when labelled pretraining already provides a sufficiently powerful representation.Rank statistics still give drastic improvement over existing methods in these cases.
A DATASET SPLITS
The experiments use benchmark-specific splits in which selected classes remain labelled and the remaining classes form the unlabelled discovery set.
- Dataset splits: CIFAR-10 and SVHN retain five labelled categories, while CIFAR-100 uses 80 labelled categories and leaves the rest unlabelled.
- Dataset splits: OmniGlot treats each of 20 evaluation alphabets as unlabelled data, with 20–47 categories and 659 characters per class.
B IMPLEMENTATION DETAILS
Implementation uses staged self-supervised, supervised, and joint training, with dataset-specific schedules and repeated benchmark evaluation.
- Training schedule: Self-supervised pretraining uses rotation prediction for 200 epochs with learning-rate drops at epochs 60, 120, and 160.The learning rate starts at 0.1 and is divided by 5 at each listed epoch.
- Training schedule: Supervised fine-tuning runs for 100 epochs, then freezes the first three convolutional blocks while updating the last block and classifier.The initial learning rate is 0.1 and is halved every 10 epochs.
- Joint training: Joint training fine-tunes on randomly sampled batches from the merged labelled and unlabelled data, using dataset-dependent epoch and learning-rate schedules.Training lasts 200, 100, or 90 epochs depending on the dataset group.
- Incremental learning: In incremental learning, the method adds a ramp-up to the unlabelled-data cross-entropy loss with coefficient 0.05.Other methods use 150 classifier-training epochs with SGD, momentum, and scheduled learning-rate reductions.
- Evaluation procedure: Experiments use PyTorch 1.1.0 on NVIDIA Tesla M40 GPUs and average results over 10 runs, except for ImageNet’s three subsets.The authors report generally stable results and publicly release the code.
C STANDARD DEVIATION OF INCREMENTAL LEARNING EXPERIMENT IN TABLE 4
Table 5 reports incremental-learning performance for labelled (“old”), unlabelled (“new”), and combined (“all”) test classes, with predictions unrestricted to their respective subsets.
- “Old” denotes ACC on labelled classes, “new” denotes ACC on unlabelled testing classes, and “all” denotes ACC across the whole testing set.Predictions are not restricted to the corresponding old or new subset.
D IMPACT OF k OVER RESULTS
Performance is evaluated across k values from 1 to 50, with k = 5 and 7 performing best overall and most other settings remaining stable.
- Results were generally stable for all tested k values except k = 1.
E RESULTS WITH AN UNKNOWN NUMBER OF CLASSES
The method is evaluated when the number of new classes is estimated rather than known in advance, reaching state-of-the-art on ImageNet and matching it on OmniGlot.
- When Cu is computed using DTC, the method reaches state-of-the-art on ImageNet and is on par with the state-of-the-art on OmniGlot.Table 6 compares ACC for KCL, DTC, and the proposed method on unlabelled OmniGlot and ImageNet splits.
- The assumption that Cu is known a priori can be restrictive in practice.