PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoNovel: Automatically Discovering and Learning Novel Visual Categories

Kai Han, Sylvestre-Alvise Rebuffi, Sébastien Ehrhardt, Andrea Vedaldi, Andrew Zisserman

arXiv:2106.15252v1cs.CV

TL;DR

The paper asks how to discover novel visual classes from unlabelled images when labels exist only for other classes. AutoNovel combines self-supervised representation learning, ranking-statistics transfer, and joint labelled–unlabelled optimization, while also estimating unknown class counts. It substantially improves novel category discovery and achieves state-of-the-art unsupervised clustering results on public benchmarks.

  • Problem

    Discovering new classes requires clustering unlabelled images, but representations trained only on labelled classes can be biased toward those classes.

  • Method

    AutoNovel learns representations self-supervised on all images, transfers labelled-class information through ranking statistics, and jointly optimizes labelled classification with unlabelled clustering.

  • Results

    AutoNovel substantially outperforms existing methods for novel category discovery and achieves state-of-the-art results for unsupervised image clustering.

  • Takeaways & Limitations

    The combined approach supports novel category discovery, estimates the number of unlabelled classes, and extends to fully unsupervised image clustering.

  • Takeaways & Limitations

    The method assumes labelled and unlabelled classes share a similar category definition and belong to the same vision dataset.

Abstract

from arXiv · show

We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. We present a new approach called AutoNovel to address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labelled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) we use ranking statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data. Moreover, we propose a method to estimate the number of classes for the case where the number of new categories is not known a priori. We evaluate AutoNovel on standard classification benchmarks and substantially outperform current methods for novel category discovery. In addition, we also show that AutoNovel can be used for fully unsupervised image clustering, achieving promising results.

1 INTRODUCTION

AutoNovel addresses novel category discovery by using labelled examples from known classes to cluster images from new, unlabelled classes. It combines self-supervised representation learning, ranking-based transfer, joint optimization, and class-number estimation, achieving strong results on novel-category and unsupervised clustering benchmarks.

  • Novel category discovery uses labelled images from known classes to partition unlabelled images from new classes into clusters.
  • AutoNovel pre-trains a CNN with self-supervision on labelled and unlabelled images, reducing bias toward the labelled classes before supervised fine-tuning.
  • Ranking statistics compare unlabelled image representations by shared maximally activated components, producing noisy pairwise pseudo-labels for similarity learning.
  • A joint objective uses labelled classification targets and unlabelled pseudo-labels, helping optimize both tasks while avoiding forgetting.
  • AutoNovel estimates unknown class counts by probing clustering with held-out known classes and evaluating probe accuracy alongside unlabelled cluster quality.
  • The method substantially outperforms existing novel-category discovery techniques and achieves state-of-the-art results in unsupervised image clustering.

2 RELATED WORK

Related work spans semi-supervised learning, transfer learning, clustering, and zero-shot learning, but AutoNovel targets transferring knowledge from labelled classes to unlabelled classes that differ from them. Its use of ranking statistics for novel category discovery is presented as novel.

  • Unlike conventional semi-supervised learning, AutoNovel addresses unlabelled data whose classes differ from those in the labelled data.
  • Unlike typical transfer learning settings, AutoNovel transfers information from a labelled dataset to an unlabelled dataset.
  • Clustering is often ill-posed because datasets admit multiple valid partitions, so AutoNovel uses labelled data to learn an appropriate class criterion.
  • Earlier novel-category methods learn embeddings from labelled data and then fine-tune or predict similarities for unlabelled clustering, whereas AutoNovel uses a different pipeline.
  • Zero-shot learning recognizes new classes but requires additional side information such as class attributes, unlike the setting described here.
  • AutoNovel applies ranking statistics to novel category discovery with deep neural networks, which the authors identify as a first use.

3 METHOD

AutoNovel learns a representation from labelled and unlabelled images, transfers labelled-class structure through ranking-based pseudo-labels, and jointly trains classification and clustering objectives. The method uses shared embeddings, consistency-aware optimization, and incremental learning to discover unlabelled classes while retaining known-class information.

  • Problem formulation: AutoNovel defines a CNN representation Φ that recognizes known classes and discovers new classes in an unlabelled dataset whose classes are disjoint from the labelled classes.The method assumes that labelled data can provide a general notion of a good class despite distributional differences between labelled and unlabelled datasets.
  • Self-supervised initialization: Supervised bootstrapping can over-specialize representations to labelled classes, so AutoNovel instead pre-trains Φ with self-supervision on the union of labelled and unlabelled images.RotNet is the default self-supervised method; SimCLR and MoCo are also evaluated, while most representation parameters remain frozen to reduce overfitting.
  • Ranking statistics: Ranking statistics create pairwise pseudo-labels by assigning s_ij = 1 when two unlabelled images share the same top-k representation dimensions, and 0 otherwise.The experiments use k = 5; this focuses comparisons on salient feature components while ignoring noisy low-magnitude components.
  • Unlabelled comparison: A new head η_u maps embeddings to C_u outputs, and the inner product between two outputs supplies a learned similarity score for unlabelled image pairs.The output dimension equals the number of new classes, allowing the maximum-output index to serve as the predicted class rather than applying k-means.
  • Joint training: AutoNovel jointly optimizes the labelled CE loss and unlabelled pairwise BCE loss over their shared embedding to avoid catastrophic forgetting from sequential training.Consistency regularization is added for labelled and unlabelled data, while incremental learning allows information to flow between the two subsets.
  • Unsupervised clustering: Compared with DAC, AutoNovel uses ranking-based pseudo-labels, BCE with consistency regularization, self-supervised lower-level features, and limited parameter updates; it outperforms DAC and IIC on several benchmarks.During training, interactions between the labelled and unlabelled heads refine predictions and generate better pseudo-labels.

4 ESTIMATING THE NUMBER OF CLASSES

AutoNovel estimates the unknown number of unlabelled categories by combining labelled probe classes with unlabelled data, repeatedly clustering them, and selecting the class count favored by two quality indices.

  • Final estimate: The estimated number of categories is selected by maximizing the quality indices, averaging their proposed optima, and rerunning k-means with that value.Outlier unlabelled clusters containing less than τ, such as 1%, of the largest-cluster mass are then dropped.
  • Data split: The method splits known classes into training and probe subsets, then divides probe classes into anchor and validation sets for estimation.The anchor probe classes constrain semi-supervised k-means, while the validation probe classes remain available for evaluating cluster quality.
  • Constrained clustering: Anchor probe images are forced to map to clusters according to their ground-truth labels during constrained k-means with the unlabelled data.The procedure sweeps candidate total category counts while clustering the combined probe and unlabelled sets.
  • Model selection: Two quality indices evaluate each candidate count: validation-probe clustering accuracy and an unlabelled-data cluster-validity index.The first uses labelled validation probes, whereas the second assesses cohesion and separation in unlabelled clusters.
  • Quality indices: The Silhouette index is used as the preferred cluster-validity measure because it captures intra-cluster cohesion versus inter-cluster separation.The paper notes that no cluster-validity metric is uniformly best, although Silhouette generally works well in this setting.

5 EXPERIMENTS

Experiments evaluate AutoNovel on standard benchmarks with predefined labelled–unlabelled class splits, controlled training procedures, ablations, and repeated runs. The study also notes that clustering remains difficult as the number of categories grows.

  • Datasets: AutoNovel is evaluated on CIFAR10, CIFAR100, SVHN, OmniGlot, and ImageNet, with unlabelled sets containing 5, 20, 5, 654, and 30 classes respectively.OmniGlot uses 20 splits and ImageNet uses three splits, with average unlabelled-set clustering accuracy reported.
  • Scope: Most experiments involve fewer than 100 unlabelled classes, a scope boundary because clustering thousands of categories remains very difficult.The paper explicitly identifies this class-count regime as a potential limitation of the evaluation setting.
  • Architectures and optimization: The experiments use ResNet-18 except on OmniGlot, where a six-layer VGG-like network preserves comparability with prior work.Batch size is 128, and k = 5 is used across experiments because it performed consistently well across datasets.
  • Training protocol: Training proceeds through self-supervised pre-training, supervised fine-tuning on labelled data, and joint fine-tuning on merged labelled and unlabelled data.The default self-supervised task is four-way rotation prediction, followed by dataset-specific joint-training schedules.
  • Evaluation protocol: Hyper-parameters are tuned using a probe validation set constructed from labelled data, with only two labelled classes used for CIFAR10 and SVHN.Results are generally averaged over 10 runs, except ImageNet, which averages its three 30-class subsets.

5.2 Ablation study

Ablation experiments show that AutoNovel’s components materially contribute to unlabelled-data clustering, while incremental learning provides an additional accuracy boost and ranking performance is generally stable across k.

  • Ablation study: Removing any AutoNovel component substantially lowers unlabelled-set ACC, indicating that all evaluated components contribute significantly.The ablation evaluates clustering because assignments are measured on the unlabelled data.
  • Incremental learning: 1.3%, 2.0% and 0.2% points are gained on CIFAR10, CIFAR100 and SVHN respectively by incorporating discovered classes into classification.This additional boost comes from the incremental-learning component.
  • Ranking statistics: k = {5, 7} gives the best ranking-statistics results overall across SVHN, CIFAR10, and CIFAR100.The reported sweep evaluates k = {1, 2, 3, 5, 7, 10, 15, 20, 50}.
  • Ranking statistics: Ranking-statistics performance is generally stable for every tested k except k = 1.The stability claim is reported across the evaluated values of k.

5.3 Novel category discovery

AutoNovel is compared across multiple novel-category-discovery benchmarks and visualized during training. The learned unlabelled representations become more separated, while incremental learning reduces overlap between old and new classes.

  • Benchmark comparisons: The experiments compare AutoNovel with k-means baselines and state-of-the-art methods for novel category discovery on CIFAR10, CIFAR100, and SVHN.The baselines differ in whether features come from a labelled-data fine-tuned model or a model trained from scratch on labelled images.
  • Representation evolution: During CIFAR10 training, t-SNE visualizations show initially overlapping unlabelled clusters becoming progressively more separated.The plotted points are coloured by their ground-truth labels and correspond to dog, frog, horse, ship, and truck instances.
  • Benchmark comparisons: AutoNovel is additionally evaluated on OmniGlot and ImageNet, averaging results over 20 alphabets and three 30-class unlabelled sets respectively.For these datasets, metric learning on labelled classes is used to pre-train the feature extractor instead of the default self-supervised learning.
  • Incremental learning: Without incremental learning, t-SNE shows greater overlap between old class 3 and new class 5 than when incremental learning is incorporated.Old classes are 0–4 and new classes are 5–9 in the CIFAR10 visualization.

5.4 Incremental learning scheme

AutoNovel incrementally learns novel classes while retaining recognition of existing labelled classes. Compared with competing methods, it performs well across both old and new classes and improves new-class cluster accuracy.

  • 5.4 Incremental learning scheme: Incremental learning lets AutoNovel learn new classes on top of old ones without additional adaptation.The method is designed to recognize both existing and novel classes jointly.
  • 5.4 Incremental learning scheme: AutoNovel achieves similar performance on old and new classes for CIFAR10 and SVHN, where the class split is balanced.The reported balance contrasts with competing representations that favor one class subset.
  • 5.4 Incremental learning scheme: KCL and MCL favor old classes, with KCL showing a 25.3% accuracy discrepancy between old and new classes on SVHN.DTC instead performs better on new classes and poorly on old classes for CIFAR10.
  • 5.4 Incremental learning scheme: With incremental learning, new-class clusters become more accurate, while the main remaining confusion is between dogs and horses.The paper attributes this confusion to similar colors or poses in the images.

5.5 Finding the number of novel categories

AutoNovel estimates the number of unlabelled categories before transfer clustering rather than assuming it in advance. The estimates are reasonably accurate, and using them improves novel-category discovery performance.

  • 5.5 Finding the number of novel categories: AutoNovel estimates the unknown number of unlabelled categories before applying its ranking-based transfer clustering algorithm.KCL and MCL instead assume a large category count of 100.
  • 5.5 Finding the number of novel categories: 4.6 average error on OmniGlot and 2.33 on ImageNet validate the effectiveness of AutoNovel’s category-number estimation.These values are reported for novel category number estimation.
  • 5.5 Finding the number of novel categories: AutoNovel outperforms MCL by 5.2% ACC on OmniGlot and 9.0% ACC on ImageNet when using the estimated category numbers.The comparison concerns novel category discovery with unknown class number.
  • 5.5 Finding the number of novel categories: Using the estimated number of clusters significantly improves KCL and MCL, while DTC beats AutoNovel by 1.6% on OmniGlot and loses by 2.9% on ImageNet.These comparisons use the estimated category number.
  • 5.5 Finding the number of novel categories: Figure 10 evaluates sensitivity to the assumed cluster count by varying it from 20 to 100 on 30-class ImageNetA.The ground-truth category count is 30.

5.6 Transferring from ImageNet pretrained model

This section examines ImageNet-pretrained representations and alternatives to ranking statistics for transferring knowledge to novel-category clustering. Ranking-based and related similarity methods generally outperform k-means, while soft rank similarity can approach hard rank similarity with tuned k.

  • 5.6 Transferring from ImageNet pretrained model: The study tests ImageNet-pretrained features as an alternative source of representation transfer for novel category discovery.The feature extractor is fine-tuned in the AutoNovel pipeline.
  • 5.6 Transferring from ImageNet pretrained model: Ranking statistics, nearest neighbor, and cosine similarity significantly outperform k-means on CIFAR10 and SVHN.On CIFAR100 and SVHN, ranking statistics and cosine similarity also outperform nearest neighbor.
  • 5.6 Transferring from ImageNet pretrained model: Cosine similarity performance depends on its threshold, with the reported best thresholds equal to 0.85, 0.8, and 0.9 for CIFAR10, CIFAR100, and SVHN.Figure 11 reports performance across thresholds from 0.1 to 0.97.
  • 5.6 Transferring from ImageNet pretrained model: Soft rank similarity counts shared elements in two images’ top-k rankings and uses that count to replace the pairwise similarity term.The method defines the soft similarity from the number of shared ranked elements.
  • 5.6 Transferring from ImageNet pretrained model: With k = 15, soft rank similarity is generally on par with hard binary rank similarity, whereas k = 5 performs poorly.The value k = 15 is selected using the paper’s validation method.

5.8 Other self-supervised learning methods

AutoNovel’s self-supervised stage can use methods beyond RotNet, and their relative performance depends on how much of the network is fine-tuned. Contrastive methods excel when only linear heads are tuned, while broader fine-tuning changes the comparison.

  • 5.8 Other self-supervised learning methods: The pipeline replaces RotNet with SimCLR, MoCo, or MoCo v2 to evaluate other self-supervised representation-learning methods.These methods are described as recent approaches for object recognition and detection.
  • 5.8 Other self-supervised learning methods: When only linear heads are fine-tuned, SimCLR, MoCo, and MoCo v2 significantly outperform RotNet on CIFAR10 novel-category accuracy.The evaluation measures ACC on the unlabelled data.
  • 5.8 Other self-supervised learning methods: RotNet’s higher-level features focus on rotation prediction, which is loosely related to novel category discovery and yields poor performance under head-only fine-tuning.The comparison is tied to the choice of frozen versus fine-tuned layers.
  • 5.8 Other self-supervised learning methods: The experiments vary which ResNet18 layers are frozen and fine-tune the selected layer together with subsequent layers.The setup ranges from tuning only the two linear heads to tuning all parameters.

5.9 Unsupervised image clustering

AutoNovel is evaluated for fully unsupervised image clustering on CIFAR10, CIFAR100-20, and STL10. It performs comparably to IIC on CIFAR10 and STL10 while outperforming IIC on CIFAR100-20.

  • AutoNovel is compared with state-of-the-art clustering methods on CIFAR10, CIFAR100-20, and STL10.The experiment uses all 10 classes in CIFAR10 and STL10, and 20 meta classes in CIFAR100-20.
  • AutoNovel performs on par with IIC on CIFAR10 and STL10.
  • 9.3%: AutoNovel significantly outperforms IIC on CIFAR100-20.
  • AutoNovel requires conventional augmentation and batch size 128, whereas IIC requires Sobel filtering and much larger batch sizes.IIC uses batch sizes of 660, 1000, and 700 on CIFAR10, CIFAR100-20, and STL10, respectively.
  • Table 13 reports unsupervised image-clustering results, with “k-means on S.S.” denoting k-means on the self-supervised model representation.

6 CONCLUSIONS

The paper concludes that AutoNovel combines self-supervised representation bootstrapping, ranking statistics, and joint incremental optimization to discover novel classes. It also supports unsupervised clustering, while assuming labelled and unlabelled classes share a similar category definition within the same vision dataset.

  • Conclusions: AutoNovel combines self-supervised bootstrapping, ranking statistics, and joint incremental optimization for novel category discovery.
  • Conclusions: Ranking statistics provide robust clustering of noisy image descriptors and transfer knowledge from labelled classes to unlabelled data.
  • Conclusions: Jointly optimizing labelled recognition and unlabelled clustering can reinforce both tasks while avoiding forgetting.
  • Conclusions: AutoNovel can also perform unsupervised image clustering after removing the labelled-data requirement, with performance on par with state-of-the-art methods.
  • Assumption: The method assumes labelled and unlabelled classes share a similar category definition and belong to the same vision dataset.The paper identifies measuring task relevance as a future research direction.
Loading 2106.15252v1…