PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Category Discovery

Sagar Vaze, Kai Han, Andrea Vedaldi, Andrew Zisserman

arXiv:2201.02609v2cs.CVcs.LG

TL;DR

Generalized Category Discovery asks how to categorize unlabelled images when they may belong to either labelled or novel classes and the number of novel classes is unknown. The paper adapts novel-category-discovery baselines, combines contrastive vision-transformer representations with semi-supervised k-means, and proposes class-number estimation. The proposed methods substantially outperform established baselines on generic and fine-grained benchmarks.

  • Problem

    Generalized Category Discovery addresses image recognition when unlabelled images may come from labelled or novel classes, without assuming the number of categories is known.

  • Method

    The paper uses contrastive representation learning with vision transformers, semi-supervised k-means clustering, adapted novel-category-discovery baselines, and an algorithm for estimating unlabelled class counts.

  • Results

    The proposed methods substantially outperform established baselines on generic object-recognition datasets and especially challenging fine-grained benchmarks.

  • Takeaways & Limitations

    Direct clustering of contrastively trained vision-transformer features is a surprisingly good classification approach in the generalized setting, whereas parametric classifiers tend to overfit labelled classes.

  • Takeaways & Limitations

    The approach assumes no domain shift between labelled and unlabelled subsets and does not address streaming data without retraining from scratch.

Abstract

from arXiv · show

In this paper, we consider a highly general image recognition setting wherein, given a labelled and unlabelled set of images, the task is to categorize all images in the unlabelled set. Here, the unlabelled images may come from labelled classes or from novel ones. Existing recognition methods are not able to deal with this setting, because they make several restrictive assumptions, such as the unlabelled instances only coming from known - or unknown - classes, and the number of unknown classes being known a-priori. We address the more unconstrained setting, naming it 'Generalized Category Discovery', and challenge all these assumptions. We first establish strong baselines by taking state-of-the-art algorithms from novel category discovery and adapting them for this task. Next, we propose the use of vision transformers with contrastive representation learning for this open-world setting. We then introduce a simple yet effective semi-supervised $k$-means method to cluster the unlabelled data into seen and unseen classes automatically, substantially outperforming the baselines. Finally, we also propose a new approach to estimate the number of classes in the unlabelled data. We thoroughly evaluate our approach on public datasets for generic object classification and on fine-grained datasets, leveraging the recent Semantic Shift Benchmark suite. Project page at https://www.robots.ox.ac.uk/~vgg/research/gcd

1. Introduction

The paper formalizes Generalized Category Discovery, where unlabelled images may belong to either labelled or novel classes. It adapts existing methods, proposes contrastive clustering, and addresses unknown class-count estimation.

  • Problem: Generalized Category Discovery assigns categories to unlabelled images that may come from labelled or novel classes.The setting is motivated by realistic applications where the origin of new images cannot be assumed in advance.
  • Problem: Existing recognition settings impose restrictive assumptions, including fully labelled training data, shared labelled and unlabelled classes, or exclusively novel unlabelled classes.These assumptions distinguish GCD from standard classification, semi-supervised learning, open-set recognition, and novel-category discovery.
  • Baselines: Adapted novel-category-discovery methods provide strong baselines but tend to overfit their classification heads to labelled classes in the generalized setting.The adaptations modify training and inference mechanisms and use a more robust backbone architecture.
  • Proposed method: Contrastive representation learning with vision transformers and semi-supervised k-means clusters unlabelled data without a parametric classifier.The method leverages vision transformers' nearest-neighbour properties and is evaluated on generic and fine-grained benchmarks.
  • Class-count estimation: The paper introduces an algorithm to estimate the number of categories in unlabelled data, avoiding the usual assumption that this number is known.This addresses an under-investigated problem in image recognition.

2. Related work

The related work spans semi-supervised learning, open-set recognition, and novel-category discovery. These settings address parts of the broader problem but differ in their assumptions and objectives.

  • Semi-supervised learning: Semi-supervised learning uses labelled and unlabelled data while assuming that both subsets contain instances from the same classes.Its objective is to learn a robust classification model from both data sources.
  • Open-set recognition: Open-set recognition classifies instances from labelled semantic classes while detecting test instances from unseen classes.Prior approaches include Extreme Value Theory, adversarial samples, reconstruction error, and prototype-based distances.
  • Novel-category discovery: Novel-category discovery studies unlabelled data from novel classes using methods such as DTC, KCL, and MCL.The paper adapts representative NCD methods as baselines for the more general GCD setting.
  • Related open-world settings: Concurrent work studies a similar open-world image-recognition setting, but does not use large-scale pretraining or evaluate on the Semantic Shift Benchmark.The comparison concerns Open World Semi-Supervised Learning and its distinction from GCD.

3. Generalized category discovery

Generalized Category Discovery assigns labels to unlabelled images that may belong to either labelled or novel classes. The paper combines contrastive representation learning, semi-supervised clustering, and class-number estimation to address this setting.

  • Problem: Generalized Category Discovery assigns labels to all unlabelled images using classes that may or may not appear in the labelled set.
  • Representation learning and clustering: The proposed representation-learning approach uses contrastive learning with a vision transformer and performs classification through semi-supervised k-means rather than a parametric head.The method clusters directly in feature space and uses labelled data within a contrastive framework.
  • Representation learning and clustering: Removing the parametric classification head is intended to reduce overfitting to labelled classes while allowing labelled and unlabelled data to be treated similarly.
  • Estimating the number of categories: The method estimates the number of unlabelled categories by clustering the entire dataset and evaluating labelled-subset accuracy across candidate cluster counts.The Hungarian algorithm evaluates assignments, while black-box optimization avoids exhaustive enumeration over all k values.
  • Baselines: The paper adapts RankStats and UNO from novel category discovery as generalized-category-discovery baselines and reports that existing NCD methods can overfit labelled classes.

4. Experiments

The experiments evaluate Generalized Category Discovery across generic and fine-grained datasets, measuring recognition of all, old, and new unlabelled instances. The proposed vision-transformer, contrastive-learning, and semi-supervised clustering approach consistently outperforms adapted NCD baselines and estimates unknown class counts with bounded error.

  • Experimental setup: The unlabelled set combines remaining images from labelled classes with all images from other classes, and evaluation reports All, Old, and New subsets.Accuracy on All instances is the main metric; Old and New subsets separately measure seen- and novel-category recognition.
  • Datasets: Experiments span CIFAR10, CIFAR100, ImageNet-100, CUB, Stanford Cars, and Herbarium19, covering generic and fine-grained recognition settings.The fine-grained benchmarks provide semantically coherent old and new categories, while Herbarium19 reflects a real-world discovery setting.
  • Comparison with baselines: 9.3% absolute and 11.5% proportional gains over RankStats+ and UNO+ appear on standard datasets, while fine-grained evaluations yield 8.9% absolute and 27.0% proportional gains.These comparisons use accuracy over all unlabelled instances.
  • Comparison with baselines: Parametric baselines can outperform the proposed method on Old classes, but this advantage reduces New-class accuracy; early stopping gives their best overall performance.Longer baseline training shifts accuracy toward New classes at the expense of Old classes.
  • Class-count estimation: The class-count estimator has at most 10% error on generic datasets and 18.9% average discrepancy on fine-grained datasets.The fine-grained setting is harder because many constituent classes are visually similar.
  • Ablation study: The full method requires the combination of a ViT backbone, contrastive fine-tuning, and semi-supervised k-means rather than any single component.Contrastive losses add 19% on CIFAR100 and 16% on Herbarium19; semi-supervised k-means further improves All accuracy by 2% and 7%, respectively.

5. Conclusion

The paper introduces Generalized Category Discovery as a challenging, realistic image-recognition setting without several assumptions of existing sub-fields. It finds that direct clustering of contrastively trained ViT features can outperform parametric classification in this setting.

  • Generalized Category Discovery is presented as a new and realistic setting for image recognition.
  • GCD removes limiting assumptions from existing settings such as novel category discovery and open-set recognition.
  • Parametric classifiers tend to overfit labelled classes in the generalized setting.
  • Direct clustering of features from contrastively trained ViTs is reported as an effective classification method for GCD.

Appendices

Figure 4 illustrates semi-supervised k-means for k = 5, initializing centroids from labelled classes and unlabelled data before assigning cluster labels.

  • For k = 5, three centroids are initialized from averages of labelled data points in their classes.
  • k-means++ is then run on unlabelled data to obtain two additional centroids.
  • The five resulting centroids are used to assign each data point a cluster label.

A. Dataset details

The dataset setup divides classes into ‘Old’ labelled classes and ‘New’ classes, generally sampling half of the classes for each group. CIFAR100 is the stated exception.

  • For most datasets, 50% of classes are sampled as ‘Old’ classes and the remainder as ‘New’ classes.
  • CIFAR100 uses 80 ‘Old’ classes instead of the general 50% split.
  • Generic object datasets assign the first classes by class index to ‘Old’ and the rest to ‘New’.
  • Semantic Shift Benchmark datasets use the data splits provided in [45].

B. Semi-supervised k-means

The appendix expands the semi-supervised k-means algorithm for Generalized Category Discovery through Figure 4.

  • Figure 4 provides an expanded description of the semi-supervised k-means algorithm for GCD.

C. Estimating the number of classes

The paper estimates the total number of categories by running k-means with varying k and selecting the value that maximizes labelled-subset clustering accuracy. Across datasets, this accuracy is approximately bell-shaped and peaks near the ground-truth class count.

  • The method varies k in k-means on the entire dataset while monitoring clustering accuracy on the labelled subset.This procedure estimates k = |Y_L ∪ Y_U|.
  • The labelled-subset accuracy follows an approximately bell-shaped curve across the evaluated datasets.
  • The accuracy maximum occurs near each dataset’s ground-truth number of classes, supporting maximization as a class-count estimator.The ground-truth counts for CIFAR10, CIFAR100, ImageNet-100, and CUB are 10, 100, 100, and 200, respectively.

D. Results on FGVC-Aircraft

The paper evaluates its baselines and method on FGVC-Aircraft using Semantic Shift Benchmark splits and clarifies how clustering accuracy is computed. It uses a single Hungarian assignment over all instances for Old and New categories, unlike Legacy NCD’s separate assignments.

  • The evaluation includes FGVC-Aircraft splits from the Semantic Shift Benchmark suite.
  • The Hungarian algorithm matches predicted cluster indices to ground-truth labels when computing clustering accuracy.
  • Legacy NCD computes Hungarian assignments independently for Old and New subsets, then calculates accuracy separately.
  • This work computes the Hungarian assignment over all instances, forcing clusters to map across Old and New categories and lowering one subset’s performance when necessary.
  • Accuracy over All categories in the unlabelled set follows the protocol used both here and in novel category discovery literature.

F. Attention maps

The attention analysis examines ViT attention heads over spatial locations and finds that fine-tuning with the proposed approach produces semantically specialized, relatively robust object-part attention. These parts can transfer between Old and New fine-grained classes.

  • Visualization construction: The visualization process examines attention from heads supporting the [CLS] token to spatial image locations.The input contains features for the image patches and the [CLS] token.
  • Visualization construction: The ViT model uses 12 attention heads, each independently attending over input features.
  • Visualization construction: Certain attention heads specialize in coherent semantic object parts, although some heads remain uninterpretable.
  • Attention visualizations: Pre-trained DINO often attends to salient regions, but a given head shows relatively little semantic consistency across images.Examples include wheels in cars and heads of birds.
  • Attention visualizations: After training with the proposed approach, heads specialize in parts such as Windshield, Headlight, Wheelhouse, Beak, Head, and Belly, robustly across pose and scale shifts.The visualizations include Old and New classes in Stanford Cars and CUB.
  • Attention visualizations: Semantic-part representations can transfer between Old and New fine-grained classes, supporting generalization from labelled to unlabelled classes.

G. Broader impact and limitations

The paper emphasizes caution when using unsupervised label assignment, especially for sensitive personal-data applications. Its practical scope excludes domain shift and streaming data, and its experiments use licensed third-party datasets that may contain people.

  • Broader impact: Because the method assigns labels unsupervisedly and discovers new labels, the authors recommend manually checking results in sensitive contexts.They specifically mention processing personal data.
  • Limitations: The method assumes no domain shift between labelled and unlabelled subsets.The paper does not address reliably classifying photographs and paintings of the same classes with one model.
  • Limitations: The paper does not consider streaming or continual learning, so new data would require retraining the model from scratch.
  • Data considerations: The experiments use standard third-party datasets compatibly with their licenses, including Internet images that may incidentally feature people.
Loading 2201.02609v2…