PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Semi-supervised Gaussian Mixture Models for Generalized Category Discovery

Bingchen Zhao, Xin Wen, Kai Han

arXiv:2305.06144v2cs.CV

TL;DR

GCD must cluster unlabelled images that may belong to known or novel classes, without assuming the class count is known. The paper alternates class and prototype estimation with representation learning, and reports state-of-the-art results across generic and fine-grained benchmarks.

  • Problem

    GCD assigns labels to unlabelled instances without assuming the number of new or total categories is known.

    Source: §3 · p. 3

  • Method

    An EM-like framework alternates semi-supervised GMM class and prototype estimation with prototypical contrastive representation learning.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The framework achieves state-of-the-art performance on multiple public benchmarks, including generic and fine-grained image-recognition datasets.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The paper concludes that class-number estimation and representation learning can facilitate more robust category discovery.

    Source: §5 · p. 9

    The setting assumes the labelled-category count is available, while the new-category and total-category counts are unknown.

    Source: §3 · p. 3

Abstract

from arXiv · show

In this paper, we address the problem of generalized category discovery (GCD), \ie, given a set of images where part of them are labelled and the rest are not, the task is to automatically cluster the images in the unlabelled data, leveraging the information from the labelled data, while the unlabelled data contain images from the labelled classes and also new ones. GCD is similar to semi-supervised learning (SSL) but is more realistic and challenging, as SSL assumes all the unlabelled images are from the same classes as the labelled ones. We also do not assume the class number in the unlabelled data is known a-priori, making the GCD problem even harder. To tackle the problem of GCD without knowing the class number, we propose an EM-like framework that alternates between representation learning and class number estimation. We propose a semi-supervised variant of the Gaussian Mixture Model (GMM) with a stochastic splitting and merging mechanism to dynamically determine the prototypes by examining the cluster compactness and separability. With these prototypes, we leverage prototypical contrastive learning for representation learning on the partially labelled data subject to the constraints imposed by the labelled data. Our framework alternates between these two steps until convergence. The cluster assignment for an unlabelled instance can then be retrieved by identifying its nearest prototype. We comprehensively evaluate our framework on both generic image classification datasets and challenging fine-grained object recognition datasets, achieving state-of-the-art performance.

1. Introduction

The introduction frames GCD as clustering unlabeled data spanning known and novel classes without a known class count, and presents GPC as jointly learning representations and estimating classes and prototypes.

1. Introduction

  • GCD extends beyond SSL by allowing unlabelled data to contain both known and novel classes, addressing the cost and limits of obtaining annotations.SSL assumes labelled instances for all classes in the unlabelled data.Source: §1 · p. 1
  • Existing NCD and GCD methods assume the unlabelled class count is known or precomputed, which can produce suboptimal solutions.The introduction notes that class counts are often unknown in practice.Source: §1 · p. 2
  • GPC alternates class-number and prototype estimation with prototypical contrastive representation learning, using a semi-supervised GMM that splits separable clusters and merges cluttered ones.The framework treats representation learning and class-number estimation as mutually reinforcing steps.Source: §1 · p. 2
  • GPC achieves state-of-the-art results on generic image-classification benchmarks and the fine-grained Semantic Shifts Benchmark suite.The reported evaluations include CIFAR10, CIFAR100, ImageNet-100, CUB-200, Stanford-Cars, and FGVC-aircrafts.Source: §1 · p. 2

2. Related work

Related work situates GCD among category-discovery, semi-supervised, contrastive-learning, and clustering methods, highlighting the challenge of unknown class counts and label-guided clustering.

2. Related work

  • NCD transfers knowledge from labelled categories to discover new ones, while GCD also allows unlabelled instances from known categories; prior approaches often require known or precomputed novel-class counts.The cited precomputation does not account for representation learning.Source: §2 · p. 2
  • Prototypical contrastive learning contrasts features against prototypes, which provide higher-level dataset abstractions and can support partially supervised GCD clustering.The paper adopts PCL because it directly considers the downstream clustering task.Source: §2 · pp. 2–3
  • Unlike GCD, standard SSL assumes labelled and unlabelled data come from the same classes, and some extensions do not discover novel categories.Those extensions focus on improving performance for labelled categories.Source: §2 · p. 3
  • Unsupervised clustering can produce multiple valid partitions without labelled-data guidance, so its clustering criteria may not match GCD’s labels.DeepDPM is cited as an approach that automatically changes the inferred number of clusters through splitting and merging.Source: §2 · p. 3

3. Method

Generalized category discovery must assign unlabelled examples from both seen and unseen classes without assuming the class count is known. The method jointly estimates prototypes and class count while learning representations, using assignments from the prototypes.

3. Method

  • The EM-like framework alternates representation learning with class-number estimation, with label assignment emerging during estimation and final assignments made by nearest prototype.The process repeats until convergence.Source: §3 · p. 3, §3.2 · p. 5

3.1. Representation learning

  • Prototypical contrastive learning uses class-center prototypes, computed from labelled-class means for seen classes and a semi-supervised GMM for unseen classes.The PCL weight is gradually increased from 0 to 1 during the first T epochs because early prototypes are not informative.Source: §3.1 · p. 4
  • PCA projects features into principal directions before contrastive learning to address dimensional collapse, where a few dimensions recover most representation-space variance.The prototypes are also computed in the projected space.Source: §3.1 · p. 4

3.2. Class number and prototypes estimation with semi-supervised Gaussian mixture model

  • The semi-supervised GMM stochastically splits separable clusters and merges cluttered clusters, while preventing changes that conflict with labelled classes.Splitting is favored when sub-components are balanced and independent; clusters containing labelled instances cannot be split, and clusters containing different labelled classes cannot be merged.Source: §3.2 · p. 5
  • After splitting and merging, the method updates prototypes and estimated class count, then alternates estimation with representation learning until convergence.Unlabelled assignments use the nearest prototype without a separate non-parametric clustering step.Source: §3.2 · p. 5

4. Experimental results

The method achieves strong results across generic, fine-grained, and long-tailed benchmarks, including when the class number is unknown. Ablations examine initialization, training cost, prototype estimation, and PCA choices.

4.1. Experimental setup

  • Evaluation covers generic and fine-grained datasets; 50% of images from labeled classes are labeled, while the unlabeled pool includes known- and novel-class images.The generic datasets are CIFAR-10/100 and ImageNet-100; the fine-grained benchmark includes CUB-200, Stanford Cars, and FGVC-Aircraft.Source: §4.1 · pp. 5–6

4.2. Comparison with the state-of-the-art

  • Our method performs best overall on generic datasets, is competitive on Semantic Shift Benchmark datasets, and ranks best on Herbarium-19 for All and New classes.On the Semantic Shift Benchmark, it again performs better when the class number is unknown.Source: Table 1 · p. 6, §4.2 · pp. 6–7, Table 3 · p. 7

4.3. Novel class number estimation

  • Class-number estimation is generally robust across initial guesses, and the authors select a simple, reliable initialization for all datasets.The reported results examine different initial guesses for the number of classes.Source: §4.3 · p. 7, Table 5 · p. 8

4.5. Ablation study

  • 128 principal directions explain most variance on both datasets and produce the best performance among tested PCA dimensions, avoiding dimensional collapse.Source: Figure 3 · p. 7, §4.5 · p. 8

4.4. Training complexity

  • Despite the extended fitting time required by GMMs, the method outperforms Vaze et al. on CUB and ImageNet-100 at similar training duration and avoids post-training SS-k-means.The comparison extends Vaze et al. to 1.5× its original training epochs.Source: §4.4 · p. 7, Table 6 · p. 8

4.5. Ablation study

  • The semi-supervised GMM achieves the best performance across three datasets versus DBSCAN, agglomerative clustering, and semi-supervised k-means, while remaining parameter-free.DBSCAN requires radius and minimum-core-point parameters whose ideal values lack a principled way to obtain in practice.Source: §4.5 · p. 8
  • Adding the GMM to Vaze et al. improves results on CUB and Stanford Cars, while the full framework performs best across datasets; class-number estimation also works atop other learned representations.The overall best results use the representation learned within the proposed framework.Source: Table 7 · p. 8, §4.5 · pp. 8–9, Table 8 · p. 9
  • Across partial-overlap settings and CUB old/new class ratios, the method outperforms Vaze et al., though estimates worsen when the initial guess is far from ground truth.Source: Table 9 · p. 9, Table 10 · p. 9, §4.5 · p. 9

5. Conclusion

The conclusion presents an EM-like framework that alternates class-number and prototype estimation with representation learning, which the authors report can reinforce category discovery. The appendix details the Gaussian priors and marginal-likelihood calculation used in cluster splitting and merging.

5. Conclusion

  • The framework alternates estimating class numbers and prototypes with robust representation learning, and the authors report that the two stages facilitate each other.It uses evolving prototypes for prototypical contrastive learning and achieves state-of-the-art performance on multiple public benchmarks.Source: §5 · p. 9

A. Details of the framework for splitting and merging clusters

  • The splitting-and-merging framework models each Gaussian component with mixture weight, mean, and covariance parameters, using a Normal Inverse Wishart prior for unknown means and covariances.The prior's conjugacy gives a closed-form posterior.Source: A. Details of the framework for… · p. 12
  • The framework computes marginal likelihood by integrating out a component's mean and covariance, supplying the quantity used in the main paper's splitting and merging equations.The passage gives the closed-form expression and says it can be used to compute the main-paper equations.Source: A. Details of the framework for… · p. 12

B. Estimating the number of clusters on validation set

The validation experiment tests initial novel-class-number guesses using a split of labelled classes with labels withheld. The reported observation is that a guess around K_l/2 often yields a good estimate.

B. Estimating the number of clusters on validation set

C. Convergence of the estimated class number in longer training

The longer-training experiment examines estimated class-number convergence on CUB after 800 training epochs. Its results are presented in Table 12.

C. Convergence of the estimated class number in longer training

D. Error bars for generalized category discovery performance

Across known and unknown class-number settings, the method consistently outperforms the previous state of the art in the reported generic and Semantic Shift Benchmark results. The authors report relatively small variation across three random-seed runs.

D. Error bars for generalized category discovery performance

E. Error bars for category number estimation

The appendix reports repeated-seed category-number estimates and additional ORCA comparisons using different pretrained data.

E. Error bars for category number estimation

F. Further comparison with ORCA

G. Qualitative results

The qualitative results contrast DINO groupings with representations learned by the proposed method and compare their category structure in two datasets.

G. Qualitative results

H. Limitation and negative societal impact

The authors identify limits to the method’s performance and the scope of the evaluation, and recommend validation and adaptation before real-world deployment.

H. Limitation and negative societal impact

  • Performance remains below fully supervised models, and curated-data results require careful validation and adaptation for specific real-world applications.The authors also note a performance gap when the category number is unknown and greater complexity in real-world data.Source: H. Limitation and negative societal… · p. 18

I. License of used datasets

The datasets have research-use permissions, with non-commercial restrictions specified for ImageNet and FGVC-aircraft.

I. License of used datasets

  • CIFAR-10, CIFAR-100, Stanford Cars, and CUB-200 permit research use; ImageNet and FGVC-aircraft are limited to non-commercial research or educational use.ImageNet permits non-commercial research and educational use; FGVC-aircraft is specified for non-commercial research.Source: §I · p. 18

Related papers

Loading 2305.06144v2…